[Met_help] [rt.rap.ucar.edu #71510] History for Integer number after LINE_TYPE

Tue Apr 21 10:30:54 MDT 2015

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello,

I’m processing the output .stat files produced by MET ENSEMBLE_STAT to do some statistics where I can add in my own observation error variances, however upon writing some MATLAB code to read in the files, i’m having this issue where there is a integer after the LINE_TYPE (here i’m showing METAR ORANK) that is either 3 or 4 digits, which affects the reading of other data after (I can’t determine why it is either 3 or 4 digits). I have tried to make some adjustments for the text reader but I can’t seem to find any explanation on what this integer numbers is from the README or online. Below are snips out of two hourly files, other hourly files each have different integers (1075 and 635). Do you happen to know what this integer is and any way to force it to remain the same number of digits each time I run ensemble stat?: 

$VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE
$FULL    UW_MEAN               1          NA         NA         NA    NA ORANK     1075

$VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE
$FULL    UW_MEAN               1          NA         NA         NA    NA ORANK     635

Thank you,

Brock

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Integer number after LINE_TYPE
From: John Halley Gotway
Time: Mon Apr 20 13:04:56 2015

Brock,

When you run ensemble_stat, if you set the ORANK output type in the
configuration file to BOTH, you'll get an output that ends with
"_orank.txt".  That file contains only the ORANK line type and has a
nice
listing of header columns on the first line.  Also, table 7-2 of the
MET
user's guide lists the contents of the ORANK line type:

http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.0.pdf

Listed below are the data columns that follow the LINE_TYPE column for
the
ORANK line type:
   TOTAL, INDEX, OBS_SID, OBS_LAT, OBS_LON, OBS_LVL, OBS_ELV, OBS,
PIT,
RANK, N_ENS_VLD, N_ENS, ENS_i (for each member), OBS_QC

The TOTAL column is the one immediately following the LINE_TYPE
column.
This lists the number of pairs that went into the creation of the
corresponding ranked histogram.  In your example, one ranked histogram
had
1075 points and another had 635.  You should have 1075 ORANK lines for
the
first case, each containing the raw observation value and all of the
corresponding ensemble members' values.  Then you should have 635
lines for
the second case.

There is no way to force MET to use the same number of digits when
writing
integers.

I strongly suspect that there's a way in MATLAB to read this data
without
requiring the same number of digits.  Is the problem perhaps that
you're
trying to read the .stat file directly which contains a mixture of
ORANK
and RHIST lines?  The .stat file is a "ragged array" meaning that the
number of columns varies from line to line depending on the line type.
That would likely cause trouble for MATLAB.

Please try re-running Ensemble-Stat but set the config file
"output_flag"
to BOTH for orank.  Then try reading the "_orank.txt" file into
Matlab.
Hopefully, that'll work better.

Alternatively, you could use "grep" to filter out the ORANK lines to a
separate file and read that into Matlab.

Thanks,
John Halley Gotway
met_help at ucar.edu

On Mon, Apr 20, 2015 at 11:58 AM, Burghardt, Brock via RT
<met_help at ucar.edu
> wrote:

>
> Mon Apr 20 11:58:25 2015: Request 71510 was acted upon.
> Transaction: Ticket created by brock.burghardt at ttu.edu
>        Queue: met_help
>      Subject: Integer number after LINE_TYPE
>        Owner: Nobody
>   Requestors: brock.burghardt at ttu.edu
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71510 >
>
>
> Hello,
>
>
> I’m processing the output .stat files produced by MET ENSEMBLE_STAT
to do
> some statistics where I can add in my own observation error
variances,
> however upon writing some MATLAB code to read in the files, i’m
having this
> issue where there is a integer after the LINE_TYPE (here i’m showing
METAR
> ORANK) that is either 3 or 4 digits, which affects the reading of
other
> data after (I can’t determine why it is either 3 or 4 digits). I
have tried
> to make some adjustments for the text reader but I can’t seem to
find any
> explanation on what this integer numbers is from the README or
online.
> Below are snips out of two hourly files, other hourly files each
have
> different integers (1075 and 635). Do you happen to know what this
integer
> is and any way to force it to remain the same number of digits each
time I
> run ensemble stat?:
>
> $VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH
ALPHA
> LINE_TYPE
> $FULL    UW_MEAN               1          NA         NA         NA
NA
> ORANK     1075
>
> $VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH
ALPHA
> LINE_TYPE
> $FULL    UW_MEAN               1          NA         NA         NA
NA
> ORANK     635
>
>
> Thank you,
>
> Brock
>
>
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #71510] Integer number after LINE_TYPE
From: Burghardt, Brock
Time: Tue Apr 21 10:23:48 2015

Hi John,

Thanks for the response, and I apologize I did not look hard enough to
see that the line after LINE_TYPE was the number of samples in the
rank histogram. Yes, I think if I can have Matlab read in the .txt, it
will be much easier than the .STAT.

I removed my UPP converted wrfout.grb files due to file space
conservation, but am rerunning it and will try rerunning met
ensemble_stat with the orank output flag to “BOTH”. I will let you
know if I have any problems, but this answers my question and was very
helpful.

Thanks again,
Brock

Brock Burghardt
Atmospheric Science PhD Student
Dept. of Geosciences
Texas Tech University

> On Apr 20, 2015, at 2:04 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Brock,
>
> When you run ensemble_stat, if you set the ORANK output type in the
> configuration file to BOTH, you'll get an output that ends with
> "_orank.txt".  That file contains only the ORANK line type and has a
nice
> listing of header columns on the first line.  Also, table 7-2 of the
MET
> user's guide lists the contents of the ORANK line type:
>
>
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.0.pdf
>
> Listed below are the data columns that follow the LINE_TYPE column
for the
> ORANK line type:
>   TOTAL, INDEX, OBS_SID, OBS_LAT, OBS_LON, OBS_LVL, OBS_ELV, OBS,
PIT,
> RANK, N_ENS_VLD, N_ENS, ENS_i (for each member), OBS_QC
>
> The TOTAL column is the one immediately following the LINE_TYPE
column.
> This lists the number of pairs that went into the creation of the
> corresponding ranked histogram.  In your example, one ranked
histogram had
> 1075 points and another had 635.  You should have 1075 ORANK lines
for the
> first case, each containing the raw observation value and all of the
> corresponding ensemble members' values.  Then you should have 635
lines for
> the second case.
>
> There is no way to force MET to use the same number of digits when
writing
> integers.
>
> I strongly suspect that there's a way in MATLAB to read this data
without
> requiring the same number of digits.  Is the problem perhaps that
you're
> trying to read the .stat file directly which contains a mixture of
ORANK
> and RHIST lines?  The .stat file is a "ragged array" meaning that
the
> number of columns varies from line to line depending on the line
type.
> That would likely cause trouble for MATLAB.
>
> Please try re-running Ensemble-Stat but set the config file
"output_flag"
> to BOTH for orank.  Then try reading the "_orank.txt" file into
Matlab.
> Hopefully, that'll work better.
>
> Alternatively, you could use "grep" to filter out the ORANK lines to
a
> separate file and read that into Matlab.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
>
>
> On Mon, Apr 20, 2015 at 11:58 AM, Burghardt, Brock via RT
<met_help at ucar.edu
>> wrote:
>
>>
>> Mon Apr 20 11:58:25 2015: Request 71510 was acted upon.
>> Transaction: Ticket created by brock.burghardt at ttu.edu
>>       Queue: met_help
>>     Subject: Integer number after LINE_TYPE
>>       Owner: Nobody
>>  Requestors: brock.burghardt at ttu.edu
>>      Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71510 >
>>
>>
>> Hello,
>>
>>
>> I’m processing the output .stat files produced by MET ENSEMBLE_STAT
to do
>> some statistics where I can add in my own observation error
variances,
>> however upon writing some MATLAB code to read in the files, i’m
having this
>> issue where there is a integer after the LINE_TYPE (here i’m
showing METAR
>> ORANK) that is either 3 or 4 digits, which affects the reading of
other
>> data after (I can’t determine why it is either 3 or 4 digits). I
have tried
>> to make some adjustments for the text reader but I can’t seem to
find any
>> explanation on what this integer numbers is from the README or
online.
>> Below are snips out of two hourly files, other hourly files each
have
>> different integers (1075 and 635). Do you happen to know what this
integer
>> is and any way to force it to remain the same number of digits each
time I
>> run ensemble stat?:
>>
>> $VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH
ALPHA
>> LINE_TYPE
>> $FULL    UW_MEAN               1          NA         NA         NA
NA
>> ORANK     1075
>>
>> $VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH
ALPHA
>> LINE_TYPE
>> $FULL    UW_MEAN               1          NA         NA         NA
NA
>> ORANK     635
>>
>>
>> Thank you,
>>
>> Brock
>>
>>
>>
>

------------------------------------------------