[Met_help] ASCII Data for MET

[John Halley Gotway]

Thomas,

I believe you question is really, how does Point-Stat do matching in the vertical, in particular for surface observations?

The answer is that we take a pretty simplistic approach inherited from the NCEP verification tools.  Point-Stat determines if an observation is a surface observation based purely on message type.  Any
observation with a message type of ADPSFC or SFCSHP is considered to be at the surface.  So when you're verifying surface variables, such as 2-meter temperature (TMP/Z2) and 10-meter winds (UWND/Z10),
only those observations with a message type of ADPSFC or SFCSHP will be used.

So even though the MET point observation format includes elevation information, Point-Stat is not currently using it.  We put it in there as a placeholder anticipating that we may need to do more
complex matching in the vertical in the future.  How would you like to see Point-Stat handle matching in the vertical?  Is there some other logic you'd prefer to see?

So for your surface observations that measure t2m, td2m, and 10-meter winds, just be sure to set the message type for them to be ADPSFC, and Point-Stat will interpret them as surface observations.

It is only when verifying upper-air variables that Point-Stat actually uses the pressure value for the observations.  For upper-air, it does matching in the vertical based on pressure.

To be clear, if you want to verify 2-meter temperature for example, use the following settings in the Point-Stat config file:
fcst_field[] = [ "TMP/Z2" ];
obs_field[]  = []; (unless set, this will default to the fcst setting)
message_type[] = [ "ADPSFC" ];

Hope that helps.
John

Thomas Schwitalla wrote:
> John,
> 
> now a few questions arised:
> 
> How to deal with missing values in the observations? Some stations
> measure t2m,td2m and 10m Wind speed and direction but not pressure. In
> this case variables are marked with "-99.9" ;  is this a problem for the
> verification with MET?
> 
> How does the point_stat tool recognizes the Temperature with GRIB code
> 11 as 2m Temperature (also in case of missing pressure)? Is it done via
> the two height records in the ascii file?
> 
> How should I specify the surface variables in "fcst_field" for the
> point_stat tool?
> 
> Sorry for my bunch of questions and thanks for your help
> Thomas
> 
> John Halley Gotway schrieb:
>> Thomas,
>>
>> Yeah, that abort message from BUFRLIB is what I expected.  BUFRLIB
>> just isn't 64-bit compatible.  You're right that blocking/unblocking
>> is an issue... but that still won't fix the problem.  To my
>> knowledge, BUFRLIB just won't work with a 64-bit compiler.
>>
>> In METv2.0, we've moved the blocking step into the PB2NC tool.  And
>> the way the blocking is performed is controlled by a compile time
>> flag.  In the top-level MET Makefile, the "ARCH_FLAGS" line has
>> the "-DBLOCK4" flag commented out for GNU compilers.  You could try
>> uncommenting that and recompiling MET, being sure to do a "make clean"
>> first.  And then you could try rerunning that "test_pb2nc.sh"
>> script.  But I'm not hopeful.  I'm guessing you'll still get that
>> abort message from BUFRLIB.
>>
>> Unfortunately, we haven't made any progress on support for the ECMWF
>> BUFR data to date.  However, I'll forward your message to the group
>> that decides our priorities for upcoming releases to let them
>> know you were asking about it.
>>
>> Lastly, I agree with you.  If you're dealing with ~2000
>> observations/day, I think it'd be better to sort them by day. 
>> Otherwise, if they were all in one file for a month, Point-Stat would
>> have to
>> sift through 60,000 observations each time you run it... which could
>> slow it down.
>>
>> Good luck,
>> John
>>
>> Thomas Schwitalla wrote:
>>  
>>> John,
>>>
>>> John Halley Gotway schrieb:
>>>    
>>>> Thomas,
>>>>
>>>> I'm glad you were able to compile MET on a 64-bit machine.  Since it
>>>> sounds like you're using the ASCII2NC tool and not the PB2NC tool, you
>>>> shouldn't have any problems.  It's the PB2NC which
>>>> interfaces with BUFRLIB where the problem exists for 64-bit
>>>> machines. Could you try running the test script
>>>> "METv2.0/scripts/test_pb2nc.sh"
>>>> and let me know if you see an error message in the output?
>>>>         
>>> I get:
>>>
>>> **************BUFR ARCHIVE LIBRARY ABORT*****************
>>>
>>> BUFRLIB: OPENBF - FIRST 4 BYTES READ FROM RECORD IN INPUT FILE CONNECTED
>>> TO UNIT  11 NOT 'BUFR', DOES NOT CONTAIN BUFR DATA
>>>
>>> **************BUFR ARCHIVE LIBRARY ABORT*****************
>>>
>>>
>>> This is obviously a problem of NCEP BUFR in combination with gfortran,
>>> because I get similar error messages when using PREPBUFR data from NCEP
>>> directly in the WRF-Var system... (currently it is not possible).
>>> Maybe it helps to unblock the unblocked data again and then reblock
>>> them?
>>>
>>> Are there any news regarding the possible use of ECMWF BUFR data as
>>> desired at the Winter MET tutorial?
>>>
>>>    
>>>> Yes, you will still need to run your WRF output through WPP prior to
>>>> running it through MET.  The purpose of this is twofold: (1) to
>>>> convert the native model sigma levels to pressure levels and (2) to
>>>> interpolate the model's staggered grid to a more regular grid.  There
>>>> is another WRF tool that performs these functions (only for ARW)
>>>> called "pinterp".  We'll be working with the ARW folks to add
>>>> support in MET for the NetCDF output of "pinterp" for the next release
>>>> of MET.
>>>>         
>>> This sounds good :-)
>>>    
>>>> For your second question, yes, you could write the whole month's
>>>> observations to a single NetCDF file if you like.  But each time you
>>>> run Point-Stat, you should use the "-valid_beg" and "-valid_end"
>>>> command line arguments (or set "beg_ds" and "end_ds" in the Point-Stat
>>>> config file) to define the time window for the observations to be used.
>>>>
>>>> Also, it depends on the number of observations we're talking about.
>>>> If you were using thousands of observations for each day, I'd suggest
>>>> breaking them out into a single file per day to make
>>>> Point-Stat run faster.  But I'm guessing you're not using all that
>>>> many observations and won't have problems with how fast Point-Stat
>>>> runs.
>>>>         
>>> Because it is a huge amount of observations (e.g. ~2000 temperature
>>> measurements per hour over central Europe), I think it is better to
>>> split the data into daily files. The run-time of the tools is not that
>>> time critical (even if it runs 1 day).
>>>
>>> Thomas
>>>    
>>>> Thanks,
>>>> John
>>>>
>>>> Thomas Schwitalla wrote:
>>>>  
>>>>      
>>>>> Hi John,
>>>>>
>>>>> now I want to start using MET for verification of my WRF runs. :-)
>>>>> As far as I know, MET currently only deals with GRIB files created
>>>>> from
>>>>> WPP!?
>>>>> Is this still valid?
>>>>>
>>>>> The second question is about the ASCII-Date used for the ascii2nc
>>>>> tool:
>>>>>
>>>>> Is it possible, to write all observations (for one month e.g.) into a
>>>>> single file or  should the timestamp of the observation files
>>>>> correspond
>>>>> to the timestep of one WRF output file (my purpose is to do a one
>>>>> month
>>>>> long statistic or timeseries of surface variables) ?
>>>>>
>>>>> Best regards
>>>>> Thomas
>>>>>
>>>>> P.S.: I had no problems compiling  MET and running the test dataset
>>>>> on a
>>>>> 64bit Linux machine using gfortran 8-)
>>>>>
>>>>>             
>>>     
> 
>