[Met_help] [rt.rap.ucar.edu #94354] History for Re: valid time from grib data
George McCabe via RT
met_help at ucar.edu
Wed Mar 11 08:59:31 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Correction: the ukmo_24h.conf in the previous email actually had the two
valid_times in FCST_PCP_COMBINE_COMMAND hardwired in. I meant to ask
whether the FCST_PCP_COMBINE_COMMAND set up in the text of the last
email is the correct way to pass on the command to METplus.
Thanks,
Ying
On 2/28/20 11:13 PM, Ying Lin wrote:
> [This follow-up inquiry is for next week. I'm just closing up the
> week here, after a 5-day work station outage].
>
> Hi George and Co.:
>
> Thank you very much for the offer to take a look at the DWD QPF
> files. Here is one:
>
> /gpfs/dell1/nco/ops/dcom/prod/20200228/qpf_verif/dwd_2020022800_012_036
>
> wgrib2 output:
> m72a1 $ wgrib2 -V dwd_2020022800_012_036
> BAD GDS:lon1=0.000000 lon2=2147.733648 should be 0..360
> 1:0:vt=2020022912:surface:Code Table 4.11=255:TPRATE Total
> Precipitation Rate [kg/m^2/s]:
> ndata=1038240:undef=0:mean=2.47516:min=-0.00390625:max=182.363
> grid_template=0:winds(N/S):
> lat-lon grid:(1440 x 721) units 1e-06 input WE:SN output WE:SN res 48
> lat -90.000000 to 90.000000 by 0.250000
> lon 0.000000 to 2147.733648 by 0.250000 #points=1038240
>
> There are several problems with this file:
> 1) the longitude range should have been [0,359.75],
> 2) The field is actually APCP (kg/m^2), not TPRATE (kg/m^2/s)
> 3) The field should not have negative values. I think MET has a way
> to deal with this. Does METplus has a way to say "do not use QPF
> values < 0"?
> 4) the time parameters might also be somewhat irregular. The
> vt=2020022912 in the wgrib2 output above is correct, but 'degrib2'
> output has "FIELD: UNKNOWN Surface (720 -2160 hr ) valid 720 minute
> after 2020022800:00:00 to 2020022912:00:00". The field is (or should
> be) 24h accumulation between 2020/02/28 12Z to 2020/02/29 12Z, so
> "valid 720 minute after 2020022800:00:00 to 2020022912:00:00" is
> correct, but the (720 -2160 hr ) should have been (720-2160 min).
> Can this somehow be fixed using pygrib?
>
> --------------------------------------------------------------
> Back to using FCST_PCP_COMBINE_COMMAND in the model config file for
> UKMO data: the command you gave me did work. The following
>
> /gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
> -v 5 -add
> /gpfs/dell1/nco/ops/dcom/prod/20200226/qpf_verif/ukmo.2020022612
> 'name="TWATP"; level="L0"; valid_time="20200227_00";'
> /gpfs/dell1/nco/ops/dcom/prod/20200226/qpf_verif/ukmo.2020022612
> 'name="TWATP"; level="L0"; valid_time="20200227_12";'
> ukmo.v2020022712_f024_a24h
> produced a correct ukmo.v2020022712_f024_a24h field. But I'm not sure
> how to use that in the FCST_PCP_COMBINE_COMMAN. In the attached
> ukmo_24h.conf, I have
>
> FCST_PCP_COMBINE_COMMAND = -add
> ${modpath}/prod/{init?fmt=%Y%m%d}/qpf_verif/ukmo.{init?fmt=%Y%m%d%H}
> 'name="TWATP"; level="L0"; valid_time="{valid?fmt=%Y%m%d}_00";'
> ${modpath}/prod/{init?fmt=%Y%m%d}/qpf_verif/ukmo.{init?fmt=%Y%m%d%H}
> 'name="TWATP"; level="L0"; valid_time="{valid?fmt=%Y%m%d}_12";'
> {OUTPUT_BASE}/ukmo/bucket/{valid?fmt=%Y%m%d}/ukmo.v{valid?fmt=%Y%m%d}12_f024_a24h
>
> Is that how it should be? Would METplus know to apply the LEAD_SEQ
> and to the above and create the appropriate 24h bucket files?
>
> Thanks again for your help -
>
> Ying
>
> On 2/20/20 10:45 AM, George McCabe wrote:
>> Also, if you would like to send us a path to a DWD file on WCOSS we
>> can take a look and see what we can do with it.
>>
>> On Thu, Feb 20, 2020 at 8:44 AM George McCabe <mccabe at ucar.edu
>> <mailto:mccabe at ucar.edu>> wrote:
>>
>> Hi Ying,
>>
>> You need to remove the valid time from after the filename and
>> before the name/level/valid_time item. Try this command:
>>
>> /gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>> -v 5 -add
>> /gpfs/dell1/nco/ops/dcom/prod/20200216/qpf_verif/ukmo.2020021612
>> 'name="TWATP"; level="L0"; valid_time="20200217_00";'
>> /gpfs/dell1/nco/ops/dcom/prod/20200216/qpf_verif/ukmo.2020021612
>> 'name="TWATP"; level="L0"; valid_time="20200217_12";'
>> /gpfs/dell2/ptmp/Ying.Lin/metplus.v3.out/ukmo/bucket/20200217/ukmo.v2020021712_f024_a24h
>>
>> On Wed, Feb 19, 2020 at 5:52 PM Ying Lin - NOAA Federal
>> <ying.lin at noaa.gov <mailto:ying.lin at noaa.gov>> wrote:
>>
>> Thanks George! Do you mean something like this:
>> /gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>> -v 5 -add
>> /gpfs/dell1/nco/ops/dcom/prod/20200216/qpf_verif/ukmo.2020021612
>> 20200217_00 'name="TWATP"; level="L0";
>> valid_time="20200217_00";'
>> /gpfs/dell1/nco/ops/dcom/prod/20200216/qpf_verif/ukmo.2020021612
>> 'name="TWATP"; level="L0"; valid_time="20200217_12";'
>> /gpfs/dell2/ptmp/Ying.Lin/metplus.v3.out/ukmo/bucket/20200217/ukmo.v2020021712_f024_a24h
>>
>>
>> I'm getting this from the above:
>> DEBUG 2: Since the "-field" command line option was not used,
>> parsing the command line arguments a list of files, each
>> followed by a configuration string.
>> ERROR :
>> ERROR : process_add_sub_derive_args() -> file name used when
>> config string expected
>> (/gpfs/dell1/nco/ops/dcom/prod/20200216/qpf_verif/ukmo.2020021612).
>> Did you forget the "-field" option?
>> ERROR :
>>
>> These are the first two records in ukmo.2020021612 that I'm
>> trying to sum up:
>> wgrib2 -V
>> /gpfs/dell1/nco/ops/dcom/prod/20200216/qpf_verif/ukmo.2020021612
>> | more
>> ** WARNING input Code Table 4.3 = 255 (undefined) **
>> ** WARNING input Code Table 4.3 = 255 (undefined) **
>> 1:0:vt=2020021700:surface:Code Table 4.11=255:TWATP Total
>> Water Precipitation [k
>> g/m^2]:
>> ndata=4915200:undef=0:mean=1.27074:min=0:max=1106.62
>> grid_template=0:winds(N/S):
>> lat-lon grid:(2560 x 1920) units 1e-06 input WE:SN output
>> WE:SN res 48
>> lat -89.953125 to 89.953125 by 0.093750
>> lon 0.070312 to 359.929687 by 0.140625 #points=4915200
>>
>> 2:14745803:vt=2020021712:surface:Code Table 4.11=255:TWATP
>> Total Water Precipita
>> tion [kg/m^2]:
>> ndata=4915200:undef=0:mean=1.1913:min=0:max=615.645
>> grid_template=0:winds(N/S):
>> lat-lon grid:(2560 x 1920) units 1e-06 input WE:SN output
>> WE:SN res 48
>> lat -89.953125 to 89.953125 by 0.093750
>> lon 0.070312 to 359.929687 by 0.140625 #points=4915200
>>
>> Tried adding "-field" to the command above:, which failed in
>> a different way:
>> /gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/8.1/bin/pcp_combine
>> -v 5 -add
>> /gpfs/dell1/nco/ops/dcom/prod/20200216/qpf_verif/ukmo.2020021612
>> 20200217_00 'name="TWATP"; level="L0";
>> valid_time="20200217_00";'
>> /gpfs/dell1/nco/ops/dcom/prod/20200216/qpf_verif/ukmo.2020021612
>> 'name="TWATP"; level="L0"; valid_time="20200217_12";' -field
>> TWATP
>> /gpfs/dell2/ptmp/Ying.Lin/metplus.v3.out/ukmo/bucket/20200217/ukmo.v2020021712_f024_a24h
>>
>> DEBUG 2: Since the "-field" command line option was used,
>> parsing the command line arguments as a list of files.
>> DEBUG 2: Performing derivation command (sum) for 5 files.
>> DEBUG 1: Reading data (TWATP) from input file:
>> /gpfs/dell1/nco/ops/dcom/prod/20200216/qpf_verif/ukmo.2020021612
>> ERROR :
>> ERROR : yyerror() -> syntax error in file "met_config_147762_0"
>> ERROR :
>> ERROR : line = 2
>> ERROR :
>> ERROR : column = 5
>>
>> Is there a way to set up the pcp_combine command differently?
>>
>> Thank you again for the very productive discussion today (and
>> going 30 minutes over the 45 min time slot!).
>>
>> I'll try John's suggestion about the DWD data later.
>>
>> Ying
>> ---
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
>> Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov <mailto:Ying.Lin at noaa.gov>
>>
>>
>>
>> On Wed, Feb 19, 2020 at 4:10 PM George McCabe
>> <mccabe at ucar.edu <mailto:mccabe at ucar.edu>> wrote:
>>
>> Hi Ying,
>>
>> I learned you are able to set the valid time after the
>> filename by adding:
>>
>> 'name="APCP"; level="L0"; valid_time="20190201_00";'
>>
>> L0 can be typically used if the level of the filename if
>> there isn't one. You can use the MET tool plot_data_plane
>> to test that doing this pulls the record you need. Let me
>> know if you have issues with this. You could point us to
>> a file path and we can take a look.
>>
>> John mentioned that you could try running plot_data_plane
>> on the DWD file to see if MET can read it and it
>> positions it correctly. If not, MET 9.0 supports using
>> python embedding, so you can use a python script to
>> preprocess the grib file using something like pygrib (as
>> you mentioned) and pass the data into PCPCombine
>> correctly. I can look into how to change the values using
>> pygrib and pass the data into MET. The script needs to
>> get a data points into a numpy array and set a python
>> dictionary with various attributes so MET knows what it
>> is reading. I can help with this part too. If you could
>> point me to one of these files, I can have someone pull
>> the data down and use it to get the script working.
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not
>> feel obliged to reply to this email outside of your
>> normal working hours.
>>
>>
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
>> obliged to reply to this email outside of your normal working hours.
>>
>>
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
>> obliged to reply to this email outside of your normal working hours.
>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: valid time from grib data
From: George McCabe
Time: Thu Mar 05 14:25:57 2020
Hi Ying,
I implemented the changes needed for PCPCombine to set the options the
way
your case does using ADD. They are available in the develop branch of
the
METplus repository on GitHub. If you would prefer to obtain a release,
we
are putting out METplus 3.0beta4 on Friday (tomorrow). I tested out
this
case using the UKMO file you sent me processing the 24 hour lead.
However,
I am not sure how it will behave with your use case that processes
other
foreact leads. I only have the single file you sent me, so I can't
test
this myself. You could get the updates and see if you are able to
process
your data this way. Here are some configurations I used to get the 24
hr
lead to run:
[config]
FCST_PCP_COMBINE_METHOD = ADD
FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
FCST_PCP_COMBINE_INPUT_NAMES = TWATP
FCST_PCP_COMBINE_INPUT_LEVELS = L0
FCST_PCP_COMBINE_INPUT_OPTIONS = valid_time="{valid?fmt=%Y%m%d_%H}";
FCST_PCP_COMBINE_CONSTANT_INIT = True
[filename_templates]
FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
FCST_PCP_COMBINE_OUTPUT_TEMPLATE = ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
> Transaction: Given to mccabe (George McCabe) by RT_System
> Queue: met_help
> Subject: Re: valid time from grib data
> Owner: mccabe
> Requestors: ying.lin at noaa.gov
> Status: deleted
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>
>
> This transaction appears to have no content
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94354] Re: valid time from grib data
From: Ying Lin
Time: Thu Mar 05 16:04:41 2020
Hi George,
That is good news. If you want to test it out, all the UKMO files for
that cycle were in that one file's parent directory:
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
I was writing you in another window earlier:
In re the UKMO:
Thank you for the github issues ticket. Having METplus handle the one
input file to eliminate the need for pre-processing is fine. Since
you
mentioned that METplus 3.0beta2 does have support for
FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try, setting in
the model config file
FCST_PCP_COMBINE_INPUT_LEVELS = L0
FCST_PCP_COMBINE_INPUT_NAMES = TWATP
It did work.
In re DWD: Wesley Ebisuzaki told me a way today to fixed the bad
longitude problem (Lon2):
-------------------------
wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid -grib
new.grb
What does -set_int 3 60 359750000 do
set 4 byte integer in Section 3, octet 60-63 with value 359750000
ref
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
You could argue that ocets 60-63 should be an unsigned integer, but
wgrib2
doesn't have a -set_uint.
-------------------------
Also, on the matter of TPRATE in the DWD data: setting the field name
to
TPRATE in the METplus config won't work because the field is actually
APCP, not TPRATE as the grib header says. It's off by a factor of
86,400. I'll do some experiment with the DWD.
Thanks,
Ying
On 3/5/20 4:25 PM, George McCabe via RT wrote:
> Hi Ying,
>
> I implemented the changes needed for PCPCombine to set the options
the way
> your case does using ADD. They are available in the develop branch
of the
> METplus repository on GitHub. If you would prefer to obtain a
release, we
> are putting out METplus 3.0beta4 on Friday (tomorrow). I tested out
this
> case using the UKMO file you sent me processing the 24 hour lead.
However,
> I am not sure how it will behave with your use case that processes
other
> foreact leads. I only have the single file you sent me, so I can't
test
> this myself. You could get the updates and see if you are able to
process
> your data this way. Here are some configurations I used to get the
24 hr
> lead to run:
>
> [config]
> FCST_PCP_COMBINE_METHOD = ADD
> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> FCST_PCP_COMBINE_INPUT_OPTIONS = valid_time="{valid?fmt=%Y%m%d_%H}";
> FCST_PCP_COMBINE_CONSTANT_INIT = True
>
> [filename_templates]
> FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
> FCST_PCP_COMBINE_OUTPUT_TEMPLATE = ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
>
> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
>> Transaction: Given to mccabe (George McCabe) by RT_System
>> Queue: met_help
>> Subject: Re: valid time from grib data
>> Owner: mccabe
>> Requestors: ying.lin at noaa.gov
>> Status: deleted
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>>
>>
>> This transaction appears to have no content
>>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Re: valid time from grib data
From: George McCabe
Time: Thu Mar 05 16:33:04 2020
Hi Ying,
I didn't realize the file you sent me had all of the data needed for
your
run in one file. I reconfigured and was able to get it to run
correctly!
Attached is the config file I used to get it to process all of the
forecast
leads. I had to change it to loop by INIT time so the init time stayed
constant through the runs, but the valid time was updated for each
run.
Regarding TPRATE, MET should read the data with the TPRATE name and
can
still treat it as APCP. Changing the output field name to APCP would
make
this more clear. I think MET only cares about which record to read in,
regardless of the incorrect name. Maybe I am misunderstanding the
issue.
That is great news about the progress using wgrib2. If pygrib has the
same
functionality used in wgrib2 (which I would think it does), then you
could
use Python Embedding in MET and write a python script to reformat the
data
and pass it into the MET tools. I can help you with this process is
that is
possible.
On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
> Hi George,
>
> That is good news. If you want to test it out, all the UKMO files
for
> that cycle were in that one file's parent directory:
> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
>
> I was writing you in another window earlier:
>
> In re the UKMO:
> Thank you for the github issues ticket. Having METplus handle the
one
> input file to eliminate the need for pre-processing is fine. Since
you
> mentioned that METplus 3.0beta2 does have support for
> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try, setting
in
> the model config file
>
> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>
> It did work.
>
> In re DWD: Wesley Ebisuzaki told me a way today to fixed the bad
> longitude problem (Lon2):
>
> -------------------------
> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid -grib
new.grb
>
> What does -set_int 3 60 359750000 do
>
> set 4 byte integer in Section 3, octet 60-63 with value 359750000
>
> ref
> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
> https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
>
> You could argue that ocets 60-63 should be an unsigned integer, but
wgrib2
> doesn't have a -set_uint.
> -------------------------
>
> Also, on the matter of TPRATE in the DWD data: setting the field
name to
> TPRATE in the METplus config won't work because the field is
actually
> APCP, not TPRATE as the grib header says. It's off by a factor of
> 86,400. I'll do some experiment with the DWD.
>
> Thanks,
>
> Ying
>
> On 3/5/20 4:25 PM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > I implemented the changes needed for PCPCombine to set the options
the
> way
> > your case does using ADD. They are available in the develop branch
of the
> > METplus repository on GitHub. If you would prefer to obtain a
release, we
> > are putting out METplus 3.0beta4 on Friday (tomorrow). I tested
out this
> > case using the UKMO file you sent me processing the 24 hour lead.
> However,
> > I am not sure how it will behave with your use case that processes
other
> > foreact leads. I only have the single file you sent me, so I can't
test
> > this myself. You could get the updates and see if you are able to
process
> > your data this way. Here are some configurations I used to get the
24 hr
> > lead to run:
> >
> > [config]
> > FCST_PCP_COMBINE_METHOD = ADD
> > FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
> > FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> > FCST_PCP_COMBINE_INPUT_LEVELS = L0
> > FCST_PCP_COMBINE_INPUT_OPTIONS =
valid_time="{valid?fmt=%Y%m%d_%H}";
> > FCST_PCP_COMBINE_CONSTANT_INIT = True
> >
> > [filename_templates]
> > FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
> > FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
> >
> > On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
> >> Transaction: Given to mccabe (George McCabe) by RT_System
> >> Queue: met_help
> >> Subject: Re: valid time from grib data
> >> Owner: mccabe
> >> Requestors: ying.lin at noaa.gov
> >> Status: deleted
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
> >
> >>
> >>
> >> This transaction appears to have no content
> >>
> >
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: valid time from grib data
From: Ying Lin
Time: Fri Mar 06 10:43:25 2020
Thanks George. You made a good point about the data name of the
forecast field vs. what it actually represented. I was thinking that
MET would have looked at the DWD's GRIB header "TPRATE Total
Precipitation Rate [kg/m^2/s]" and convert that into 24h accumulation
(in the case of 24h verification) by multiplying it by 86,400 (which
would have translated, e.g. a one-day global max rainfall of 236mm to
20,390m). But come to think of it what you said had to be the case,
it
seems highly unlikely that MET would have examined the unit encoded in
the grib header and then do a conversion accordingly, without
additional
user input specified in the config file.
Which led to a related but separate question: if the input data is
"TP=Total precipitation [m]" (ECMWF), then I ought to give
instructions
in the config file telling GridStat that the unit of forecast is m
while
the unit of analysis (i.e. observation) is kg/m^2 (equivalent to mm).
Is there something in the METplus config file instructing GridStat to
first multiply the amounts in input forecast field by a factor of,
say,
1,000? For the contingency table scores, I can see doing this via the
thresholds:
BOTH_VAR1_LEVELS = A24
FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635, gt0.0127,
gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762, gt0.1016
OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05, gt25.4,
gt38.1, gt50.8, gt76.2, gt101.6
That ought to do it for the scores with thresholds. But what about
the
SL1L2 scores (e.g. the Fbar vs. Obar - comparing the average forecast
precip vs. observed precip)? How would MET know that the FCST field
need to be multiplied by 1,000 before being compared to the OBS?
Attached are 1) the general GridStat config for the CTC and SL1L2
stats
and 2) config for a specific model (ECMWF).
This has branched out pretty far from the original subject of the
met_help ticket. Do you prefer keeping it all a long-running ticket,
or
should I open up a separate met_help ticket, with a more relevant subj
line?
Thank you -
Ying
On 3/5/20 6:33 PM, George McCabe via RT wrote:
> Hi Ying,
>
> I didn't realize the file you sent me had all of the data needed for
your
> run in one file. I reconfigured and was able to get it to run
correctly!
> Attached is the config file I used to get it to process all of the
forecast
> leads. I had to change it to loop by INIT time so the init time
stayed
> constant through the runs, but the valid time was updated for each
run.
>
> Regarding TPRATE, MET should read the data with the TPRATE name and
can
> still treat it as APCP. Changing the output field name to APCP would
make
> this more clear. I think MET only cares about which record to read
in,
> regardless of the incorrect name. Maybe I am misunderstanding the
issue.
>
> That is great news about the progress using wgrib2. If pygrib has
the same
> functionality used in wgrib2 (which I would think it does), then you
could
> use Python Embedding in MET and write a python script to reformat
the data
> and pass it into the MET tools. I can help you with this process is
that is
> possible.
>
> On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> Hi George,
>>
>> That is good news. If you want to test it out, all the UKMO files
for
>> that cycle were in that one file's parent directory:
>> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
>>
>> I was writing you in another window earlier:
>>
>> In re the UKMO:
>> Thank you for the github issues ticket. Having METplus handle the
one
>> input file to eliminate the need for pre-processing is fine. Since
you
>> mentioned that METplus 3.0beta2 does have support for
>> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try, setting
in
>> the model config file
>>
>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>
>> It did work.
>>
>> In re DWD: Wesley Ebisuzaki told me a way today to fixed the bad
>> longitude problem (Lon2):
>>
>> -------------------------
>> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid -grib
new.grb
>>
>> What does -set_int 3 60 359750000 do
>>
>> set 4 byte integer in Section 3, octet 60-63 with value 359750000
>>
>> ref
>> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
>> https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
>>
>> You could argue that ocets 60-63 should be an unsigned integer, but
wgrib2
>> doesn't have a -set_uint.
>> -------------------------
>>
>> Also, on the matter of TPRATE in the DWD data: setting the field
name to
>> TPRATE in the METplus config won't work because the field is
actually
>> APCP, not TPRATE as the grib header says. It's off by a factor of
>> 86,400. I'll do some experiment with the DWD.
>>
>> Thanks,
>>
>> Ying
>>
>> On 3/5/20 4:25 PM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> I implemented the changes needed for PCPCombine to set the options
the
>> way
>>> your case does using ADD. They are available in the develop branch
of the
>>> METplus repository on GitHub. If you would prefer to obtain a
release, we
>>> are putting out METplus 3.0beta4 on Friday (tomorrow). I tested
out this
>>> case using the UKMO file you sent me processing the 24 hour lead.
>> However,
>>> I am not sure how it will behave with your use case that processes
other
>>> foreact leads. I only have the single file you sent me, so I can't
test
>>> this myself. You could get the updates and see if you are able to
process
>>> your data this way. Here are some configurations I used to get the
24 hr
>>> lead to run:
>>>
>>> [config]
>>> FCST_PCP_COMBINE_METHOD = ADD
>>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>>> FCST_PCP_COMBINE_INPUT_OPTIONS =
valid_time="{valid?fmt=%Y%m%d_%H}";
>>> FCST_PCP_COMBINE_CONSTANT_INIT = True
>>>
>>> [filename_templates]
>>> FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
>>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
>>>
>>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
>>>> Transaction: Given to mccabe (George McCabe) by RT_System
>>>> Queue: met_help
>>>> Subject: Re: valid time from grib data
>>>> Owner: mccabe
>>>> Requestors: ying.lin at noaa.gov
>>>> Status: deleted
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
>>>>
>>>> This transaction appears to have no content
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Re: valid time from grib data
From: Ying Lin
Time: Fri Mar 06 10:43:25 2020
////////////////////////////////////////////////////////////////////////////////
//
// Grid-Stat configuration file.
//
// For additional information, see the MET_BASE/config/README file.
//
////////////////////////////////////////////////////////////////////////////////
//
// Output model name to be written
//
model = "${MODEL}";
//
// Output description to be written
// May be set separately in each "obs.field" entry
//
desc = "NA";
//
// Output observation type to be written
//
obtype = "${OBTYPE}";
////////////////////////////////////////////////////////////////////////////////
//
// Verification grid
//
regrid = {
to_grid = OBS;
method = NEAREST;
width = 1;
vld_thresh = 0.5;
shape = SQUARE;
}
////////////////////////////////////////////////////////////////////////////////
censor_thresh = [];
censor_val = [];
cat_thresh = [ NA ];
cnt_thresh = [ NA ];
cnt_logic = UNION;
wind_thresh = [ NA ];
wind_logic = UNION;
eclv_points = 0.05;
nc_pairs_var_str = "";
rank_corr_flag = FALSE;
//
// Forecast and observation fields to be verified
//
fcst = {
field = [ ${FCST_FIELD} ];
}
obs = {
field = [ ${OBS_FIELD} ];
}
////////////////////////////////////////////////////////////////////////////////
//
// Climatology mean data
//
climo_mean = {
file_name = [];
field = [];
regrid = {
method = NEAREST;
width = 1;
vld_thresh = 0.5;
shape = SQUARE;
}
time_interp_method = DW_MEAN;
match_month = TRUE;
match_day = FALSE;
time_step = 21600;
}
climo_stdev = climo_mean;
climo_stdev = {
file_name = [];
}
climo_cdf_bins = 1;
////////////////////////////////////////////////////////////////////////////////
//
// Verification masking regions
//
mask = {
grid = [];
poly = [ "${polydir}/CONUS.nc", "${polydir}/APL.nc",
"${polydir}/GMC.nc", "${polydir}/GRB.nc", "${polydir}/LMV.nc",
"${polydir}/MDW.nc", "${polydir}/NEC.nc", "${polydir}/NMT.nc",
"${polydir}/NPL.nc", "${polydir}/NWC.nc", "${polydir}/SEC.nc",
"${polydir}/SMT.nc", "${polydir}/SPL.nc", "${polydir}/SWC.nc",
"${polydir}/SWD.nc"];
}
////////////////////////////////////////////////////////////////////////////////
//
// Confidence interval settings
//
ci_alpha = [ 0.05 ];
boot = {
interval = PCTILE;
rep_prop = 1.0;
n_rep = 0;
rng = "mt19937";
seed = "";
}
////////////////////////////////////////////////////////////////////////////////
//
// Data smoothing methods
//
interp = {
field = BOTH;
vld_thresh = 1.0;
shape = SQUARE;
type = [
{
method = NEAREST;
width = 1;
}
];
}
////////////////////////////////////////////////////////////////////////////////
//
// Neighborhood methods
//
nbrhd = {
field = BOTH;
shape = SQUARE;
width = [ 1 ];
cov_thresh = [ >=0.5 ];
vld_thresh = 1.0;
}
////////////////////////////////////////////////////////////////////////////////
//
// Fourier decomposition
// May be set separately in each "obs.field" entry
//
fourier = {
wave_1d_beg = [];
wave_1d_end = [];
}
////////////////////////////////////////////////////////////////////////////////
//
// Gradient statistics
// May be set separately in each "obs.field" entry
//
gradient = {
dx = [ 1 ];
dy = [ 1 ];
}
////////////////////////////////////////////////////////////////////////////////
//
// Statistical output types
//
output_flag = {
fho = NONE;
ctc = BOTH;
cts = BOTH;
mctc = NONE;
mcts = NONE;
cnt = NONE;
sl1l2 = BOTH;
sal1l2 = NONE;
vl1l2 = NONE;
val1l2 = NONE;
vcnt = NONE;
pct = NONE;
pstd = NONE;
pjc = NONE;
prc = NONE;
eclv = NONE;
nbrctc = NONE;
nbrcts = NONE;
nbrcnt = NONE;
grad = NONE;
}
//
// NetCDF matched pairs output file
// May be set separately in each "obs.field" entry
//
nc_pairs_flag = {
latlon = FALSE;
raw = FALSE;
diff = FALSE;
climo = FALSE;
weight = FALSE;
nbrhd = FALSE;
fourier = FALSE;
gradient = FALSE;
apply_mask = FALSE;
}
////////////////////////////////////////////////////////////////////////////////
grid_weight_flag = NONE;
tmp_dir = "/tmp";
output_prefix = "ctc_sl1l2_${acc}_${MODEL}_${OBTYPE}";
//version = "V8.1";
////////////////////////////////////////////////////////////////////////////////
------------------------------------------------
Subject: Re: valid time from grib data
From: Ying Lin
Time: Fri Mar 06 10:43:25 2020
# Grid to Grid Precipitation Example
[config]
# time looping - options are INIT, VALID, RETRO, and REALTIME
LOOP_BY = VALID
# Format of VALID_BEG and VALID_END
VALID_TIME_FMT = %Y%m%d%H
# Start time for METplus run
VALID_BEG = {ENV[vday]}12
#
# End time for METplus run
VALID_END = {ENV[vday]}12
#VALID_END = {now?fmt=%Y%m%d}12
YLMETPLUS_PATH = {ENV[YLMETPLUS_PATH]}
# Increment between METplus runs in seconds. Must be >= 60
# 86400 sec=24h
VALID_INCREMENT = 86400
# Options are times, processes
# times = run all items in the PROCESS_LIST for a single
initialization
# time, then repeat until all times have been evaluated.
# processes = run each item in the PROCESS_LIST for all times
# specified, then repeat for the next item in the PROCESS_LIST.
LOOP_ORDER = times
# List of applications to run
PROCESS_LIST = PcpCombine, RegridDataPlane, GridStat
# run pcp_combine on forecast/obs data?
FCST_PCP_COMBINE_RUN = False
OBS_PCP_COMBINE_RUN = True
OBS_REGRID_DATA_PLANE_RUN = True
# mode of pcp_combine to use (SUM, ADD, SUBTRACT)
OBS_PCP_COMBINE_METHOD = ADD
# If 'bucket' or 'regrid' output already exists, skip the PcpCombine
and Regrid
# for the data (e.g. verifying analysis only processed once when
multiple
# models are being verified)
PCP_COMBINE_SKIP_IF_OUTPUT_EXISTS = True
REGRID_DATA_PLANE_SKIP_IF_OUTPUT_EXISTS = True
# list of variables to compare
FCST_VAR1_NAME = TP
OBS_VAR1_NAME = APCP
# Can specify 'record number in a grib file' -
FCST_VAR1_LEVELS = R001
OBS_VAR1_LEVELS = A24
BOTH_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05, gt25.4,
gt38.1, gt50.8, gt76.2, gt101.6
LEAD_SEQ = 24,48,72
# description of data to be processed
# used in output file path
MODEL = {ENV[MODEL]}
model = {ENV[model]}
modpath = {ENV[modpath]}
ccpapath = {ENV[ccpapath]}
OBTYPE = CCPA
# Used by regrid_data_plane to remap data
REGRID_DATA_PLANE_VERIF_GRID={INPUT_BASE}/qpf/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
# method to run regrid_data_plane, not setting this will default to
NEAREST
REGRID_DATA_PLANE_METHOD = BUDGET
# regridding width used in regrid_data_plane, not setting this will
default to 1
REGRID_DATA_PLANE_WIDTH = 2
# location of grid_stat MET config file
GRID_STAT_CONFIG_FILE =
{YLMETPLUS_PATH}/yl/parm/GridStatConfig_ctc_sl1l2
# Forecast data description variables
FCST_PCP_COMBINE_INPUT_DATATYPE = GRIB
FCST_IS_PROB = false
# Set to true if data is only available once per day
FCST_PCP_COMBINE_IS_DAILY_FILE = false
# File format. Options are GRIB, NETCDF, or GEMPAK
OBS_PCP_COMBINE_INPUT_DATATYPE = GRIB
# Set to true if data is only available once per day
OBS_PCP_COMBINE_IS_DAILY_FILE = false
# Accumulation interval available in observation data
OBS_PCP_COMBINE_INPUT_ACCUMS = 06
[dir]
# input and output data directories
OBS_PCP_COMBINE_INPUT_DIR = {ccpapath}
OBS_PCP_COMBINE_OUTPUT_DIR = {OUTPUT_BASE}/ccpa/bucket
OBS_REGRID_DATA_PLANE_INPUT_DIR = {OBS_PCP_COMBINE_OUTPUT_DIR}
OBS_REGRID_DATA_PLANE_OUTPUT_DIR = {OUTPUT_BASE}/ccpa/regrid
OBS_GRID_STAT_INPUT_DIR = {OBS_REGRID_DATA_PLANE_OUTPUT_DIR}
GRID_STAT_OUTPUT_DIR = {STATS_BASE}
# location of configuration files used by MET applications
CONFIG_DIR={PARM_BASE}/use_cases/grid_to_grid/met_config
[filename_templates]
# format of filenames
FCST_GRID_STAT_INPUT_TEMPLATE =
{modpath}/{init?fmt=%Y%m%d}/qpf_verif/UWD{init?fmt=%Y%m%d%H}00{valid?fmt=%m%d%H}001
# ANLYS
OBS_PCP_COMBINE_INPUT_TEMPLATE =
ccpa.{valid?fmt=%Y%m%d}/{valid?fmt=%H}/ccpa.t{valid?fmt=%H}z.06h.hrap.conus.gb2
OBS_PCP_COMBINE_OUTPUT_TEMPLATE =
ccpa.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h
OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE =
{OBS_PCP_COMBINE_OUTPUT_TEMPLATE}
OBS_REGRID_DATA_PLANE_OUTPUT_TEMPLATE =
ccpa.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h.g212
OBS_GRID_STAT_INPUT_TEMPLATE = {OBS_REGRID_DATA_PLANE_OUTPUT_TEMPLATE}
# grid_stat output:
GRID_STAT_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d}
------------------------------------------------
Subject: Re: valid time from grib data
From: George McCabe
Time: Fri Mar 06 11:06:20 2020
Hi Ying,
MET has the capability to convert units within the config files. See
this
excerpt from the MET User's Guide (Chapter 3 - MET Data I/O):
// - The "convert" entry is a user-defined function of a single
variable
// for processing input data values. Any input values that are not bad
// data are replaced by the value of this function. The convert
function
// is applied prior to regridding or thresholding. This function may
// include any of the built-in math functions (e.g. sqrt, log10)
// described above.
// Several standard unit conversion functions are already defined in
// data/config/ConfigConstants.
// Examples of user-defined conversion functions include:
// convert(x) = 2*x;
// convert(x) = x^2;
// convert(a) = log10(a);
// convert(a) = a^10;
// convert(t) = max(1, sqrt(abs(t)));
// convert(x) = K_to_C(x); where K_to_C(x) is defined in
// ConfigConstants
You can set the conversion via METplus by setting:
FCST_VAR1_OPTIONS = convert(x) = x*1000;
Let me know if that doesn't work.
Regarding the long ticket, I am not sure what the preferred protocol
of the
team for the ticking system. I personally don't mind continuing work
on a
single thread so someone else don't have to go through the steps to
read it
and assign it to me each time. I will make a note to bring it up in
the
next meeting and I'll see what they say.
Thanks,
George
On Fri, Mar 6, 2020 at 10:44 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>
> Thanks George. You made a good point about the data name of the
> forecast field vs. what it actually represented. I was thinking
that
> MET would have looked at the DWD's GRIB header "TPRATE Total
> Precipitation Rate [kg/m^2/s]" and convert that into 24h
accumulation
> (in the case of 24h verification) by multiplying it by 86,400 (which
> would have translated, e.g. a one-day global max rainfall of 236mm
to
> 20,390m). But come to think of it what you said had to be the case,
it
> seems highly unlikely that MET would have examined the unit encoded
in
> the grib header and then do a conversion accordingly, without
additional
> user input specified in the config file.
>
> Which led to a related but separate question: if the input data is
> "TP=Total precipitation [m]" (ECMWF), then I ought to give
instructions
> in the config file telling GridStat that the unit of forecast is m
while
> the unit of analysis (i.e. observation) is kg/m^2 (equivalent to
mm).
> Is there something in the METplus config file instructing GridStat
to
> first multiply the amounts in input forecast field by a factor of,
say,
> 1,000? For the contingency table scores, I can see doing this via
the
> thresholds:
>
> BOTH_VAR1_LEVELS = A24
> FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635, gt0.0127,
> gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762, gt0.1016
> OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05, gt25.4,
> gt38.1, gt50.8, gt76.2, gt101.6
>
> That ought to do it for the scores with thresholds. But what about
the
> SL1L2 scores (e.g. the Fbar vs. Obar - comparing the average
forecast
> precip vs. observed precip)? How would MET know that the FCST field
> need to be multiplied by 1,000 before being compared to the OBS?
>
> Attached are 1) the general GridStat config for the CTC and SL1L2
stats
> and 2) config for a specific model (ECMWF).
>
> This has branched out pretty far from the original subject of the
> met_help ticket. Do you prefer keeping it all a long-running
ticket, or
> should I open up a separate met_help ticket, with a more relevant
subj
> line?
>
> Thank you -
>
> Ying
>
>
> On 3/5/20 6:33 PM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > I didn't realize the file you sent me had all of the data needed
for your
> > run in one file. I reconfigured and was able to get it to run
correctly!
> > Attached is the config file I used to get it to process all of the
> forecast
> > leads. I had to change it to loop by INIT time so the init time
stayed
> > constant through the runs, but the valid time was updated for each
run.
> >
> > Regarding TPRATE, MET should read the data with the TPRATE name
and can
> > still treat it as APCP. Changing the output field name to APCP
would make
> > this more clear. I think MET only cares about which record to read
in,
> > regardless of the incorrect name. Maybe I am misunderstanding the
issue.
> >
> > That is great news about the progress using wgrib2. If pygrib has
the
> same
> > functionality used in wgrib2 (which I would think it does), then
you
> could
> > use Python Embedding in MET and write a python script to reformat
the
> data
> > and pass it into the MET tools. I can help you with this process
is that
> is
> > possible.
> >
> > On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> Hi George,
> >>
> >> That is good news. If you want to test it out, all the UKMO
files for
> >> that cycle were in that one file's parent directory:
> >> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
> >>
> >> I was writing you in another window earlier:
> >>
> >> In re the UKMO:
> >> Thank you for the github issues ticket. Having METplus handle
the one
> >> input file to eliminate the need for pre-processing is fine.
Since you
> >> mentioned that METplus 3.0beta2 does have support for
> >> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try,
setting in
> >> the model config file
> >>
> >> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> >> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> >>
> >> It did work.
> >>
> >> In re DWD: Wesley Ebisuzaki told me a way today to fixed the bad
> >> longitude problem (Lon2):
> >>
> >> -------------------------
> >> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid
-grib
> new.grb
> >>
> >> What does -set_int 3 60 359750000 do
> >>
> >> set 4 byte integer in Section 3, octet 60-63 with value 359750000
> >>
> >> ref
> >>
> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
> >> https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
> >>
> >> You could argue that ocets 60-63 should be an unsigned integer,
but
> wgrib2
> >> doesn't have a -set_uint.
> >> -------------------------
> >>
> >> Also, on the matter of TPRATE in the DWD data: setting the field
name to
> >> TPRATE in the METplus config won't work because the field is
actually
> >> APCP, not TPRATE as the grib header says. It's off by a factor of
> >> 86,400. I'll do some experiment with the DWD.
> >>
> >> Thanks,
> >>
> >> Ying
> >>
> >> On 3/5/20 4:25 PM, George McCabe via RT wrote:
> >>> Hi Ying,
> >>>
> >>> I implemented the changes needed for PCPCombine to set the
options the
> >> way
> >>> your case does using ADD. They are available in the develop
branch of
> the
> >>> METplus repository on GitHub. If you would prefer to obtain a
release,
> we
> >>> are putting out METplus 3.0beta4 on Friday (tomorrow). I tested
out
> this
> >>> case using the UKMO file you sent me processing the 24 hour
lead.
> >> However,
> >>> I am not sure how it will behave with your use case that
processes
> other
> >>> foreact leads. I only have the single file you sent me, so I
can't test
> >>> this myself. You could get the updates and see if you are able
to
> process
> >>> your data this way. Here are some configurations I used to get
the 24
> hr
> >>> lead to run:
> >>>
> >>> [config]
> >>> FCST_PCP_COMBINE_METHOD = ADD
> >>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
> >>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> >>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> >>> FCST_PCP_COMBINE_INPUT_OPTIONS =
valid_time="{valid?fmt=%Y%m%d_%H}";
> >>> FCST_PCP_COMBINE_CONSTANT_INIT = True
> >>>
> >>> [filename_templates]
> >>> FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
> >>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
> >>>
> >>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT <
> >>> met_help at ucar.edu> wrote:
> >>>
> >>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
> >>>> Transaction: Given to mccabe (George McCabe) by RT_System
> >>>> Queue: met_help
> >>>> Subject: Re: valid time from grib data
> >>>> Owner: mccabe
> >>>> Requestors: ying.lin at noaa.gov
> >>>> Status: deleted
> >>>> Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
> >>>>
> >>>> This transaction appears to have no content
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94354] Re: valid time from grib data
From: Ying Lin
Time: Fri Mar 06 13:13:28 2020
Thank you very much George! This is really helpful. "
FCST_VAR1_OPTIONS = convert(x) = x*1000
does work.
One more question: for ECMWF, I have
FCST_VAR1_NAME = TP
for UKMO, it is
FCST_VAR1_NAME = TWATP
All the other models have
BOTH_VAR1_NAME = APCP
(the analysis/obs is always APCP, and for now I'm converting DWD
forecast to APCP rather than using the original wrongly named TPRATE).
After loading the stats to METviewer server, all the scores look
reasonable individually, but they can't all be compared together,
since
the "Y1 Dependent (Forecast) Variables" are APCP_24, TWATP_24 and TP,
there doesn't seem to be a way to combine them. This might be a
METviewer issue, but I'm wondering: is there a way in METplus config
to
specify that, even though the FCST_VAR1_NAME is something other than
APCP, the FCST_VAR in the GridStat ought to be APCP? This is not a
request for METplus to accommodate yet another oddball input feature,
just wondering if there's already something in the existing config set
up for such a thing.
Thanks again -
Ying
On 3/6/20 1:06 PM, George McCabe via RT wrote:
> Hi Ying,
>
> MET has the capability to convert units within the config files. See
this
> excerpt from the MET User's Guide (Chapter 3 - MET Data I/O):
>
> // - The "convert" entry is a user-defined function of a single
variable
> // for processing input data values. Any input values that are not
bad
> // data are replaced by the value of this function. The convert
function
> // is applied prior to regridding or thresholding. This function may
> // include any of the built-in math functions (e.g. sqrt, log10)
> // described above.
> // Several standard unit conversion functions are already defined in
> // data/config/ConfigConstants.
> // Examples of user-defined conversion functions include:
> // convert(x) = 2*x;
> // convert(x) = x^2;
> // convert(a) = log10(a);
> // convert(a) = a^10;
> // convert(t) = max(1, sqrt(abs(t)));
> // convert(x) = K_to_C(x); where K_to_C(x) is defined in
> // ConfigConstants
>
> You can set the conversion via METplus by setting:
>
> FCST_VAR1_OPTIONS = convert(x) = x*1000;
>
> Let me know if that doesn't work.
>
> Regarding the long ticket, I am not sure what the preferred protocol
of the
> team for the ticking system. I personally don't mind continuing work
on a
> single thread so someone else don't have to go through the steps to
read it
> and assign it to me each time. I will make a note to bring it up in
the
> next meeting and I'll see what they say.
>
> Thanks,
> George
>
> On Fri, Mar 6, 2020 at 10:44 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>>
>> Thanks George. You made a good point about the data name of the
>> forecast field vs. what it actually represented. I was thinking
that
>> MET would have looked at the DWD's GRIB header "TPRATE Total
>> Precipitation Rate [kg/m^2/s]" and convert that into 24h
accumulation
>> (in the case of 24h verification) by multiplying it by 86,400
(which
>> would have translated, e.g. a one-day global max rainfall of 236mm
to
>> 20,390m). But come to think of it what you said had to be the
case, it
>> seems highly unlikely that MET would have examined the unit encoded
in
>> the grib header and then do a conversion accordingly, without
additional
>> user input specified in the config file.
>>
>> Which led to a related but separate question: if the input data
is
>> "TP=Total precipitation [m]" (ECMWF), then I ought to give
instructions
>> in the config file telling GridStat that the unit of forecast is m
while
>> the unit of analysis (i.e. observation) is kg/m^2 (equivalent to
mm).
>> Is there something in the METplus config file instructing GridStat
to
>> first multiply the amounts in input forecast field by a factor of,
say,
>> 1,000? For the contingency table scores, I can see doing this via
the
>> thresholds:
>>
>> BOTH_VAR1_LEVELS = A24
>> FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635, gt0.0127,
>> gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762, gt0.1016
>> OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05, gt25.4,
>> gt38.1, gt50.8, gt76.2, gt101.6
>>
>> That ought to do it for the scores with thresholds. But what about
the
>> SL1L2 scores (e.g. the Fbar vs. Obar - comparing the average
forecast
>> precip vs. observed precip)? How would MET know that the FCST
field
>> need to be multiplied by 1,000 before being compared to the OBS?
>>
>> Attached are 1) the general GridStat config for the CTC and SL1L2
stats
>> and 2) config for a specific model (ECMWF).
>>
>> This has branched out pretty far from the original subject of the
>> met_help ticket. Do you prefer keeping it all a long-running
ticket, or
>> should I open up a separate met_help ticket, with a more relevant
subj
>> line?
>>
>> Thank you -
>>
>> Ying
>>
>>
>> On 3/5/20 6:33 PM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> I didn't realize the file you sent me had all of the data needed
for your
>>> run in one file. I reconfigured and was able to get it to run
correctly!
>>> Attached is the config file I used to get it to process all of the
>> forecast
>>> leads. I had to change it to loop by INIT time so the init time
stayed
>>> constant through the runs, but the valid time was updated for each
run.
>>>
>>> Regarding TPRATE, MET should read the data with the TPRATE name
and can
>>> still treat it as APCP. Changing the output field name to APCP
would make
>>> this more clear. I think MET only cares about which record to read
in,
>>> regardless of the incorrect name. Maybe I am misunderstanding the
issue.
>>>
>>> That is great news about the progress using wgrib2. If pygrib has
the
>> same
>>> functionality used in wgrib2 (which I would think it does), then
you
>> could
>>> use Python Embedding in MET and write a python script to reformat
the
>> data
>>> and pass it into the MET tools. I can help you with this process
is that
>> is
>>> possible.
>>>
>>> On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> Hi George,
>>>>
>>>> That is good news. If you want to test it out, all the UKMO
files for
>>>> that cycle were in that one file's parent directory:
>>>> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
>>>>
>>>> I was writing you in another window earlier:
>>>>
>>>> In re the UKMO:
>>>> Thank you for the github issues ticket. Having METplus handle
the one
>>>> input file to eliminate the need for pre-processing is fine.
Since you
>>>> mentioned that METplus 3.0beta2 does have support for
>>>> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try,
setting in
>>>> the model config file
>>>>
>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>>>
>>>> It did work.
>>>>
>>>> In re DWD: Wesley Ebisuzaki told me a way today to fixed the bad
>>>> longitude problem (Lon2):
>>>>
>>>> -------------------------
>>>> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid
-grib
>> new.grb
>>>> What does -set_int 3 60 359750000 do
>>>>
>>>> set 4 byte integer in Section 3, octet 60-63 with value 359750000
>>>>
>>>> ref
>>>>
>> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
>>>> https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
>>>>
>>>> You could argue that ocets 60-63 should be an unsigned integer,
but
>> wgrib2
>>>> doesn't have a -set_uint.
>>>> -------------------------
>>>>
>>>> Also, on the matter of TPRATE in the DWD data: setting the field
name to
>>>> TPRATE in the METplus config won't work because the field is
actually
>>>> APCP, not TPRATE as the grib header says. It's off by a factor of
>>>> 86,400. I'll do some experiment with the DWD.
>>>>
>>>> Thanks,
>>>>
>>>> Ying
>>>>
>>>> On 3/5/20 4:25 PM, George McCabe via RT wrote:
>>>>> Hi Ying,
>>>>>
>>>>> I implemented the changes needed for PCPCombine to set the
options the
>>>> way
>>>>> your case does using ADD. They are available in the develop
branch of
>> the
>>>>> METplus repository on GitHub. If you would prefer to obtain a
release,
>> we
>>>>> are putting out METplus 3.0beta4 on Friday (tomorrow). I tested
out
>> this
>>>>> case using the UKMO file you sent me processing the 24 hour
lead.
>>>> However,
>>>>> I am not sure how it will behave with your use case that
processes
>> other
>>>>> foreact leads. I only have the single file you sent me, so I
can't test
>>>>> this myself. You could get the updates and see if you are able
to
>> process
>>>>> your data this way. Here are some configurations I used to get
the 24
>> hr
>>>>> lead to run:
>>>>>
>>>>> [config]
>>>>> FCST_PCP_COMBINE_METHOD = ADD
>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>>>>> FCST_PCP_COMBINE_INPUT_OPTIONS =
valid_time="{valid?fmt=%Y%m%d_%H}";
>>>>> FCST_PCP_COMBINE_CONSTANT_INIT = True
>>>>>
>>>>> [filename_templates]
>>>>> FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
>>>>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
>>>>>
>>>>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT <
>>>>> met_help at ucar.edu> wrote:
>>>>>
>>>>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
>>>>>> Transaction: Given to mccabe (George McCabe) by RT_System
>>>>>> Queue: met_help
>>>>>> Subject: Re: valid time from grib data
>>>>>> Owner: mccabe
>>>>>> Requestors: ying.lin at noaa.gov
>>>>>> Status: deleted
>>>>>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
>>>>>> This transaction appears to have no content
>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94354] Re: valid time from grib data
From: Ying Lin
Time: Fri Mar 06 14:47:27 2020
Hi George,
Found out that METviewer server could indeed include multiple
FCST_VAR1_NAMEs, though it doesn't quite do what I meant for it to do:
it was searching for stats for all models/all FCST_VAR1_NAME combos,
not
just the existing combos.
But turns out it's very easy to do a 'sed' string substitution in the
output stats before loading to the METviewer server, that solves the
problem.
I might add this to the Monday MET user meeting discussion list to see
if there is a METviewer solution. Though so far the 'sed' solution
seems
to work fine. Anyway, this doesn't to be a METplus issue.
Thanks,
Ying
On 3/6/20 3:13 PM, Ying Lin wrote:
> Thank you very much George! This is really helpful. "
>
> FCST_VAR1_OPTIONS = convert(x) = x*1000
>
> does work.
>
> One more question: for ECMWF, I have
> FCST_VAR1_NAME = TP
>
> for UKMO, it is
> FCST_VAR1_NAME = TWATP
>
> All the other models have
> BOTH_VAR1_NAME = APCP
>
> (the analysis/obs is always APCP, and for now I'm converting DWD
> forecast to APCP rather than using the original wrongly named
TPRATE).
>
> After loading the stats to METviewer server, all the scores look
> reasonable individually, but they can't all be compared together,
> since the "Y1 Dependent (Forecast) Variables" are APCP_24, TWATP_24
> and TP, there doesn't seem to be a way to combine them. This might
be
> a METviewer issue, but I'm wondering: is there a way in METplus
config
> to specify that, even though the FCST_VAR1_NAME is something other
> than APCP, the FCST_VAR in the GridStat ought to be APCP? This is
not
> a request for METplus to accommodate yet another oddball input
> feature, just wondering if there's already something in the existing
> config set up for such a thing.
>
> Thanks again -
>
> Ying
>
> On 3/6/20 1:06 PM, George McCabe via RT wrote:
>> Hi Ying,
>>
>> MET has the capability to convert units within the config files.
See
>> this
>> excerpt from the MET User's Guide (Chapter 3 - MET Data I/O):
>>
>> // - The "convert" entry is a user-defined function of a single
variable
>> // for processing input data values. Any input values that are not
bad
>> // data are replaced by the value of this function. The convert
function
>> // is applied prior to regridding or thresholding. This function
may
>> // include any of the built-in math functions (e.g. sqrt, log10)
>> // described above.
>> // Several standard unit conversion functions are already defined
in
>> // data/config/ConfigConstants.
>> // Examples of user-defined conversion functions include:
>> // convert(x) = 2*x;
>> // convert(x) = x^2;
>> // convert(a) = log10(a);
>> // convert(a) = a^10;
>> // convert(t) = max(1, sqrt(abs(t)));
>> // convert(x) = K_to_C(x); where K_to_C(x) is defined in
>> // ConfigConstants
>>
>> You can set the conversion via METplus by setting:
>>
>> FCST_VAR1_OPTIONS = convert(x) = x*1000;
>>
>> Let me know if that doesn't work.
>>
>> Regarding the long ticket, I am not sure what the preferred
protocol
>> of the
>> team for the ticking system. I personally don't mind continuing
work
>> on a
>> single thread so someone else don't have to go through the steps to
>> read it
>> and assign it to me each time. I will make a note to bring it up in
the
>> next meeting and I'll see what they say.
>>
>> Thanks,
>> George
>>
>> On Fri, Mar 6, 2020 at 10:44 AM Ying Lin via RT <met_help at ucar.edu>
>> wrote:
>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>>>
>>> Thanks George. You made a good point about the data name of the
>>> forecast field vs. what it actually represented. I was thinking
that
>>> MET would have looked at the DWD's GRIB header "TPRATE Total
>>> Precipitation Rate [kg/m^2/s]" and convert that into 24h
accumulation
>>> (in the case of 24h verification) by multiplying it by 86,400
(which
>>> would have translated, e.g. a one-day global max rainfall of 236mm
to
>>> 20,390m). But come to think of it what you said had to be the
case, it
>>> seems highly unlikely that MET would have examined the unit
encoded in
>>> the grib header and then do a conversion accordingly, without
>>> additional
>>> user input specified in the config file.
>>>
>>> Which led to a related but separate question: if the input data
is
>>> "TP=Total precipitation [m]" (ECMWF), then I ought to give
instructions
>>> in the config file telling GridStat that the unit of forecast is m
>>> while
>>> the unit of analysis (i.e. observation) is kg/m^2 (equivalent to
mm).
>>> Is there something in the METplus config file instructing GridStat
to
>>> first multiply the amounts in input forecast field by a factor of,
say,
>>> 1,000? For the contingency table scores, I can see doing this via
the
>>> thresholds:
>>>
>>> BOTH_VAR1_LEVELS = A24
>>> FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635, gt0.0127,
>>> gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762, gt0.1016
>>> OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05,
gt25.4,
>>> gt38.1, gt50.8, gt76.2, gt101.6
>>>
>>> That ought to do it for the scores with thresholds. But what
about the
>>> SL1L2 scores (e.g. the Fbar vs. Obar - comparing the average
forecast
>>> precip vs. observed precip)? How would MET know that the FCST
field
>>> need to be multiplied by 1,000 before being compared to the OBS?
>>>
>>> Attached are 1) the general GridStat config for the CTC and SL1L2
stats
>>> and 2) config for a specific model (ECMWF).
>>>
>>> This has branched out pretty far from the original subject of the
>>> met_help ticket. Do you prefer keeping it all a long-running
>>> ticket, or
>>> should I open up a separate met_help ticket, with a more relevant
subj
>>> line?
>>>
>>> Thank you -
>>>
>>> Ying
>>>
>>>
>>> On 3/5/20 6:33 PM, George McCabe via RT wrote:
>>>> Hi Ying,
>>>>
>>>> I didn't realize the file you sent me had all of the data needed
>>>> for your
>>>> run in one file. I reconfigured and was able to get it to run
>>>> correctly!
>>>> Attached is the config file I used to get it to process all of
the
>>> forecast
>>>> leads. I had to change it to loop by INIT time so the init time
stayed
>>>> constant through the runs, but the valid time was updated for
each
>>>> run.
>>>>
>>>> Regarding TPRATE, MET should read the data with the TPRATE name
and
>>>> can
>>>> still treat it as APCP. Changing the output field name to APCP
>>>> would make
>>>> this more clear. I think MET only cares about which record to
read in,
>>>> regardless of the incorrect name. Maybe I am misunderstanding the
>>>> issue.
>>>>
>>>> That is great news about the progress using wgrib2. If pygrib has
the
>>> same
>>>> functionality used in wgrib2 (which I would think it does), then
you
>>> could
>>>> use Python Embedding in MET and write a python script to reformat
the
>>> data
>>>> and pass it into the MET tools. I can help you with this process
is
>>>> that
>>> is
>>>> possible.
>>>>
>>>> On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT
<met_help at ucar.edu>
>>> wrote:
>>>>> Hi George,
>>>>>
>>>>> That is good news. If you want to test it out, all the UKMO
files
>>>>> for
>>>>> that cycle were in that one file's parent directory:
>>>>> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
>>>>>
>>>>> I was writing you in another window earlier:
>>>>>
>>>>> In re the UKMO:
>>>>> Thank you for the github issues ticket. Having METplus handle
the
>>>>> one
>>>>> input file to eliminate the need for pre-processing is fine.
>>>>> Since you
>>>>> mentioned that METplus 3.0beta2 does have support for
>>>>> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try,
setting in
>>>>> the model config file
>>>>>
>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>>>>
>>>>> It did work.
>>>>>
>>>>> In re DWD: Wesley Ebisuzaki told me a way today to fixed the bad
>>>>> longitude problem (Lon2):
>>>>>
>>>>> -------------------------
>>>>> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid
-grib
>>> new.grb
>>>>> What does -set_int 3 60 359750000 do
>>>>>
>>>>> set 4 byte integer in Section 3, octet 60-63 with value
359750000
>>>>>
>>>>> ref
>>>>>
>>>
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
>>>
>>>>>
https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
>>>>>
>>>>> You could argue that ocets 60-63 should be an unsigned integer,
but
>>> wgrib2
>>>>> doesn't have a -set_uint.
>>>>> -------------------------
>>>>>
>>>>> Also, on the matter of TPRATE in the DWD data: setting the field
>>>>> name to
>>>>> TPRATE in the METplus config won't work because the field is
actually
>>>>> APCP, not TPRATE as the grib header says. It's off by a factor
of
>>>>> 86,400. I'll do some experiment with the DWD.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Ying
>>>>>
>>>>> On 3/5/20 4:25 PM, George McCabe via RT wrote:
>>>>>> Hi Ying,
>>>>>>
>>>>>> I implemented the changes needed for PCPCombine to set the
>>>>>> options the
>>>>> way
>>>>>> your case does using ADD. They are available in the develop
>>>>>> branch of
>>> the
>>>>>> METplus repository on GitHub. If you would prefer to obtain a
>>>>>> release,
>>> we
>>>>>> are putting out METplus 3.0beta4 on Friday (tomorrow). I tested
out
>>> this
>>>>>> case using the UKMO file you sent me processing the 24 hour
lead.
>>>>> However,
>>>>>> I am not sure how it will behave with your use case that
processes
>>> other
>>>>>> foreact leads. I only have the single file you sent me, so I
>>>>>> can't test
>>>>>> this myself. You could get the updates and see if you are able
to
>>> process
>>>>>> your data this way. Here are some configurations I used to get
>>>>>> the 24
>>> hr
>>>>>> lead to run:
>>>>>>
>>>>>> [config]
>>>>>> FCST_PCP_COMBINE_METHOD = ADD
>>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>>>>>> FCST_PCP_COMBINE_INPUT_OPTIONS =
valid_time="{valid?fmt=%Y%m%d_%H}";
>>>>>> FCST_PCP_COMBINE_CONSTANT_INIT = True
>>>>>>
>>>>>> [filename_templates]
>>>>>> FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
>>>>>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
>>>>>>
>>>>>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
>>>>>>> Transaction: Given to mccabe (George McCabe) by RT_System
>>>>>>> Queue: met_help
>>>>>>> Subject: Re: valid time from grib data
>>>>>>> Owner: mccabe
>>>>>>> Requestors: ying.lin at noaa.gov
>>>>>>> Status: deleted
>>>>>>> Ticket <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
>>>>>>> This transaction appears to have no content
>>>>>>>
>>>>> --
>>>>> Ying Lin
>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>> NCWCP Cubicle No. 2015
>>>>> Ying.Lin at noaa.gov
>>>>>
>>>>>
>>>>>
>>>>>
>>> --
>>> Ying Lin
>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>> NCWCP Cubicle No. 2015
>>> Ying.Lin at noaa.gov
>>>
>>>
>>>
>>>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Re: valid time from grib data
From: George McCabe
Time: Fri Mar 06 15:22:02 2020
Hi Ying,
I may be misunderstanding your question, but if you are processing all
3
data sets with PCPCombine before calling GridStat, you can define the
FCST_PCP_COMBINE_OUTPUT_NAME in the config files to set the output
field
name of that tool. You can set them all to write out APCP, then read
in
APCP for all data sets in GridStat with FCST_VAR1_NAME = APCP. Does
this
solve the problem? If not, you can send me your configuration files
and I
can try to take a closer look. As far as I know the FCST_VAR value in
the
GridStat output is tied to the input to GridStat field name.
On Fri, Mar 6, 2020 at 2:47 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>
> Hi George,
>
> Found out that METviewer server could indeed include multiple
> FCST_VAR1_NAMEs, though it doesn't quite do what I meant for it to
do:
> it was searching for stats for all models/all FCST_VAR1_NAME combos,
not
> just the existing combos.
>
> But turns out it's very easy to do a 'sed' string substitution in
the
> output stats before loading to the METviewer server, that solves the
> problem.
>
> I might add this to the Monday MET user meeting discussion list to
see
> if there is a METviewer solution. Though so far the 'sed' solution
seems
> to work fine. Anyway, this doesn't to be a METplus issue.
>
> Thanks,
>
> Ying
>
>
> On 3/6/20 3:13 PM, Ying Lin wrote:
> > Thank you very much George! This is really helpful. "
> >
> > FCST_VAR1_OPTIONS = convert(x) = x*1000
> >
> > does work.
> >
> > One more question: for ECMWF, I have
> > FCST_VAR1_NAME = TP
> >
> > for UKMO, it is
> > FCST_VAR1_NAME = TWATP
> >
> > All the other models have
> > BOTH_VAR1_NAME = APCP
> >
> > (the analysis/obs is always APCP, and for now I'm converting DWD
> > forecast to APCP rather than using the original wrongly named
TPRATE).
> >
> > After loading the stats to METviewer server, all the scores look
> > reasonable individually, but they can't all be compared together,
> > since the "Y1 Dependent (Forecast) Variables" are APCP_24,
TWATP_24
> > and TP, there doesn't seem to be a way to combine them. This might
be
> > a METviewer issue, but I'm wondering: is there a way in METplus
config
> > to specify that, even though the FCST_VAR1_NAME is something other
> > than APCP, the FCST_VAR in the GridStat ought to be APCP? This is
not
> > a request for METplus to accommodate yet another oddball input
> > feature, just wondering if there's already something in the
existing
> > config set up for such a thing.
> >
> > Thanks again -
> >
> > Ying
> >
> > On 3/6/20 1:06 PM, George McCabe via RT wrote:
> >> Hi Ying,
> >>
> >> MET has the capability to convert units within the config files.
See
> >> this
> >> excerpt from the MET User's Guide (Chapter 3 - MET Data I/O):
> >>
> >> // - The "convert" entry is a user-defined function of a single
variable
> >> // for processing input data values. Any input values that are
not bad
> >> // data are replaced by the value of this function. The convert
function
> >> // is applied prior to regridding or thresholding. This function
may
> >> // include any of the built-in math functions (e.g. sqrt, log10)
> >> // described above.
> >> // Several standard unit conversion functions are already defined
in
> >> // data/config/ConfigConstants.
> >> // Examples of user-defined conversion functions include:
> >> // convert(x) = 2*x;
> >> // convert(x) = x^2;
> >> // convert(a) = log10(a);
> >> // convert(a) = a^10;
> >> // convert(t) = max(1, sqrt(abs(t)));
> >> // convert(x) = K_to_C(x); where K_to_C(x) is defined in
> >> // ConfigConstants
> >>
> >> You can set the conversion via METplus by setting:
> >>
> >> FCST_VAR1_OPTIONS = convert(x) = x*1000;
> >>
> >> Let me know if that doesn't work.
> >>
> >> Regarding the long ticket, I am not sure what the preferred
protocol
> >> of the
> >> team for the ticking system. I personally don't mind continuing
work
> >> on a
> >> single thread so someone else don't have to go through the steps
to
> >> read it
> >> and assign it to me each time. I will make a note to bring it up
in the
> >> next meeting and I'll see what they say.
> >>
> >> Thanks,
> >> George
> >>
> >> On Fri, Mar 6, 2020 at 10:44 AM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
> >>>
> >>> Thanks George. You made a good point about the data name of the
> >>> forecast field vs. what it actually represented. I was thinking
that
> >>> MET would have looked at the DWD's GRIB header "TPRATE Total
> >>> Precipitation Rate [kg/m^2/s]" and convert that into 24h
accumulation
> >>> (in the case of 24h verification) by multiplying it by 86,400
(which
> >>> would have translated, e.g. a one-day global max rainfall of
236mm to
> >>> 20,390m). But come to think of it what you said had to be the
case, it
> >>> seems highly unlikely that MET would have examined the unit
encoded in
> >>> the grib header and then do a conversion accordingly, without
> >>> additional
> >>> user input specified in the config file.
> >>>
> >>> Which led to a related but separate question: if the input
data is
> >>> "TP=Total precipitation [m]" (ECMWF), then I ought to give
instructions
> >>> in the config file telling GridStat that the unit of forecast is
m
> >>> while
> >>> the unit of analysis (i.e. observation) is kg/m^2 (equivalent to
mm).
> >>> Is there something in the METplus config file instructing
GridStat to
> >>> first multiply the amounts in input forecast field by a factor
of, say,
> >>> 1,000? For the contingency table scores, I can see doing this
via the
> >>> thresholds:
> >>>
> >>> BOTH_VAR1_LEVELS = A24
> >>> FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635, gt0.0127,
> >>> gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762, gt0.1016
> >>> OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05,
gt25.4,
> >>> gt38.1, gt50.8, gt76.2, gt101.6
> >>>
> >>> That ought to do it for the scores with thresholds. But what
about the
> >>> SL1L2 scores (e.g. the Fbar vs. Obar - comparing the average
forecast
> >>> precip vs. observed precip)? How would MET know that the FCST
field
> >>> need to be multiplied by 1,000 before being compared to the OBS?
> >>>
> >>> Attached are 1) the general GridStat config for the CTC and
SL1L2 stats
> >>> and 2) config for a specific model (ECMWF).
> >>>
> >>> This has branched out pretty far from the original subject of
the
> >>> met_help ticket. Do you prefer keeping it all a long-running
> >>> ticket, or
> >>> should I open up a separate met_help ticket, with a more
relevant subj
> >>> line?
> >>>
> >>> Thank you -
> >>>
> >>> Ying
> >>>
> >>>
> >>> On 3/5/20 6:33 PM, George McCabe via RT wrote:
> >>>> Hi Ying,
> >>>>
> >>>> I didn't realize the file you sent me had all of the data
needed
> >>>> for your
> >>>> run in one file. I reconfigured and was able to get it to run
> >>>> correctly!
> >>>> Attached is the config file I used to get it to process all of
the
> >>> forecast
> >>>> leads. I had to change it to loop by INIT time so the init time
stayed
> >>>> constant through the runs, but the valid time was updated for
each
> >>>> run.
> >>>>
> >>>> Regarding TPRATE, MET should read the data with the TPRATE name
and
> >>>> can
> >>>> still treat it as APCP. Changing the output field name to APCP
> >>>> would make
> >>>> this more clear. I think MET only cares about which record to
read in,
> >>>> regardless of the incorrect name. Maybe I am misunderstanding
the
> >>>> issue.
> >>>>
> >>>> That is great news about the progress using wgrib2. If pygrib
has the
> >>> same
> >>>> functionality used in wgrib2 (which I would think it does),
then you
> >>> could
> >>>> use Python Embedding in MET and write a python script to
reformat the
> >>> data
> >>>> and pass it into the MET tools. I can help you with this
process is
> >>>> that
> >>> is
> >>>> possible.
> >>>>
> >>>> On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT
<met_help at ucar.edu>
> >>> wrote:
> >>>>> Hi George,
> >>>>>
> >>>>> That is good news. If you want to test it out, all the UKMO
files
> >>>>> for
> >>>>> that cycle were in that one file's parent directory:
> >>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
> >>>>>
> >>>>> I was writing you in another window earlier:
> >>>>>
> >>>>> In re the UKMO:
> >>>>> Thank you for the github issues ticket. Having METplus handle
the
> >>>>> one
> >>>>> input file to eliminate the need for pre-processing is fine.
> >>>>> Since you
> >>>>> mentioned that METplus 3.0beta2 does have support for
> >>>>> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try,
setting in
> >>>>> the model config file
> >>>>>
> >>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> >>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> >>>>>
> >>>>> It did work.
> >>>>>
> >>>>> In re DWD: Wesley Ebisuzaki told me a way today to fixed the
bad
> >>>>> longitude problem (Lon2):
> >>>>>
> >>>>> -------------------------
> >>>>> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid
-grib
> >>> new.grb
> >>>>> What does -set_int 3 60 359750000 do
> >>>>>
> >>>>> set 4 byte integer in Section 3, octet 60-63 with value
359750000
> >>>>>
> >>>>> ref
> >>>>>
> >>>
> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
> >>>
> >>>>>
https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
> >>>>>
> >>>>> You could argue that ocets 60-63 should be an unsigned
integer, but
> >>> wgrib2
> >>>>> doesn't have a -set_uint.
> >>>>> -------------------------
> >>>>>
> >>>>> Also, on the matter of TPRATE in the DWD data: setting the
field
> >>>>> name to
> >>>>> TPRATE in the METplus config won't work because the field is
actually
> >>>>> APCP, not TPRATE as the grib header says. It's off by a factor
of
> >>>>> 86,400. I'll do some experiment with the DWD.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Ying
> >>>>>
> >>>>> On 3/5/20 4:25 PM, George McCabe via RT wrote:
> >>>>>> Hi Ying,
> >>>>>>
> >>>>>> I implemented the changes needed for PCPCombine to set the
> >>>>>> options the
> >>>>> way
> >>>>>> your case does using ADD. They are available in the develop
> >>>>>> branch of
> >>> the
> >>>>>> METplus repository on GitHub. If you would prefer to obtain a
> >>>>>> release,
> >>> we
> >>>>>> are putting out METplus 3.0beta4 on Friday (tomorrow). I
tested out
> >>> this
> >>>>>> case using the UKMO file you sent me processing the 24 hour
lead.
> >>>>> However,
> >>>>>> I am not sure how it will behave with your use case that
processes
> >>> other
> >>>>>> foreact leads. I only have the single file you sent me, so I
> >>>>>> can't test
> >>>>>> this myself. You could get the updates and see if you are
able to
> >>> process
> >>>>>> your data this way. Here are some configurations I used to
get
> >>>>>> the 24
> >>> hr
> >>>>>> lead to run:
> >>>>>>
> >>>>>> [config]
> >>>>>> FCST_PCP_COMBINE_METHOD = ADD
> >>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
> >>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> >>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> >>>>>> FCST_PCP_COMBINE_INPUT_OPTIONS =
valid_time="{valid?fmt=%Y%m%d_%H}";
> >>>>>> FCST_PCP_COMBINE_CONSTANT_INIT = True
> >>>>>>
> >>>>>> [filename_templates]
> >>>>>> FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
> >>>>>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
> >>>>>>
> >>>>>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT <
> >>>>>> met_help at ucar.edu> wrote:
> >>>>>>
> >>>>>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
> >>>>>>> Transaction: Given to mccabe (George McCabe) by RT_System
> >>>>>>> Queue: met_help
> >>>>>>> Subject: Re: valid time from grib data
> >>>>>>> Owner: mccabe
> >>>>>>> Requestors: ying.lin at noaa.gov
> >>>>>>> Status: deleted
> >>>>>>> Ticket <URL:
> >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
> >>>>>>> This transaction appears to have no content
> >>>>>>>
> >>>>> --
> >>>>> Ying Lin
> >>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>>> NCWCP Cubicle No. 2015
> >>>>> Ying.Lin at noaa.gov
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>> --
> >>> Ying Lin
> >>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>> NCWCP Cubicle No. 2015
> >>> Ying.Lin at noaa.gov
> >>>
> >>>
> >>>
> >>>
> >
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: valid time from grib data
From: Ying Lin
Time: Fri Mar 06 17:33:42 2020
Hi George,
No, you didn't misunderstand my question, and setting
FCST_PCP_COMBINE_OUTPUT_NAME = APCP
did solve my problem in the case of UKMO verification, the FCST_VAR in
the output stats did end up being APCP_24. This is great.
One more question: the above works very nicely when
"FCST_PCP_COMBINE_RUN = True". The ECMWF input is already a 24h total
and its config file (attached ecmwf_24h.conf) has
FCST_VAR1_NAME = TP
FCST_VAR1_LEVELS = R001
and its "FCST_VAR FCST_UNITS" fields in the output stats are "TP
m". I seem to recall MET's pcp_combine has an option to run on a
single
field (i.e. without adding/subtracting) with an irregular variable
name,
level etc. and convert it into a netCDF with more convectional
metadata.
Does the current METplus has such a feature?
Thanks again. There is no rush to answer this.
Ying
On 3/6/20 5:22 PM, George McCabe via RT wrote:
> Hi Ying,
>
> I may be misunderstanding your question, but if you are processing
all 3
> data sets with PCPCombine before calling GridStat, you can define
the
> FCST_PCP_COMBINE_OUTPUT_NAME in the config files to set the output
field
> name of that tool. You can set them all to write out APCP, then read
in
> APCP for all data sets in GridStat with FCST_VAR1_NAME = APCP. Does
this
> solve the problem? If not, you can send me your configuration files
and I
> can try to take a closer look. As far as I know the FCST_VAR value
in the
> GridStat output is tied to the input to GridStat field name.
>
> On Fri, Mar 6, 2020 at 2:47 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>>
>> Hi George,
>>
>> Found out that METviewer server could indeed include multiple
>> FCST_VAR1_NAMEs, though it doesn't quite do what I meant for it to
do:
>> it was searching for stats for all models/all FCST_VAR1_NAME
combos, not
>> just the existing combos.
>>
>> But turns out it's very easy to do a 'sed' string substitution in
the
>> output stats before loading to the METviewer server, that solves
the
>> problem.
>>
>> I might add this to the Monday MET user meeting discussion list to
see
>> if there is a METviewer solution. Though so far the 'sed' solution
seems
>> to work fine. Anyway, this doesn't to be a METplus issue.
>>
>> Thanks,
>>
>> Ying
>>
>>
>> On 3/6/20 3:13 PM, Ying Lin wrote:
>>> Thank you very much George! This is really helpful. "
>>>
>>> FCST_VAR1_OPTIONS = convert(x) = x*1000
>>>
>>> does work.
>>>
>>> One more question: for ECMWF, I have
>>> FCST_VAR1_NAME = TP
>>>
>>> for UKMO, it is
>>> FCST_VAR1_NAME = TWATP
>>>
>>> All the other models have
>>> BOTH_VAR1_NAME = APCP
>>>
>>> (the analysis/obs is always APCP, and for now I'm converting DWD
>>> forecast to APCP rather than using the original wrongly named
TPRATE).
>>>
>>> After loading the stats to METviewer server, all the scores look
>>> reasonable individually, but they can't all be compared together,
>>> since the "Y1 Dependent (Forecast) Variables" are APCP_24,
TWATP_24
>>> and TP, there doesn't seem to be a way to combine them. This might
be
>>> a METviewer issue, but I'm wondering: is there a way in METplus
config
>>> to specify that, even though the FCST_VAR1_NAME is something other
>>> than APCP, the FCST_VAR in the GridStat ought to be APCP? This is
not
>>> a request for METplus to accommodate yet another oddball input
>>> feature, just wondering if there's already something in the
existing
>>> config set up for such a thing.
>>>
>>> Thanks again -
>>>
>>> Ying
>>>
>>> On 3/6/20 1:06 PM, George McCabe via RT wrote:
>>>> Hi Ying,
>>>>
>>>> MET has the capability to convert units within the config files.
See
>>>> this
>>>> excerpt from the MET User's Guide (Chapter 3 - MET Data I/O):
>>>>
>>>> // - The "convert" entry is a user-defined function of a single
variable
>>>> // for processing input data values. Any input values that are
not bad
>>>> // data are replaced by the value of this function. The convert
function
>>>> // is applied prior to regridding or thresholding. This function
may
>>>> // include any of the built-in math functions (e.g. sqrt, log10)
>>>> // described above.
>>>> // Several standard unit conversion functions are already defined
in
>>>> // data/config/ConfigConstants.
>>>> // Examples of user-defined conversion functions include:
>>>> // convert(x) = 2*x;
>>>> // convert(x) = x^2;
>>>> // convert(a) = log10(a);
>>>> // convert(a) = a^10;
>>>> // convert(t) = max(1, sqrt(abs(t)));
>>>> // convert(x) = K_to_C(x); where K_to_C(x) is defined in
>>>> // ConfigConstants
>>>>
>>>> You can set the conversion via METplus by setting:
>>>>
>>>> FCST_VAR1_OPTIONS = convert(x) = x*1000;
>>>>
>>>> Let me know if that doesn't work.
>>>>
>>>> Regarding the long ticket, I am not sure what the preferred
protocol
>>>> of the
>>>> team for the ticking system. I personally don't mind continuing
work
>>>> on a
>>>> single thread so someone else don't have to go through the steps
to
>>>> read it
>>>> and assign it to me each time. I will make a note to bring it up
in the
>>>> next meeting and I'll see what they say.
>>>>
>>>> Thanks,
>>>> George
>>>>
>>>> On Fri, Mar 6, 2020 at 10:44 AM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>>>>>
>>>>> Thanks George. You made a good point about the data name of the
>>>>> forecast field vs. what it actually represented. I was thinking
that
>>>>> MET would have looked at the DWD's GRIB header "TPRATE Total
>>>>> Precipitation Rate [kg/m^2/s]" and convert that into 24h
accumulation
>>>>> (in the case of 24h verification) by multiplying it by 86,400
(which
>>>>> would have translated, e.g. a one-day global max rainfall of
236mm to
>>>>> 20,390m). But come to think of it what you said had to be the
case, it
>>>>> seems highly unlikely that MET would have examined the unit
encoded in
>>>>> the grib header and then do a conversion accordingly, without
>>>>> additional
>>>>> user input specified in the config file.
>>>>>
>>>>> Which led to a related but separate question: if the input
data is
>>>>> "TP=Total precipitation [m]" (ECMWF), then I ought to give
instructions
>>>>> in the config file telling GridStat that the unit of forecast is
m
>>>>> while
>>>>> the unit of analysis (i.e. observation) is kg/m^2 (equivalent to
mm).
>>>>> Is there something in the METplus config file instructing
GridStat to
>>>>> first multiply the amounts in input forecast field by a factor
of, say,
>>>>> 1,000? For the contingency table scores, I can see doing this
via the
>>>>> thresholds:
>>>>>
>>>>> BOTH_VAR1_LEVELS = A24
>>>>> FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635, gt0.0127,
>>>>> gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762, gt0.1016
>>>>> OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05,
gt25.4,
>>>>> gt38.1, gt50.8, gt76.2, gt101.6
>>>>>
>>>>> That ought to do it for the scores with thresholds. But what
about the
>>>>> SL1L2 scores (e.g. the Fbar vs. Obar - comparing the average
forecast
>>>>> precip vs. observed precip)? How would MET know that the FCST
field
>>>>> need to be multiplied by 1,000 before being compared to the OBS?
>>>>>
>>>>> Attached are 1) the general GridStat config for the CTC and
SL1L2 stats
>>>>> and 2) config for a specific model (ECMWF).
>>>>>
>>>>> This has branched out pretty far from the original subject of
the
>>>>> met_help ticket. Do you prefer keeping it all a long-running
>>>>> ticket, or
>>>>> should I open up a separate met_help ticket, with a more
relevant subj
>>>>> line?
>>>>>
>>>>> Thank you -
>>>>>
>>>>> Ying
>>>>>
>>>>>
>>>>> On 3/5/20 6:33 PM, George McCabe via RT wrote:
>>>>>> Hi Ying,
>>>>>>
>>>>>> I didn't realize the file you sent me had all of the data
needed
>>>>>> for your
>>>>>> run in one file. I reconfigured and was able to get it to run
>>>>>> correctly!
>>>>>> Attached is the config file I used to get it to process all of
the
>>>>> forecast
>>>>>> leads. I had to change it to loop by INIT time so the init time
stayed
>>>>>> constant through the runs, but the valid time was updated for
each
>>>>>> run.
>>>>>>
>>>>>> Regarding TPRATE, MET should read the data with the TPRATE name
and
>>>>>> can
>>>>>> still treat it as APCP. Changing the output field name to APCP
>>>>>> would make
>>>>>> this more clear. I think MET only cares about which record to
read in,
>>>>>> regardless of the incorrect name. Maybe I am misunderstanding
the
>>>>>> issue.
>>>>>>
>>>>>> That is great news about the progress using wgrib2. If pygrib
has the
>>>>> same
>>>>>> functionality used in wgrib2 (which I would think it does),
then you
>>>>> could
>>>>>> use Python Embedding in MET and write a python script to
reformat the
>>>>> data
>>>>>> and pass it into the MET tools. I can help you with this
process is
>>>>>> that
>>>>> is
>>>>>> possible.
>>>>>>
>>>>>> On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT
<met_help at ucar.edu>
>>>>> wrote:
>>>>>>> Hi George,
>>>>>>>
>>>>>>> That is good news. If you want to test it out, all the UKMO
files
>>>>>>> for
>>>>>>> that cycle were in that one file's parent directory:
>>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
>>>>>>>
>>>>>>> I was writing you in another window earlier:
>>>>>>>
>>>>>>> In re the UKMO:
>>>>>>> Thank you for the github issues ticket. Having METplus handle
the
>>>>>>> one
>>>>>>> input file to eliminate the need for pre-processing is fine.
>>>>>>> Since you
>>>>>>> mentioned that METplus 3.0beta2 does have support for
>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try,
setting in
>>>>>>> the model config file
>>>>>>>
>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>>>>>>
>>>>>>> It did work.
>>>>>>>
>>>>>>> In re DWD: Wesley Ebisuzaki told me a way today to fixed the
bad
>>>>>>> longitude problem (Lon2):
>>>>>>>
>>>>>>> -------------------------
>>>>>>> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid
-grib
>>>>> new.grb
>>>>>>> What does -set_int 3 60 359750000 do
>>>>>>>
>>>>>>> set 4 byte integer in Section 3, octet 60-63 with value
359750000
>>>>>>>
>>>>>>> ref
>>>>>>>
>> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
>>>>>>>
https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
>>>>>>>
>>>>>>> You could argue that ocets 60-63 should be an unsigned
integer, but
>>>>> wgrib2
>>>>>>> doesn't have a -set_uint.
>>>>>>> -------------------------
>>>>>>>
>>>>>>> Also, on the matter of TPRATE in the DWD data: setting the
field
>>>>>>> name to
>>>>>>> TPRATE in the METplus config won't work because the field is
actually
>>>>>>> APCP, not TPRATE as the grib header says. It's off by a factor
of
>>>>>>> 86,400. I'll do some experiment with the DWD.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Ying
>>>>>>>
>>>>>>> On 3/5/20 4:25 PM, George McCabe via RT wrote:
>>>>>>>> Hi Ying,
>>>>>>>>
>>>>>>>> I implemented the changes needed for PCPCombine to set the
>>>>>>>> options the
>>>>>>> way
>>>>>>>> your case does using ADD. They are available in the develop
>>>>>>>> branch of
>>>>> the
>>>>>>>> METplus repository on GitHub. If you would prefer to obtain a
>>>>>>>> release,
>>>>> we
>>>>>>>> are putting out METplus 3.0beta4 on Friday (tomorrow). I
tested out
>>>>> this
>>>>>>>> case using the UKMO file you sent me processing the 24 hour
lead.
>>>>>>> However,
>>>>>>>> I am not sure how it will behave with your use case that
processes
>>>>> other
>>>>>>>> foreact leads. I only have the single file you sent me, so I
>>>>>>>> can't test
>>>>>>>> this myself. You could get the updates and see if you are
able to
>>>>> process
>>>>>>>> your data this way. Here are some configurations I used to
get
>>>>>>>> the 24
>>>>> hr
>>>>>>>> lead to run:
>>>>>>>>
>>>>>>>> [config]
>>>>>>>> FCST_PCP_COMBINE_METHOD = ADD
>>>>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
>>>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>>>>>>>> FCST_PCP_COMBINE_INPUT_OPTIONS =
valid_time="{valid?fmt=%Y%m%d_%H}";
>>>>>>>> FCST_PCP_COMBINE_CONSTANT_INIT = True
>>>>>>>>
>>>>>>>> [filename_templates]
>>>>>>>> FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
>>>>>>>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
>>>>>>>>
>>>>>>>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT <
>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>
>>>>>>>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
>>>>>>>>> Transaction: Given to mccabe (George McCabe) by RT_System
>>>>>>>>> Queue: met_help
>>>>>>>>> Subject: Re: valid time from grib data
>>>>>>>>> Owner: mccabe
>>>>>>>>> Requestors: ying.lin at noaa.gov
>>>>>>>>> Status: deleted
>>>>>>>>> Ticket <URL:
>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
>>>>>>>>> This transaction appears to have no content
>>>>>>>>>
>>>>>>> --
>>>>>>> Ying Lin
>>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>>> NCWCP Cubicle No. 2015
>>>>>>> Ying.Lin at noaa.gov
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> --
>>>>> Ying Lin
>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>> NCWCP Cubicle No. 2015
>>>>> Ying.Lin at noaa.gov
>>>>>
>>>>>
>>>>>
>>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Re: valid time from grib data
From: Ying Lin
Time: Fri Mar 06 17:33:42 2020
# Grid to Grid Precipitation Example
[config]
# time looping - options are INIT, VALID, RETRO, and REALTIME
LOOP_BY = VALID
# Format of VALID_BEG and VALID_END
VALID_TIME_FMT = %Y%m%d%H
# Start time for METplus run
VALID_BEG = {ENV[vday]}12
#
# End time for METplus run
VALID_END = {ENV[vday]}12
#VALID_END = {now?fmt=%Y%m%d}12
YLMETPLUS_PATH = {ENV[YLMETPLUS_PATH]}
# Increment between METplus runs in seconds. Must be >= 60
# 86400 sec=24h
VALID_INCREMENT = 86400
# Options are times, processes
# times = run all items in the PROCESS_LIST for a single
initialization
# time, then repeat until all times have been evaluated.
# processes = run each item in the PROCESS_LIST for all times
# specified, then repeat for the next item in the PROCESS_LIST.
LOOP_ORDER = times
# List of applications to run
PROCESS_LIST = PcpCombine, RegridDataPlane, GridStat
# run pcp_combine on forecast/obs data?
FCST_PCP_COMBINE_RUN = False
OBS_PCP_COMBINE_RUN = True
OBS_REGRID_DATA_PLANE_RUN = True
# mode of pcp_combine to use (SUM, ADD, SUBTRACT)
OBS_PCP_COMBINE_METHOD = ADD
# If 'bucket' or 'regrid' output already exists, skip the PcpCombine
and Regrid
# for the data (e.g. verifying analysis only processed once when
multiple
# models are being verified)
PCP_COMBINE_SKIP_IF_OUTPUT_EXISTS = True
REGRID_DATA_PLANE_SKIP_IF_OUTPUT_EXISTS = True
# list of variables to compare
FCST_VAR1_NAME = TP
OBS_VAR1_NAME = APCP
# Can specify 'record number in a grib file' -
FCST_VAR1_LEVELS = R001
OBS_VAR1_LEVELS = A24
BOTH_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05, gt25.4,
gt38.1, gt50.8, gt76.2, gt101.6
LEAD_SEQ = 24,48,72
# description of data to be processed
# used in output file path
MODEL = {ENV[MODEL]}
model = {ENV[model]}
modpath = {ENV[modpath]}
ccpapath = {ENV[ccpapath]}
OBTYPE = CCPA
# Used by regrid_data_plane to remap data
REGRID_DATA_PLANE_VERIF_GRID={INPUT_BASE}/qpf/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
# method to run regrid_data_plane, not setting this will default to
NEAREST
REGRID_DATA_PLANE_METHOD = BUDGET
# regridding width used in regrid_data_plane, not setting this will
default to 1
REGRID_DATA_PLANE_WIDTH = 2
# location of grid_stat MET config file
GRID_STAT_CONFIG_FILE =
{YLMETPLUS_PATH}/yl/parm/GridStatConfig_ctc_sl1l2
# Forecast data description variables
FCST_PCP_COMBINE_INPUT_DATATYPE = GRIB
FCST_IS_PROB = false
# Set to true if data is only available once per day
FCST_PCP_COMBINE_IS_DAILY_FILE = false
# File format. Options are GRIB, NETCDF, or GEMPAK
OBS_PCP_COMBINE_INPUT_DATATYPE = GRIB
# Set to true if data is only available once per day
OBS_PCP_COMBINE_IS_DAILY_FILE = false
# Accumulation interval available in observation data
OBS_PCP_COMBINE_INPUT_ACCUMS = 06
[dir]
# input and output data directories
OBS_PCP_COMBINE_INPUT_DIR = {ccpapath}
OBS_PCP_COMBINE_OUTPUT_DIR = {OUTPUT_BASE}/ccpa/bucket
OBS_REGRID_DATA_PLANE_INPUT_DIR = {OBS_PCP_COMBINE_OUTPUT_DIR}
OBS_REGRID_DATA_PLANE_OUTPUT_DIR = {OUTPUT_BASE}/ccpa/regrid
OBS_GRID_STAT_INPUT_DIR = {OBS_REGRID_DATA_PLANE_OUTPUT_DIR}
GRID_STAT_OUTPUT_DIR = {STATS_BASE}
# location of configuration files used by MET applications
CONFIG_DIR={PARM_BASE}/use_cases/grid_to_grid/met_config
[filename_templates]
# format of filenames
FCST_GRID_STAT_INPUT_TEMPLATE =
{modpath}/{init?fmt=%Y%m%d}/qpf_verif/UWD{init?fmt=%Y%m%d%H}00{valid?fmt=%m%d%H}001
# ANLYS
OBS_PCP_COMBINE_INPUT_TEMPLATE =
ccpa.{valid?fmt=%Y%m%d}/{valid?fmt=%H}/ccpa.t{valid?fmt=%H}z.06h.hrap.conus.gb2
OBS_PCP_COMBINE_OUTPUT_TEMPLATE =
ccpa.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h
OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE =
{OBS_PCP_COMBINE_OUTPUT_TEMPLATE}
OBS_REGRID_DATA_PLANE_OUTPUT_TEMPLATE =
ccpa.{valid?fmt=%Y%m%d%H}_a{level?fmt=%HH}h.g212
OBS_GRID_STAT_INPUT_TEMPLATE = {OBS_REGRID_DATA_PLANE_OUTPUT_TEMPLATE}
# grid_stat output:
GRID_STAT_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d}
------------------------------------------------
Subject: Re: valid time from grib data
From: George McCabe
Time: Mon Mar 09 09:06:19 2020
Hi Ying,
If you set the input and output accumulations to the same value, i.e.
24H
for both, then PCPCombine will get a single file and process it. This
works
fine, but it is sort of a hack to obtain what you need. If the data
you are
comparing are on different grids, I would recommend processing the
ECMWF
files with RegridDataPlane to get the data in the correct naming
format and
on the correct grid for processing with GridStat. This way you won't
need
to set any regrid information for GridStat and that tool will run
faster.
On Fri, Mar 6, 2020 at 5:34 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>
> Hi George,
>
> No, you didn't misunderstand my question, and setting
>
> FCST_PCP_COMBINE_OUTPUT_NAME = APCP
>
> did solve my problem in the case of UKMO verification, the FCST_VAR
in
> the output stats did end up being APCP_24. This is great.
>
> One more question: the above works very nicely when
> "FCST_PCP_COMBINE_RUN = True". The ECMWF input is already a 24h
total
> and its config file (attached ecmwf_24h.conf) has
>
> FCST_VAR1_NAME = TP
> FCST_VAR1_LEVELS = R001
>
> and its "FCST_VAR FCST_UNITS" fields in the output stats are "TP
> m". I seem to recall MET's pcp_combine has an option to run on a
single
> field (i.e. without adding/subtracting) with an irregular variable
name,
> level etc. and convert it into a netCDF with more convectional
metadata.
> Does the current METplus has such a feature?
>
> Thanks again. There is no rush to answer this.
>
> Ying
>
>
>
> On 3/6/20 5:22 PM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > I may be misunderstanding your question, but if you are processing
all 3
> > data sets with PCPCombine before calling GridStat, you can define
the
> > FCST_PCP_COMBINE_OUTPUT_NAME in the config files to set the output
field
> > name of that tool. You can set them all to write out APCP, then
read in
> > APCP for all data sets in GridStat with FCST_VAR1_NAME = APCP.
Does this
> > solve the problem? If not, you can send me your configuration
files and I
> > can try to take a closer look. As far as I know the FCST_VAR value
in the
> > GridStat output is tied to the input to GridStat field name.
> >
> > On Fri, Mar 6, 2020 at 2:47 PM Ying Lin via RT <met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
> >>
> >> Hi George,
> >>
> >> Found out that METviewer server could indeed include multiple
> >> FCST_VAR1_NAMEs, though it doesn't quite do what I meant for it
to do:
> >> it was searching for stats for all models/all FCST_VAR1_NAME
combos, not
> >> just the existing combos.
> >>
> >> But turns out it's very easy to do a 'sed' string substitution in
the
> >> output stats before loading to the METviewer server, that solves
the
> >> problem.
> >>
> >> I might add this to the Monday MET user meeting discussion list
to see
> >> if there is a METviewer solution. Though so far the 'sed'
solution seems
> >> to work fine. Anyway, this doesn't to be a METplus issue.
> >>
> >> Thanks,
> >>
> >> Ying
> >>
> >>
> >> On 3/6/20 3:13 PM, Ying Lin wrote:
> >>> Thank you very much George! This is really helpful. "
> >>>
> >>> FCST_VAR1_OPTIONS = convert(x) = x*1000
> >>>
> >>> does work.
> >>>
> >>> One more question: for ECMWF, I have
> >>> FCST_VAR1_NAME = TP
> >>>
> >>> for UKMO, it is
> >>> FCST_VAR1_NAME = TWATP
> >>>
> >>> All the other models have
> >>> BOTH_VAR1_NAME = APCP
> >>>
> >>> (the analysis/obs is always APCP, and for now I'm converting DWD
> >>> forecast to APCP rather than using the original wrongly named
TPRATE).
> >>>
> >>> After loading the stats to METviewer server, all the scores look
> >>> reasonable individually, but they can't all be compared
together,
> >>> since the "Y1 Dependent (Forecast) Variables" are APCP_24,
TWATP_24
> >>> and TP, there doesn't seem to be a way to combine them. This
might be
> >>> a METviewer issue, but I'm wondering: is there a way in METplus
config
> >>> to specify that, even though the FCST_VAR1_NAME is something
other
> >>> than APCP, the FCST_VAR in the GridStat ought to be APCP? This
is not
> >>> a request for METplus to accommodate yet another oddball input
> >>> feature, just wondering if there's already something in the
existing
> >>> config set up for such a thing.
> >>>
> >>> Thanks again -
> >>>
> >>> Ying
> >>>
> >>> On 3/6/20 1:06 PM, George McCabe via RT wrote:
> >>>> Hi Ying,
> >>>>
> >>>> MET has the capability to convert units within the config
files. See
> >>>> this
> >>>> excerpt from the MET User's Guide (Chapter 3 - MET Data I/O):
> >>>>
> >>>> // - The "convert" entry is a user-defined function of a single
> variable
> >>>> // for processing input data values. Any input values that are
not bad
> >>>> // data are replaced by the value of this function. The convert
> function
> >>>> // is applied prior to regridding or thresholding. This
function may
> >>>> // include any of the built-in math functions (e.g. sqrt,
log10)
> >>>> // described above.
> >>>> // Several standard unit conversion functions are already
defined in
> >>>> // data/config/ConfigConstants.
> >>>> // Examples of user-defined conversion functions include:
> >>>> // convert(x) = 2*x;
> >>>> // convert(x) = x^2;
> >>>> // convert(a) = log10(a);
> >>>> // convert(a) = a^10;
> >>>> // convert(t) = max(1, sqrt(abs(t)));
> >>>> // convert(x) = K_to_C(x); where K_to_C(x) is defined in
> >>>> // ConfigConstants
> >>>>
> >>>> You can set the conversion via METplus by setting:
> >>>>
> >>>> FCST_VAR1_OPTIONS = convert(x) = x*1000;
> >>>>
> >>>> Let me know if that doesn't work.
> >>>>
> >>>> Regarding the long ticket, I am not sure what the preferred
protocol
> >>>> of the
> >>>> team for the ticking system. I personally don't mind continuing
work
> >>>> on a
> >>>> single thread so someone else don't have to go through the
steps to
> >>>> read it
> >>>> and assign it to me each time. I will make a note to bring it
up in
> the
> >>>> next meeting and I'll see what they say.
> >>>>
> >>>> Thanks,
> >>>> George
> >>>>
> >>>> On Fri, Mar 6, 2020 at 10:44 AM Ying Lin via RT
<met_help at ucar.edu>
> >>>> wrote:
> >>>>
> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
>
> >>>>>
> >>>>> Thanks George. You made a good point about the data name of
the
> >>>>> forecast field vs. what it actually represented. I was
thinking that
> >>>>> MET would have looked at the DWD's GRIB header "TPRATE Total
> >>>>> Precipitation Rate [kg/m^2/s]" and convert that into 24h
accumulation
> >>>>> (in the case of 24h verification) by multiplying it by 86,400
(which
> >>>>> would have translated, e.g. a one-day global max rainfall of
236mm to
> >>>>> 20,390m). But come to think of it what you said had to be the
case,
> it
> >>>>> seems highly unlikely that MET would have examined the unit
encoded
> in
> >>>>> the grib header and then do a conversion accordingly, without
> >>>>> additional
> >>>>> user input specified in the config file.
> >>>>>
> >>>>> Which led to a related but separate question: if the input
data is
> >>>>> "TP=Total precipitation [m]" (ECMWF), then I ought to give
> instructions
> >>>>> in the config file telling GridStat that the unit of forecast
is m
> >>>>> while
> >>>>> the unit of analysis (i.e. observation) is kg/m^2 (equivalent
to mm).
> >>>>> Is there something in the METplus config file instructing
GridStat to
> >>>>> first multiply the amounts in input forecast field by a factor
of,
> say,
> >>>>> 1,000? For the contingency table scores, I can see doing this
via
> the
> >>>>> thresholds:
> >>>>>
> >>>>> BOTH_VAR1_LEVELS = A24
> >>>>> FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635, gt0.0127,
> >>>>> gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762, gt0.1016
> >>>>> OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05,
gt25.4,
> >>>>> gt38.1, gt50.8, gt76.2, gt101.6
> >>>>>
> >>>>> That ought to do it for the scores with thresholds. But what
about
> the
> >>>>> SL1L2 scores (e.g. the Fbar vs. Obar - comparing the average
forecast
> >>>>> precip vs. observed precip)? How would MET know that the FCST
field
> >>>>> need to be multiplied by 1,000 before being compared to the
OBS?
> >>>>>
> >>>>> Attached are 1) the general GridStat config for the CTC and
SL1L2
> stats
> >>>>> and 2) config for a specific model (ECMWF).
> >>>>>
> >>>>> This has branched out pretty far from the original subject of
the
> >>>>> met_help ticket. Do you prefer keeping it all a long-running
> >>>>> ticket, or
> >>>>> should I open up a separate met_help ticket, with a more
relevant
> subj
> >>>>> line?
> >>>>>
> >>>>> Thank you -
> >>>>>
> >>>>> Ying
> >>>>>
> >>>>>
> >>>>> On 3/5/20 6:33 PM, George McCabe via RT wrote:
> >>>>>> Hi Ying,
> >>>>>>
> >>>>>> I didn't realize the file you sent me had all of the data
needed
> >>>>>> for your
> >>>>>> run in one file. I reconfigured and was able to get it to run
> >>>>>> correctly!
> >>>>>> Attached is the config file I used to get it to process all
of the
> >>>>> forecast
> >>>>>> leads. I had to change it to loop by INIT time so the init
time
> stayed
> >>>>>> constant through the runs, but the valid time was updated for
each
> >>>>>> run.
> >>>>>>
> >>>>>> Regarding TPRATE, MET should read the data with the TPRATE
name and
> >>>>>> can
> >>>>>> still treat it as APCP. Changing the output field name to
APCP
> >>>>>> would make
> >>>>>> this more clear. I think MET only cares about which record to
read
> in,
> >>>>>> regardless of the incorrect name. Maybe I am misunderstanding
the
> >>>>>> issue.
> >>>>>>
> >>>>>> That is great news about the progress using wgrib2. If pygrib
has
> the
> >>>>> same
> >>>>>> functionality used in wgrib2 (which I would think it does),
then you
> >>>>> could
> >>>>>> use Python Embedding in MET and write a python script to
reformat
> the
> >>>>> data
> >>>>>> and pass it into the MET tools. I can help you with this
process is
> >>>>>> that
> >>>>> is
> >>>>>> possible.
> >>>>>>
> >>>>>> On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT
<met_help at ucar.edu>
> >>>>> wrote:
> >>>>>>> Hi George,
> >>>>>>>
> >>>>>>> That is good news. If you want to test it out, all the UKMO
files
> >>>>>>> for
> >>>>>>> that cycle were in that one file's parent directory:
> >>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
> >>>>>>>
> >>>>>>> I was writing you in another window earlier:
> >>>>>>>
> >>>>>>> In re the UKMO:
> >>>>>>> Thank you for the github issues ticket. Having METplus
handle the
> >>>>>>> one
> >>>>>>> input file to eliminate the need for pre-processing is fine.
> >>>>>>> Since you
> >>>>>>> mentioned that METplus 3.0beta2 does have support for
> >>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try,
setting
> in
> >>>>>>> the model config file
> >>>>>>>
> >>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> >>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> >>>>>>>
> >>>>>>> It did work.
> >>>>>>>
> >>>>>>> In re DWD: Wesley Ebisuzaki told me a way today to fixed the
bad
> >>>>>>> longitude problem (Lon2):
> >>>>>>>
> >>>>>>> -------------------------
> >>>>>>> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid
-grib
> >>>>> new.grb
> >>>>>>> What does -set_int 3 60 359750000 do
> >>>>>>>
> >>>>>>> set 4 byte integer in Section 3, octet 60-63 with value
359750000
> >>>>>>>
> >>>>>>> ref
> >>>>>>>
> >>
> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
> >>>>>>>
https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
> >>>>>>>
> >>>>>>> You could argue that ocets 60-63 should be an unsigned
integer, but
> >>>>> wgrib2
> >>>>>>> doesn't have a -set_uint.
> >>>>>>> -------------------------
> >>>>>>>
> >>>>>>> Also, on the matter of TPRATE in the DWD data: setting the
field
> >>>>>>> name to
> >>>>>>> TPRATE in the METplus config won't work because the field is
> actually
> >>>>>>> APCP, not TPRATE as the grib header says. It's off by a
factor of
> >>>>>>> 86,400. I'll do some experiment with the DWD.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Ying
> >>>>>>>
> >>>>>>> On 3/5/20 4:25 PM, George McCabe via RT wrote:
> >>>>>>>> Hi Ying,
> >>>>>>>>
> >>>>>>>> I implemented the changes needed for PCPCombine to set the
> >>>>>>>> options the
> >>>>>>> way
> >>>>>>>> your case does using ADD. They are available in the develop
> >>>>>>>> branch of
> >>>>> the
> >>>>>>>> METplus repository on GitHub. If you would prefer to obtain
a
> >>>>>>>> release,
> >>>>> we
> >>>>>>>> are putting out METplus 3.0beta4 on Friday (tomorrow). I
tested
> out
> >>>>> this
> >>>>>>>> case using the UKMO file you sent me processing the 24 hour
lead.
> >>>>>>> However,
> >>>>>>>> I am not sure how it will behave with your use case that
processes
> >>>>> other
> >>>>>>>> foreact leads. I only have the single file you sent me, so
I
> >>>>>>>> can't test
> >>>>>>>> this myself. You could get the updates and see if you are
able to
> >>>>> process
> >>>>>>>> your data this way. Here are some configurations I used to
get
> >>>>>>>> the 24
> >>>>> hr
> >>>>>>>> lead to run:
> >>>>>>>>
> >>>>>>>> [config]
> >>>>>>>> FCST_PCP_COMBINE_METHOD = ADD
> >>>>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
> >>>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> >>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> >>>>>>>> FCST_PCP_COMBINE_INPUT_OPTIONS =
> valid_time="{valid?fmt=%Y%m%d_%H}";
> >>>>>>>> FCST_PCP_COMBINE_CONSTANT_INIT = True
> >>>>>>>>
> >>>>>>>> [filename_templates]
> >>>>>>>> FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
> >>>>>>>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
> ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
> >>>>>>>>
> >>>>>>>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT
<
> >>>>>>>> met_help at ucar.edu> wrote:
> >>>>>>>>
> >>>>>>>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
> >>>>>>>>> Transaction: Given to mccabe (George McCabe) by RT_System
> >>>>>>>>> Queue: met_help
> >>>>>>>>> Subject: Re: valid time from grib data
> >>>>>>>>> Owner: mccabe
> >>>>>>>>> Requestors: ying.lin at noaa.gov
> >>>>>>>>> Status: deleted
> >>>>>>>>> Ticket <URL:
> >>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
> >>>>>>>>> This transaction appears to have no content
> >>>>>>>>>
> >>>>>>> --
> >>>>>>> Ying Lin
> >>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
> Branch
> >>>>>>> NCWCP Cubicle No. 2015
> >>>>>>> Ying.Lin at noaa.gov
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>> --
> >>>>> Ying Lin
> >>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>>> NCWCP Cubicle No. 2015
> >>>>> Ying.Lin at noaa.gov
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94354] Re: valid time from grib data
From: Ying Lin
Time: Mon Mar 09 09:25:50 2020
Thanks George. You mean set
FCST_REGRID_DATA_PLANE_RUN = True
and something like
FCST_REGRID_DATA_PLANE_INPUT_DIR = {MODDIR}
FCST_REGRID_DATA_PLANE_OUTPUT_DIR = {OUTPUT_BASE}/{MODEL}/regrid
FCST_GRID_STAT_INPUT_DIR = {FCST_REGRID_DATA_PLANE_OUTPUT_DIR}
FCST_REGRID_DATA_PLANE_OUTPUT_TEMPLATE = ...
so the forecast data will be regrid'd to the same
REGRID_DATA_PLANE_VERIF_GRID (G212 in this case) as the OBS field? Is
there a parameter to specify in the config file to tell METplus that
the
output of FCST_REGRID_DATA_PLANE_RUN should be APCP?
Thanks again -
Ying
On 3/9/20 11:06 AM, George McCabe via RT wrote:
> Hi Ying,
>
> If you set the input and output accumulations to the same value,
i.e. 24H
> for both, then PCPCombine will get a single file and process it.
This works
> fine, but it is sort of a hack to obtain what you need. If the data
you are
> comparing are on different grids, I would recommend processing the
ECMWF
> files with RegridDataPlane to get the data in the correct naming
format and
> on the correct grid for processing with GridStat. This way you won't
need
> to set any regrid information for GridStat and that tool will run
faster.
>
> On Fri, Mar 6, 2020 at 5:34 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>>
>> Hi George,
>>
>> No, you didn't misunderstand my question, and setting
>>
>> FCST_PCP_COMBINE_OUTPUT_NAME = APCP
>>
>> did solve my problem in the case of UKMO verification, the FCST_VAR
in
>> the output stats did end up being APCP_24. This is great.
>>
>> One more question: the above works very nicely when
>> "FCST_PCP_COMBINE_RUN = True". The ECMWF input is already a 24h
total
>> and its config file (attached ecmwf_24h.conf) has
>>
>> FCST_VAR1_NAME = TP
>> FCST_VAR1_LEVELS = R001
>>
>> and its "FCST_VAR FCST_UNITS" fields in the output stats are "TP
>> m". I seem to recall MET's pcp_combine has an option to run on a
single
>> field (i.e. without adding/subtracting) with an irregular variable
name,
>> level etc. and convert it into a netCDF with more convectional
metadata.
>> Does the current METplus has such a feature?
>>
>> Thanks again. There is no rush to answer this.
>>
>> Ying
>>
>>
>>
>> On 3/6/20 5:22 PM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> I may be misunderstanding your question, but if you are processing
all 3
>>> data sets with PCPCombine before calling GridStat, you can define
the
>>> FCST_PCP_COMBINE_OUTPUT_NAME in the config files to set the output
field
>>> name of that tool. You can set them all to write out APCP, then
read in
>>> APCP for all data sets in GridStat with FCST_VAR1_NAME = APCP.
Does this
>>> solve the problem? If not, you can send me your configuration
files and I
>>> can try to take a closer look. As far as I know the FCST_VAR value
in the
>>> GridStat output is tied to the input to GridStat field name.
>>>
>>> On Fri, Mar 6, 2020 at 2:47 PM Ying Lin via RT <met_help at ucar.edu>
>> wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>>>>
>>>> Hi George,
>>>>
>>>> Found out that METviewer server could indeed include multiple
>>>> FCST_VAR1_NAMEs, though it doesn't quite do what I meant for it
to do:
>>>> it was searching for stats for all models/all FCST_VAR1_NAME
combos, not
>>>> just the existing combos.
>>>>
>>>> But turns out it's very easy to do a 'sed' string substitution in
the
>>>> output stats before loading to the METviewer server, that solves
the
>>>> problem.
>>>>
>>>> I might add this to the Monday MET user meeting discussion list
to see
>>>> if there is a METviewer solution. Though so far the 'sed'
solution seems
>>>> to work fine. Anyway, this doesn't to be a METplus issue.
>>>>
>>>> Thanks,
>>>>
>>>> Ying
>>>>
>>>>
>>>> On 3/6/20 3:13 PM, Ying Lin wrote:
>>>>> Thank you very much George! This is really helpful. "
>>>>>
>>>>> FCST_VAR1_OPTIONS = convert(x) = x*1000
>>>>>
>>>>> does work.
>>>>>
>>>>> One more question: for ECMWF, I have
>>>>> FCST_VAR1_NAME = TP
>>>>>
>>>>> for UKMO, it is
>>>>> FCST_VAR1_NAME = TWATP
>>>>>
>>>>> All the other models have
>>>>> BOTH_VAR1_NAME = APCP
>>>>>
>>>>> (the analysis/obs is always APCP, and for now I'm converting DWD
>>>>> forecast to APCP rather than using the original wrongly named
TPRATE).
>>>>>
>>>>> After loading the stats to METviewer server, all the scores look
>>>>> reasonable individually, but they can't all be compared
together,
>>>>> since the "Y1 Dependent (Forecast) Variables" are APCP_24,
TWATP_24
>>>>> and TP, there doesn't seem to be a way to combine them. This
might be
>>>>> a METviewer issue, but I'm wondering: is there a way in METplus
config
>>>>> to specify that, even though the FCST_VAR1_NAME is something
other
>>>>> than APCP, the FCST_VAR in the GridStat ought to be APCP? This
is not
>>>>> a request for METplus to accommodate yet another oddball input
>>>>> feature, just wondering if there's already something in the
existing
>>>>> config set up for such a thing.
>>>>>
>>>>> Thanks again -
>>>>>
>>>>> Ying
>>>>>
>>>>> On 3/6/20 1:06 PM, George McCabe via RT wrote:
>>>>>> Hi Ying,
>>>>>>
>>>>>> MET has the capability to convert units within the config
files. See
>>>>>> this
>>>>>> excerpt from the MET User's Guide (Chapter 3 - MET Data I/O):
>>>>>>
>>>>>> // - The "convert" entry is a user-defined function of a single
>> variable
>>>>>> // for processing input data values. Any input values that are
not bad
>>>>>> // data are replaced by the value of this function. The convert
>> function
>>>>>> // is applied prior to regridding or thresholding. This
function may
>>>>>> // include any of the built-in math functions (e.g. sqrt,
log10)
>>>>>> // described above.
>>>>>> // Several standard unit conversion functions are already
defined in
>>>>>> // data/config/ConfigConstants.
>>>>>> // Examples of user-defined conversion functions include:
>>>>>> // convert(x) = 2*x;
>>>>>> // convert(x) = x^2;
>>>>>> // convert(a) = log10(a);
>>>>>> // convert(a) = a^10;
>>>>>> // convert(t) = max(1, sqrt(abs(t)));
>>>>>> // convert(x) = K_to_C(x); where K_to_C(x) is defined in
>>>>>> // ConfigConstants
>>>>>>
>>>>>> You can set the conversion via METplus by setting:
>>>>>>
>>>>>> FCST_VAR1_OPTIONS = convert(x) = x*1000;
>>>>>>
>>>>>> Let me know if that doesn't work.
>>>>>>
>>>>>> Regarding the long ticket, I am not sure what the preferred
protocol
>>>>>> of the
>>>>>> team for the ticking system. I personally don't mind continuing
work
>>>>>> on a
>>>>>> single thread so someone else don't have to go through the
steps to
>>>>>> read it
>>>>>> and assign it to me each time. I will make a note to bring it
up in
>> the
>>>>>> next meeting and I'll see what they say.
>>>>>>
>>>>>> Thanks,
>>>>>> George
>>>>>>
>>>>>> On Fri, Mar 6, 2020 at 10:44 AM Ying Lin via RT
<met_help at ucar.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
>
>>>>>>>
>>>>>>> Thanks George. You made a good point about the data name of
the
>>>>>>> forecast field vs. what it actually represented. I was
thinking that
>>>>>>> MET would have looked at the DWD's GRIB header "TPRATE Total
>>>>>>> Precipitation Rate [kg/m^2/s]" and convert that into 24h
accumulation
>>>>>>> (in the case of 24h verification) by multiplying it by 86,400
(which
>>>>>>> would have translated, e.g. a one-day global max rainfall of
236mm to
>>>>>>> 20,390m). But come to think of it what you said had to be the
case,
>> it
>>>>>>> seems highly unlikely that MET would have examined the unit
encoded
>> in
>>>>>>> the grib header and then do a conversion accordingly, without
>>>>>>> additional
>>>>>>> user input specified in the config file.
>>>>>>>
>>>>>>> Which led to a related but separate question: if the input
data is
>>>>>>> "TP=Total precipitation [m]" (ECMWF), then I ought to give
>> instructions
>>>>>>> in the config file telling GridStat that the unit of forecast
is m
>>>>>>> while
>>>>>>> the unit of analysis (i.e. observation) is kg/m^2 (equivalent
to mm).
>>>>>>> Is there something in the METplus config file instructing
GridStat to
>>>>>>> first multiply the amounts in input forecast field by a factor
of,
>> say,
>>>>>>> 1,000? For the contingency table scores, I can see doing this
via
>> the
>>>>>>> thresholds:
>>>>>>>
>>>>>>> BOTH_VAR1_LEVELS = A24
>>>>>>> FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635, gt0.0127,
>>>>>>> gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762, gt0.1016
>>>>>>> OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05,
gt25.4,
>>>>>>> gt38.1, gt50.8, gt76.2, gt101.6
>>>>>>>
>>>>>>> That ought to do it for the scores with thresholds. But what
about
>> the
>>>>>>> SL1L2 scores (e.g. the Fbar vs. Obar - comparing the average
forecast
>>>>>>> precip vs. observed precip)? How would MET know that the FCST
field
>>>>>>> need to be multiplied by 1,000 before being compared to the
OBS?
>>>>>>>
>>>>>>> Attached are 1) the general GridStat config for the CTC and
SL1L2
>> stats
>>>>>>> and 2) config for a specific model (ECMWF).
>>>>>>>
>>>>>>> This has branched out pretty far from the original subject of
the
>>>>>>> met_help ticket. Do you prefer keeping it all a long-running
>>>>>>> ticket, or
>>>>>>> should I open up a separate met_help ticket, with a more
relevant
>> subj
>>>>>>> line?
>>>>>>>
>>>>>>> Thank you -
>>>>>>>
>>>>>>> Ying
>>>>>>>
>>>>>>>
>>>>>>> On 3/5/20 6:33 PM, George McCabe via RT wrote:
>>>>>>>> Hi Ying,
>>>>>>>>
>>>>>>>> I didn't realize the file you sent me had all of the data
needed
>>>>>>>> for your
>>>>>>>> run in one file. I reconfigured and was able to get it to run
>>>>>>>> correctly!
>>>>>>>> Attached is the config file I used to get it to process all
of the
>>>>>>> forecast
>>>>>>>> leads. I had to change it to loop by INIT time so the init
time
>> stayed
>>>>>>>> constant through the runs, but the valid time was updated for
each
>>>>>>>> run.
>>>>>>>>
>>>>>>>> Regarding TPRATE, MET should read the data with the TPRATE
name and
>>>>>>>> can
>>>>>>>> still treat it as APCP. Changing the output field name to
APCP
>>>>>>>> would make
>>>>>>>> this more clear. I think MET only cares about which record to
read
>> in,
>>>>>>>> regardless of the incorrect name. Maybe I am misunderstanding
the
>>>>>>>> issue.
>>>>>>>>
>>>>>>>> That is great news about the progress using wgrib2. If pygrib
has
>> the
>>>>>>> same
>>>>>>>> functionality used in wgrib2 (which I would think it does),
then you
>>>>>>> could
>>>>>>>> use Python Embedding in MET and write a python script to
reformat
>> the
>>>>>>> data
>>>>>>>> and pass it into the MET tools. I can help you with this
process is
>>>>>>>> that
>>>>>>> is
>>>>>>>> possible.
>>>>>>>>
>>>>>>>> On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT
<met_help at ucar.edu>
>>>>>>> wrote:
>>>>>>>>> Hi George,
>>>>>>>>>
>>>>>>>>> That is good news. If you want to test it out, all the UKMO
files
>>>>>>>>> for
>>>>>>>>> that cycle were in that one file's parent directory:
>>>>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
>>>>>>>>>
>>>>>>>>> I was writing you in another window earlier:
>>>>>>>>>
>>>>>>>>> In re the UKMO:
>>>>>>>>> Thank you for the github issues ticket. Having METplus
handle the
>>>>>>>>> one
>>>>>>>>> input file to eliminate the need for pre-processing is fine.
>>>>>>>>> Since you
>>>>>>>>> mentioned that METplus 3.0beta2 does have support for
>>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try,
setting
>> in
>>>>>>>>> the model config file
>>>>>>>>>
>>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>>>>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>>>>>>>>
>>>>>>>>> It did work.
>>>>>>>>>
>>>>>>>>> In re DWD: Wesley Ebisuzaki told me a way today to fixed the
bad
>>>>>>>>> longitude problem (Lon2):
>>>>>>>>>
>>>>>>>>> -------------------------
>>>>>>>>> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000 -grid
-grib
>>>>>>> new.grb
>>>>>>>>> What does -set_int 3 60 359750000 do
>>>>>>>>>
>>>>>>>>> set 4 byte integer in Section 3, octet 60-63 with value
359750000
>>>>>>>>>
>>>>>>>>> ref
>>>>>>>>>
>> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
>>>>>>>>>
https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
>>>>>>>>>
>>>>>>>>> You could argue that ocets 60-63 should be an unsigned
integer, but
>>>>>>> wgrib2
>>>>>>>>> doesn't have a -set_uint.
>>>>>>>>> -------------------------
>>>>>>>>>
>>>>>>>>> Also, on the matter of TPRATE in the DWD data: setting the
field
>>>>>>>>> name to
>>>>>>>>> TPRATE in the METplus config won't work because the field is
>> actually
>>>>>>>>> APCP, not TPRATE as the grib header says. It's off by a
factor of
>>>>>>>>> 86,400. I'll do some experiment with the DWD.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Ying
>>>>>>>>>
>>>>>>>>> On 3/5/20 4:25 PM, George McCabe via RT wrote:
>>>>>>>>>> Hi Ying,
>>>>>>>>>>
>>>>>>>>>> I implemented the changes needed for PCPCombine to set the
>>>>>>>>>> options the
>>>>>>>>> way
>>>>>>>>>> your case does using ADD. They are available in the develop
>>>>>>>>>> branch of
>>>>>>> the
>>>>>>>>>> METplus repository on GitHub. If you would prefer to obtain
a
>>>>>>>>>> release,
>>>>>>> we
>>>>>>>>>> are putting out METplus 3.0beta4 on Friday (tomorrow). I
tested
>> out
>>>>>>> this
>>>>>>>>>> case using the UKMO file you sent me processing the 24 hour
lead.
>>>>>>>>> However,
>>>>>>>>>> I am not sure how it will behave with your use case that
processes
>>>>>>> other
>>>>>>>>>> foreact leads. I only have the single file you sent me, so
I
>>>>>>>>>> can't test
>>>>>>>>>> this myself. You could get the updates and see if you are
able to
>>>>>>> process
>>>>>>>>>> your data this way. Here are some configurations I used to
get
>>>>>>>>>> the 24
>>>>>>> hr
>>>>>>>>>> lead to run:
>>>>>>>>>>
>>>>>>>>>> [config]
>>>>>>>>>> FCST_PCP_COMBINE_METHOD = ADD
>>>>>>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
>>>>>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
>>>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
>>>>>>>>>> FCST_PCP_COMBINE_INPUT_OPTIONS =
>> valid_time="{valid?fmt=%Y%m%d_%H}";
>>>>>>>>>> FCST_PCP_COMBINE_CONSTANT_INIT = True
>>>>>>>>>>
>>>>>>>>>> [filename_templates]
>>>>>>>>>> FCST_PCP_COMBINE_INPUT_TEMPLATE = ukmo.{init?fmt=%Y%m%d}12
>>>>>>>>>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
>> ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
>>>>>>>>>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via RT
<
>>>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
>>>>>>>>>>> Transaction: Given to mccabe (George McCabe) by RT_System
>>>>>>>>>>> Queue: met_help
>>>>>>>>>>> Subject: Re: valid time from grib data
>>>>>>>>>>> Owner: mccabe
>>>>>>>>>>> Requestors: ying.lin at noaa.gov
>>>>>>>>>>> Status: deleted
>>>>>>>>>>> Ticket <URL:
>>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
>>>>>>>>>>> This transaction appears to have no content
>>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ying Lin
>>>>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
>> Branch
>>>>>>>>> NCWCP Cubicle No. 2015
>>>>>>>>> Ying.Lin at noaa.gov
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>> --
>>>>>>> Ying Lin
>>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>>> NCWCP Cubicle No. 2015
>>>>>>> Ying.Lin at noaa.gov
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Re: valid time from grib data
From: George McCabe
Time: Mon Mar 09 09:35:29 2020
Hi Ying,
Yes, it will use the same VERIF_GRID to regrid.
Take a look at this page for a list of all the possible variables for
RegridDataPlane wrapper:
https://ncar.github.io/METplus/Users_Guide/wrappers.html#regriddataplane
Specifically, FCST_REGRID_DATA_PLANE_VAR<n>_OUTPUT_FIELD_NAME can be
used
to set the output field name.
Then I believe you can set the regridding to NONE in the GridStat
config
file. I can't see your GridStatConfig_ctc_sl1l2 file so I'm not sure
what
it is set to. If you referenced ${REGRID_TO_GRID} in that file, then
you
could set GRID_STAT_REGRID_TO_GRID in your METplus config file to set
it,
but it will default to NONE if you leave it out.
See
https://ncar.github.io/METplus/Users_Guide/met_tool_wrapper/GridStat/GridStat.html#met-
configuration
to see all of the variables you can use in the MET config file and how
they
relate to the METplus config variables.
On Mon, Mar 9, 2020 at 9:25 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
>
> Thanks George. You mean set
> FCST_REGRID_DATA_PLANE_RUN = True
> and something like
> FCST_REGRID_DATA_PLANE_INPUT_DIR = {MODDIR}
> FCST_REGRID_DATA_PLANE_OUTPUT_DIR = {OUTPUT_BASE}/{MODEL}/regrid
> FCST_GRID_STAT_INPUT_DIR = {FCST_REGRID_DATA_PLANE_OUTPUT_DIR}
> FCST_REGRID_DATA_PLANE_OUTPUT_TEMPLATE = ...
>
> so the forecast data will be regrid'd to the same
> REGRID_DATA_PLANE_VERIF_GRID (G212 in this case) as the OBS field?
Is
> there a parameter to specify in the config file to tell METplus that
the
> output of FCST_REGRID_DATA_PLANE_RUN should be APCP?
>
> Thanks again -
>
> Ying
>
> On 3/9/20 11:06 AM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > If you set the input and output accumulations to the same value,
i.e. 24H
> > for both, then PCPCombine will get a single file and process it.
This
> works
> > fine, but it is sort of a hack to obtain what you need. If the
data you
> are
> > comparing are on different grids, I would recommend processing the
ECMWF
> > files with RegridDataPlane to get the data in the correct naming
format
> and
> > on the correct grid for processing with GridStat. This way you
won't need
> > to set any regrid information for GridStat and that tool will run
faster.
> >
> > On Fri, Mar 6, 2020 at 5:34 PM Ying Lin via RT <met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
> >>
> >> Hi George,
> >>
> >> No, you didn't misunderstand my question, and setting
> >>
> >> FCST_PCP_COMBINE_OUTPUT_NAME = APCP
> >>
> >> did solve my problem in the case of UKMO verification, the
FCST_VAR in
> >> the output stats did end up being APCP_24. This is great.
> >>
> >> One more question: the above works very nicely when
> >> "FCST_PCP_COMBINE_RUN = True". The ECMWF input is already a 24h
total
> >> and its config file (attached ecmwf_24h.conf) has
> >>
> >> FCST_VAR1_NAME = TP
> >> FCST_VAR1_LEVELS = R001
> >>
> >> and its "FCST_VAR FCST_UNITS" fields in the output stats are "TP
> >> m". I seem to recall MET's pcp_combine has an option to run on a
single
> >> field (i.e. without adding/subtracting) with an irregular
variable name,
> >> level etc. and convert it into a netCDF with more convectional
metadata.
> >> Does the current METplus has such a feature?
> >>
> >> Thanks again. There is no rush to answer this.
> >>
> >> Ying
> >>
> >>
> >>
> >> On 3/6/20 5:22 PM, George McCabe via RT wrote:
> >>> Hi Ying,
> >>>
> >>> I may be misunderstanding your question, but if you are
processing all
> 3
> >>> data sets with PCPCombine before calling GridStat, you can
define the
> >>> FCST_PCP_COMBINE_OUTPUT_NAME in the config files to set the
output
> field
> >>> name of that tool. You can set them all to write out APCP, then
read in
> >>> APCP for all data sets in GridStat with FCST_VAR1_NAME = APCP.
Does
> this
> >>> solve the problem? If not, you can send me your configuration
files
> and I
> >>> can try to take a closer look. As far as I know the FCST_VAR
value in
> the
> >>> GridStat output is tied to the input to GridStat field name.
> >>>
> >>> On Fri, Mar 6, 2020 at 2:47 PM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
> >>>>
> >>>> Hi George,
> >>>>
> >>>> Found out that METviewer server could indeed include multiple
> >>>> FCST_VAR1_NAMEs, though it doesn't quite do what I meant for it
to do:
> >>>> it was searching for stats for all models/all FCST_VAR1_NAME
combos,
> not
> >>>> just the existing combos.
> >>>>
> >>>> But turns out it's very easy to do a 'sed' string substitution
in the
> >>>> output stats before loading to the METviewer server, that
solves the
> >>>> problem.
> >>>>
> >>>> I might add this to the Monday MET user meeting discussion list
to see
> >>>> if there is a METviewer solution. Though so far the 'sed'
solution
> seems
> >>>> to work fine. Anyway, this doesn't to be a METplus issue.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Ying
> >>>>
> >>>>
> >>>> On 3/6/20 3:13 PM, Ying Lin wrote:
> >>>>> Thank you very much George! This is really helpful. "
> >>>>>
> >>>>> FCST_VAR1_OPTIONS = convert(x) = x*1000
> >>>>>
> >>>>> does work.
> >>>>>
> >>>>> One more question: for ECMWF, I have
> >>>>> FCST_VAR1_NAME = TP
> >>>>>
> >>>>> for UKMO, it is
> >>>>> FCST_VAR1_NAME = TWATP
> >>>>>
> >>>>> All the other models have
> >>>>> BOTH_VAR1_NAME = APCP
> >>>>>
> >>>>> (the analysis/obs is always APCP, and for now I'm converting
DWD
> >>>>> forecast to APCP rather than using the original wrongly named
> TPRATE).
> >>>>>
> >>>>> After loading the stats to METviewer server, all the scores
look
> >>>>> reasonable individually, but they can't all be compared
together,
> >>>>> since the "Y1 Dependent (Forecast) Variables" are APCP_24,
TWATP_24
> >>>>> and TP, there doesn't seem to be a way to combine them. This
might be
> >>>>> a METviewer issue, but I'm wondering: is there a way in
METplus
> config
> >>>>> to specify that, even though the FCST_VAR1_NAME is something
other
> >>>>> than APCP, the FCST_VAR in the GridStat ought to be APCP?
This is
> not
> >>>>> a request for METplus to accommodate yet another oddball input
> >>>>> feature, just wondering if there's already something in the
existing
> >>>>> config set up for such a thing.
> >>>>>
> >>>>> Thanks again -
> >>>>>
> >>>>> Ying
> >>>>>
> >>>>> On 3/6/20 1:06 PM, George McCabe via RT wrote:
> >>>>>> Hi Ying,
> >>>>>>
> >>>>>> MET has the capability to convert units within the config
files. See
> >>>>>> this
> >>>>>> excerpt from the MET User's Guide (Chapter 3 - MET Data I/O):
> >>>>>>
> >>>>>> // - The "convert" entry is a user-defined function of a
single
> >> variable
> >>>>>> // for processing input data values. Any input values that
are not
> bad
> >>>>>> // data are replaced by the value of this function. The
convert
> >> function
> >>>>>> // is applied prior to regridding or thresholding. This
function may
> >>>>>> // include any of the built-in math functions (e.g. sqrt,
log10)
> >>>>>> // described above.
> >>>>>> // Several standard unit conversion functions are already
defined in
> >>>>>> // data/config/ConfigConstants.
> >>>>>> // Examples of user-defined conversion functions include:
> >>>>>> // convert(x) = 2*x;
> >>>>>> // convert(x) = x^2;
> >>>>>> // convert(a) = log10(a);
> >>>>>> // convert(a) = a^10;
> >>>>>> // convert(t) = max(1, sqrt(abs(t)));
> >>>>>> // convert(x) = K_to_C(x); where K_to_C(x) is defined in
> >>>>>> // ConfigConstants
> >>>>>>
> >>>>>> You can set the conversion via METplus by setting:
> >>>>>>
> >>>>>> FCST_VAR1_OPTIONS = convert(x) = x*1000;
> >>>>>>
> >>>>>> Let me know if that doesn't work.
> >>>>>>
> >>>>>> Regarding the long ticket, I am not sure what the preferred
protocol
> >>>>>> of the
> >>>>>> team for the ticking system. I personally don't mind
continuing work
> >>>>>> on a
> >>>>>> single thread so someone else don't have to go through the
steps to
> >>>>>> read it
> >>>>>> and assign it to me each time. I will make a note to bring it
up in
> >> the
> >>>>>> next meeting and I'll see what they say.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> George
> >>>>>>
> >>>>>> On Fri, Mar 6, 2020 at 10:44 AM Ying Lin via RT
<met_help at ucar.edu>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
> >>>>>>>
> >>>>>>> Thanks George. You made a good point about the data name of
the
> >>>>>>> forecast field vs. what it actually represented. I was
thinking
> that
> >>>>>>> MET would have looked at the DWD's GRIB header "TPRATE Total
> >>>>>>> Precipitation Rate [kg/m^2/s]" and convert that into 24h
> accumulation
> >>>>>>> (in the case of 24h verification) by multiplying it by
86,400
> (which
> >>>>>>> would have translated, e.g. a one-day global max rainfall of
236mm
> to
> >>>>>>> 20,390m). But come to think of it what you said had to be
the
> case,
> >> it
> >>>>>>> seems highly unlikely that MET would have examined the unit
encoded
> >> in
> >>>>>>> the grib header and then do a conversion accordingly,
without
> >>>>>>> additional
> >>>>>>> user input specified in the config file.
> >>>>>>>
> >>>>>>> Which led to a related but separate question: if the input
data
> is
> >>>>>>> "TP=Total precipitation [m]" (ECMWF), then I ought to give
> >> instructions
> >>>>>>> in the config file telling GridStat that the unit of
forecast is m
> >>>>>>> while
> >>>>>>> the unit of analysis (i.e. observation) is kg/m^2
(equivalent to
> mm).
> >>>>>>> Is there something in the METplus config file instructing
GridStat
> to
> >>>>>>> first multiply the amounts in input forecast field by a
factor of,
> >> say,
> >>>>>>> 1,000? For the contingency table scores, I can see doing
this via
> >> the
> >>>>>>> thresholds:
> >>>>>>>
> >>>>>>> BOTH_VAR1_LEVELS = A24
> >>>>>>> FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635,
gt0.0127,
> >>>>>>> gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762, gt0.1016
> >>>>>>> OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7, gt19.05,
gt25.4,
> >>>>>>> gt38.1, gt50.8, gt76.2, gt101.6
> >>>>>>>
> >>>>>>> That ought to do it for the scores with thresholds. But
what about
> >> the
> >>>>>>> SL1L2 scores (e.g. the Fbar vs. Obar - comparing the average
> forecast
> >>>>>>> precip vs. observed precip)? How would MET know that the
FCST
> field
> >>>>>>> need to be multiplied by 1,000 before being compared to the
OBS?
> >>>>>>>
> >>>>>>> Attached are 1) the general GridStat config for the CTC and
SL1L2
> >> stats
> >>>>>>> and 2) config for a specific model (ECMWF).
> >>>>>>>
> >>>>>>> This has branched out pretty far from the original subject
of the
> >>>>>>> met_help ticket. Do you prefer keeping it all a long-
running
> >>>>>>> ticket, or
> >>>>>>> should I open up a separate met_help ticket, with a more
relevant
> >> subj
> >>>>>>> line?
> >>>>>>>
> >>>>>>> Thank you -
> >>>>>>>
> >>>>>>> Ying
> >>>>>>>
> >>>>>>>
> >>>>>>> On 3/5/20 6:33 PM, George McCabe via RT wrote:
> >>>>>>>> Hi Ying,
> >>>>>>>>
> >>>>>>>> I didn't realize the file you sent me had all of the data
needed
> >>>>>>>> for your
> >>>>>>>> run in one file. I reconfigured and was able to get it to
run
> >>>>>>>> correctly!
> >>>>>>>> Attached is the config file I used to get it to process all
of the
> >>>>>>> forecast
> >>>>>>>> leads. I had to change it to loop by INIT time so the init
time
> >> stayed
> >>>>>>>> constant through the runs, but the valid time was updated
for each
> >>>>>>>> run.
> >>>>>>>>
> >>>>>>>> Regarding TPRATE, MET should read the data with the TPRATE
name
> and
> >>>>>>>> can
> >>>>>>>> still treat it as APCP. Changing the output field name to
APCP
> >>>>>>>> would make
> >>>>>>>> this more clear. I think MET only cares about which record
to read
> >> in,
> >>>>>>>> regardless of the incorrect name. Maybe I am
misunderstanding the
> >>>>>>>> issue.
> >>>>>>>>
> >>>>>>>> That is great news about the progress using wgrib2. If
pygrib has
> >> the
> >>>>>>> same
> >>>>>>>> functionality used in wgrib2 (which I would think it does),
then
> you
> >>>>>>> could
> >>>>>>>> use Python Embedding in MET and write a python script to
reformat
> >> the
> >>>>>>> data
> >>>>>>>> and pass it into the MET tools. I can help you with this
process
> is
> >>>>>>>> that
> >>>>>>> is
> >>>>>>>> possible.
> >>>>>>>>
> >>>>>>>> On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT <
> met_help at ucar.edu>
> >>>>>>> wrote:
> >>>>>>>>> Hi George,
> >>>>>>>>>
> >>>>>>>>> That is good news. If you want to test it out, all the
UKMO
> files
> >>>>>>>>> for
> >>>>>>>>> that cycle were in that one file's parent directory:
> >>>>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
> >>>>>>>>>
> >>>>>>>>> I was writing you in another window earlier:
> >>>>>>>>>
> >>>>>>>>> In re the UKMO:
> >>>>>>>>> Thank you for the github issues ticket. Having METplus
handle
> the
> >>>>>>>>> one
> >>>>>>>>> input file to eliminate the need for pre-processing is
fine.
> >>>>>>>>> Since you
> >>>>>>>>> mentioned that METplus 3.0beta2 does have support for
> >>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a try,
> setting
> >> in
> >>>>>>>>> the model config file
> >>>>>>>>>
> >>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> >>>>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> >>>>>>>>>
> >>>>>>>>> It did work.
> >>>>>>>>>
> >>>>>>>>> In re DWD: Wesley Ebisuzaki told me a way today to fixed
the bad
> >>>>>>>>> longitude problem (Lon2):
> >>>>>>>>>
> >>>>>>>>> -------------------------
> >>>>>>>>> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000
-grid -grib
> >>>>>>> new.grb
> >>>>>>>>> What does -set_int 3 60 359750000 do
> >>>>>>>>>
> >>>>>>>>> set 4 byte integer in Section 3, octet 60-63 with value
359750000
> >>>>>>>>>
> >>>>>>>>> ref
> >>>>>>>>>
> >>
> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
0.shtml
> >>>>>>>>>
> https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
> >>>>>>>>>
> >>>>>>>>> You could argue that ocets 60-63 should be an unsigned
integer,
> but
> >>>>>>> wgrib2
> >>>>>>>>> doesn't have a -set_uint.
> >>>>>>>>> -------------------------
> >>>>>>>>>
> >>>>>>>>> Also, on the matter of TPRATE in the DWD data: setting the
field
> >>>>>>>>> name to
> >>>>>>>>> TPRATE in the METplus config won't work because the field
is
> >> actually
> >>>>>>>>> APCP, not TPRATE as the grib header says. It's off by a
factor of
> >>>>>>>>> 86,400. I'll do some experiment with the DWD.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Ying
> >>>>>>>>>
> >>>>>>>>> On 3/5/20 4:25 PM, George McCabe via RT wrote:
> >>>>>>>>>> Hi Ying,
> >>>>>>>>>>
> >>>>>>>>>> I implemented the changes needed for PCPCombine to set
the
> >>>>>>>>>> options the
> >>>>>>>>> way
> >>>>>>>>>> your case does using ADD. They are available in the
develop
> >>>>>>>>>> branch of
> >>>>>>> the
> >>>>>>>>>> METplus repository on GitHub. If you would prefer to
obtain a
> >>>>>>>>>> release,
> >>>>>>> we
> >>>>>>>>>> are putting out METplus 3.0beta4 on Friday (tomorrow). I
tested
> >> out
> >>>>>>> this
> >>>>>>>>>> case using the UKMO file you sent me processing the 24
hour
> lead.
> >>>>>>>>> However,
> >>>>>>>>>> I am not sure how it will behave with your use case that
> processes
> >>>>>>> other
> >>>>>>>>>> foreact leads. I only have the single file you sent me,
so I
> >>>>>>>>>> can't test
> >>>>>>>>>> this myself. You could get the updates and see if you are
able
> to
> >>>>>>> process
> >>>>>>>>>> your data this way. Here are some configurations I used
to get
> >>>>>>>>>> the 24
> >>>>>>> hr
> >>>>>>>>>> lead to run:
> >>>>>>>>>>
> >>>>>>>>>> [config]
> >>>>>>>>>> FCST_PCP_COMBINE_METHOD = ADD
> >>>>>>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
> >>>>>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> >>>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> >>>>>>>>>> FCST_PCP_COMBINE_INPUT_OPTIONS =
> >> valid_time="{valid?fmt=%Y%m%d_%H}";
> >>>>>>>>>> FCST_PCP_COMBINE_CONSTANT_INIT = True
> >>>>>>>>>>
> >>>>>>>>>> [filename_templates]
> >>>>>>>>>> FCST_PCP_COMBINE_INPUT_TEMPLATE =
ukmo.{init?fmt=%Y%m%d}12
> >>>>>>>>>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
> >> ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
> >>>>>>>>>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via
RT <
> >>>>>>>>>> met_help at ucar.edu> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted upon.
> >>>>>>>>>>> Transaction: Given to mccabe (George McCabe) by
RT_System
> >>>>>>>>>>> Queue: met_help
> >>>>>>>>>>> Subject: Re: valid time from grib data
> >>>>>>>>>>> Owner: mccabe
> >>>>>>>>>>> Requestors: ying.lin at noaa.gov
> >>>>>>>>>>> Status: deleted
> >>>>>>>>>>> Ticket <URL:
> >>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
> >>>>>>>>>>> This transaction appears to have no content
> >>>>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Ying Lin
> >>>>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
> >> Branch
> >>>>>>>>> NCWCP Cubicle No. 2015
> >>>>>>>>> Ying.Lin at noaa.gov
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>> --
> >>>>>>> Ying Lin
> >>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
> Branch
> >>>>>>> NCWCP Cubicle No. 2015
> >>>>>>> Ying.Lin at noaa.gov
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>> --
> >>>> Ying Lin
> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>> NCWCP Cubicle No. 2015
> >>>> Ying.Lin at noaa.gov
> >>>>
> >>>>
> >>>>
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: valid time from grib data
From: George McCabe
Time: Wed Mar 11 08:59:16 2020
Hi Ying,
I spoke with the team and the proper protocol for MET Help is to
resolve and create a new ticket for each issue so we can better track
how many issues we have handled. I am closing this ticket now, so
please do not reply to this email and create a new met-help ticket for
any future requests.
Thanks,
George
On Mon Mar 09 09:35:29 2020, mccabe wrote:
> Hi Ying,
>
> Yes, it will use the same VERIF_GRID to regrid.
>
> Take a look at this page for a list of all the possible variables
for
> RegridDataPlane wrapper:
>
>
https://ncar.github.io/METplus/Users_Guide/wrappers.html#regriddataplane
>
> Specifically, FCST_REGRID_DATA_PLANE_VAR<n>_OUTPUT_FIELD_NAME can be
> used
> to set the output field name.
>
> Then I believe you can set the regridding to NONE in the GridStat
> config
> file. I can't see your GridStatConfig_ctc_sl1l2 file so I'm not sure
> what
> it is set to. If you referenced ${REGRID_TO_GRID} in that file, then
> you
> could set GRID_STAT_REGRID_TO_GRID in your METplus config file to
set
> it,
> but it will default to NONE if you leave it out.
>
> See
>
https://ncar.github.io/METplus/Users_Guide/met_tool_wrapper/GridStat/GridStat.html#met-
> configuration
> to see all of the variables you can use in the MET config file and
how
> they
> relate to the METplus config variables.
>
> On Mon, Mar 9, 2020 at 9:25 AM Ying Lin via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
> >
> > Thanks George. You mean set
> > FCST_REGRID_DATA_PLANE_RUN = True
> > and something like
> > FCST_REGRID_DATA_PLANE_INPUT_DIR = {MODDIR}
> > FCST_REGRID_DATA_PLANE_OUTPUT_DIR = {OUTPUT_BASE}/{MODEL}/regrid
> > FCST_GRID_STAT_INPUT_DIR = {FCST_REGRID_DATA_PLANE_OUTPUT_DIR}
> > FCST_REGRID_DATA_PLANE_OUTPUT_TEMPLATE = ...
> >
> > so the forecast data will be regrid'd to the same
> > REGRID_DATA_PLANE_VERIF_GRID (G212 in this case) as the OBS field?
> > Is
> > there a parameter to specify in the config file to tell METplus
that
> > the
> > output of FCST_REGRID_DATA_PLANE_RUN should be APCP?
> >
> > Thanks again -
> >
> > Ying
> >
> > On 3/9/20 11:06 AM, George McCabe via RT wrote:
> > > Hi Ying,
> > >
> > > If you set the input and output accumulations to the same value,
> > > i.e. 24H
> > > for both, then PCPCombine will get a single file and process it.
> > > This
> > works
> > > fine, but it is sort of a hack to obtain what you need. If the
data
> > > you
> > are
> > > comparing are on different grids, I would recommend processing
the
> > > ECMWF
> > > files with RegridDataPlane to get the data in the correct naming
> > > format
> > and
> > > on the correct grid for processing with GridStat. This way you
> > > won't need
> > > to set any regrid information for GridStat and that tool will
run
> > > faster.
> > >
> > > On Fri, Mar 6, 2020 at 5:34 PM Ying Lin via RT
<met_help at ucar.edu>
> > wrote:
> > >
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354 >
> > >>
> > >> Hi George,
> > >>
> > >> No, you didn't misunderstand my question, and setting
> > >>
> > >> FCST_PCP_COMBINE_OUTPUT_NAME = APCP
> > >>
> > >> did solve my problem in the case of UKMO verification, the
> > >> FCST_VAR in
> > >> the output stats did end up being APCP_24. This is great.
> > >>
> > >> One more question: the above works very nicely when
> > >> "FCST_PCP_COMBINE_RUN = True". The ECMWF input is already a
24h
> > >> total
> > >> and its config file (attached ecmwf_24h.conf) has
> > >>
> > >> FCST_VAR1_NAME = TP
> > >> FCST_VAR1_LEVELS = R001
> > >>
> > >> and its "FCST_VAR FCST_UNITS" fields in the output stats are
"TP
> > >> m". I seem to recall MET's pcp_combine has an option to run on
a
> > >> single
> > >> field (i.e. without adding/subtracting) with an irregular
variable
> > >> name,
> > >> level etc. and convert it into a netCDF with more convectional
> > >> metadata.
> > >> Does the current METplus has such a feature?
> > >>
> > >> Thanks again. There is no rush to answer this.
> > >>
> > >> Ying
> > >>
> > >>
> > >>
> > >> On 3/6/20 5:22 PM, George McCabe via RT wrote:
> > >>> Hi Ying,
> > >>>
> > >>> I may be misunderstanding your question, but if you are
> > >>> processing all
> > 3
> > >>> data sets with PCPCombine before calling GridStat, you can
define
> > >>> the
> > >>> FCST_PCP_COMBINE_OUTPUT_NAME in the config files to set the
> > >>> output
> > field
> > >>> name of that tool. You can set them all to write out APCP,
then
> > >>> read in
> > >>> APCP for all data sets in GridStat with FCST_VAR1_NAME = APCP.
> > >>> Does
> > this
> > >>> solve the problem? If not, you can send me your configuration
> > >>> files
> > and I
> > >>> can try to take a closer look. As far as I know the FCST_VAR
> > >>> value in
> > the
> > >>> GridStat output is tied to the input to GridStat field name.
> > >>>
> > >>> On Fri, Mar 6, 2020 at 2:47 PM Ying Lin via RT
> > >>> <met_help at ucar.edu>
> > >> wrote:
> > >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
>
> > >>>>
> > >>>> Hi George,
> > >>>>
> > >>>> Found out that METviewer server could indeed include multiple
> > >>>> FCST_VAR1_NAMEs, though it doesn't quite do what I meant for
it
> > >>>> to do:
> > >>>> it was searching for stats for all models/all FCST_VAR1_NAME
> > >>>> combos,
> > not
> > >>>> just the existing combos.
> > >>>>
> > >>>> But turns out it's very easy to do a 'sed' string
substitution
> > >>>> in the
> > >>>> output stats before loading to the METviewer server, that
solves
> > >>>> the
> > >>>> problem.
> > >>>>
> > >>>> I might add this to the Monday MET user meeting discussion
list
> > >>>> to see
> > >>>> if there is a METviewer solution. Though so far the 'sed'
> > >>>> solution
> > seems
> > >>>> to work fine. Anyway, this doesn't to be a METplus issue.
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Ying
> > >>>>
> > >>>>
> > >>>> On 3/6/20 3:13 PM, Ying Lin wrote:
> > >>>>> Thank you very much George! This is really helpful. "
> > >>>>>
> > >>>>> FCST_VAR1_OPTIONS = convert(x) = x*1000
> > >>>>>
> > >>>>> does work.
> > >>>>>
> > >>>>> One more question: for ECMWF, I have
> > >>>>> FCST_VAR1_NAME = TP
> > >>>>>
> > >>>>> for UKMO, it is
> > >>>>> FCST_VAR1_NAME = TWATP
> > >>>>>
> > >>>>> All the other models have
> > >>>>> BOTH_VAR1_NAME = APCP
> > >>>>>
> > >>>>> (the analysis/obs is always APCP, and for now I'm converting
> > >>>>> DWD
> > >>>>> forecast to APCP rather than using the original wrongly
named
> > TPRATE).
> > >>>>>
> > >>>>> After loading the stats to METviewer server, all the scores
> > >>>>> look
> > >>>>> reasonable individually, but they can't all be compared
> > >>>>> together,
> > >>>>> since the "Y1 Dependent (Forecast) Variables" are APCP_24,
> > >>>>> TWATP_24
> > >>>>> and TP, there doesn't seem to be a way to combine them. This
> > >>>>> might be
> > >>>>> a METviewer issue, but I'm wondering: is there a way in
METplus
> > config
> > >>>>> to specify that, even though the FCST_VAR1_NAME is something
> > >>>>> other
> > >>>>> than APCP, the FCST_VAR in the GridStat ought to be APCP?
This
> > >>>>> is
> > not
> > >>>>> a request for METplus to accommodate yet another oddball
input
> > >>>>> feature, just wondering if there's already something in the
> > >>>>> existing
> > >>>>> config set up for such a thing.
> > >>>>>
> > >>>>> Thanks again -
> > >>>>>
> > >>>>> Ying
> > >>>>>
> > >>>>> On 3/6/20 1:06 PM, George McCabe via RT wrote:
> > >>>>>> Hi Ying,
> > >>>>>>
> > >>>>>> MET has the capability to convert units within the config
> > >>>>>> files. See
> > >>>>>> this
> > >>>>>> excerpt from the MET User's Guide (Chapter 3 - MET Data
I/O):
> > >>>>>>
> > >>>>>> // - The "convert" entry is a user-defined function of a
> > >>>>>> single
> > >> variable
> > >>>>>> // for processing input data values. Any input values that
are
> > >>>>>> not
> > bad
> > >>>>>> // data are replaced by the value of this function. The
> > >>>>>> convert
> > >> function
> > >>>>>> // is applied prior to regridding or thresholding. This
> > >>>>>> function may
> > >>>>>> // include any of the built-in math functions (e.g. sqrt,
> > >>>>>> log10)
> > >>>>>> // described above.
> > >>>>>> // Several standard unit conversion functions are already
> > >>>>>> defined in
> > >>>>>> // data/config/ConfigConstants.
> > >>>>>> // Examples of user-defined conversion functions include:
> > >>>>>> // convert(x) = 2*x;
> > >>>>>> // convert(x) = x^2;
> > >>>>>> // convert(a) = log10(a);
> > >>>>>> // convert(a) = a^10;
> > >>>>>> // convert(t) = max(1, sqrt(abs(t)));
> > >>>>>> // convert(x) = K_to_C(x); where K_to_C(x) is defined in
> > >>>>>> // ConfigConstants
> > >>>>>>
> > >>>>>> You can set the conversion via METplus by setting:
> > >>>>>>
> > >>>>>> FCST_VAR1_OPTIONS = convert(x) = x*1000;
> > >>>>>>
> > >>>>>> Let me know if that doesn't work.
> > >>>>>>
> > >>>>>> Regarding the long ticket, I am not sure what the preferred
> > >>>>>> protocol
> > >>>>>> of the
> > >>>>>> team for the ticking system. I personally don't mind
> > >>>>>> continuing work
> > >>>>>> on a
> > >>>>>> single thread so someone else don't have to go through the
> > >>>>>> steps to
> > >>>>>> read it
> > >>>>>> and assign it to me each time. I will make a note to bring
it
> > >>>>>> up in
> > >> the
> > >>>>>> next meeting and I'll see what they say.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> George
> > >>>>>>
> > >>>>>> On Fri, Mar 6, 2020 at 10:44 AM Ying Lin via RT
> > >>>>>> <met_help at ucar.edu>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
> > >>>>>>> >
> > >>>>>>>
> > >>>>>>> Thanks George. You made a good point about the data name
of
> > >>>>>>> the
> > >>>>>>> forecast field vs. what it actually represented. I was
> > >>>>>>> thinking
> > that
> > >>>>>>> MET would have looked at the DWD's GRIB header "TPRATE
Total
> > >>>>>>> Precipitation Rate [kg/m^2/s]" and convert that into 24h
> > accumulation
> > >>>>>>> (in the case of 24h verification) by multiplying it by
86,400
> > (which
> > >>>>>>> would have translated, e.g. a one-day global max rainfall
of
> > >>>>>>> 236mm
> > to
> > >>>>>>> 20,390m). But come to think of it what you said had to be
> > >>>>>>> the
> > case,
> > >> it
> > >>>>>>> seems highly unlikely that MET would have examined the
unit
> > >>>>>>> encoded
> > >> in
> > >>>>>>> the grib header and then do a conversion accordingly,
without
> > >>>>>>> additional
> > >>>>>>> user input specified in the config file.
> > >>>>>>>
> > >>>>>>> Which led to a related but separate question: if the
input
> > >>>>>>> data
> > is
> > >>>>>>> "TP=Total precipitation [m]" (ECMWF), then I ought to give
> > >> instructions
> > >>>>>>> in the config file telling GridStat that the unit of
forecast
> > >>>>>>> is m
> > >>>>>>> while
> > >>>>>>> the unit of analysis (i.e. observation) is kg/m^2
(equivalent
> > >>>>>>> to
> > mm).
> > >>>>>>> Is there something in the METplus config file instructing
> > >>>>>>> GridStat
> > to
> > >>>>>>> first multiply the amounts in input forecast field by a
> > >>>>>>> factor of,
> > >> say,
> > >>>>>>> 1,000? For the contingency table scores, I can see doing
> > >>>>>>> this via
> > >> the
> > >>>>>>> thresholds:
> > >>>>>>>
> > >>>>>>> BOTH_VAR1_LEVELS = A24
> > >>>>>>> FCST_VAR1_THRESH = gt0.000254, gt0.00254, gt0.00635,
> > >>>>>>> gt0.0127,
> > >>>>>>> gt0.01905, gt0.0254, gt0.0381, gt0.0508, gt0.0762,
gt0.1016
> > >>>>>>> OBS_VAR1_THRESH = gt0.254, gt2.54, gt6.35, gt12.7,
gt19.05,
> > >>>>>>> gt25.4,
> > >>>>>>> gt38.1, gt50.8, gt76.2, gt101.6
> > >>>>>>>
> > >>>>>>> That ought to do it for the scores with thresholds. But
what
> > >>>>>>> about
> > >> the
> > >>>>>>> SL1L2 scores (e.g. the Fbar vs. Obar - comparing the
average
> > forecast
> > >>>>>>> precip vs. observed precip)? How would MET know that the
> > >>>>>>> FCST
> > field
> > >>>>>>> need to be multiplied by 1,000 before being compared to
the
> > >>>>>>> OBS?
> > >>>>>>>
> > >>>>>>> Attached are 1) the general GridStat config for the CTC
and
> > >>>>>>> SL1L2
> > >> stats
> > >>>>>>> and 2) config for a specific model (ECMWF).
> > >>>>>>>
> > >>>>>>> This has branched out pretty far from the original subject
of
> > >>>>>>> the
> > >>>>>>> met_help ticket. Do you prefer keeping it all a long-
running
> > >>>>>>> ticket, or
> > >>>>>>> should I open up a separate met_help ticket, with a more
> > >>>>>>> relevant
> > >> subj
> > >>>>>>> line?
> > >>>>>>>
> > >>>>>>> Thank you -
> > >>>>>>>
> > >>>>>>> Ying
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On 3/5/20 6:33 PM, George McCabe via RT wrote:
> > >>>>>>>> Hi Ying,
> > >>>>>>>>
> > >>>>>>>> I didn't realize the file you sent me had all of the data
> > >>>>>>>> needed
> > >>>>>>>> for your
> > >>>>>>>> run in one file. I reconfigured and was able to get it to
> > >>>>>>>> run
> > >>>>>>>> correctly!
> > >>>>>>>> Attached is the config file I used to get it to process
all
> > >>>>>>>> of the
> > >>>>>>> forecast
> > >>>>>>>> leads. I had to change it to loop by INIT time so the
init
> > >>>>>>>> time
> > >> stayed
> > >>>>>>>> constant through the runs, but the valid time was updated
> > >>>>>>>> for each
> > >>>>>>>> run.
> > >>>>>>>>
> > >>>>>>>> Regarding TPRATE, MET should read the data with the
TPRATE
> > >>>>>>>> name
> > and
> > >>>>>>>> can
> > >>>>>>>> still treat it as APCP. Changing the output field name to
> > >>>>>>>> APCP
> > >>>>>>>> would make
> > >>>>>>>> this more clear. I think MET only cares about which
record
> > >>>>>>>> to read
> > >> in,
> > >>>>>>>> regardless of the incorrect name. Maybe I am
> > >>>>>>>> misunderstanding the
> > >>>>>>>> issue.
> > >>>>>>>>
> > >>>>>>>> That is great news about the progress using wgrib2. If
> > >>>>>>>> pygrib has
> > >> the
> > >>>>>>> same
> > >>>>>>>> functionality used in wgrib2 (which I would think it
does),
> > >>>>>>>> then
> > you
> > >>>>>>> could
> > >>>>>>>> use Python Embedding in MET and write a python script to
> > >>>>>>>> reformat
> > >> the
> > >>>>>>> data
> > >>>>>>>> and pass it into the MET tools. I can help you with this
> > >>>>>>>> process
> > is
> > >>>>>>>> that
> > >>>>>>> is
> > >>>>>>>> possible.
> > >>>>>>>>
> > >>>>>>>> On Thu, Mar 5, 2020 at 11:05 PM Ying Lin via RT <
> > met_help at ucar.edu>
> > >>>>>>> wrote:
> > >>>>>>>>> Hi George,
> > >>>>>>>>>
> > >>>>>>>>> That is good news. If you want to test it out, all the
> > >>>>>>>>> UKMO
> > files
> > >>>>>>>>> for
> > >>>>>>>>> that cycle were in that one file's parent directory:
> > >>>>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/ukmo/.
> > >>>>>>>>>
> > >>>>>>>>> I was writing you in another window earlier:
> > >>>>>>>>>
> > >>>>>>>>> In re the UKMO:
> > >>>>>>>>> Thank you for the github issues ticket. Having METplus
> > >>>>>>>>> handle
> > the
> > >>>>>>>>> one
> > >>>>>>>>> input file to eliminate the need for pre-processing is
> > >>>>>>>>> fine.
> > >>>>>>>>> Since you
> > >>>>>>>>> mentioned that METplus 3.0beta2 does have support for
> > >>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS, I decided to give it a
try,
> > setting
> > >> in
> > >>>>>>>>> the model config file
> > >>>>>>>>>
> > >>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> > >>>>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> > >>>>>>>>>
> > >>>>>>>>> It did work.
> > >>>>>>>>>
> > >>>>>>>>> In re DWD: Wesley Ebisuzaki told me a way today to fixed
> > >>>>>>>>> the bad
> > >>>>>>>>> longitude problem (Lon2):
> > >>>>>>>>>
> > >>>>>>>>> -------------------------
> > >>>>>>>>> wgrib2 dwd_2020030100_012_036 -set_int 3 60 359750000
-grid
> > >>>>>>>>> -grib
> > >>>>>>> new.grb
> > >>>>>>>>> What does -set_int 3 60 359750000 do
> > >>>>>>>>>
> > >>>>>>>>> set 4 byte integer in Section 3, octet 60-63 with value
> > >>>>>>>>> 359750000
> > >>>>>>>>>
> > >>>>>>>>> ref
> > >>>>>>>>>
> > >>
> >
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp3-
> > 0.shtml
> > >>>>>>>>>
> > https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/set_int.html
> > >>>>>>>>>
> > >>>>>>>>> You could argue that ocets 60-63 should be an unsigned
> > >>>>>>>>> integer,
> > but
> > >>>>>>> wgrib2
> > >>>>>>>>> doesn't have a -set_uint.
> > >>>>>>>>> -------------------------
> > >>>>>>>>>
> > >>>>>>>>> Also, on the matter of TPRATE in the DWD data: setting
the
> > >>>>>>>>> field
> > >>>>>>>>> name to
> > >>>>>>>>> TPRATE in the METplus config won't work because the
field
> > >>>>>>>>> is
> > >> actually
> > >>>>>>>>> APCP, not TPRATE as the grib header says. It's off by a
> > >>>>>>>>> factor of
> > >>>>>>>>> 86,400. I'll do some experiment with the DWD.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>>
> > >>>>>>>>> Ying
> > >>>>>>>>>
> > >>>>>>>>> On 3/5/20 4:25 PM, George McCabe via RT wrote:
> > >>>>>>>>>> Hi Ying,
> > >>>>>>>>>>
> > >>>>>>>>>> I implemented the changes needed for PCPCombine to set
the
> > >>>>>>>>>> options the
> > >>>>>>>>> way
> > >>>>>>>>>> your case does using ADD. They are available in the
> > >>>>>>>>>> develop
> > >>>>>>>>>> branch of
> > >>>>>>> the
> > >>>>>>>>>> METplus repository on GitHub. If you would prefer to
> > >>>>>>>>>> obtain a
> > >>>>>>>>>> release,
> > >>>>>>> we
> > >>>>>>>>>> are putting out METplus 3.0beta4 on Friday (tomorrow).
I
> > >>>>>>>>>> tested
> > >> out
> > >>>>>>> this
> > >>>>>>>>>> case using the UKMO file you sent me processing the 24
> > >>>>>>>>>> hour
> > lead.
> > >>>>>>>>> However,
> > >>>>>>>>>> I am not sure how it will behave with your use case
that
> > processes
> > >>>>>>> other
> > >>>>>>>>>> foreact leads. I only have the single file you sent me,
so
> > >>>>>>>>>> I
> > >>>>>>>>>> can't test
> > >>>>>>>>>> this myself. You could get the updates and see if you
are
> > >>>>>>>>>> able
> > to
> > >>>>>>> process
> > >>>>>>>>>> your data this way. Here are some configurations I used
to
> > >>>>>>>>>> get
> > >>>>>>>>>> the 24
> > >>>>>>> hr
> > >>>>>>>>>> lead to run:
> > >>>>>>>>>>
> > >>>>>>>>>> [config]
> > >>>>>>>>>> FCST_PCP_COMBINE_METHOD = ADD
> > >>>>>>>>>> FCST_PCP_COMBINE_INPUT_ACCUMS = 12H
> > >>>>>>>>>> FCST_PCP_COMBINE_INPUT_NAMES = TWATP
> > >>>>>>>>>> FCST_PCP_COMBINE_INPUT_LEVELS = L0
> > >>>>>>>>>> FCST_PCP_COMBINE_INPUT_OPTIONS =
> > >> valid_time="{valid?fmt=%Y%m%d_%H}";
> > >>>>>>>>>> FCST_PCP_COMBINE_CONSTANT_INIT = True
> > >>>>>>>>>>
> > >>>>>>>>>> [filename_templates]
> > >>>>>>>>>> FCST_PCP_COMBINE_INPUT_TEMPLATE =
ukmo.{init?fmt=%Y%m%d}12
> > >>>>>>>>>> FCST_PCP_COMBINE_OUTPUT_TEMPLATE =
> > >> ukmo.{valid?fmt=%Y%m%d%H}_A24.nc
> > >>>>>>>>>> On Mon, Mar 2, 2020 at 4:08 PM The RT System itself via
RT
> > >>>>>>>>>> <
> > >>>>>>>>>> met_help at ucar.edu> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Mon Mar 02 09:08:18 2020: Request 94354 was acted
upon.
> > >>>>>>>>>>> Transaction: Given to mccabe (George McCabe) by
RT_System
> > >>>>>>>>>>> Queue: met_help
> > >>>>>>>>>>> Subject: Re: valid time from grib data
> > >>>>>>>>>>> Owner: mccabe
> > >>>>>>>>>>> Requestors: ying.lin at noaa.gov
> > >>>>>>>>>>> Status: deleted
> > >>>>>>>>>>> Ticket <URL:
> > >>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94354
> > >>>>>>>>>>> This transaction appears to have no content
> > >>>>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> Ying Lin
> > >>>>>>>>> NCEP/EMC/Verification, Post-processing and Product
> > >>>>>>>>> Generation
> > >> Branch
> > >>>>>>>>> NCWCP Cubicle No. 2015
> > >>>>>>>>> Ying.Lin at noaa.gov
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>> --
> > >>>>>>> Ying Lin
> > >>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
> > Branch
> > >>>>>>> NCWCP Cubicle No. 2015
> > >>>>>>> Ying.Lin at noaa.gov
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>> --
> > >>>> Ying Lin
> > >>>> NCEP/EMC/Verification, Post-processing and Product Generation
> > >>>> Branch
> > >>>> NCWCP Cubicle No. 2015
> > >>>> Ying.Lin at noaa.gov
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >> --
> > >> Ying Lin
> > >> NCEP/EMC/Verification, Post-processing and Product Generation
> > >> Branch
> > >> NCWCP Cubicle No. 2015
> > >> Ying.Lin at noaa.gov
> > >>
> > >>
> > >>
> > >>
> >
> > --
> > Ying Lin
> > NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> > NCWCP Cubicle No. 2015
> > Ying.Lin at noaa.gov
> >
> >
> >
> >
------------------------------------------------
More information about the Met_help
mailing list