[Met_help] [rt.rap.ucar.edu #87101] History for pcp_combine for MRMS QPE01H
John Halley Gotway via RT
met_help at ucar.edu
Tue Sep 25 14:05:12 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
This is about today's MET+ users meeting, the issue of how to use
pcp_combine on MRMS QPE01H, where missing data is denoted by '-3'. I
tried to follow John's suggestion, with this:
pcp_combine -sum 00000000_000000 1 \
${vday}_120000 24 \
mrms_mm.${vday}12.24h.nc \
-field 'name="GaugeCorrQPE01H"; level="Z0";
censor_thresh=[==-3]; censor_value=[-9999];' \
-pcpdir data \
-v 5
And got this err msg:
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of
type "F
ileType_Gb2".
ERROR :
ERROR : VarInfo::set_dict() -> The number of censor thresholds in
"censor_thres
h" (1) must match the number of replacement values in "censor_val" (0).
ERROR :
Did I do something wrong? I can't figure why it would consider
censor_value=[-9999] doesn't count as a value.
Thanks -
Ying
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: pcp_combine for MRMS QPE01H
From: John Halley Gotway
Time: Mon Sep 24 14:19:10 2018
Ying,
Sorry, my mistake! I told you to use "censor_value" but it should be
"censor_val". The answer's there in that ERROR message:
must match the number of replacement values in "censor_val"
Please try again using...
censor_thresh = [eq-3]; censor_val = [-9999];
Also, note that when defining arrays in MET config settings, if the
array
is of length 1, you don't have to use the square brackets:
censor_thresh = eq-3; censor_val = -9999;
Also, note that 'eq' works just as well as '=='. So the following
works
too:
censor_thresh = ==-3; censor_val = -9999;
You just need to have a space between the single '=' and the double
'=='.
Generally, when defining thresholds on the command line, I prefer the
Fortran syntax (eq, lt, le, and so on) so as to avoid using greater
than
(>) and less than (<) characters since they can cause trouble if the
shell
interprets them.
Thanks,
John
On Mon, Sep 24, 2018 at 11:49 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> Mon Sep 24 11:49:14 2018: Request 87101 was acted upon.
> Transaction: Ticket created by ying.lin at noaa.gov
> Queue: met_help
> Subject: pcp_combine for MRMS QPE01H
> Owner: Nobody
> Requestors: ying.lin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
>
>
> This is about today's MET+ users meeting, the issue of how to use
> pcp_combine on MRMS QPE01H, where missing data is denoted by '-3'.
I
> tried to follow John's suggestion, with this:
>
> pcp_combine -sum 00000000_000000 1 \
> ${vday}_120000 24 \
> mrms_mm.${vday}12.24h.nc \
> -field 'name="GaugeCorrQPE01H"; level="Z0";
> censor_thresh=[==-3]; censor_value=[-9999];' \
> -pcpdir data \
> -v 5
> And got this err msg:
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "F
> ileType_Gb2".
> ERROR :
> ERROR : VarInfo::set_dict() -> The number of censor thresholds in
> "censor_thres
> h" (1) must match the number of replacement values in "censor_val"
(0).
> ERROR :
>
> Did I do something wrong? I can't figure why it would consider
> censor_value=[-9999] doesn't count as a value.
>
> Thanks -
>
> Ying
>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87101] pcp_combine for MRMS QPE01H
From: Ying Lin
Time: Mon Sep 24 19:54:05 2018
Thanks very much John! I was staring at the "censor_thresh = [...};
censor_val = [...]" and it didn't occur to me that it should have been
"censor_val" not "censor_value".
It seemed to work on Tide (/gpfs/tp1/ptmp/Ying.Lin/mrms.sum), but when
I
tried to look at the 24h nc file with plot_data_plane the field
appears
to be empty. I copied the data and script files to Theia
(/scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/)
thinking that it would be easier for you or someone else at met_help
to
look at a Theia directory. After running pcp_combine and
plot_data_plane on Theia, the resulting image seems to be the right
24h
accumulation. I then scp'd Theia's
/scratch4/NCEPDEV/fv3-
cam/save/Ying.Lin/met_question/mrms_sum/mrms_mm.2018092112.24h.nc
to Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/ and renamed it
"mrms_24h_theia.nc", and plot_data_plane of this theia file on Tide
also
seems to be the right image.
So something appears to be wrong when I ran pcp_combine on Tide
(likely
something I did, but I can't figure out what). I did a 'ncdump' on
mrms_mm.2018092112.24h.nc (pcp_combine'd on Tide) and
mrms_24h_theia.nc
- see Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/dump.tide and dump.theia
and
they are different. The Tide dump file is smaller. I'm totally
stymied
now. Any suggestions on how I might proceed?
Thank you again -
Ying
On 09/24/2018 04:19 PM, John Halley Gotway via RT wrote:
> Ying,
>
> Sorry, my mistake! I told you to use "censor_value" but it should
be
> "censor_val". The answer's there in that ERROR message:
> must match the number of replacement values in "censor_val"
>
> Please try again using...
> censor_thresh = [eq-3]; censor_val = [-9999];
>
> Also, note that when defining arrays in MET config settings, if the
array
> is of length 1, you don't have to use the square brackets:
> censor_thresh = eq-3; censor_val = -9999;
>
> Also, note that 'eq' works just as well as '=='. So the following
works
> too:
> censor_thresh = ==-3; censor_val = -9999;
>
> You just need to have a space between the single '=' and the double
'=='.
> Generally, when defining thresholds on the command line, I prefer
the
> Fortran syntax (eq, lt, le, and so on) so as to avoid using greater
than
> (>) and less than (<) characters since they can cause trouble if the
shell
> interprets them.
>
> Thanks,
> John
>
>
>
>
>
> On Mon, Sep 24, 2018 at 11:49 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> Mon Sep 24 11:49:14 2018: Request 87101 was acted upon.
>> Transaction: Ticket created by ying.lin at noaa.gov
>> Queue: met_help
>> Subject: pcp_combine for MRMS QPE01H
>> Owner: Nobody
>> Requestors: ying.lin at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
>>
>>
>> This is about today's MET+ users meeting, the issue of how to use
>> pcp_combine on MRMS QPE01H, where missing data is denoted by '-3'.
I
>> tried to follow John's suggestion, with this:
>>
>> pcp_combine -sum 00000000_000000 1 \
>> ${vday}_120000 24 \
>> mrms_mm.${vday}12.24h.nc \
>> -field 'name="GaugeCorrQPE01H"; level="Z0";
>> censor_thresh=[==-3]; censor_value=[-9999];' \
>> -pcpdir data \
>> -v 5
>> And got this err msg:
>>
>> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
>> type "F
>> ileType_Gb2".
>> ERROR :
>> ERROR : VarInfo::set_dict() -> The number of censor thresholds in
>> "censor_thres
>> h" (1) must match the number of replacement values in "censor_val"
(0).
>> ERROR :
>>
>> Did I do something wrong? I can't figure why it would consider
>> censor_value=[-9999] doesn't count as a value.
>>
>> Thanks -
>>
>> Ying
>>
>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: pcp_combine for MRMS QPE01H
From: John Halley Gotway
Time: Tue Sep 25 09:37:22 2018
Ying,
Thanks for pointing me to the data you're testing. To summarize, it
sounds
like you're getting different behavior from pcp_combine when running
on
theia (output looks good) versus tide (output looks bad).
As a side note, I'm surprised you're doing this in the first place...
adding up 24 1-hourly MRMS files to a make a 24-hour accumulation. I
thought that MRMS would already be available as a 24-hour
accumulation. Is
that not the case?
Next, we really should talk to folks generating the MRMS GRIB2 files.
Frankly its silly that they aren't encoding the bad data values well.
If
we could modify a couple lines in the GRIB2 generation process, we
should
be able to encoded missing data values properly and save everybody
using
this data a lot of trouble downstream.
But on to the problem at hand. Looking at your script, I see that
you've
turned up the verbosity level to 5 (-v 5) to see more logging output.
That's a great idea... and you can also use the "-log" option to write
the
output to a log file (-log run_pcp_combine_on_theia.log).
Here's some ways you could approach this problem:
(1) Run the same pcp_combine on theia and tide using verbosity level 5
and
writing a log file. Rather than diffing the NetCDF output files, diff
the
log files. In particular, look to see how many values were censored
by
comparing these types of lines:
DEBUG 3: Censored values for 8793570 of 24500000 grid points.
Also compare the min/max values, as reported by MET by comparing these
types of lines:
DEBUG 4: plane min: -3
DEBUG 4: plane max: 28.9
If those numbers differ, then there's a problem in how MET is reading
the
data from the GRIB2 files.
(2) My guess is that you'll see a difference in the numbers in (1).
And
that'll indicate that there's a problem in the installation on tide.
If
the problem is in reading data from GRIB2 files, then the problem is
most
likely in how the GRIB2C library was compiled. If that's the case,
I'll
work with Julie Prestopnik to recompile MET and the GRIB2-related
libraries.
Lastly, while pcp_combine should work on this, I've found that running
theia is painfully slow. Computing the sum over 24 files took 14
minutes!
We should use this example to profile the MET code and look for
opportunities to speed it up.
Thanks,
John
On Mon, Sep 24, 2018 at 7:54 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
>
> Thanks very much John! I was staring at the "censor_thresh = [...};
> censor_val = [...]" and it didn't occur to me that it should have
been
> "censor_val" not "censor_value".
>
> It seemed to work on Tide (/gpfs/tp1/ptmp/Ying.Lin/mrms.sum), but
when I
> tried to look at the 24h nc file with plot_data_plane the field
appears
> to be empty. I copied the data and script files to Theia
> (/scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/)
> thinking that it would be easier for you or someone else at met_help
to
> look at a Theia directory. After running pcp_combine and
> plot_data_plane on Theia, the resulting image seems to be the right
24h
> accumulation. I then scp'd Theia's
> /scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/
> mrms_mm.2018092112.24h.nc
> to Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/ and renamed it
> "mrms_24h_theia.nc", and plot_data_plane of this theia file on Tide
also
> seems to be the right image.
>
> So something appears to be wrong when I ran pcp_combine on Tide
(likely
> something I did, but I can't figure out what). I did a 'ncdump' on
> mrms_mm.2018092112.24h.nc (pcp_combine'd on Tide) and
mrms_24h_theia.nc
> - see Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/dump.tide and dump.theia
and
> they are different. The Tide dump file is smaller. I'm totally
stymied
> now. Any suggestions on how I might proceed?
>
> Thank you again -
>
> Ying
>
> On 09/24/2018 04:19 PM, John Halley Gotway via RT wrote:
> > Ying,
> >
> > Sorry, my mistake! I told you to use "censor_value" but it should
be
> > "censor_val". The answer's there in that ERROR message:
> > must match the number of replacement values in "censor_val"
> >
> > Please try again using...
> > censor_thresh = [eq-3]; censor_val = [-9999];
> >
> > Also, note that when defining arrays in MET config settings, if
the array
> > is of length 1, you don't have to use the square brackets:
> > censor_thresh = eq-3; censor_val = -9999;
> >
> > Also, note that 'eq' works just as well as '=='. So the following
works
> > too:
> > censor_thresh = ==-3; censor_val = -9999;
> >
> > You just need to have a space between the single '=' and the
double '=='.
> > Generally, when defining thresholds on the command line, I prefer
the
> > Fortran syntax (eq, lt, le, and so on) so as to avoid using
greater than
> > (>) and less than (<) characters since they can cause trouble if
the
> shell
> > interprets them.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> > On Mon, Sep 24, 2018 at 11:49 AM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> Mon Sep 24 11:49:14 2018: Request 87101 was acted upon.
> >> Transaction: Ticket created by ying.lin at noaa.gov
> >> Queue: met_help
> >> Subject: pcp_combine for MRMS QPE01H
> >> Owner: Nobody
> >> Requestors: ying.lin at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101
> >
> >>
> >>
> >> This is about today's MET+ users meeting, the issue of how to use
> >> pcp_combine on MRMS QPE01H, where missing data is denoted by '-
3'. I
> >> tried to follow John's suggestion, with this:
> >>
> >> pcp_combine -sum 00000000_000000 1 \
> >> ${vday}_120000 24 \
> >> mrms_mm.${vday}12.24h.nc \
> >> -field 'name="GaugeCorrQPE01H"; level="Z0";
> >> censor_thresh=[==-3]; censor_value=[-9999];' \
> >> -pcpdir data \
> >> -v 5
> >> And got this err msg:
> >>
> >> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> >> type "F
> >> ileType_Gb2".
> >> ERROR :
> >> ERROR : VarInfo::set_dict() -> The number of censor thresholds
in
> >> "censor_thres
> >> h" (1) must match the number of replacement values in
"censor_val" (0).
> >> ERROR :
> >>
> >> Did I do something wrong? I can't figure why it would consider
> >> censor_value=[-9999] doesn't count as a value.
> >>
> >> Thanks -
> >>
> >> Ying
> >>
> >>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87101] pcp_combine for MRMS QPE01H
From: Ying Lin
Time: Tue Sep 25 10:52:51 2018
Hi John,
Thanks for the great tip about looking at the log of pcp_combine
log. I saved the run output of runs on both machines. While the
'plane
min' on are '-3' for all 24h on both machines, 'plane max' on Tide are
also all '-3', while on Theia they are 52.6, 77.5, 63.8.
Yes, my pcp_combine on Theia took 12-13 minutes.
You're right about GaugeCorr_QPE already having 24h output (and
3h,
6h, 12h). We don't get this on wcoss:/dcom but the short-term
mrms.ncep
web site has them, and (more usefully) they're also available on the
http://mtarchive.geol.iastate.edu/ site. I was doing this mainly as a
learning opportunity since I was so taken by the nifty "censor_thresh
=
==-3; censor_val = -9999" in MET, compared to my previous 'wgrib2'
manipulation.
Anyway, thanks much for your answer to my question and for the
great tips, I'm learning a lot. I'll save this test case/data in case
of a future test of a recompiled pcp_combine on Tide, but there is no
urgency - I can just go wget the QPE_24H instead of pcp_combine.
Please feel free to close this ticket as you see fit.
Ying
On 09/25/2018 11:37 AM, John Halley Gotway via RT wrote:
> Ying,
>
> Thanks for pointing me to the data you're testing. To summarize, it
sounds
> like you're getting different behavior from pcp_combine when running
on
> theia (output looks good) versus tide (output looks bad).
>
> As a side note, I'm surprised you're doing this in the first
place...
> adding up 24 1-hourly MRMS files to a make a 24-hour accumulation.
I
> thought that MRMS would already be available as a 24-hour
accumulation. Is
> that not the case?
>
> Next, we really should talk to folks generating the MRMS GRIB2
files.
> Frankly its silly that they aren't encoding the bad data values
well. If
> we could modify a couple lines in the GRIB2 generation process, we
should
> be able to encoded missing data values properly and save everybody
using
> this data a lot of trouble downstream.
>
> But on to the problem at hand. Looking at your script, I see that
you've
> turned up the verbosity level to 5 (-v 5) to see more logging
output.
> That's a great idea... and you can also use the "-log" option to
write the
> output to a log file (-log run_pcp_combine_on_theia.log).
>
> Here's some ways you could approach this problem:
>
> (1) Run the same pcp_combine on theia and tide using verbosity level
5 and
> writing a log file. Rather than diffing the NetCDF output files,
diff the
> log files. In particular, look to see how many values were censored
by
> comparing these types of lines:
> DEBUG 3: Censored values for 8793570 of 24500000 grid points.
>
> Also compare the min/max values, as reported by MET by comparing
these
> types of lines:
> DEBUG 4: plane min: -3
> DEBUG 4: plane max: 28.9
>
> If those numbers differ, then there's a problem in how MET is
reading the
> data from the GRIB2 files.
>
> (2) My guess is that you'll see a difference in the numbers in (1).
And
> that'll indicate that there's a problem in the installation on tide.
If
> the problem is in reading data from GRIB2 files, then the problem is
most
> likely in how the GRIB2C library was compiled. If that's the case,
I'll
> work with Julie Prestopnik to recompile MET and the GRIB2-related
libraries.
>
> Lastly, while pcp_combine should work on this, I've found that
running
> theia is painfully slow. Computing the sum over 24 files took 14
minutes!
>
> We should use this example to profile the MET code and look for
> opportunities to speed it up.
>
> Thanks,
> John
>
> On Mon, Sep 24, 2018 at 7:54 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
>>
>> Thanks very much John! I was staring at the "censor_thresh =
[...};
>> censor_val = [...]" and it didn't occur to me that it should have
been
>> "censor_val" not "censor_value".
>>
>> It seemed to work on Tide (/gpfs/tp1/ptmp/Ying.Lin/mrms.sum), but
when I
>> tried to look at the 24h nc file with plot_data_plane the field
appears
>> to be empty. I copied the data and script files to Theia
>> (/scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/)
>> thinking that it would be easier for you or someone else at
met_help to
>> look at a Theia directory. After running pcp_combine and
>> plot_data_plane on Theia, the resulting image seems to be the right
24h
>> accumulation. I then scp'd Theia's
>> /scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/
>> mrms_mm.2018092112.24h.nc
>> to Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/ and renamed it
>> "mrms_24h_theia.nc", and plot_data_plane of this theia file on Tide
also
>> seems to be the right image.
>>
>> So something appears to be wrong when I ran pcp_combine on Tide
(likely
>> something I did, but I can't figure out what). I did a 'ncdump' on
>> mrms_mm.2018092112.24h.nc (pcp_combine'd on Tide) and
mrms_24h_theia.nc
>> - see Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/dump.tide and
dump.theia and
>> they are different. The Tide dump file is smaller. I'm totally
stymied
>> now. Any suggestions on how I might proceed?
>>
>> Thank you again -
>>
>> Ying
>>
>> On 09/24/2018 04:19 PM, John Halley Gotway via RT wrote:
>>> Ying,
>>>
>>> Sorry, my mistake! I told you to use "censor_value" but it should
be
>>> "censor_val". The answer's there in that ERROR message:
>>> must match the number of replacement values in "censor_val"
>>>
>>> Please try again using...
>>> censor_thresh = [eq-3]; censor_val = [-9999];
>>>
>>> Also, note that when defining arrays in MET config settings, if
the array
>>> is of length 1, you don't have to use the square brackets:
>>> censor_thresh = eq-3; censor_val = -9999;
>>>
>>> Also, note that 'eq' works just as well as '=='. So the following
works
>>> too:
>>> censor_thresh = ==-3; censor_val = -9999;
>>>
>>> You just need to have a space between the single '=' and the
double '=='.
>>> Generally, when defining thresholds on the command line, I prefer
the
>>> Fortran syntax (eq, lt, le, and so on) so as to avoid using
greater than
>>> (>) and less than (<) characters since they can cause trouble if
the
>> shell
>>> interprets them.
>>>
>>> Thanks,
>>> John
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Sep 24, 2018 at 11:49 AM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> Mon Sep 24 11:49:14 2018: Request 87101 was acted upon.
>>>> Transaction: Ticket created by ying.lin at noaa.gov
>>>> Queue: met_help
>>>> Subject: pcp_combine for MRMS QPE01H
>>>> Owner: Nobody
>>>> Requestors: ying.lin at noaa.gov
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101
>>>>
>>>> This is about today's MET+ users meeting, the issue of how to use
>>>> pcp_combine on MRMS QPE01H, where missing data is denoted by '-
3'. I
>>>> tried to follow John's suggestion, with this:
>>>>
>>>> pcp_combine -sum 00000000_000000 1 \
>>>> ${vday}_120000 24 \
>>>> mrms_mm.${vday}12.24h.nc \
>>>> -field 'name="GaugeCorrQPE01H"; level="Z0";
>>>> censor_thresh=[==-3]; censor_value=[-9999];' \
>>>> -pcpdir data \
>>>> -v 5
>>>> And got this err msg:
>>>>
>>>> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
>>>> type "F
>>>> ileType_Gb2".
>>>> ERROR :
>>>> ERROR : VarInfo::set_dict() -> The number of censor thresholds
in
>>>> "censor_thres
>>>> h" (1) must match the number of replacement values in
"censor_val" (0).
>>>> ERROR :
>>>>
>>>> Did I do something wrong? I can't figure why it would consider
>>>> censor_value=[-9999] doesn't count as a value.
>>>>
>>>> Thanks -
>>>>
>>>> Ying
>>>>
>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: pcp_combine for MRMS QPE01H
From: John Halley Gotway
Time: Tue Sep 25 11:28:10 2018
Ying (and Julie),
Thanks for letting us know that the min/max values are reported on
Tide as
-3/-3. That furthers my suspicion that the build on Tide is
problematic.
Julie, can you please take a look at this? Just run plot_data_plane
on one
of the MRMS files in /gpfs/tp1/ptmp/Ying.Lin/mrms.sum/data at
verbosity
level 4:
plot_data_plane
/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/data/GaugeCorr_QPE_01H_00.00_20180920-
180000.grib2
plot.ps 'name="GaugeCorrQPE01H"; level="Z0";' -v 4
The data values *should* reported in the log output as:
DEBUG 4: plane min: -3
DEBUG 4: plane max: 63.4
If they're -3 and -3, then we'll need to recompile the GRIB2C library
and
MET. Keep debugging and reruning this test to until those values are
correctly listed in the log output.
Thanks,
John
On Tue, Sep 25, 2018 at 10:53 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
>
> Hi John,
>
> Thanks for the great tip about looking at the log of
pcp_combine
> log. I saved the run output of runs on both machines. While the
'plane
> min' on are '-3' for all 24h on both machines, 'plane max' on Tide
are
> also all '-3', while on Theia they are 52.6, 77.5, 63.8.
>
> Yes, my pcp_combine on Theia took 12-13 minutes.
>
> You're right about GaugeCorr_QPE already having 24h output (and
3h,
> 6h, 12h). We don't get this on wcoss:/dcom but the short-term
mrms.ncep
> web site has them, and (more usefully) they're also available on the
> http://mtarchive.geol.iastate.edu/ site. I was doing this mainly as
a
> learning opportunity since I was so taken by the nifty
"censor_thresh =
> ==-3; censor_val = -9999" in MET, compared to my previous 'wgrib2'
> manipulation.
>
> Anyway, thanks much for your answer to my question and for the
> great tips, I'm learning a lot. I'll save this test case/data in
case
> of a future test of a recompiled pcp_combine on Tide, but there is
no
> urgency - I can just go wget the QPE_24H instead of pcp_combine.
>
> Please feel free to close this ticket as you see fit.
>
> Ying
>
> On 09/25/2018 11:37 AM, John Halley Gotway via RT wrote:
> > Ying,
> >
> > Thanks for pointing me to the data you're testing. To summarize,
it
> sounds
> > like you're getting different behavior from pcp_combine when
running on
> > theia (output looks good) versus tide (output looks bad).
> >
> > As a side note, I'm surprised you're doing this in the first
place...
> > adding up 24 1-hourly MRMS files to a make a 24-hour accumulation.
I
> > thought that MRMS would already be available as a 24-hour
accumulation.
> Is
> > that not the case?
> >
> > Next, we really should talk to folks generating the MRMS GRIB2
files.
> > Frankly its silly that they aren't encoding the bad data values
well. If
> > we could modify a couple lines in the GRIB2 generation process, we
should
> > be able to encoded missing data values properly and save everybody
using
> > this data a lot of trouble downstream.
> >
> > But on to the problem at hand. Looking at your script, I see that
you've
> > turned up the verbosity level to 5 (-v 5) to see more logging
output.
> > That's a great idea... and you can also use the "-log" option to
write
> the
> > output to a log file (-log run_pcp_combine_on_theia.log).
> >
> > Here's some ways you could approach this problem:
> >
> > (1) Run the same pcp_combine on theia and tide using verbosity
level 5
> and
> > writing a log file. Rather than diffing the NetCDF output files,
diff
> the
> > log files. In particular, look to see how many values were
censored by
> > comparing these types of lines:
> > DEBUG 3: Censored values for 8793570 of 24500000 grid points.
> >
> > Also compare the min/max values, as reported by MET by comparing
these
> > types of lines:
> > DEBUG 4: plane min: -3
> > DEBUG 4: plane max: 28.9
> >
> > If those numbers differ, then there's a problem in how MET is
reading the
> > data from the GRIB2 files.
> >
> > (2) My guess is that you'll see a difference in the numbers in
(1). And
> > that'll indicate that there's a problem in the installation on
tide. If
> > the problem is in reading data from GRIB2 files, then the problem
is most
> > likely in how the GRIB2C library was compiled. If that's the
case, I'll
> > work with Julie Prestopnik to recompile MET and the GRIB2-related
> libraries.
> >
> > Lastly, while pcp_combine should work on this, I've found that
running
> > theia is painfully slow. Computing the sum over 24 files took 14
> minutes!
> >
> > We should use this example to profile the MET code and look for
> > opportunities to speed it up.
> >
> > Thanks,
> > John
> >
> > On Mon, Sep 24, 2018 at 7:54 PM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
> >>
> >> Thanks very much John! I was staring at the "censor_thresh =
[...};
> >> censor_val = [...]" and it didn't occur to me that it should have
been
> >> "censor_val" not "censor_value".
> >>
> >> It seemed to work on Tide (/gpfs/tp1/ptmp/Ying.Lin/mrms.sum), but
when I
> >> tried to look at the 24h nc file with plot_data_plane the field
appears
> >> to be empty. I copied the data and script files to Theia
> >> (/scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/)
> >> thinking that it would be easier for you or someone else at
met_help to
> >> look at a Theia directory. After running pcp_combine and
> >> plot_data_plane on Theia, the resulting image seems to be the
right 24h
> >> accumulation. I then scp'd Theia's
> >> /scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/
> >> mrms_mm.2018092112.24h.nc
> >> to Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/ and renamed it
> >> "mrms_24h_theia.nc", and plot_data_plane of this theia file on
Tide
> also
> >> seems to be the right image.
> >>
> >> So something appears to be wrong when I ran pcp_combine on Tide
(likely
> >> something I did, but I can't figure out what). I did a 'ncdump'
on
> >> mrms_mm.2018092112.24h.nc (pcp_combine'd on Tide) and
mrms_24h_theia.nc
> >> - see Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/dump.tide and
dump.theia and
> >> they are different. The Tide dump file is smaller. I'm totally
stymied
> >> now. Any suggestions on how I might proceed?
> >>
> >> Thank you again -
> >>
> >> Ying
> >>
> >> On 09/24/2018 04:19 PM, John Halley Gotway via RT wrote:
> >>> Ying,
> >>>
> >>> Sorry, my mistake! I told you to use "censor_value" but it
should be
> >>> "censor_val". The answer's there in that ERROR message:
> >>> must match the number of replacement values in "censor_val"
> >>>
> >>> Please try again using...
> >>> censor_thresh = [eq-3]; censor_val = [-9999];
> >>>
> >>> Also, note that when defining arrays in MET config settings, if
the
> array
> >>> is of length 1, you don't have to use the square brackets:
> >>> censor_thresh = eq-3; censor_val = -9999;
> >>>
> >>> Also, note that 'eq' works just as well as '=='. So the
following
> works
> >>> too:
> >>> censor_thresh = ==-3; censor_val = -9999;
> >>>
> >>> You just need to have a space between the single '=' and the
double
> '=='.
> >>> Generally, when defining thresholds on the command line, I
prefer the
> >>> Fortran syntax (eq, lt, le, and so on) so as to avoid using
greater
> than
> >>> (>) and less than (<) characters since they can cause trouble if
the
> >> shell
> >>> interprets them.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Sep 24, 2018 at 11:49 AM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>> Mon Sep 24 11:49:14 2018: Request 87101 was acted upon.
> >>>> Transaction: Ticket created by ying.lin at noaa.gov
> >>>> Queue: met_help
> >>>> Subject: pcp_combine for MRMS QPE01H
> >>>> Owner: Nobody
> >>>> Requestors: ying.lin at noaa.gov
> >>>> Status: new
> >>>> Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101
> >>>>
> >>>> This is about today's MET+ users meeting, the issue of how to
use
> >>>> pcp_combine on MRMS QPE01H, where missing data is denoted by '-
3'. I
> >>>> tried to follow John's suggestion, with this:
> >>>>
> >>>> pcp_combine -sum 00000000_000000 1 \
> >>>> ${vday}_120000 24 \
> >>>> mrms_mm.${vday}12.24h.nc \
> >>>> -field 'name="GaugeCorrQPE01H";
level="Z0";
> >>>> censor_thresh=[==-3]; censor_value=[-9999];' \
> >>>> -pcpdir data \
> >>>> -v 5
> >>>> And got this err msg:
> >>>>
> >>>> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of
> >>>> type "F
> >>>> ileType_Gb2".
> >>>> ERROR :
> >>>> ERROR : VarInfo::set_dict() -> The number of censor thresholds
in
> >>>> "censor_thres
> >>>> h" (1) must match the number of replacement values in
"censor_val"
> (0).
> >>>> ERROR :
> >>>>
> >>>> Did I do something wrong? I can't figure why it would consider
> >>>> censor_value=[-9999] doesn't count as a value.
> >>>>
> >>>> Thanks -
> >>>>
> >>>> Ying
> >>>>
> >>>>
> >>>> --
> >>>> Ying Lin
> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>> NCWCP Cubicle No. 2015
> >>>> Ying.Lin at noaa.gov
> >>>>
> >>>>
> >>>>
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: pcp_combine for MRMS QPE01H
From: Julie Prestopnik
Time: Tue Sep 25 12:59:39 2018
Hi John and Ying.
John, thank you for giving me the direct command to run and for
telling me
what the output should be. As you suspected, I did get -3 and -3 and
received a libpng warning. I went ahead and recompiled zlib, libpng,
jasper, GRIB2C, and met-7.0 (with the latest patches).
I now get a successful run on tide with the expected values.
Ying, please let me know how it goes. Also, I assumed you were using
met-7.0. If that is not the case, please let me know.
Thanks,
Julie
On Tue, Sep 25, 2018 at 11:28 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Ying (and Julie),
>
> Thanks for letting us know that the min/max values are reported on
Tide as
> -3/-3. That furthers my suspicion that the build on Tide is
problematic.
>
> Julie, can you please take a look at this? Just run plot_data_plane
on
> one of the MRMS files in /gpfs/tp1/ptmp/Ying.Lin/mrms.sum/data at
verbosity
> level 4:
>
> plot_data_plane
>
/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/data/GaugeCorr_QPE_01H_00.00_20180920-
180000.grib2
> plot.ps 'name="GaugeCorrQPE01H"; level="Z0";' -v 4
>
> The data values *should* reported in the log output as:
> DEBUG 4: plane min: -3
> DEBUG 4: plane max: 63.4
>
> If they're -3 and -3, then we'll need to recompile the GRIB2C
library and
> MET. Keep debugging and reruning this test to until those values
are
> correctly listed in the log output.
>
> Thanks,
> John
>
> On Tue, Sep 25, 2018 at 10:53 AM Ying Lin via RT <met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
>>
>> Hi John,
>>
>> Thanks for the great tip about looking at the log of
pcp_combine
>> log. I saved the run output of runs on both machines. While the
'plane
>> min' on are '-3' for all 24h on both machines, 'plane max' on Tide
are
>> also all '-3', while on Theia they are 52.6, 77.5, 63.8.
>>
>> Yes, my pcp_combine on Theia took 12-13 minutes.
>>
>> You're right about GaugeCorr_QPE already having 24h output
(and 3h,
>> 6h, 12h). We don't get this on wcoss:/dcom but the short-term
mrms.ncep
>> web site has them, and (more usefully) they're also available on
the
>> http://mtarchive.geol.iastate.edu/ site. I was doing this mainly
as a
>> learning opportunity since I was so taken by the nifty
"censor_thresh =
>> ==-3; censor_val = -9999" in MET, compared to my previous 'wgrib2'
>> manipulation.
>>
>> Anyway, thanks much for your answer to my question and for the
>> great tips, I'm learning a lot. I'll save this test case/data in
case
>> of a future test of a recompiled pcp_combine on Tide, but there is
no
>> urgency - I can just go wget the QPE_24H instead of pcp_combine.
>>
>> Please feel free to close this ticket as you see fit.
>>
>> Ying
>>
>> On 09/25/2018 11:37 AM, John Halley Gotway via RT wrote:
>> > Ying,
>> >
>> > Thanks for pointing me to the data you're testing. To summarize,
it
>> sounds
>> > like you're getting different behavior from pcp_combine when
running on
>> > theia (output looks good) versus tide (output looks bad).
>> >
>> > As a side note, I'm surprised you're doing this in the first
place...
>> > adding up 24 1-hourly MRMS files to a make a 24-hour
accumulation. I
>> > thought that MRMS would already be available as a 24-hour
>> accumulation. Is
>> > that not the case?
>> >
>> > Next, we really should talk to folks generating the MRMS GRIB2
files.
>> > Frankly its silly that they aren't encoding the bad data values
well.
>> If
>> > we could modify a couple lines in the GRIB2 generation process,
we
>> should
>> > be able to encoded missing data values properly and save
everybody using
>> > this data a lot of trouble downstream.
>> >
>> > But on to the problem at hand. Looking at your script, I see
that
>> you've
>> > turned up the verbosity level to 5 (-v 5) to see more logging
output.
>> > That's a great idea... and you can also use the "-log" option to
write
>> the
>> > output to a log file (-log run_pcp_combine_on_theia.log).
>> >
>> > Here's some ways you could approach this problem:
>> >
>> > (1) Run the same pcp_combine on theia and tide using verbosity
level 5
>> and
>> > writing a log file. Rather than diffing the NetCDF output files,
diff
>> the
>> > log files. In particular, look to see how many values were
censored by
>> > comparing these types of lines:
>> > DEBUG 3: Censored values for 8793570 of 24500000 grid points.
>> >
>> > Also compare the min/max values, as reported by MET by comparing
these
>> > types of lines:
>> > DEBUG 4: plane min: -3
>> > DEBUG 4: plane max: 28.9
>> >
>> > If those numbers differ, then there's a problem in how MET is
reading
>> the
>> > data from the GRIB2 files.
>> >
>> > (2) My guess is that you'll see a difference in the numbers in
(1). And
>> > that'll indicate that there's a problem in the installation on
tide.
>> If
>> > the problem is in reading data from GRIB2 files, then the problem
is
>> most
>> > likely in how the GRIB2C library was compiled. If that's the
case,
>> I'll
>> > work with Julie Prestopnik to recompile MET and the GRIB2-related
>> libraries.
>> >
>> > Lastly, while pcp_combine should work on this, I've found that
running
>> > theia is painfully slow. Computing the sum over 24 files took 14
>> minutes!
>> >
>> > We should use this example to profile the MET code and look for
>> > opportunities to speed it up.
>> >
>> > Thanks,
>> > John
>> >
>> > On Mon, Sep 24, 2018 at 7:54 PM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>> >
>> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
>> >>
>> >> Thanks very much John! I was staring at the "censor_thresh =
[...};
>> >> censor_val = [...]" and it didn't occur to me that it should
have been
>> >> "censor_val" not "censor_value".
>> >>
>> >> It seemed to work on Tide (/gpfs/tp1/ptmp/Ying.Lin/mrms.sum),
but when
>> I
>> >> tried to look at the 24h nc file with plot_data_plane the field
appears
>> >> to be empty. I copied the data and script files to Theia
>> >> (/scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/)
>> >> thinking that it would be easier for you or someone else at
met_help to
>> >> look at a Theia directory. After running pcp_combine and
>> >> plot_data_plane on Theia, the resulting image seems to be the
right 24h
>> >> accumulation. I then scp'd Theia's
>> >> /scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/
>> >> mrms_mm.2018092112.24h.nc
>> >> to Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/ and renamed it
>> >> "mrms_24h_theia.nc", and plot_data_plane of this theia file on
Tide
>> also
>> >> seems to be the right image.
>> >>
>> >> So something appears to be wrong when I ran pcp_combine on Tide
(likely
>> >> something I did, but I can't figure out what). I did a 'ncdump'
on
>> >> mrms_mm.2018092112.24h.nc (pcp_combine'd on Tide) and
>> mrms_24h_theia.nc
>> >> - see Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/dump.tide and
dump.theia
>> and
>> >> they are different. The Tide dump file is smaller. I'm totally
>> stymied
>> >> now. Any suggestions on how I might proceed?
>> >>
>> >> Thank you again -
>> >>
>> >> Ying
>> >>
>> >> On 09/24/2018 04:19 PM, John Halley Gotway via RT wrote:
>> >>> Ying,
>> >>>
>> >>> Sorry, my mistake! I told you to use "censor_value" but it
should be
>> >>> "censor_val". The answer's there in that ERROR message:
>> >>> must match the number of replacement values in
"censor_val"
>> >>>
>> >>> Please try again using...
>> >>> censor_thresh = [eq-3]; censor_val = [-9999];
>> >>>
>> >>> Also, note that when defining arrays in MET config settings, if
the
>> array
>> >>> is of length 1, you don't have to use the square brackets:
>> >>> censor_thresh = eq-3; censor_val = -9999;
>> >>>
>> >>> Also, note that 'eq' works just as well as '=='. So the
following
>> works
>> >>> too:
>> >>> censor_thresh = ==-3; censor_val = -9999;
>> >>>
>> >>> You just need to have a space between the single '=' and the
double
>> '=='.
>> >>> Generally, when defining thresholds on the command line, I
prefer the
>> >>> Fortran syntax (eq, lt, le, and so on) so as to avoid using
greater
>> than
>> >>> (>) and less than (<) characters since they can cause trouble
if the
>> >> shell
>> >>> interprets them.
>> >>>
>> >>> Thanks,
>> >>> John
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Mon, Sep 24, 2018 at 11:49 AM Ying Lin via RT
<met_help at ucar.edu>
>> >> wrote:
>> >>>> Mon Sep 24 11:49:14 2018: Request 87101 was acted upon.
>> >>>> Transaction: Ticket created by ying.lin at noaa.gov
>> >>>> Queue: met_help
>> >>>> Subject: pcp_combine for MRMS QPE01H
>> >>>> Owner: Nobody
>> >>>> Requestors: ying.lin at noaa.gov
>> >>>> Status: new
>> >>>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101
>> >>>>
>> >>>> This is about today's MET+ users meeting, the issue of how to
use
>> >>>> pcp_combine on MRMS QPE01H, where missing data is denoted by
'-3'. I
>> >>>> tried to follow John's suggestion, with this:
>> >>>>
>> >>>> pcp_combine -sum 00000000_000000 1 \
>> >>>> ${vday}_120000 24 \
>> >>>> mrms_mm.${vday}12.24h.nc \
>> >>>> -field 'name="GaugeCorrQPE01H";
level="Z0";
>> >>>> censor_thresh=[==-3]; censor_value=[-9999];' \
>> >>>> -pcpdir data \
>> >>>> -v 5
>> >>>> And got this err msg:
>> >>>>
>> >>>> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
>> object of
>> >>>> type "F
>> >>>> ileType_Gb2".
>> >>>> ERROR :
>> >>>> ERROR : VarInfo::set_dict() -> The number of censor
thresholds in
>> >>>> "censor_thres
>> >>>> h" (1) must match the number of replacement values in
"censor_val"
>> (0).
>> >>>> ERROR :
>> >>>>
>> >>>> Did I do something wrong? I can't figure why it would
consider
>> >>>> censor_value=[-9999] doesn't count as a value.
>> >>>>
>> >>>> Thanks -
>> >>>>
>> >>>> Ying
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Ying Lin
>> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> >>>> NCWCP Cubicle No. 2015
>> >>>> Ying.Lin at noaa.gov
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >> --
>> >> Ying Lin
>> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> >> NCWCP Cubicle No. 2015
>> >> Ying.Lin at noaa.gov
>> >>
>> >>
>> >>
>> >>
>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87101] pcp_combine for MRMS QPE01H
From: Ying Lin
Time: Tue Sep 25 13:36:24 2018
Hi Julie and John,
Yes, I've been using MET-7.0. pcp_combine on Tide seems to work
with MRMS now - "plane max" now has "normal" values rather "-3" and
plot_data_plane for the resulting 24h.nc file looks normal.
Thank you very much for the quick diagnosis and fix!
Ying
On 09/25/2018 02:59 PM, Julie Prestopnik via RT wrote:
> Hi John and Ying.
>
> John, thank you for giving me the direct command to run and for
telling me
> what the output should be. As you suspected, I did get -3 and -3
and
> received a libpng warning. I went ahead and recompiled zlib,
libpng,
> jasper, GRIB2C, and met-7.0 (with the latest patches).
>
> I now get a successful run on tide with the expected values.
>
> Ying, please let me know how it goes. Also, I assumed you were
using
> met-7.0. If that is not the case, please let me know.
>
> Thanks,
> Julie
>
> On Tue, Sep 25, 2018 at 11:28 AM John Halley Gotway
<johnhg at ucar.edu> wrote:
>
>> Ying (and Julie),
>>
>> Thanks for letting us know that the min/max values are reported on
Tide as
>> -3/-3. That furthers my suspicion that the build on Tide is
problematic.
>>
>> Julie, can you please take a look at this? Just run
plot_data_plane on
>> one of the MRMS files in /gpfs/tp1/ptmp/Ying.Lin/mrms.sum/data at
verbosity
>> level 4:
>>
>> plot_data_plane
>>
/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/data/GaugeCorr_QPE_01H_00.00_20180920-
180000.grib2
>> plot.ps 'name="GaugeCorrQPE01H"; level="Z0";' -v 4
>>
>> The data values *should* reported in the log output as:
>> DEBUG 4: plane min: -3
>> DEBUG 4: plane max: 63.4
>>
>> If they're -3 and -3, then we'll need to recompile the GRIB2C
library and
>> MET. Keep debugging and reruning this test to until those values
are
>> correctly listed in the log output.
>>
>> Thanks,
>> John
>>
>> On Tue, Sep 25, 2018 at 10:53 AM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
>>>
>>> Hi John,
>>>
>>> Thanks for the great tip about looking at the log of
pcp_combine
>>> log. I saved the run output of runs on both machines. While the
'plane
>>> min' on are '-3' for all 24h on both machines, 'plane max' on Tide
are
>>> also all '-3', while on Theia they are 52.6, 77.5, 63.8.
>>>
>>> Yes, my pcp_combine on Theia took 12-13 minutes.
>>>
>>> You're right about GaugeCorr_QPE already having 24h output
(and 3h,
>>> 6h, 12h). We don't get this on wcoss:/dcom but the short-term
mrms.ncep
>>> web site has them, and (more usefully) they're also available on
the
>>> http://mtarchive.geol.iastate.edu/ site. I was doing this mainly
as a
>>> learning opportunity since I was so taken by the nifty
"censor_thresh =
>>> ==-3; censor_val = -9999" in MET, compared to my previous 'wgrib2'
>>> manipulation.
>>>
>>> Anyway, thanks much for your answer to my question and for
the
>>> great tips, I'm learning a lot. I'll save this test case/data in
case
>>> of a future test of a recompiled pcp_combine on Tide, but there is
no
>>> urgency - I can just go wget the QPE_24H instead of pcp_combine.
>>>
>>> Please feel free to close this ticket as you see fit.
>>>
>>> Ying
>>>
>>> On 09/25/2018 11:37 AM, John Halley Gotway via RT wrote:
>>>> Ying,
>>>>
>>>> Thanks for pointing me to the data you're testing. To summarize,
it
>>> sounds
>>>> like you're getting different behavior from pcp_combine when
running on
>>>> theia (output looks good) versus tide (output looks bad).
>>>>
>>>> As a side note, I'm surprised you're doing this in the first
place...
>>>> adding up 24 1-hourly MRMS files to a make a 24-hour
accumulation. I
>>>> thought that MRMS would already be available as a 24-hour
>>> accumulation. Is
>>>> that not the case?
>>>>
>>>> Next, we really should talk to folks generating the MRMS GRIB2
files.
>>>> Frankly its silly that they aren't encoding the bad data values
well.
>>> If
>>>> we could modify a couple lines in the GRIB2 generation process,
we
>>> should
>>>> be able to encoded missing data values properly and save
everybody using
>>>> this data a lot of trouble downstream.
>>>>
>>>> But on to the problem at hand. Looking at your script, I see
that
>>> you've
>>>> turned up the verbosity level to 5 (-v 5) to see more logging
output.
>>>> That's a great idea... and you can also use the "-log" option to
write
>>> the
>>>> output to a log file (-log run_pcp_combine_on_theia.log).
>>>>
>>>> Here's some ways you could approach this problem:
>>>>
>>>> (1) Run the same pcp_combine on theia and tide using verbosity
level 5
>>> and
>>>> writing a log file. Rather than diffing the NetCDF output files,
diff
>>> the
>>>> log files. In particular, look to see how many values were
censored by
>>>> comparing these types of lines:
>>>> DEBUG 3: Censored values for 8793570 of 24500000 grid
points.
>>>>
>>>> Also compare the min/max values, as reported by MET by comparing
these
>>>> types of lines:
>>>> DEBUG 4: plane min: -3
>>>> DEBUG 4: plane max: 28.9
>>>>
>>>> If those numbers differ, then there's a problem in how MET is
reading
>>> the
>>>> data from the GRIB2 files.
>>>>
>>>> (2) My guess is that you'll see a difference in the numbers in
(1). And
>>>> that'll indicate that there's a problem in the installation on
tide.
>>> If
>>>> the problem is in reading data from GRIB2 files, then the problem
is
>>> most
>>>> likely in how the GRIB2C library was compiled. If that's the
case,
>>> I'll
>>>> work with Julie Prestopnik to recompile MET and the GRIB2-related
>>> libraries.
>>>> Lastly, while pcp_combine should work on this, I've found that
running
>>>> theia is painfully slow. Computing the sum over 24 files took 14
>>> minutes!
>>>> We should use this example to profile the MET code and look for
>>>> opportunities to speed it up.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On Mon, Sep 24, 2018 at 7:54 PM Ying Lin via RT
<met_help at ucar.edu>
>>> wrote:
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101 >
>>>>>
>>>>> Thanks very much John! I was staring at the "censor_thresh =
[...};
>>>>> censor_val = [...]" and it didn't occur to me that it should
have been
>>>>> "censor_val" not "censor_value".
>>>>>
>>>>> It seemed to work on Tide (/gpfs/tp1/ptmp/Ying.Lin/mrms.sum),
but when
>>> I
>>>>> tried to look at the 24h nc file with plot_data_plane the field
appears
>>>>> to be empty. I copied the data and script files to Theia
>>>>> (/scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/)
>>>>> thinking that it would be easier for you or someone else at
met_help to
>>>>> look at a Theia directory. After running pcp_combine and
>>>>> plot_data_plane on Theia, the resulting image seems to be the
right 24h
>>>>> accumulation. I then scp'd Theia's
>>>>> /scratch4/NCEPDEV/fv3-cam/save/Ying.Lin/met_question/mrms_sum/
>>>>> mrms_mm.2018092112.24h.nc
>>>>> to Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/ and renamed it
>>>>> "mrms_24h_theia.nc", and plot_data_plane of this theia file on
Tide
>>> also
>>>>> seems to be the right image.
>>>>>
>>>>> So something appears to be wrong when I ran pcp_combine on Tide
(likely
>>>>> something I did, but I can't figure out what). I did a 'ncdump'
on
>>>>> mrms_mm.2018092112.24h.nc (pcp_combine'd on Tide) and
>>> mrms_24h_theia.nc
>>>>> - see Tide:/gpfs/tp1/ptmp/Ying.Lin/mrms.sum/dump.tide and
dump.theia
>>> and
>>>>> they are different. The Tide dump file is smaller. I'm totally
>>> stymied
>>>>> now. Any suggestions on how I might proceed?
>>>>>
>>>>> Thank you again -
>>>>>
>>>>> Ying
>>>>>
>>>>> On 09/24/2018 04:19 PM, John Halley Gotway via RT wrote:
>>>>>> Ying,
>>>>>>
>>>>>> Sorry, my mistake! I told you to use "censor_value" but it
should be
>>>>>> "censor_val". The answer's there in that ERROR message:
>>>>>> must match the number of replacement values in
"censor_val"
>>>>>>
>>>>>> Please try again using...
>>>>>> censor_thresh = [eq-3]; censor_val = [-9999];
>>>>>>
>>>>>> Also, note that when defining arrays in MET config settings, if
the
>>> array
>>>>>> is of length 1, you don't have to use the square brackets:
>>>>>> censor_thresh = eq-3; censor_val = -9999;
>>>>>>
>>>>>> Also, note that 'eq' works just as well as '=='. So the
following
>>> works
>>>>>> too:
>>>>>> censor_thresh = ==-3; censor_val = -9999;
>>>>>>
>>>>>> You just need to have a space between the single '=' and the
double
>>> '=='.
>>>>>> Generally, when defining thresholds on the command line, I
prefer the
>>>>>> Fortran syntax (eq, lt, le, and so on) so as to avoid using
greater
>>> than
>>>>>> (>) and less than (<) characters since they can cause trouble
if the
>>>>> shell
>>>>>> interprets them.
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 24, 2018 at 11:49 AM Ying Lin via RT
<met_help at ucar.edu>
>>>>> wrote:
>>>>>>> Mon Sep 24 11:49:14 2018: Request 87101 was acted upon.
>>>>>>> Transaction: Ticket created by ying.lin at noaa.gov
>>>>>>> Queue: met_help
>>>>>>> Subject: pcp_combine for MRMS QPE01H
>>>>>>> Owner: Nobody
>>>>>>> Requestors: ying.lin at noaa.gov
>>>>>>> Status: new
>>>>>>> Ticket <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87101
>>>>>>> This is about today's MET+ users meeting, the issue of how to
use
>>>>>>> pcp_combine on MRMS QPE01H, where missing data is denoted by
'-3'. I
>>>>>>> tried to follow John's suggestion, with this:
>>>>>>>
>>>>>>> pcp_combine -sum 00000000_000000 1 \
>>>>>>> ${vday}_120000 24 \
>>>>>>> mrms_mm.${vday}12.24h.nc \
>>>>>>> -field 'name="GaugeCorrQPE01H";
level="Z0";
>>>>>>> censor_thresh=[==-3]; censor_value=[-9999];' \
>>>>>>> -pcpdir data \
>>>>>>> -v 5
>>>>>>> And got this err msg:
>>>>>>>
>>>>>>> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
>>> object of
>>>>>>> type "F
>>>>>>> ileType_Gb2".
>>>>>>> ERROR :
>>>>>>> ERROR : VarInfo::set_dict() -> The number of censor
thresholds in
>>>>>>> "censor_thres
>>>>>>> h" (1) must match the number of replacement values in
"censor_val"
>>> (0).
>>>>>>> ERROR :
>>>>>>>
>>>>>>> Did I do something wrong? I can't figure why it would
consider
>>>>>>> censor_value=[-9999] doesn't count as a value.
>>>>>>>
>>>>>>> Thanks -
>>>>>>>
>>>>>>> Ying
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ying Lin
>>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>>> NCWCP Cubicle No. 2015
>>>>>>> Ying.Lin at noaa.gov
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> --
>>>>> Ying Lin
>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>> NCWCP Cubicle No. 2015
>>>>> Ying.Lin at noaa.gov
>>>>>
>>>>>
>>>>>
>>>>>
>>> --
>>> Ying Lin
>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>> NCWCP Cubicle No. 2015
>>> Ying.Lin at noaa.gov
>>>
>>>
>>>
>>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list