[Met_help] [rt.rap.ucar.edu #94703] History for mask specification for grid_stat
George McCabe via RT
met_help at ucar.edu
Wed Apr 1 12:38:15 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I’m trying to have grid_stat for two gridded model output over certain area, but it shows no matched pairs for the mask area in my log file.
I setup a poly file for tropical region, like this
TROP
20.0 -180.0
20.0 180.0
-20.0 180.0
-20.0 -180.0
20.0 -180.0
I’m wondering if the longitude value is matter, because one of my output is 0 to 360, another is -180 to 180 for longitude.
Also, I’m not sure if I use the mask in the right way.
Please give me some guidance.
Thanks,
Shih-Wei
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: mask specification for grid_stat
From: John Halley Gotway
Time: Tue Mar 24 16:48:31 2020
Shih,
I'd recommend that you run the gen_vx_mask tool to create this masking
region. It looks like you really just want to include -20 to 20
latitude.
Rather than defining a polyline region to do this, I think it's
simpler to
use the latitude masking option in gen_vx_mask. To illustrate:
(1) pull a sample global GFS file:
wget
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
(2) run gen_vx_mask to generate the masking region:
gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
tropics_mask.nc
-type lat -thresh '>=-20&&<=20' -name tropics
(3) plot the result to show the mask
plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
level="(*,*)";'
See the attached pdf.
When you run grid-stat you'll just setup the config file like this:
mask = {
grid = [];
poly = ["/path/to/tropics_mask.nc" ];
}
Where you replace /path/to with the actual path that file on your
machine.
Thanks,
John
On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
>
> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> Transaction: Ticket created by swei at albany.edu
> Queue: met_help
> Subject: mask specification for grid_stat
> Owner: Nobody
> Requestors: swei at albany.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
>
> Hi,
>
> I’m trying to have grid_stat for two gridded model output over
certain
> area, but it shows no matched pairs for the mask area in my log
file.
> I setup a poly file for tropical region, like this
> TROP
> 20.0 -180.0
> 20.0 180.0
> -20.0 180.0
> -20.0 -180.0
> 20.0 -180.0
> I’m wondering if the longitude value is matter, because one of my
output
> is 0 to 360, another is -180 to 180 for longitude.
> Also, I’m not sure if I use the mask in the right way.
> Please give me some guidance.
>
> Thanks,
> Shih-Wei
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94703] mask specification for grid_stat
From: Wei, Shih Wei
Time: Tue Mar 24 17:01:38 2020
Hi John,
Thank you.
Topical region is my test run, I may need couple regions once I can
get the statistics.
You would recommend people use gen_vx_mask if the boundary of mask
area is not too complicated, right?
I encountered another problem in the stat_analysis, could I also
consult this with you?
Best,
Shih-Wei
> On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Shih,
>
> I'd recommend that you run the gen_vx_mask tool to create this
masking
> region. It looks like you really just want to include -20 to 20
latitude.
> Rather than defining a polyline region to do this, I think it's
simpler to
> use the latitude masking option in gen_vx_mask. To illustrate:
>
> (1) pull a sample global GFS file:
> wget
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
>
> (2) run gen_vx_mask to generate the masking region:
> gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
tropics_mask.nc
> -type lat -thresh '>=-20&&<=20' -name tropics
>
> (3) plot the result to show the mask
> plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> level="(*,*)";'
>
> See the attached pdf.
>
> When you run grid-stat you'll just setup the config file like this:
>
> mask = {
> grid = [];
> poly = ["/path/to/tropics_mask.nc" ];
> }
>
> Where you replace /path/to with the actual path that file on your
machine.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
>> Transaction: Ticket created by swei at albany.edu
>> Queue: met_help
>> Subject: mask specification for grid_stat
>> Owner: Nobody
>> Requestors: swei at albany.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>>
>>
>> Hi,
>>
>> I’m trying to have grid_stat for two gridded model output over
certain
>> area, but it shows no matched pairs for the mask area in my log
file.
>> I setup a poly file for tropical region, like this
>> TROP
>> 20.0 -180.0
>> 20.0 180.0
>> -20.0 180.0
>> -20.0 -180.0
>> 20.0 -180.0
>> I’m wondering if the longitude value is matter, because one of my
output
>> is 0 to 360, another is -180 to 180 for longitude.
>> Also, I’m not sure if I use the mask in the right way.
>> Please give me some guidance.
>>
>> Thanks,
>> Shih-Wei
>>
>>
>
> <tropics_mask.pdf>
------------------------------------------------
Subject: mask specification for grid_stat
From: John Halley Gotway
Time: Tue Mar 24 17:07:03 2020
Shih-Wei,
Yes, we typically setup the masking regions first using gen_vx_mask
and
then reference those in the Grid-Stat config file.
Sure, I can answer questions about stat_analysis.
John
On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you.
> Topical region is my test run, I may need couple regions once I can
get
> the statistics.
> You would recommend people use gen_vx_mask if the boundary of mask
area is
> not too complicated, right?
>
> I encountered another problem in the stat_analysis, could I also
consult
> this with you?
>
> Best,
> Shih-Wei
>
> > On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
> >
> > Shih,
> >
> > I'd recommend that you run the gen_vx_mask tool to create this
masking
> > region. It looks like you really just want to include -20 to 20
> latitude.
> > Rather than defining a polyline region to do this, I think it's
simpler
> to
> > use the latitude masking option in gen_vx_mask. To illustrate:
> >
> > (1) pull a sample global GFS file:
> > wget
> >
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
> >
> > (2) run gen_vx_mask to generate the masking region:
> > gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
> tropics_mask.nc
> > -type lat -thresh '>=-20&&<=20' -name tropics
> >
> > (3) plot the result to show the mask
> > plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> > level="(*,*)";'
> >
> > See the attached pdf.
> >
> > When you run grid-stat you'll just setup the config file like
this:
> >
> > mask = {
> > grid = [];
> > poly = ["/path/to/tropics_mask.nc" ];
> > }
> >
> > Where you replace /path/to with the actual path that file on your
> machine.
> >
> > Thanks,
> > John
> >
> > On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
> > wrote:
> >
> >>
> >> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> >> Transaction: Ticket created by swei at albany.edu
> >> Queue: met_help
> >> Subject: mask specification for grid_stat
> >> Owner: Nobody
> >> Requestors: swei at albany.edu
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >>
> >>
> >> Hi,
> >>
> >> I’m trying to have grid_stat for two gridded model output over
certain
> >> area, but it shows no matched pairs for the mask area in my log
file.
> >> I setup a poly file for tropical region, like this
> >> TROP
> >> 20.0 -180.0
> >> 20.0 180.0
> >> -20.0 180.0
> >> -20.0 -180.0
> >> 20.0 -180.0
> >> I’m wondering if the longitude value is matter, because one of my
output
> >> is 0 to 360, another is -180 to 180 for longitude.
> >> Also, I’m not sure if I use the mask in the right way.
> >> Please give me some guidance.
> >>
> >> Thanks,
> >> Shih-Wei
> >>
> >>
> >
> > <tropics_mask.pdf>
>
>
>
>
------------------------------------------------
Subject: mask specification for grid_stat
From: Wei, Shih Wei
Time: Tue Mar 24 17:16:28 2020
John,
Thank you, I’ll try it.
I’m still learning how to use stat_analysis.
Now I can get the stat file for “FULL” domain from grid_stat, when I
point the file into stat_analysis the log file shows
DEBUG 2: Processing 1 STAT files.
DEBUG 2: STAT Lines read = 15
DEBUG 2: STAT Lines retained = 0
What I confused is none of lines in the stat file be retained, then I
got empty output file of stat_analysis.
My log file is
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
Thanks,
Shih-Wei
On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
Yes, we typically setup the masking regions first using gen_vx_mask
and
then reference those in the Grid-Stat config file.
Sure, I can answer questions about stat_analysis.
John
On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you.
Topical region is my test run, I may need couple regions once I can
get
the statistics.
You would recommend people use gen_vx_mask if the boundary of mask
area is
not too complicated, right?
I encountered another problem in the stat_analysis, could I also
consult
this with you?
Best,
Shih-Wei
On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
Shih,
I'd recommend that you run the gen_vx_mask tool to create this masking
region. It looks like you really just want to include -20 to 20
latitude.
Rather than defining a polyline region to do this, I think it's
simpler
to
use the latitude masking option in gen_vx_mask. To illustrate:
(1) pull a sample global GFS file:
wget
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
(2) run gen_vx_mask to generate the masking region:
gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
tropics_mask.nc
-type lat -thresh '>=-20&&<=20' -name tropics
(3) plot the result to show the mask
plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
level="(*,*)";'
See the attached pdf.
When you run grid-stat you'll just setup the config file like this:
mask = {
grid = [];
poly = ["/path/to/tropics_mask.nc" ];
}
Where you replace /path/to with the actual path that file on your
machine.
Thanks,
John
On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
Transaction: Ticket created by swei at albany.edu
Queue: met_help
Subject: mask specification for grid_stat
Owner: Nobody
Requestors: swei at albany.edu
Status: new
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi,
I’m trying to have grid_stat for two gridded model output over certain
area, but it shows no matched pairs for the mask area in my log file.
I setup a poly file for tropical region, like this
TROP
20.0 -180.0
20.0 180.0
-20.0 180.0
-20.0 -180.0
20.0 -180.0
I’m wondering if the longitude value is matter, because one of my
output
is 0 to 360, another is -180 to 180 for longitude.
Also, I’m not sure if I use the mask in the right way.
Please give me some guidance.
Thanks,
Shih-Wei
<tropics_mask.pdf>
------------------------------------------------
Subject: mask specification for grid_stat
From: John Halley Gotway
Time: Tue Mar 24 17:38:10 2020
Shih-Wei,
The critical information you didn't send is what is the job command
that
you're running? I'd recommend getting familiar with running the
STAT-Analysis tool first on the command line, and then once you
understand
the functionality, you can use METplus to automate calls to it.
John
On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> John,
>
> Thank you, I’ll try it.
> I’m still learning how to use stat_analysis.
> Now I can get the stat file for “FULL” domain from grid_stat, when I
point
> the file into stat_analysis the log file shows
> DEBUG 2: Processing 1 STAT files.
> DEBUG 2: STAT Lines read = 15
> DEBUG 2: STAT Lines retained = 0
> What I confused is none of lines in the stat file be retained, then
I got
> empty output file of stat_analysis.
>
> My log file is
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> Thanks,
> Shih-Wei
>
>
> On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> Yes, we typically setup the masking regions first using gen_vx_mask
and
> then reference those in the Grid-Stat config file.
>
> Sure, I can answer questions about stat_analysis.
>
> John
>
> On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you.
> Topical region is my test run, I may need couple regions once I can
get
> the statistics.
> You would recommend people use gen_vx_mask if the boundary of mask
area is
> not too complicated, right?
>
> I encountered another problem in the stat_analysis, could I also
consult
> this with you?
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> Shih,
>
> I'd recommend that you run the gen_vx_mask tool to create this
masking
> region. It looks like you really just want to include -20 to 20
> latitude.
> Rather than defining a polyline region to do this, I think it's
simpler
> to
> use the latitude masking option in gen_vx_mask. To illustrate:
>
> (1) pull a sample global GFS file:
> wget
>
>
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
>
> (2) run gen_vx_mask to generate the masking region:
> gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
> tropics_mask.nc
> -type lat -thresh '>=-20&&<=20' -name tropics
>
> (3) plot the result to show the mask
> plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> level="(*,*)";'
>
> See the attached pdf.
>
> When you run grid-stat you'll just setup the config file like this:
>
> mask = {
> grid = [];
> poly = ["/path/to/tropics_mask.nc" ];
> }
>
> Where you replace /path/to with the actual path that file on your
> machine.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
> wrote:
>
>
> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> Transaction: Ticket created by swei at albany.edu
> Queue: met_help
> Subject: mask specification for grid_stat
> Owner: Nobody
> Requestors: swei at albany.edu
> Status: new
> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703
>
>
>
> Hi,
>
> I’m trying to have grid_stat for two gridded model output over
certain
> area, but it shows no matched pairs for the mask area in my log
file.
> I setup a poly file for tropical region, like this
> TROP
> 20.0 -180.0
> 20.0 180.0
> -20.0 180.0
> -20.0 -180.0
> 20.0 -180.0
> I’m wondering if the longitude value is matter, because one of my
output
> is 0 to 360, another is -180 to 180 for longitude.
> Also, I’m not sure if I use the mask in the right way.
> Please give me some guidance.
>
> Thanks,
> Shih-Wei
>
>
>
> <tropics_mask.pdf>
>
>
>
>
>
>
>
>
------------------------------------------------
Subject: mask specification for grid_stat
From: Wei, Shih Wei
Time: Tue Mar 24 22:13:24 2020
Hi John,
I’m trying to have filter, but I did try “summary” and “aggregate”.
Both show the same message.
I’ll test in command line first.
BTW, the mask file is work, thanks for your suggestion.
Best,
Shih-Wei
On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
The critical information you didn't send is what is the job command
that
you're running? I'd recommend getting familiar with running the
STAT-Analysis tool first on the command line, and then once you
understand
the functionality, you can use METplus to automate calls to it.
John
On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
John,
Thank you, I’ll try it.
I’m still learning how to use stat_analysis.
Now I can get the stat file for “FULL” domain from grid_stat, when I
point
the file into stat_analysis the log file shows
DEBUG 2: Processing 1 STAT files.
DEBUG 2: STAT Lines read = 15
DEBUG 2: STAT Lines retained = 0
What I confused is none of lines in the stat file be retained, then I
got
empty output file of stat_analysis.
My log file is
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
Thanks,
Shih-Wei
On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
Yes, we typically setup the masking regions first using gen_vx_mask
and
then reference those in the Grid-Stat config file.
Sure, I can answer questions about stat_analysis.
John
On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you.
Topical region is my test run, I may need couple regions once I can
get
the statistics.
You would recommend people use gen_vx_mask if the boundary of mask
area is
not too complicated, right?
I encountered another problem in the stat_analysis, could I also
consult
this with you?
Best,
Shih-Wei
On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
wrote:
Shih,
I'd recommend that you run the gen_vx_mask tool to create this masking
region. It looks like you really just want to include -20 to 20
latitude.
Rather than defining a polyline region to do this, I think it's
simpler
to
use the latitude masking option in gen_vx_mask. To illustrate:
(1) pull a sample global GFS file:
wget
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
(2) run gen_vx_mask to generate the masking region:
gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
tropics_mask.nc
-type lat -thresh '>=-20&&<=20' -name tropics
(3) plot the result to show the mask
plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
level="(*,*)";'
See the attached pdf.
When you run grid-stat you'll just setup the config file like this:
mask = {
grid = [];
poly = ["/path/to/tropics_mask.nc" ];
}
Where you replace /path/to with the actual path that file on your
machine.
Thanks,
John
On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
Transaction: Ticket created by swei at albany.edu
Queue: met_help
Subject: mask specification for grid_stat
Owner: Nobody
Requestors: swei at albany.edu
Status: new
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi,
I’m trying to have grid_stat for two gridded model output over certain
area, but it shows no matched pairs for the mask area in my log file.
I setup a poly file for tropical region, like this
TROP
20.0 -180.0
20.0 180.0
-20.0 180.0
-20.0 -180.0
20.0 -180.0
I’m wondering if the longitude value is matter, because one of my
output
is 0 to 360, another is -180 to 180 for longitude.
Also, I’m not sure if I use the mask in the right way.
Please give me some guidance.
Thanks,
Shih-Wei
<tropics_mask.pdf>
------------------------------------------------
Subject: mask specification for grid_stat
From: John Halley Gotway
Time: Wed Mar 25 10:09:37 2020
Shih-Wei,
OK, I logged on to Cheyenne and looked in the log file you sent:
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis can
either run with a config file (as METplus does it) or directly on the
command line. I used the log messages to piece together the command
line
job shown below:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
-obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
-dump_row CAMSiRA_.stat*
This job runs and still gets 0 matching lines:
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
Rerunning but at a higher verbosity level (-v 3), I see the .stat file
it's
reading:
*DEBUG 3: Processing STAT file
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
... 1 of 1*
Comparing the contents of that file to the job command options,
there's a
very slight difference in the OBS_LEV column:
'0,4,*,*' != '(0,4,*,*)'
'0,7,*,*' != '(0,7,*,*)'
'0,11,*,*' != '(0,11,*,*)'
Rerunning that job without the enclosing parentheses results in 3
matching
lines:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
-obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type SL1L2
-dump_row CAMSiRA_.stat*
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
and OBS_LEV settings. I'll talk to George McCabe to let him know
about
this issue.
Thanks,
John
On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> I’m trying to have filter, but I did try “summary” and “aggregate”.
> Both show the same message.
> I’ll test in command line first.
> BTW, the mask file is work, thanks for your suggestion.
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> The critical information you didn't send is what is the job command
that
> you're running? I'd recommend getting familiar with running the
> STAT-Analysis tool first on the command line, and then once you
understand
> the functionality, you can use METplus to automate calls to it.
>
> John
>
> On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> John,
>
> Thank you, I’ll try it.
> I’m still learning how to use stat_analysis.
> Now I can get the stat file for “FULL” domain from grid_stat, when I
point
> the file into stat_analysis the log file shows
> DEBUG 2: Processing 1 STAT files.
> DEBUG 2: STAT Lines read = 15
> DEBUG 2: STAT Lines retained = 0
> What I confused is none of lines in the stat file be retained, then
I got
> empty output file of stat_analysis.
>
> My log file is
>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> Thanks,
> Shih-Wei
>
>
> On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> Yes, we typically setup the masking regions first using gen_vx_mask
and
> then reference those in the Grid-Stat config file.
>
> Sure, I can answer questions about stat_analysis.
>
> John
>
> On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you.
> Topical region is my test run, I may need couple regions once I can
get
> the statistics.
> You would recommend people use gen_vx_mask if the boundary of mask
area is
> not too complicated, right?
>
> I encountered another problem in the stat_analysis, could I also
consult
> this with you?
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
> wrote:
>
> Shih,
>
> I'd recommend that you run the gen_vx_mask tool to create this
masking
> region. It looks like you really just want to include -20 to 20
> latitude.
> Rather than defining a polyline region to do this, I think it's
simpler
> to
> use the latitude masking option in gen_vx_mask. To illustrate:
>
> (1) pull a sample global GFS file:
> wget
>
>
>
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
>
> (2) run gen_vx_mask to generate the masking region:
> gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
> tropics_mask.nc
> -type lat -thresh '>=-20&&<=20' -name tropics
>
> (3) plot the result to show the mask
> plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> level="(*,*)";'
>
> See the attached pdf.
>
> When you run grid-stat you'll just setup the config file like this:
>
> mask = {
> grid = [];
> poly = ["/path/to/tropics_mask.nc" ];
> }
>
> Where you replace /path/to with the actual path that file on your
> machine.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
> wrote:
>
>
> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> Transaction: Ticket created by swei at albany.edu
> Queue: met_help
> Subject: mask specification for grid_stat
> Owner: Nobody
> Requestors: swei at albany.edu
> Status: new
> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703
>
>
>
> Hi,
>
> I’m trying to have grid_stat for two gridded model output over
certain
> area, but it shows no matched pairs for the mask area in my log
file.
> I setup a poly file for tropical region, like this
> TROP
> 20.0 -180.0
> 20.0 180.0
> -20.0 180.0
> -20.0 -180.0
> 20.0 -180.0
> I’m wondering if the longitude value is matter, because one of my
output
> is 0 to 360, another is -180 to 180 for longitude.
> Also, I’m not sure if I use the mask in the right way.
> Please give me some guidance.
>
> Thanks,
> Shih-Wei
>
>
>
> <tropics_mask.pdf>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94703] mask specification for grid_stat
From: Wei, Shih Wei
Time: Wed Mar 25 10:22:21 2020
Hi John,
Thank you so much.
That’s the reason.
How many verbosity level that user can set?
I did notice that is different to what I set in the configuration
files, but I thought it will handle by METplus properly.
Best,
Shih-We
> On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Shih-Wei,
>
> OK, I logged on to Cheyenne and looked in the log file you sent:
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis
can
> either run with a config file (as METplus does it) or directly on
the
> command line. I used the log messages to piece together the command
line
> job shown below:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
> -obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
> This job runs and still gets 0 matching lines:
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
>
> Rerunning but at a higher verbosity level (-v 3), I see the .stat
file it's
> reading:
> *DEBUG 3: Processing STAT file
>
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
> ... 1 of 1*
>
> Comparing the contents of that file to the job command options,
there's a
> very slight difference in the OBS_LEV column:
> '0,4,*,*' != '(0,4,*,*)'
> '0,7,*,*' != '(0,7,*,*)'
> '0,11,*,*' != '(0,11,*,*)'
>
> Rerunning that job without the enclosing parentheses results in 3
matching
> lines:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
> -obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
>
> So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
> and OBS_LEV settings. I'll talk to George McCabe to let him know
about
> this issue.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>>
>> Hi John,
>>
>> I’m trying to have filter, but I did try “summary” and “aggregate”.
>> Both show the same message.
>> I’ll test in command line first.
>> BTW, the mask file is work, thanks for your suggestion.
>>
>> Best,
>> Shih-Wei
>>
>> On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT
<met_help at ucar.edu
>> <mailto:met_help at ucar.edu>> wrote:
>>
>> Shih-Wei,
>>
>> The critical information you didn't send is what is the job command
that
>> you're running? I'd recommend getting familiar with running the
>> STAT-Analysis tool first on the command line, and then once you
understand
>> the functionality, you can use METplus to automate calls to it.
>>
>> John
>>
>> On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu
>> <mailto:met_help at ucar.edu>>
>> wrote:
>>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>>
>> John,
>>
>> Thank you, I’ll try it.
>> I’m still learning how to use stat_analysis.
>> Now I can get the stat file for “FULL” domain from grid_stat, when
I point
>> the file into stat_analysis the log file shows
>> DEBUG 2: Processing 1 STAT files.
>> DEBUG 2: STAT Lines read = 15
>> DEBUG 2: STAT Lines retained = 0
>> What I confused is none of lines in the stat file be retained, then
I got
>> empty output file of stat_analysis.
>>
>> My log file is
>>
>>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>>
>> Thanks,
>> Shih-Wei
>>
>>
>> On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT
<met_help at ucar.edu
>> <mailto:met_help at ucar.edu>
>> <mailto:met_help at ucar.edu>> wrote:
>>
>> Shih-Wei,
>>
>> Yes, we typically setup the masking regions first using gen_vx_mask
and
>> then reference those in the Grid-Stat config file.
>>
>> Sure, I can answer questions about stat_analysis.
>>
>> John
>>
>> On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
>> <mailto:met_help at ucar.edu>
>> <mailto:met_help at ucar.edu>>
>> wrote:
>>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>>
>> Hi John,
>>
>> Thank you.
>> Topical region is my test run, I may need couple regions once I can
get
>> the statistics.
>> You would recommend people use gen_vx_mask if the boundary of mask
area is
>> not too complicated, right?
>>
>> I encountered another problem in the stat_analysis, could I also
consult
>> this with you?
>>
>> Best,
>> Shih-Wei
>>
>> On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
>>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
>> wrote:
>>
>> Shih,
>>
>> I'd recommend that you run the gen_vx_mask tool to create this
masking
>> region. It looks like you really just want to include -20 to 20
>> latitude.
>> Rather than defining a polyline region to do this, I think it's
simpler
>> to
>> use the latitude masking option in gen_vx_mask. To illustrate:
>>
>> (1) pull a sample global GFS file:
>> wget
>>
>>
>>
>>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
>>
>> (2) run gen_vx_mask to generate the masking region:
>> gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
>> tropics_mask.nc
>> -type lat -thresh '>=-20&&<=20' -name tropics
>>
>> (3) plot the result to show the mask
>> plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
>> level="(*,*)";'
>>
>> See the attached pdf.
>>
>> When you run grid-stat you'll just setup the config file like this:
>>
>> mask = {
>> grid = [];
>> poly = ["/path/to/tropics_mask.nc" ];
>> }
>>
>> Where you replace /path/to with the actual path that file on your
>> machine.
>>
>> Thanks,
>> John
>>
>> On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
>> wrote:
>>
>>
>> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
>> Transaction: Ticket created by swei at albany.edu
>> Queue: met_help
>> Subject: mask specification for grid_stat
>> Owner: Nobody
>> Requestors: swei at albany.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>>
>>
>> Hi,
>>
>> I’m trying to have grid_stat for two gridded model output over
certain
>> area, but it shows no matched pairs for the mask area in my log
file.
>> I setup a poly file for tropical region, like this
>> TROP
>> 20.0 -180.0
>> 20.0 180.0
>> -20.0 180.0
>> -20.0 -180.0
>> 20.0 -180.0
>> I’m wondering if the longitude value is matter, because one of my
output
>> is 0 to 360, another is -180 to 180 for longitude.
>> Also, I’m not sure if I use the mask in the right way.
>> Please give me some guidance.
>>
>> Thanks,
>> Shih-Wei
>>
>>
>>
>> <tropics_mask.pdf>
>>
>>
>>
>
------------------------------------------------
Subject: mask specification for grid_stat
From: John Halley Gotway
Time: Wed Mar 25 10:31:11 2020
Shih-Wei,
I just wrote up a bug report for this issue:
https://github.com/NCAR/METplus/issues/407
The MET tools write log messages up to about level 10 (for the most
detailed debugging). The default is 2 which prints the input/output
files
as well as a brief description of what the tool is doing.
MET's verbosity level is controlled via the METplus wrappers using the
LOG_VERBOSITY level. Or you can set it separately for each MET tool,
as
shown below:
# Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
LOG_STAT_ANALYSIS_VERBOSITY = 2
Thanks,
John
On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you so much.
> That’s the reason.
> How many verbosity level that user can set?
> I did notice that is different to what I set in the configuration
files,
> but I thought it will handle by METplus properly.
>
> Best,
> Shih-We
>
> > On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
> >
> > Shih-Wei,
> >
> > OK, I logged on to Cheyenne and looked in the log file you sent:
> >
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
> >
> > And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis
can
> > either run with a config file (as METplus does it) or directly on
the
> > command line. I used the log messages to piece together the
command line
> > job shown below:
> >
> > */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
> >
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> > -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4
-fcst_lev P500
> > -fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
> > -obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> > -dump_row CAMSiRA_.stat*
> >
> > This job runs and still gets 0 matching lines:
> >
> > *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained =
0*
> >
> > Rerunning but at a higher verbosity level (-v 3), I see the .stat
file
> it's
> > reading:
> > *DEBUG 3: Processing STAT file
> >
>
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
> > ... 1 of 1*
> >
> > Comparing the contents of that file to the job command options,
there's a
> > very slight difference in the OBS_LEV column:
> > '0,4,*,*' != '(0,4,*,*)'
> > '0,7,*,*' != '(0,7,*,*)'
> > '0,11,*,*' != '(0,11,*,*)'
> >
> > Rerunning that job without the enclosing parentheses results in 3
> matching
> > lines:
> >
> > */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
> >
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> > -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4
-fcst_lev P500
> > -fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev
'0,7,*,*'
> > -obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> > -dump_row CAMSiRA_.stat*
> >
> >
> > *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained =
3*
> >
> > So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
> > and OBS_LEV settings. I'll talk to George McCabe to let him know
about
> > this issue.
> >
> > Thanks,
> > John
> >
> > On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >>
> >> Hi John,
> >>
> >> I’m trying to have filter, but I did try “summary” and
“aggregate”.
> >> Both show the same message.
> >> I’ll test in command line first.
> >> BTW, the mask file is work, thanks for your suggestion.
> >>
> >> Best,
> >> Shih-Wei
> >>
> >> On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> >> <mailto:met_help at ucar.edu>> wrote:
> >>
> >> Shih-Wei,
> >>
> >> The critical information you didn't send is what is the job
command that
> >> you're running? I'd recommend getting familiar with running the
> >> STAT-Analysis tool first on the command line, and then once you
> understand
> >> the functionality, you can use METplus to automate calls to it.
> >>
> >> John
> >>
> >> On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> >> <mailto:met_help at ucar.edu>>
> >> wrote:
> >>
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >>
> >> John,
> >>
> >> Thank you, I’ll try it.
> >> I’m still learning how to use stat_analysis.
> >> Now I can get the stat file for “FULL” domain from grid_stat,
when I
> point
> >> the file into stat_analysis the log file shows
> >> DEBUG 2: Processing 1 STAT files.
> >> DEBUG 2: STAT Lines read = 15
> >> DEBUG 2: STAT Lines retained = 0
> >> What I confused is none of lines in the stat file be retained,
then I
> got
> >> empty output file of stat_analysis.
> >>
> >> My log file is
> >>
> >>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
> >>
> >> Thanks,
> >> Shih-Wei
> >>
> >>
> >> On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> >> <mailto:met_help at ucar.edu>
> >> <mailto:met_help at ucar.edu>> wrote:
> >>
> >> Shih-Wei,
> >>
> >> Yes, we typically setup the masking regions first using
gen_vx_mask and
> >> then reference those in the Grid-Stat config file.
> >>
> >> Sure, I can answer questions about stat_analysis.
> >>
> >> John
> >>
> >> On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> >> <mailto:met_help at ucar.edu>
> >> <mailto:met_help at ucar.edu>>
> >> wrote:
> >>
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >>
> >> Hi John,
> >>
> >> Thank you.
> >> Topical region is my test run, I may need couple regions once I
can get
> >> the statistics.
> >> You would recommend people use gen_vx_mask if the boundary of
mask area
> is
> >> not too complicated, right?
> >>
> >> I encountered another problem in the stat_analysis, could I also
consult
> >> this with you?
> >>
> >> Best,
> >> Shih-Wei
> >>
> >> On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
> >>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
> >> wrote:
> >>
> >> Shih,
> >>
> >> I'd recommend that you run the gen_vx_mask tool to create this
masking
> >> region. It looks like you really just want to include -20 to 20
> >> latitude.
> >> Rather than defining a polyline region to do this, I think it's
simpler
> >> to
> >> use the latitude masking option in gen_vx_mask. To illustrate:
> >>
> >> (1) pull a sample global GFS file:
> >> wget
> >>
> >>
> >>
> >>
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
> >>
> >> (2) run gen_vx_mask to generate the masking region:
> >> gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
> >> tropics_mask.nc
> >> -type lat -thresh '>=-20&&<=20' -name tropics
> >>
> >> (3) plot the result to show the mask
> >> plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> >> level="(*,*)";'
> >>
> >> See the attached pdf.
> >>
> >> When you run grid-stat you'll just setup the config file like
this:
> >>
> >> mask = {
> >> grid = [];
> >> poly = ["/path/to/tropics_mask.nc" ];
> >> }
> >>
> >> Where you replace /path/to with the actual path that file on your
> >> machine.
> >>
> >> Thanks,
> >> John
> >>
> >> On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> >
> >> wrote:
> >>
> >>
> >> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> >> Transaction: Ticket created by swei at albany.edu
> >> Queue: met_help
> >> Subject: mask specification for grid_stat
> >> Owner: Nobody
> >> Requestors: swei at albany.edu
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >>
> >>
> >> Hi,
> >>
> >> I’m trying to have grid_stat for two gridded model output over
certain
> >> area, but it shows no matched pairs for the mask area in my log
file.
> >> I setup a poly file for tropical region, like this
> >> TROP
> >> 20.0 -180.0
> >> 20.0 180.0
> >> -20.0 180.0
> >> -20.0 -180.0
> >> 20.0 -180.0
> >> I’m wondering if the longitude value is matter, because one of my
output
> >> is 0 to 360, another is -180 to 180 for longitude.
> >> Also, I’m not sure if I use the mask in the right way.
> >> Please give me some guidance.
> >>
> >> Thanks,
> >> Shih-Wei
> >>
> >>
> >>
> >> <tropics_mask.pdf>
> >>
> >>
> >>
> >
>
>
>
>
------------------------------------------------
Subject: mask specification for grid_stat
From: Wei, Shih Wei
Time: Wed Mar 25 10:49:38 2020
Hi John,
Thank you so much.
Another question in stat_analysis of METplus, I setup the output
within METplus conf file
MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
{MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
The MODEL1 is CAMSiRA in my case.
But when I check the output folder, the time flag is not shown
properly.
The file name become “CAMSiRA_.stat"
Did I setup the time flag wrong?
Best,
Shih-Wei
On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
I just wrote up a bug report for this issue:
https://github.com/NCAR/METplus/issues/407
The MET tools write log messages up to about level 10 (for the most
detailed debugging). The default is 2 which prints the input/output
files
as well as a brief description of what the tool is doing.
MET's verbosity level is controlled via the METplus wrappers using the
LOG_VERBOSITY level. Or you can set it separately for each MET tool,
as
shown below:
# Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
LOG_STAT_ANALYSIS_VERBOSITY = 2
Thanks,
John
On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you so much.
That’s the reason.
How many verbosity level that user can set?
I did notice that is different to what I set in the configuration
files,
but I thought it will handle by METplus properly.
Best,
Shih-We
On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
Shih-Wei,
OK, I logged on to Cheyenne and looked in the log file you sent:
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis can
either run with a config file (as METplus does it) or directly on the
command line. I used the log messages to piece together the command
line
job shown below:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
-obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
-dump_row CAMSiRA_.stat*
This job runs and still gets 0 matching lines:
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
Rerunning but at a higher verbosity level (-v 3), I see the .stat file
it's
reading:
*DEBUG 3: Processing STAT file
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
... 1 of 1*
Comparing the contents of that file to the job command options,
there's a
very slight difference in the OBS_LEV column:
'0,4,*,*' != '(0,4,*,*)'
'0,7,*,*' != '(0,7,*,*)'
'0,11,*,*' != '(0,11,*,*)'
Rerunning that job without the enclosing parentheses results in 3
matching
lines:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
-obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type SL1L2
-dump_row CAMSiRA_.stat*
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
and OBS_LEV settings. I'll talk to George McCabe to let him know
about
this issue.
Thanks,
John
On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
I’m trying to have filter, but I did try “summary” and “aggregate”.
Both show the same message.
I’ll test in command line first.
BTW, the mask file is work, thanks for your suggestion.
Best,
Shih-Wei
On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
met_help at ucar.edu
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
The critical information you didn't send is what is the job command
that
you're running? I'd recommend getting familiar with running the
STAT-Analysis tool first on the command line, and then once you
understand
the functionality, you can use METplus to automate calls to it.
John
On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
John,
Thank you, I’ll try it.
I’m still learning how to use stat_analysis.
Now I can get the stat file for “FULL” domain from grid_stat, when I
point
the file into stat_analysis the log file shows
DEBUG 2: Processing 1 STAT files.
DEBUG 2: STAT Lines read = 15
DEBUG 2: STAT Lines retained = 0
What I confused is none of lines in the stat file be retained, then I
got
empty output file of stat_analysis.
My log file is
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
Thanks,
Shih-Wei
On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
met_help at ucar.edu
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
Yes, we typically setup the masking regions first using gen_vx_mask
and
then reference those in the Grid-Stat config file.
Sure, I can answer questions about stat_analysis.
John
On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you.
Topical region is my test run, I may need couple regions once I can
get
the statistics.
You would recommend people use gen_vx_mask if the boundary of mask
area
is
not too complicated, right?
I encountered another problem in the stat_analysis, could I also
consult
this with you?
Best,
Shih-Wei
On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
wrote:
Shih,
I'd recommend that you run the gen_vx_mask tool to create this masking
region. It looks like you really just want to include -20 to 20
latitude.
Rather than defining a polyline region to do this, I think it's
simpler
to
use the latitude masking option in gen_vx_mask. To illustrate:
(1) pull a sample global GFS file:
wget
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
(2) run gen_vx_mask to generate the masking region:
gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
tropics_mask.nc
-type lat -thresh '>=-20&&<=20' -name tropics
(3) plot the result to show the mask
plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
level="(*,*)";'
See the attached pdf.
When you run grid-stat you'll just setup the config file like this:
mask = {
grid = [];
poly = ["/path/to/tropics_mask.nc" ];
}
Where you replace /path/to with the actual path that file on your
machine.
Thanks,
John
On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu
wrote:
Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
Transaction: Ticket created by swei at albany.edu
Queue: met_help
Subject: mask specification for grid_stat
Owner: Nobody
Requestors: swei at albany.edu
Status: new
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi,
I’m trying to have grid_stat for two gridded model output over certain
area, but it shows no matched pairs for the mask area in my log file.
I setup a poly file for tropical region, like this
TROP
20.0 -180.0
20.0 180.0
-20.0 180.0
-20.0 -180.0
20.0 -180.0
I’m wondering if the longitude value is matter, because one of my
output
is 0 to 360, another is -180 to 180 for longitude.
Also, I’m not sure if I use the mask in the right way.
Please give me some guidance.
Thanks,
Shih-Wei
<tropics_mask.pdf>
------------------------------------------------
Subject: mask specification for grid_stat
From: John Halley Gotway
Time: Wed Mar 25 10:54:35 2020
Shih,
I'm going to reassign this ticket to George. I think he'd be better
equipped to answer that question.
John
On Wed, Mar 25, 2020 at 10:50 AM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you so much.
> Another question in stat_analysis of METplus, I setup the output
within
> METplus conf file
> MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
> {MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
> The MODEL1 is CAMSiRA in my case.
> But when I check the output folder, the time flag is not shown
properly.
> The file name become “CAMSiRA_.stat"
> Did I setup the time flag wrong?
>
> Best,
> Shih-Wei
>
> On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> I just wrote up a bug report for this issue:
> https://github.com/NCAR/METplus/issues/407
>
> The MET tools write log messages up to about level 10 (for the most
> detailed debugging). The default is 2 which prints the input/output
files
> as well as a brief description of what the tool is doing.
>
> MET's verbosity level is controlled via the METplus wrappers using
the
> LOG_VERBOSITY level. Or you can set it separately for each MET
tool, as
> shown below:
>
> # Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
> LOG_STAT_ANALYSIS_VERBOSITY = 2
>
>
> Thanks,
> John
>
> On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you so much.
> That’s the reason.
> How many verbosity level that user can set?
> I did notice that is different to what I set in the configuration
files,
> but I thought it will handle by METplus properly.
>
> Best,
> Shih-We
>
> On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> Shih-Wei,
>
> OK, I logged on to Cheyenne and looked in the log file you sent:
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis
can
> either run with a config file (as METplus does it) or directly on
the
> command line. I used the log messages to piece together the command
line
> job shown below:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
> -obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
> This job runs and still gets 0 matching lines:
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
>
> Rerunning but at a higher verbosity level (-v 3), I see the .stat
file
> it's
> reading:
> *DEBUG 3: Processing STAT file
>
>
>
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
> ... 1 of 1*
>
> Comparing the contents of that file to the job command options,
there's a
> very slight difference in the OBS_LEV column:
> '0,4,*,*' != '(0,4,*,*)'
> '0,7,*,*' != '(0,7,*,*)'
> '0,11,*,*' != '(0,11,*,*)'
>
> Rerunning that job without the enclosing parentheses results in 3
> matching
> lines:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
> -obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
>
> So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
> and OBS_LEV settings. I'll talk to George McCabe to let him know
about
> this issue.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu
>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> I’m trying to have filter, but I did try “summary” and “aggregate”.
> Both show the same message.
> I’ll test in command line first.
> BTW, the mask file is work, thanks for your suggestion.
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> The critical information you didn't send is what is the job command
that
> you're running? I'd recommend getting familiar with running the
> STAT-Analysis tool first on the command line, and then once you
> understand
> the functionality, you can use METplus to automate calls to it.
>
> John
>
> On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> John,
>
> Thank you, I’ll try it.
> I’m still learning how to use stat_analysis.
> Now I can get the stat file for “FULL” domain from grid_stat, when I
> point
> the file into stat_analysis the log file shows
> DEBUG 2: Processing 1 STAT files.
> DEBUG 2: STAT Lines read = 15
> DEBUG 2: STAT Lines retained = 0
> What I confused is none of lines in the stat file be retained, then
I
> got
> empty output file of stat_analysis.
>
> My log file is
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> Thanks,
> Shih-Wei
>
>
> On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> Yes, we typically setup the masking regions first using gen_vx_mask
and
> then reference those in the Grid-Stat config file.
>
> Sure, I can answer questions about stat_analysis.
>
> John
>
> On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you.
> Topical region is my test run, I may need couple regions once I can
get
> the statistics.
> You would recommend people use gen_vx_mask if the boundary of mask
area
> is
> not too complicated, right?
>
> I encountered another problem in the stat_analysis, could I also
consult
> this with you?
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
> wrote:
>
> Shih,
>
> I'd recommend that you run the gen_vx_mask tool to create this
masking
> region. It looks like you really just want to include -20 to 20
> latitude.
> Rather than defining a polyline region to do this, I think it's
simpler
> to
> use the latitude masking option in gen_vx_mask. To illustrate:
>
> (1) pull a sample global GFS file:
> wget
>
>
>
>
>
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
>
> (2) run gen_vx_mask to generate the masking region:
> gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
> tropics_mask.nc
> -type lat -thresh '>=-20&&<=20' -name tropics
>
> (3) plot the result to show the mask
> plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> level="(*,*)";'
>
> See the attached pdf.
>
> When you run grid-stat you'll just setup the config file like this:
>
> mask = {
> grid = [];
> poly = ["/path/to/tropics_mask.nc" ];
> }
>
> Where you replace /path/to with the actual path that file on your
> machine.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu
>
> wrote:
>
>
> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> Transaction: Ticket created by swei at albany.edu
> Queue: met_help
> Subject: mask specification for grid_stat
> Owner: Nobody
> Requestors: swei at albany.edu
> Status: new
> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703
>
>
>
> Hi,
>
> I’m trying to have grid_stat for two gridded model output over
certain
> area, but it shows no matched pairs for the mask area in my log
file.
> I setup a poly file for tropical region, like this
> TROP
> 20.0 -180.0
> 20.0 180.0
> -20.0 180.0
> -20.0 -180.0
> 20.0 -180.0
> I’m wondering if the longitude value is matter, because one of my
output
> is 0 to 360, another is -180 to 180 for longitude.
> Also, I’m not sure if I use the mask in the right way.
> Please give me some guidance.
>
> Thanks,
> Shih-Wei
>
>
>
> <tropics_mask.pdf>
>
>
>
>
>
>
>
>
>
>
>
>
------------------------------------------------
Subject: mask specification for grid_stat
From: Wei, Shih Wei
Time: Wed Mar 25 10:55:42 2020
Sure, thank you so much, John.
Shih-Wei
On Mar 25, 2020, at 10:54 AM, John Halley Gotway via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
Shih,
I'm going to reassign this ticket to George. I think he'd be better
equipped to answer that question.
John
On Wed, Mar 25, 2020 at 10:50 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you so much.
Another question in stat_analysis of METplus, I setup the output
within
METplus conf file
MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
{MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
The MODEL1 is CAMSiRA in my case.
But when I check the output folder, the time flag is not shown
properly.
The file name become “CAMSiRA_.stat"
Did I setup the time flag wrong?
Best,
Shih-Wei
On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
I just wrote up a bug report for this issue:
https://github.com/NCAR/METplus/issues/407
The MET tools write log messages up to about level 10 (for the most
detailed debugging). The default is 2 which prints the input/output
files
as well as a brief description of what the tool is doing.
MET's verbosity level is controlled via the METplus wrappers using the
LOG_VERBOSITY level. Or you can set it separately for each MET tool,
as
shown below:
# Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
LOG_STAT_ANALYSIS_VERBOSITY = 2
Thanks,
John
On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you so much.
That’s the reason.
How many verbosity level that user can set?
I did notice that is different to what I set in the configuration
files,
but I thought it will handle by METplus properly.
Best,
Shih-We
On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
OK, I logged on to Cheyenne and looked in the log file you sent:
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis can
either run with a config file (as METplus does it) or directly on the
command line. I used the log messages to piece together the command
line
job shown below:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
-obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
-dump_row CAMSiRA_.stat*
This job runs and still gets 0 matching lines:
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
Rerunning but at a higher verbosity level (-v 3), I see the .stat file
it's
reading:
*DEBUG 3: Processing STAT file
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
... 1 of 1*
Comparing the contents of that file to the job command options,
there's a
very slight difference in the OBS_LEV column:
'0,4,*,*' != '(0,4,*,*)'
'0,7,*,*' != '(0,7,*,*)'
'0,11,*,*' != '(0,11,*,*)'
Rerunning that job without the enclosing parentheses results in 3
matching
lines:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
-obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type SL1L2
-dump_row CAMSiRA_.stat*
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
and OBS_LEV settings. I'll talk to George McCabe to let him know
about
this issue.
Thanks,
John
On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
I’m trying to have filter, but I did try “summary” and “aggregate”.
Both show the same message.
I’ll test in command line first.
BTW, the mask file is work, thanks for your suggestion.
Best,
Shih-Wei
On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
The critical information you didn't send is what is the job command
that
you're running? I'd recommend getting familiar with running the
STAT-Analysis tool first on the command line, and then once you
understand
the functionality, you can use METplus to automate calls to it.
John
On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
John,
Thank you, I’ll try it.
I’m still learning how to use stat_analysis.
Now I can get the stat file for “FULL” domain from grid_stat, when I
point
the file into stat_analysis the log file shows
DEBUG 2: Processing 1 STAT files.
DEBUG 2: STAT Lines read = 15
DEBUG 2: STAT Lines retained = 0
What I confused is none of lines in the stat file be retained, then I
got
empty output file of stat_analysis.
My log file is
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
Thanks,
Shih-Wei
On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
Yes, we typically setup the masking regions first using gen_vx_mask
and
then reference those in the Grid-Stat config file.
Sure, I can answer questions about stat_analysis.
John
On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you.
Topical region is my test run, I may need couple regions once I can
get
the statistics.
You would recommend people use gen_vx_mask if the boundary of mask
area
is
not too complicated, right?
I encountered another problem in the stat_analysis, could I also
consult
this with you?
Best,
Shih-Wei
On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
wrote:
Shih,
I'd recommend that you run the gen_vx_mask tool to create this masking
region. It looks like you really just want to include -20 to 20
latitude.
Rather than defining a polyline region to do this, I think it's
simpler
to
use the latitude masking option in gen_vx_mask. To illustrate:
(1) pull a sample global GFS file:
wget
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
(2) run gen_vx_mask to generate the masking region:
gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
tropics_mask.nc
-type lat -thresh '>=-20&&<=20' -name tropics
(3) plot the result to show the mask
plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
level="(*,*)";'
See the attached pdf.
When you run grid-stat you'll just setup the config file like this:
mask = {
grid = [];
poly = ["/path/to/tropics_mask.nc" ];
}
Where you replace /path/to with the actual path that file on your
machine.
Thanks,
John
On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu
wrote:
Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
Transaction: Ticket created by swei at albany.edu
Queue: met_help
Subject: mask specification for grid_stat
Owner: Nobody
Requestors: swei at albany.edu
Status: new
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi,
I’m trying to have grid_stat for two gridded model output over certain
area, but it shows no matched pairs for the mask area in my log file.
I setup a poly file for tropical region, like this
TROP
20.0 -180.0
20.0 180.0
-20.0 180.0
-20.0 -180.0
20.0 -180.0
I’m wondering if the longitude value is matter, because one of my
output
is 0 to 360, another is -180 to 180 for longitude.
Also, I’m not sure if I use the mask in the right way.
Please give me some guidance.
Thanks,
Shih-Wei
<tropics_mask.pdf>
------------------------------------------------
Subject: mask specification for grid_stat
From: John Halley Gotway
Time: Wed Mar 25 11:03:58 2020
Shih-Wei,
The Stat-Analysis wrapper was written by one of our NOAA colleagues.
George's guess is that the dump row template doesn't support filename
template syntax, so the init tag won't be translated correctly. I'm
guessing this will ultimately require a bugfix or enhancement to that
wrapper. I'm reassigning the ticket now and adding our NOAA colleague
to
it.
John
On Wed, Mar 25, 2020 at 10:55 AM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Sure, thank you so much, John.
>
> Shih-Wei
>
> On Mar 25, 2020, at 10:54 AM, John Halley Gotway via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih,
>
> I'm going to reassign this ticket to George. I think he'd be better
> equipped to answer that question.
>
> John
>
> On Wed, Mar 25, 2020 at 10:50 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you so much.
> Another question in stat_analysis of METplus, I setup the output
within
> METplus conf file
> MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
> {MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
> The MODEL1 is CAMSiRA in my case.
> But when I check the output folder, the time flag is not shown
properly.
> The file name become “CAMSiRA_.stat"
> Did I setup the time flag wrong?
>
> Best,
> Shih-Wei
>
> On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> I just wrote up a bug report for this issue:
> https://github.com/NCAR/METplus/issues/407
>
> The MET tools write log messages up to about level 10 (for the most
> detailed debugging). The default is 2 which prints the input/output
files
> as well as a brief description of what the tool is doing.
>
> MET's verbosity level is controlled via the METplus wrappers using
the
> LOG_VERBOSITY level. Or you can set it separately for each MET
tool, as
> shown below:
>
> # Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
> LOG_STAT_ANALYSIS_VERBOSITY = 2
>
>
> Thanks,
> John
>
> On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you so much.
> That’s the reason.
> How many verbosity level that user can set?
> I did notice that is different to what I set in the configuration
files,
> but I thought it will handle by METplus properly.
>
> Best,
> Shih-We
>
> On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> OK, I logged on to Cheyenne and looked in the log file you sent:
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis
can
> either run with a config file (as METplus does it) or directly on
the
> command line. I used the log messages to piece together the command
line
> job shown below:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
> -obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
> This job runs and still gets 0 matching lines:
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
>
> Rerunning but at a higher verbosity level (-v 3), I see the .stat
file
> it's
> reading:
> *DEBUG 3: Processing STAT file
>
>
>
>
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
> ... 1 of 1*
>
> Comparing the contents of that file to the job command options,
there's a
> very slight difference in the OBS_LEV column:
> '0,4,*,*' != '(0,4,*,*)'
> '0,7,*,*' != '(0,7,*,*)'
> '0,11,*,*' != '(0,11,*,*)'
>
> Rerunning that job without the enclosing parentheses results in 3
> matching
> lines:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
> -obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
>
> So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
> and OBS_LEV settings. I'll talk to George McCabe to let him know
about
> this issue.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> I’m trying to have filter, but I did try “summary” and “aggregate”.
> Both show the same message.
> I’ll test in command line first.
> BTW, the mask file is work, thanks for your suggestion.
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> The critical information you didn't send is what is the job command
that
> you're running? I'd recommend getting familiar with running the
> STAT-Analysis tool first on the command line, and then once you
> understand
> the functionality, you can use METplus to automate calls to it.
>
> John
>
> On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> John,
>
> Thank you, I’ll try it.
> I’m still learning how to use stat_analysis.
> Now I can get the stat file for “FULL” domain from grid_stat, when I
> point
> the file into stat_analysis the log file shows
> DEBUG 2: Processing 1 STAT files.
> DEBUG 2: STAT Lines read = 15
> DEBUG 2: STAT Lines retained = 0
> What I confused is none of lines in the stat file be retained, then
I
> got
> empty output file of stat_analysis.
>
> My log file is
>
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> Thanks,
> Shih-Wei
>
>
> On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> Yes, we typically setup the masking regions first using gen_vx_mask
and
> then reference those in the Grid-Stat config file.
>
> Sure, I can answer questions about stat_analysis.
>
> John
>
> On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you.
> Topical region is my test run, I may need couple regions once I can
get
> the statistics.
> You would recommend people use gen_vx_mask if the boundary of mask
area
> is
> not too complicated, right?
>
> I encountered another problem in the stat_analysis, could I also
consult
> this with you?
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
> wrote:
>
> Shih,
>
> I'd recommend that you run the gen_vx_mask tool to create this
masking
> region. It looks like you really just want to include -20 to 20
> latitude.
> Rather than defining a polyline region to do this, I think it's
simpler
> to
> use the latitude masking option in gen_vx_mask. To illustrate:
>
> (1) pull a sample global GFS file:
> wget
>
>
>
>
>
>
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
>
> (2) run gen_vx_mask to generate the masking region:
> gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
> tropics_mask.nc
> -type lat -thresh '>=-20&&<=20' -name tropics
>
> (3) plot the result to show the mask
> plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> level="(*,*)";'
>
> See the attached pdf.
>
> When you run grid-stat you'll just setup the config file like this:
>
> mask = {
> grid = [];
> poly = ["/path/to/tropics_mask.nc" ];
> }
>
> Where you replace /path/to with the actual path that file on your
> machine.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu
>
> wrote:
>
>
> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> Transaction: Ticket created by swei at albany.edu
> Queue: met_help
> Subject: mask specification for grid_stat
> Owner: Nobody
> Requestors: swei at albany.edu
> Status: new
> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703
>
>
>
> Hi,
>
> I’m trying to have grid_stat for two gridded model output over
certain
> area, but it shows no matched pairs for the mask area in my log
file.
> I setup a poly file for tropical region, like this
> TROP
> 20.0 -180.0
> 20.0 180.0
> -20.0 180.0
> -20.0 -180.0
> 20.0 -180.0
> I’m wondering if the longitude value is matter, because one of my
output
> is 0 to 360, another is -180 to 180 for longitude.
> Also, I’m not sure if I use the mask in the right way.
> Please give me some guidance.
>
> Thanks,
> Shih-Wei
>
>
>
> <tropics_mask.pdf>
>
>
>
------------------------------------------------
Subject: mask specification for grid_stat
From: George McCabe
Time: Wed Mar 25 11:18:11 2020
Hi Shih-Wei,
My best guess is that if you are looping over VALID times, you won't
be
able to use 'init' in the template. You could use 'valid' instead if
that
works for you. Many of the other wrappers support using 'valid' and
'init'
regardless of the LOOP_BY value, but this wrapper may not have that
functionality implemented yet. Please let me know if using 'valid'
fixes
the issue.
Thanks,
George
On Wed, Mar 25, 2020 at 11:04 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Shih-Wei,
>
> The Stat-Analysis wrapper was written by one of our NOAA colleagues.
> George's guess is that the dump row template doesn't support
filename
> template syntax, so the init tag won't be translated correctly. I'm
> guessing this will ultimately require a bugfix or enhancement to
that
> wrapper. I'm reassigning the ticket now and adding our NOAA
colleague to
> it.
>
> John
>
> On Wed, Mar 25, 2020 at 10:55 AM Wei, Shih Wei via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >
> > Sure, thank you so much, John.
> >
> > Shih-Wei
> >
> > On Mar 25, 2020, at 10:54 AM, John Halley Gotway via RT <
> met_help at ucar.edu
> > <mailto:met_help at ucar.edu>> wrote:
> >
> > Shih,
> >
> > I'm going to reassign this ticket to George. I think he'd be
better
> > equipped to answer that question.
> >
> > John
> >
> > On Wed, Mar 25, 2020 at 10:50 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> > <mailto:met_help at ucar.edu>>
> > wrote:
> >
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >
> > Hi John,
> >
> > Thank you so much.
> > Another question in stat_analysis of METplus, I setup the output
within
> > METplus conf file
> > MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
> > {MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
> > The MODEL1 is CAMSiRA in my case.
> > But when I check the output folder, the time flag is not shown
properly.
> > The file name become “CAMSiRA_.stat"
> > Did I setup the time flag wrong?
> >
> > Best,
> > Shih-Wei
> >
> > On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT <
> met_help at ucar.edu
> > <mailto:met_help at ucar.edu>
> > <mailto:met_help at ucar.edu>> wrote:
> >
> > Shih-Wei,
> >
> > I just wrote up a bug report for this issue:
> > https://github.com/NCAR/METplus/issues/407
> >
> > The MET tools write log messages up to about level 10 (for the
most
> > detailed debugging). The default is 2 which prints the
input/output
> files
> > as well as a brief description of what the tool is doing.
> >
> > MET's verbosity level is controlled via the METplus wrappers using
the
> > LOG_VERBOSITY level. Or you can set it separately for each MET
tool, as
> > shown below:
> >
> > # Verbosity of MET output - overrides LOG_VERBOSITY for
StatAnalysis only
> > LOG_STAT_ANALYSIS_VERBOSITY = 2
> >
> >
> > Thanks,
> > John
> >
> > On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> > <mailto:met_help at ucar.edu>>
> > wrote:
> >
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >
> > Hi John,
> >
> > Thank you so much.
> > That’s the reason.
> > How many verbosity level that user can set?
> > I did notice that is different to what I set in the configuration
files,
> > but I thought it will handle by METplus properly.
> >
> > Best,
> > Shih-We
> >
> > On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
> > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
> >
> > Shih-Wei,
> >
> > OK, I logged on to Cheyenne and looked in the log file you sent:
> >
> >
> >
> >
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
> >
> > And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis
can
> > either run with a config file (as METplus does it) or directly on
the
> > command line. I used the log messages to piece together the
command line
> > job shown below:
> >
> > */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
> >
> >
> >
> >
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> > -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4
-fcst_lev P500
> > -fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
> > -obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> > -dump_row CAMSiRA_.stat*
> >
> > This job runs and still gets 0 matching lines:
> >
> > *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained =
0*
> >
> > Rerunning but at a higher verbosity level (-v 3), I see the .stat
file
> > it's
> > reading:
> > *DEBUG 3: Processing STAT file
> >
> >
> >
> >
>
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
> > ... 1 of 1*
> >
> > Comparing the contents of that file to the job command options,
there's a
> > very slight difference in the OBS_LEV column:
> > '0,4,*,*' != '(0,4,*,*)'
> > '0,7,*,*' != '(0,7,*,*)'
> > '0,11,*,*' != '(0,11,*,*)'
> >
> > Rerunning that job without the enclosing parentheses results in 3
> > matching
> > lines:
> >
> > */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
> >
> >
> >
> >
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> > -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4
-fcst_lev P500
> > -fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev
'0,7,*,*'
> > -obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> > -dump_row CAMSiRA_.stat*
> >
> >
> > *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained =
3*
> >
> > So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
> > and OBS_LEV settings. I'll talk to George McCabe to let him know
about
> > this issue.
> >
> > Thanks,
> > John
> >
> > On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> > <mailto:met_help at ucar.edu>
> >
> > wrote:
> >
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >
> > Hi John,
> >
> > I’m trying to have filter, but I did try “summary” and
“aggregate”.
> > Both show the same message.
> > I’ll test in command line first.
> > BTW, the mask file is work, thanks for your suggestion.
> >
> > Best,
> > Shih-Wei
> >
> > On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
> > met_help at ucar.edu<mailto:met_help at ucar.edu>
> > <mailto:met_help at ucar.edu>> wrote:
> >
> > Shih-Wei,
> >
> > The critical information you didn't send is what is the job
command that
> > you're running? I'd recommend getting familiar with running the
> > STAT-Analysis tool first on the command line, and then once you
> > understand
> > the functionality, you can use METplus to automate calls to it.
> >
> > John
> >
> > On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> > <mailto:met_help at ucar.edu>
> > <mailto:met_help at ucar.edu>>
> > wrote:
> >
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >
> > John,
> >
> > Thank you, I’ll try it.
> > I’m still learning how to use stat_analysis.
> > Now I can get the stat file for “FULL” domain from grid_stat, when
I
> > point
> > the file into stat_analysis the log file shows
> > DEBUG 2: Processing 1 STAT files.
> > DEBUG 2: STAT Lines read = 15
> > DEBUG 2: STAT Lines retained = 0
> > What I confused is none of lines in the stat file be retained,
then I
> > got
> > empty output file of stat_analysis.
> >
> > My log file is
> >
> >
> >
> >
> >
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
> >
> > Thanks,
> > Shih-Wei
> >
> >
> > On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
> > met_help at ucar.edu<mailto:met_help at ucar.edu>
> > <mailto:met_help at ucar.edu>
> > <mailto:met_help at ucar.edu>> wrote:
> >
> > Shih-Wei,
> >
> > Yes, we typically setup the masking regions first using
gen_vx_mask and
> > then reference those in the Grid-Stat config file.
> >
> > Sure, I can answer questions about stat_analysis.
> >
> > John
> >
> > On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> > <mailto:met_help at ucar.edu>
> > <mailto:met_help at ucar.edu>>
> > wrote:
> >
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >
> > Hi John,
> >
> > Thank you.
> > Topical region is my test run, I may need couple regions once I
can get
> > the statistics.
> > You would recommend people use gen_vx_mask if the boundary of mask
area
> > is
> > not too complicated, right?
> >
> > I encountered another problem in the stat_analysis, could I also
consult
> > this with you?
> >
> > Best,
> > Shih-Wei
> >
> > On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
> >
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
> > wrote:
> >
> > Shih,
> >
> > I'd recommend that you run the gen_vx_mask tool to create this
masking
> > region. It looks like you really just want to include -20 to 20
> > latitude.
> > Rather than defining a polyline region to do this, I think it's
simpler
> > to
> > use the latitude masking option in gen_vx_mask. To illustrate:
> >
> > (1) pull a sample global GFS file:
> > wget
> >
> >
> >
> >
> >
> >
> >
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
> >
> > (2) run gen_vx_mask to generate the masking region:
> > gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
> > tropics_mask.nc
> > -type lat -thresh '>=-20&&<=20' -name tropics
> >
> > (3) plot the result to show the mask
> > plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> > level="(*,*)";'
> >
> > See the attached pdf.
> >
> > When you run grid-stat you'll just setup the config file like
this:
> >
> > mask = {
> > grid = [];
> > poly = ["/path/to/tropics_mask.nc" ];
> > }
> >
> > Where you replace /path/to with the actual path that file on your
> > machine.
> >
> > Thanks,
> > John
> >
> > On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> >
> > Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> > Transaction: Ticket created by swei at albany.edu
> > Queue: met_help
> > Subject: mask specification for grid_stat
> > Owner: Nobody
> > Requestors: swei at albany.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
> >
> >
> > Hi,
> >
> > I’m trying to have grid_stat for two gridded model output over
certain
> > area, but it shows no matched pairs for the mask area in my log
file.
> > I setup a poly file for tropical region, like this
> > TROP
> > 20.0 -180.0
> > 20.0 180.0
> > -20.0 180.0
> > -20.0 -180.0
> > 20.0 -180.0
> > I’m wondering if the longitude value is matter, because one of my
output
> > is 0 to 360, another is -180 to 180 for longitude.
> > Also, I’m not sure if I use the mask in the right way.
> > Please give me some guidance.
> >
> > Thanks,
> > Shih-Wei
> >
> >
> >
> > <tropics_mask.pdf>
> >
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: mask specification for grid_stat
From: Wei, Shih Wei
Time: Wed Mar 25 14:47:19 2020
Hi John and George,
Thanks for your information.
I changed the looping time variables from VALID to INIT, I thought it
would work.
Looks like it’s definitely not.
I just tested with modifying LOOP_BY back to VALID and corresponding
variables.
And set the dump file with
MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
{MODEL1}/{MODEL1}_{valid?fmt=%Y%m%d%H}.stat
The file name still doesn’t include the time flag properly.
But it works with the list that be looped over, e.g.
{MODEL1}/{MODEL1}_{vx_mask?fmt=%s}.stat
In my case, I looped over FULL and tropics.
Then it can generate two stat files, _FULL.stat and _tropics.stat,
respectively.
Best,
Shih-Wei
On Mar 25, 2020, at 11:18 AM, George McCabe via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
Hi Shih-Wei,
My best guess is that if you are looping over VALID times, you won't
be
able to use 'init' in the template. You could use 'valid' instead if
that
works for you. Many of the other wrappers support using 'valid' and
'init'
regardless of the LOOP_BY value, but this wrapper may not have that
functionality implemented yet. Please let me know if using 'valid'
fixes
the issue.
Thanks,
George
On Wed, Mar 25, 2020 at 11:04 AM John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Shih-Wei,
The Stat-Analysis wrapper was written by one of our NOAA colleagues.
George's guess is that the dump row template doesn't support filename
template syntax, so the init tag won't be translated correctly. I'm
guessing this will ultimately require a bugfix or enhancement to that
wrapper. I'm reassigning the ticket now and adding our NOAA colleague
to
it.
John
On Wed, Mar 25, 2020 at 10:55 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Sure, thank you so much, John.
Shih-Wei
On Mar 25, 2020, at 10:54 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih,
I'm going to reassign this ticket to George. I think he'd be better
equipped to answer that question.
John
On Wed, Mar 25, 2020 at 10:50 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you so much.
Another question in stat_analysis of METplus, I setup the output
within
METplus conf file
MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
{MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
The MODEL1 is CAMSiRA in my case.
But when I check the output folder, the time flag is not shown
properly.
The file name become “CAMSiRA_.stat"
Did I setup the time flag wrong?
Best,
Shih-Wei
On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
I just wrote up a bug report for this issue:
https://github.com/NCAR/METplus/issues/407
The MET tools write log messages up to about level 10 (for the most
detailed debugging). The default is 2 which prints the input/output
files
as well as a brief description of what the tool is doing.
MET's verbosity level is controlled via the METplus wrappers using the
LOG_VERBOSITY level. Or you can set it separately for each MET tool,
as
shown below:
# Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
LOG_STAT_ANALYSIS_VERBOSITY = 2
Thanks,
John
On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you so much.
That’s the reason.
How many verbosity level that user can set?
I did notice that is different to what I set in the configuration
files,
but I thought it will handle by METplus properly.
Best,
Shih-We
On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
wrote:
Shih-Wei,
OK, I logged on to Cheyenne and looked in the log file you sent:
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis can
either run with a config file (as METplus does it) or directly on the
command line. I used the log messages to piece together the command
line
job shown below:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
-obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
-dump_row CAMSiRA_.stat*
This job runs and still gets 0 matching lines:
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
Rerunning but at a higher verbosity level (-v 3), I see the .stat file
it's
reading:
*DEBUG 3: Processing STAT file
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
... 1 of 1*
Comparing the contents of that file to the job command options,
there's a
very slight difference in the OBS_LEV column:
'0,4,*,*' != '(0,4,*,*)'
'0,7,*,*' != '(0,7,*,*)'
'0,11,*,*' != '(0,11,*,*)'
Rerunning that job without the enclosing parentheses results in 3
matching
lines:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
-obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type SL1L2
-dump_row CAMSiRA_.stat*
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
and OBS_LEV settings. I'll talk to George McCabe to let him know
about
this issue.
Thanks,
John
On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
I’m trying to have filter, but I did try “summary” and “aggregate”.
Both show the same message.
I’ll test in command line first.
BTW, the mask file is work, thanks for your suggestion.
Best,
Shih-Wei
On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
The critical information you didn't send is what is the job command
that
you're running? I'd recommend getting familiar with running the
STAT-Analysis tool first on the command line, and then once you
understand
the functionality, you can use METplus to automate calls to it.
John
On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
John,
Thank you, I’ll try it.
I’m still learning how to use stat_analysis.
Now I can get the stat file for “FULL” domain from grid_stat, when I
point
the file into stat_analysis the log file shows
DEBUG 2: Processing 1 STAT files.
DEBUG 2: STAT Lines read = 15
DEBUG 2: STAT Lines retained = 0
What I confused is none of lines in the stat file be retained, then I
got
empty output file of stat_analysis.
My log file is
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
Thanks,
Shih-Wei
On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
Yes, we typically setup the masking regions first using gen_vx_mask
and
then reference those in the Grid-Stat config file.
Sure, I can answer questions about stat_analysis.
John
On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you.
Topical region is my test run, I may need couple regions once I can
get
the statistics.
You would recommend people use gen_vx_mask if the boundary of mask
area
is
not too complicated, right?
I encountered another problem in the stat_analysis, could I also
consult
this with you?
Best,
Shih-Wei
On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
wrote:
Shih,
I'd recommend that you run the gen_vx_mask tool to create this masking
region. It looks like you really just want to include -20 to 20
latitude.
Rather than defining a polyline region to do this, I think it's
simpler
to
use the latitude masking option in gen_vx_mask. To illustrate:
(1) pull a sample global GFS file:
wget
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
(2) run gen_vx_mask to generate the masking region:
gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
tropics_mask.nc
-type lat -thresh '>=-20&&<=20' -name tropics
(3) plot the result to show the mask
plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
level="(*,*)";'
See the attached pdf.
When you run grid-stat you'll just setup the config file like this:
mask = {
grid = [];
poly = ["/path/to/tropics_mask.nc" ];
}
Where you replace /path/to with the actual path that file on your
machine.
Thanks,
John
On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu
wrote:
Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
Transaction: Ticket created by swei at albany.edu
Queue: met_help
Subject: mask specification for grid_stat
Owner: Nobody
Requestors: swei at albany.edu
Status: new
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi,
I’m trying to have grid_stat for two gridded model output over certain
area, but it shows no matched pairs for the mask area in my log file.
I setup a poly file for tropical region, like this
TROP
20.0 -180.0
20.0 180.0
-20.0 180.0
-20.0 -180.0
20.0 -180.0
I’m wondering if the longitude value is matter, because one of my
output
is 0 to 360, another is -180 to 180 for longitude.
Also, I’m not sure if I use the mask in the right way.
Please give me some guidance.
Thanks,
Shih-Wei
<tropics_mask.pdf>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: mask specification for grid_stat
From: George McCabe
Time: Wed Mar 25 14:53:23 2020
Hi Shih-Wei,
That is odd that it still doesn't substitute the valid time. Could you
send
me your metplus_final.conf file (located in the directory you set
OUTPUT_BASE to)?
The following use case that is found in the repository appears to use
valid
time and correctly substitutes it in the dump file.
parm/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis.conf
You could compare your configuration to this file to see if something
is
obviously different between them that would cause this to happen.
On Wed, Mar 25, 2020 at 2:47 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John and George,
>
> Thanks for your information.
> I changed the looping time variables from VALID to INIT, I thought
it
> would work.
> Looks like it’s definitely not.
>
> I just tested with modifying LOOP_BY back to VALID and corresponding
> variables.
> And set the dump file with
> MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
> {MODEL1}/{MODEL1}_{valid?fmt=%Y%m%d%H}.stat
> The file name still doesn’t include the time flag properly.
> But it works with the list that be looped over, e.g.
> {MODEL1}/{MODEL1}_{vx_mask?fmt=%s}.stat
> In my case, I looped over FULL and tropics.
> Then it can generate two stat files, _FULL.stat and _tropics.stat,
> respectively.
>
> Best,
> Shih-Wei
>
>
> On Mar 25, 2020, at 11:18 AM, George McCabe via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>> wrote:
>
> Hi Shih-Wei,
>
> My best guess is that if you are looping over VALID times, you won't
be
> able to use 'init' in the template. You could use 'valid' instead if
that
> works for you. Many of the other wrappers support using 'valid' and
'init'
> regardless of the LOOP_BY value, but this wrapper may not have that
> functionality implemented yet. Please let me know if using 'valid'
fixes
> the issue.
>
> Thanks,
> George
>
> On Wed, Mar 25, 2020 at 11:04 AM John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Shih-Wei,
>
> The Stat-Analysis wrapper was written by one of our NOAA colleagues.
> George's guess is that the dump row template doesn't support
filename
> template syntax, so the init tag won't be translated correctly. I'm
> guessing this will ultimately require a bugfix or enhancement to
that
> wrapper. I'm reassigning the ticket now and adding our NOAA
colleague to
> it.
>
> John
>
> On Wed, Mar 25, 2020 at 10:55 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Sure, thank you so much, John.
>
> Shih-Wei
>
> On Mar 25, 2020, at 10:54 AM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih,
>
> I'm going to reassign this ticket to George. I think he'd be better
> equipped to answer that question.
>
> John
>
> On Wed, Mar 25, 2020 at 10:50 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you so much.
> Another question in stat_analysis of METplus, I setup the output
within
> METplus conf file
> MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
> {MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
> The MODEL1 is CAMSiRA in my case.
> But when I check the output folder, the time flag is not shown
properly.
> The file name become “CAMSiRA_.stat"
> Did I setup the time flag wrong?
>
> Best,
> Shih-Wei
>
> On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> I just wrote up a bug report for this issue:
> https://github.com/NCAR/METplus/issues/407
>
> The MET tools write log messages up to about level 10 (for the most
> detailed debugging). The default is 2 which prints the input/output
> files
> as well as a brief description of what the tool is doing.
>
> MET's verbosity level is controlled via the METplus wrappers using
the
> LOG_VERBOSITY level. Or you can set it separately for each MET
tool, as
> shown below:
>
> # Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
> LOG_STAT_ANALYSIS_VERBOSITY = 2
>
>
> Thanks,
> John
>
> On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you so much.
> That’s the reason.
> How many verbosity level that user can set?
> I did notice that is different to what I set in the configuration
files,
> but I thought it will handle by METplus properly.
>
> Best,
> Shih-We
>
> On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
> wrote:
>
> Shih-Wei,
>
> OK, I logged on to Cheyenne and looked in the log file you sent:
>
>
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis
can
> either run with a config file (as METplus does it) or directly on
the
> command line. I used the log messages to piece together the command
line
> job shown below:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
>
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
> -obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
> This job runs and still gets 0 matching lines:
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
>
> Rerunning but at a higher verbosity level (-v 3), I see the .stat
file
> it's
> reading:
> *DEBUG 3: Processing STAT file
>
>
>
>
>
>
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
> ... 1 of 1*
>
> Comparing the contents of that file to the job command options,
there's a
> very slight difference in the OBS_LEV column:
> '0,4,*,*' != '(0,4,*,*)'
> '0,7,*,*' != '(0,7,*,*)'
> '0,11,*,*' != '(0,11,*,*)'
>
> Rerunning that job without the enclosing parentheses results in 3
> matching
> lines:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
>
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
> -obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
>
> So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
> and OBS_LEV settings. I'll talk to George McCabe to let him know
about
> this issue.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> I’m trying to have filter, but I did try “summary” and “aggregate”.
> Both show the same message.
> I’ll test in command line first.
> BTW, the mask file is work, thanks for your suggestion.
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> The critical information you didn't send is what is the job command
that
> you're running? I'd recommend getting familiar with running the
> STAT-Analysis tool first on the command line, and then once you
> understand
> the functionality, you can use METplus to automate calls to it.
>
> John
>
> On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> John,
>
> Thank you, I’ll try it.
> I’m still learning how to use stat_analysis.
> Now I can get the stat file for “FULL” domain from grid_stat, when I
> point
> the file into stat_analysis the log file shows
> DEBUG 2: Processing 1 STAT files.
> DEBUG 2: STAT Lines read = 15
> DEBUG 2: STAT Lines retained = 0
> What I confused is none of lines in the stat file be retained, then
I
> got
> empty output file of stat_analysis.
>
> My log file is
>
>
>
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> Thanks,
> Shih-Wei
>
>
> On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> Yes, we typically setup the masking regions first using gen_vx_mask
and
> then reference those in the Grid-Stat config file.
>
> Sure, I can answer questions about stat_analysis.
>
> John
>
> On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you.
> Topical region is my test run, I may need couple regions once I can
get
> the statistics.
> You would recommend people use gen_vx_mask if the boundary of mask
area
> is
> not too complicated, right?
>
> I encountered another problem in the stat_analysis, could I also
consult
> this with you?
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
> ><mailto:met_help at ucar.edu>>
> wrote:
>
> Shih,
>
> I'd recommend that you run the gen_vx_mask tool to create this
masking
> region. It looks like you really just want to include -20 to 20
> latitude.
> Rather than defining a polyline region to do this, I think it's
simpler
> to
> use the latitude masking option in gen_vx_mask. To illustrate:
>
> (1) pull a sample global GFS file:
> wget
>
>
>
>
>
>
>
>
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
>
> (2) run gen_vx_mask to generate the masking region:
> gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
> tropics_mask.nc
> -type lat -thresh '>=-20&&<=20' -name tropics
>
> (3) plot the result to show the mask
> plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> level="(*,*)";'
>
> See the attached pdf.
>
> When you run grid-stat you'll just setup the config file like this:
>
> mask = {
> grid = [];
> poly = ["/path/to/tropics_mask.nc" ];
> }
>
> Where you replace /path/to with the actual path that file on your
> machine.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu
>
> wrote:
>
>
> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> Transaction: Ticket created by swei at albany.edu
> Queue: met_help
> Subject: mask specification for grid_stat
> Owner: Nobody
> Requestors: swei at albany.edu
> Status: new
> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703
>
>
>
> Hi,
>
> I’m trying to have grid_stat for two gridded model output over
certain
> area, but it shows no matched pairs for the mask area in my log
file.
> I setup a poly file for tropical region, like this
> TROP
> 20.0 -180.0
> 20.0 180.0
> -20.0 180.0
> -20.0 -180.0
> 20.0 -180.0
> I’m wondering if the longitude value is matter, because one of my
output
> is 0 to 360, another is -180 to 180 for longitude.
> Also, I’m not sure if I use the mask in the right way.
> Please give me some guidance.
>
> Thanks,
> Shih-Wei
>
>
>
> <tropics_mask.pdf>
>
>
>
>
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: mask specification for grid_stat
From: Wei, Shih Wei
Time: Wed Mar 25 15:08:11 2020
Hi George,
I’m working on Cheyenne, please check the file if it’s easier.
I also attached the same file here.
/glade/work/swei/MAPP/METplus_wk/output2/metplus_final.conf.george
and the corresponding log file is
/glade/work/swei/MAPP/METplus_wk/output2/log/master_metplus.log.20200325145922.george
I actually copied that StatAnalysis.conf to my local folder and
modified it.
The file I’m using is
/glade/work/swei/MAPP/METplus_wk/workdir_bk/StatAnalysis.conf
Best,
Shih-Wei
On Mar 25, 2020, at 2:53 PM, George McCabe via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
Hi Shih-Wei,
That is odd that it still doesn't substitute the valid time. Could you
send
me your metplus_final.conf file (located in the directory you set
OUTPUT_BASE to)?
The following use case that is found in the repository appears to use
valid
time and correctly substitutes it in the dump file.
parm/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis.conf
You could compare your configuration to this file to see if something
is
obviously different between them that would cause this to happen.
On Wed, Mar 25, 2020 at 2:47 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John and George,
Thanks for your information.
I changed the looping time variables from VALID to INIT, I thought it
would work.
Looks like it’s definitely not.
I just tested with modifying LOOP_BY back to VALID and corresponding
variables.
And set the dump file with
MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
{MODEL1}/{MODEL1}_{valid?fmt=%Y%m%d%H}.stat
The file name still doesn’t include the time flag properly.
But it works with the list that be looped over, e.g.
{MODEL1}/{MODEL1}_{vx_mask?fmt=%s}.stat
In my case, I looped over FULL and tropics.
Then it can generate two stat files, _FULL.stat and _tropics.stat,
respectively.
Best,
Shih-Wei
On Mar 25, 2020, at 11:18 AM, George McCabe via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Hi Shih-Wei,
My best guess is that if you are looping over VALID times, you won't
be
able to use 'init' in the template. You could use 'valid' instead if
that
works for you. Many of the other wrappers support using 'valid' and
'init'
regardless of the LOOP_BY value, but this wrapper may not have that
functionality implemented yet. Please let me know if using 'valid'
fixes
the issue.
Thanks,
George
On Wed, Mar 25, 2020 at 11:04 AM John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Shih-Wei,
The Stat-Analysis wrapper was written by one of our NOAA colleagues.
George's guess is that the dump row template doesn't support filename
template syntax, so the init tag won't be translated correctly. I'm
guessing this will ultimately require a bugfix or enhancement to that
wrapper. I'm reassigning the ticket now and adding our NOAA colleague
to
it.
John
On Wed, Mar 25, 2020 at 10:55 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Sure, thank you so much, John.
Shih-Wei
On Mar 25, 2020, at 10:54 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih,
I'm going to reassign this ticket to George. I think he'd be better
equipped to answer that question.
John
On Wed, Mar 25, 2020 at 10:50 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you so much.
Another question in stat_analysis of METplus, I setup the output
within
METplus conf file
MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
{MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
The MODEL1 is CAMSiRA in my case.
But when I check the output folder, the time flag is not shown
properly.
The file name become “CAMSiRA_.stat"
Did I setup the time flag wrong?
Best,
Shih-Wei
On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
I just wrote up a bug report for this issue:
https://github.com/NCAR/METplus/issues/407
The MET tools write log messages up to about level 10 (for the most
detailed debugging). The default is 2 which prints the input/output
files
as well as a brief description of what the tool is doing.
MET's verbosity level is controlled via the METplus wrappers using the
LOG_VERBOSITY level. Or you can set it separately for each MET tool,
as
shown below:
# Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
LOG_STAT_ANALYSIS_VERBOSITY = 2
Thanks,
John
On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you so much.
That’s the reason.
How many verbosity level that user can set?
I did notice that is different to what I set in the configuration
files,
but I thought it will handle by METplus properly.
Best,
Shih-We
On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
wrote:
Shih-Wei,
OK, I logged on to Cheyenne and looked in the log file you sent:
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis can
either run with a config file (as METplus does it) or directly on the
command line. I used the log messages to piece together the command
line
job shown below:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
-obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
-dump_row CAMSiRA_.stat*
This job runs and still gets 0 matching lines:
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
Rerunning but at a higher verbosity level (-v 3), I see the .stat file
it's
reading:
*DEBUG 3: Processing STAT file
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
... 1 of 1*
Comparing the contents of that file to the job command options,
there's a
very slight difference in the OBS_LEV column:
'0,4,*,*' != '(0,4,*,*)'
'0,7,*,*' != '(0,7,*,*)'
'0,11,*,*' != '(0,11,*,*)'
Rerunning that job without the enclosing parentheses results in 3
matching
lines:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
-obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type SL1L2
-dump_row CAMSiRA_.stat*
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
and OBS_LEV settings. I'll talk to George McCabe to let him know
about
this issue.
Thanks,
John
On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
I’m trying to have filter, but I did try “summary” and “aggregate”.
Both show the same message.
I’ll test in command line first.
BTW, the mask file is work, thanks for your suggestion.
Best,
Shih-Wei
On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
The critical information you didn't send is what is the job command
that
you're running? I'd recommend getting familiar with running the
STAT-Analysis tool first on the command line, and then once you
understand
the functionality, you can use METplus to automate calls to it.
John
On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
John,
Thank you, I’ll try it.
I’m still learning how to use stat_analysis.
Now I can get the stat file for “FULL” domain from grid_stat, when I
point
the file into stat_analysis the log file shows
DEBUG 2: Processing 1 STAT files.
DEBUG 2: STAT Lines read = 15
DEBUG 2: STAT Lines retained = 0
What I confused is none of lines in the stat file be retained, then I
got
empty output file of stat_analysis.
My log file is
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
Thanks,
Shih-Wei
On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
Yes, we typically setup the masking regions first using gen_vx_mask
and
then reference those in the Grid-Stat config file.
Sure, I can answer questions about stat_analysis.
John
On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you.
Topical region is my test run, I may need couple regions once I can
get
the statistics.
You would recommend people use gen_vx_mask if the boundary of mask
area
is
not too complicated, right?
I encountered another problem in the stat_analysis, could I also
consult
this with you?
Best,
Shih-Wei
On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
<mailto:met_help at ucar.edu>>
wrote:
Shih,
I'd recommend that you run the gen_vx_mask tool to create this masking
region. It looks like you really just want to include -20 to 20
latitude.
Rather than defining a polyline region to do this, I think it's
simpler
to
use the latitude masking option in gen_vx_mask. To illustrate:
(1) pull a sample global GFS file:
wget
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
(2) run gen_vx_mask to generate the masking region:
gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
tropics_mask.nc
-type lat -thresh '>=-20&&<=20' -name tropics
(3) plot the result to show the mask
plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
level="(*,*)";'
See the attached pdf.
When you run grid-stat you'll just setup the config file like this:
mask = {
grid = [];
poly = ["/path/to/tropics_mask.nc" ];
}
Where you replace /path/to with the actual path that file on your
machine.
Thanks,
John
On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
wrote:
Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
Transaction: Ticket created by swei at albany.edu<mailto:swei at albany.edu>
Queue: met_help
Subject: mask specification for grid_stat
Owner: Nobody
Requestors: swei at albany.edu<mailto:swei at albany.edu>
Status: new
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi,
I’m trying to have grid_stat for two gridded model output over certain
area, but it shows no matched pairs for the mask area in my log file.
I setup a poly file for tropical region, like this
TROP
20.0 -180.0
20.0 180.0
-20.0 180.0
-20.0 -180.0
20.0 -180.0
I’m wondering if the longitude value is matter, because one of my
output
is 0 to 360, another is -180 to 180 for longitude.
Also, I’m not sure if I use the mask in the right way.
Please give me some guidance.
Thanks,
Shih-Wei
<tropics_mask.pdf>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: mask specification for grid_stat
From: George McCabe
Time: Wed Mar 25 15:16:49 2020
Hi,
I took a look at your configuration, and I did notice a difference
between
that and the StatAnalysis.conf file that may be the cause.
In yours:
FCST_VALID_HOUR_LIST =
In StatAnalysis.conf
FCST_VALID_HOUR_LIST = 12
I did not write this wrapper and it contains some logic that differs
from
the other wrappers regarding time. I am not sure what these settings
control, but I did notice this difference, so it could have an affect
on
the output. If this does not fix the issue, I will have to defer to
the
author of the wrapper to provide more information.
Thanks,
George
On Wed, Mar 25, 2020 at 3:08 PM Wei, Shih Wei via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi George,
>
> I’m working on Cheyenne, please check the file if it’s easier.
> I also attached the same file here.
> /glade/work/swei/MAPP/METplus_wk/output2/metplus_final.conf.george
> and the corresponding log file is
>
>
/glade/work/swei/MAPP/METplus_wk/output2/log/master_metplus.log.20200325145922.george
>
> I actually copied that StatAnalysis.conf to my local folder and
modified
> it.
> The file I’m using is
> /glade/work/swei/MAPP/METplus_wk/workdir_bk/StatAnalysis.conf
>
> Best,
> Shih-Wei
>
>
> On Mar 25, 2020, at 2:53 PM, George McCabe via RT <met_help at ucar.edu
> <mailto:met_help at ucar.edu>> wrote:
>
> Hi Shih-Wei,
>
> That is odd that it still doesn't substitute the valid time. Could
you send
> me your metplus_final.conf file (located in the directory you set
> OUTPUT_BASE to)?
>
> The following use case that is found in the repository appears to
use valid
> time and correctly substitutes it in the dump file.
>
> parm/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis.conf
>
> You could compare your configuration to this file to see if
something is
> obviously different between them that would cause this to happen.
>
> On Wed, Mar 25, 2020 at 2:47 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John and George,
>
> Thanks for your information.
> I changed the looping time variables from VALID to INIT, I thought
it
> would work.
> Looks like it’s definitely not.
>
> I just tested with modifying LOOP_BY back to VALID and corresponding
> variables.
> And set the dump file with
> MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
> {MODEL1}/{MODEL1}_{valid?fmt=%Y%m%d%H}.stat
> The file name still doesn’t include the time flag properly.
> But it works with the list that be looped over, e.g.
> {MODEL1}/{MODEL1}_{vx_mask?fmt=%s}.stat
> In my case, I looped over FULL and tropics.
> Then it can generate two stat files, _FULL.stat and _tropics.stat,
> respectively.
>
> Best,
> Shih-Wei
>
>
> On Mar 25, 2020, at 11:18 AM, George McCabe via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Hi Shih-Wei,
>
> My best guess is that if you are looping over VALID times, you won't
be
> able to use 'init' in the template. You could use 'valid' instead if
that
> works for you. Many of the other wrappers support using 'valid' and
'init'
> regardless of the LOOP_BY value, but this wrapper may not have that
> functionality implemented yet. Please let me know if using 'valid'
fixes
> the issue.
>
> Thanks,
> George
>
> On Wed, Mar 25, 2020 at 11:04 AM John Halley Gotway via RT <
>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Shih-Wei,
>
> The Stat-Analysis wrapper was written by one of our NOAA colleagues.
> George's guess is that the dump row template doesn't support
filename
> template syntax, so the init tag won't be translated correctly. I'm
> guessing this will ultimately require a bugfix or enhancement to
that
> wrapper. I'm reassigning the ticket now and adding our NOAA
colleague to
> it.
>
> John
>
> On Wed, Mar 25, 2020 at 10:55 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Sure, thank you so much, John.
>
> Shih-Wei
>
> On Mar 25, 2020, at 10:54 AM, John Halley Gotway via RT <
>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih,
>
> I'm going to reassign this ticket to George. I think he'd be better
> equipped to answer that question.
>
> John
>
> On Wed, Mar 25, 2020 at 10:50 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you so much.
> Another question in stat_analysis of METplus, I setup the output
within
> METplus conf file
> MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
> {MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
> The MODEL1 is CAMSiRA in my case.
> But when I check the output folder, the time flag is not shown
properly.
> The file name become “CAMSiRA_.stat"
> Did I setup the time flag wrong?
>
> Best,
> Shih-Wei
>
> On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT <
>
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> I just wrote up a bug report for this issue:
> https://github.com/NCAR/METplus/issues/407
>
> The MET tools write log messages up to about level 10 (for the most
> detailed debugging). The default is 2 which prints the input/output
> files
> as well as a brief description of what the tool is doing.
>
> MET's verbosity level is controlled via the METplus wrappers using
the
> LOG_VERBOSITY level. Or you can set it separately for each MET
tool, as
> shown below:
>
> # Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
> LOG_STAT_ANALYSIS_VERBOSITY = 2
>
>
> Thanks,
> John
>
> On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you so much.
> That’s the reason.
> How many verbosity level that user can set?
> I did notice that is different to what I set in the configuration
files,
> but I thought it will handle by METplus properly.
>
> Best,
> Shih-We
>
> On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
> ><mailto:met_help at ucar.edu>>
> wrote:
>
> Shih-Wei,
>
> OK, I logged on to Cheyenne and looked in the log file you sent:
>
>
>
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis
can
> either run with a config file (as METplus does it) or directly on
the
> command line. I used the log messages to piece together the command
line
> job shown below:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
>
>
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
> -obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
> This job runs and still gets 0 matching lines:
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
>
> Rerunning but at a higher verbosity level (-v 3), I see the .stat
file
> it's
> reading:
> *DEBUG 3: Processing STAT file
>
>
>
>
>
>
>
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
> ... 1 of 1*
>
> Comparing the contents of that file to the job command options,
there's a
> very slight difference in the OBS_LEV column:
> '0,4,*,*' != '(0,4,*,*)'
> '0,7,*,*' != '(0,7,*,*)'
> '0,11,*,*' != '(0,11,*,*)'
>
> Rerunning that job without the enclosing parentheses results in 3
> matching
> lines:
>
> */glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
>
>
>
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
> -job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
> -fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
> -obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
> -dump_row CAMSiRA_.stat*
>
>
> *DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
>
> So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
> and OBS_LEV settings. I'll talk to George McCabe to let him know
about
> this issue.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> I’m trying to have filter, but I did try “summary” and “aggregate”.
> Both show the same message.
> I’ll test in command line first.
> BTW, the mask file is work, thanks for your suggestion.
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
> ><mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> The critical information you didn't send is what is the job command
that
> you're running? I'd recommend getting familiar with running the
> STAT-Analysis tool first on the command line, and then once you
> understand
> the functionality, you can use METplus to automate calls to it.
>
> John
>
> On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> John,
>
> Thank you, I’ll try it.
> I’m still learning how to use stat_analysis.
> Now I can get the stat file for “FULL” domain from grid_stat, when I
> point
> the file into stat_analysis the log file shows
> DEBUG 2: Processing 1 STAT files.
> DEBUG 2: STAT Lines read = 15
> DEBUG 2: STAT Lines retained = 0
> What I confused is none of lines in the stat file be retained, then
I
> got
> empty output file of stat_analysis.
>
> My log file is
>
>
>
>
>
>
>
>
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
>
> Thanks,
> Shih-Wei
>
>
> On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
> ><mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>> wrote:
>
> Shih-Wei,
>
> Yes, we typically setup the masking regions first using gen_vx_mask
and
> then reference those in the Grid-Stat config file.
>
> Sure, I can answer questions about stat_analysis.
>
> John
>
> On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>
> <mailto:met_help at ucar.edu>>
> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
>
> Hi John,
>
> Thank you.
> Topical region is my test run, I may need couple regions once I can
get
> the statistics.
> You would recommend people use gen_vx_mask if the boundary of mask
area
> is
> not too complicated, right?
>
> I encountered another problem in the stat_analysis, could I also
consult
> this with you?
>
> Best,
> Shih-Wei
>
> On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
> met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
> ><mailto:met_help at ucar.edu
> <mailto:met_help at ucar.edu>>
> wrote:
>
> Shih,
>
> I'd recommend that you run the gen_vx_mask tool to create this
masking
> region. It looks like you really just want to include -20 to 20
> latitude.
> Rather than defining a polyline region to do this, I think it's
simpler
> to
> use the latitude masking option in gen_vx_mask. To illustrate:
>
> (1) pull a sample global GFS file:
> wget
>
>
>
>
>
>
>
>
>
>
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
>
> (2) run gen_vx_mask to generate the masking region:
> gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
> tropics_mask.nc
> -type lat -thresh '>=-20&&<=20' -name tropics
>
> (3) plot the result to show the mask
> plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
> level="(*,*)";'
>
> See the attached pdf.
>
> When you run grid-stat you'll just setup the config file like this:
>
> mask = {
> grid = [];
> poly = ["/path/to/tropics_mask.nc" ];
> }
>
> Where you replace /path/to with the actual path that file on your
> machine.
>
> Thanks,
> John
>
> On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu
> <mailto:met_help at ucar.edu>
>
> wrote:
>
>
> Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
> Transaction: Ticket created by
swei at albany.edu<mailto:swei at albany.edu>
> Queue: met_help
> Subject: mask specification for grid_stat
> Owner: Nobody
> Requestors: swei at albany.edu<mailto:swei at albany.edu>
> Status: new
> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703
>
>
>
> Hi,
>
> I’m trying to have grid_stat for two gridded model output over
certain
> area, but it shows no matched pairs for the mask area in my log
file.
> I setup a poly file for tropical region, like this
> TROP
> 20.0 -180.0
> 20.0 180.0
> -20.0 180.0
> -20.0 -180.0
> 20.0 -180.0
> I’m wondering if the longitude value is matter, because one of my
output
> is 0 to 360, another is -180 to 180 for longitude.
> Also, I’m not sure if I use the mask in the right way.
> Please give me some guidance.
>
> Thanks,
> Shih-Wei
>
>
>
> <tropics_mask.pdf>
>
>
>
>
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: mask specification for grid_stat
From: Wei, Shih Wei
Time: Wed Mar 25 15:29:20 2020
Hi George,
That’s the point.
It also works for INIT, just need to setup FCST_INIT_HOUR_LIST.
Thank you so much.
Best,
Shih-Wei
On Mar 25, 2020, at 3:16 PM, George McCabe via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
Hi,
I took a look at your configuration, and I did notice a difference
between
that and the StatAnalysis.conf file that may be the cause.
In yours:
FCST_VALID_HOUR_LIST =
In StatAnalysis.conf
FCST_VALID_HOUR_LIST = 12
I did not write this wrapper and it contains some logic that differs
from
the other wrappers regarding time. I am not sure what these settings
control, but I did notice this difference, so it could have an affect
on
the output. If this does not fix the issue, I will have to defer to
the
author of the wrapper to provide more information.
Thanks,
George
On Wed, Mar 25, 2020 at 3:08 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi George,
I’m working on Cheyenne, please check the file if it’s easier.
I also attached the same file here.
/glade/work/swei/MAPP/METplus_wk/output2/metplus_final.conf.george
and the corresponding log file is
/glade/work/swei/MAPP/METplus_wk/output2/log/master_metplus.log.20200325145922.george
I actually copied that StatAnalysis.conf to my local folder and
modified
it.
The file I’m using is
/glade/work/swei/MAPP/METplus_wk/workdir_bk/StatAnalysis.conf
Best,
Shih-Wei
On Mar 25, 2020, at 2:53 PM, George McCabe via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Hi Shih-Wei,
That is odd that it still doesn't substitute the valid time. Could you
send
me your metplus_final.conf file (located in the directory you set
OUTPUT_BASE to)?
The following use case that is found in the repository appears to use
valid
time and correctly substitutes it in the dump file.
parm/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis.conf
You could compare your configuration to this file to see if something
is
obviously different between them that would cause this to happen.
On Wed, Mar 25, 2020 at 2:47 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John and George,
Thanks for your information.
I changed the looping time variables from VALID to INIT, I thought it
would work.
Looks like it’s definitely not.
I just tested with modifying LOOP_BY back to VALID and corresponding
variables.
And set the dump file with
MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
{MODEL1}/{MODEL1}_{valid?fmt=%Y%m%d%H}.stat
The file name still doesn’t include the time flag properly.
But it works with the list that be looped over, e.g.
{MODEL1}/{MODEL1}_{vx_mask?fmt=%s}.stat
In my case, I looped over FULL and tropics.
Then it can generate two stat files, _FULL.stat and _tropics.stat,
respectively.
Best,
Shih-Wei
On Mar 25, 2020, at 11:18 AM, George McCabe via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Hi Shih-Wei,
My best guess is that if you are looping over VALID times, you won't
be
able to use 'init' in the template. You could use 'valid' instead if
that
works for you. Many of the other wrappers support using 'valid' and
'init'
regardless of the LOOP_BY value, but this wrapper may not have that
functionality implemented yet. Please let me know if using 'valid'
fixes
the issue.
Thanks,
George
On Wed, Mar 25, 2020 at 11:04 AM John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Shih-Wei,
The Stat-Analysis wrapper was written by one of our NOAA colleagues.
George's guess is that the dump row template doesn't support filename
template syntax, so the init tag won't be translated correctly. I'm
guessing this will ultimately require a bugfix or enhancement to that
wrapper. I'm reassigning the ticket now and adding our NOAA colleague
to
it.
John
On Wed, Mar 25, 2020 at 10:55 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Sure, thank you so much, John.
Shih-Wei
On Mar 25, 2020, at 10:54 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih,
I'm going to reassign this ticket to George. I think he'd be better
equipped to answer that question.
John
On Wed, Mar 25, 2020 at 10:50 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you so much.
Another question in stat_analysis of METplus, I setup the output
within
METplus conf file
MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE =
{MODEL1}/{MODEL1}_{init?fmt=%Y%m%d%H}.stat
The MODEL1 is CAMSiRA in my case.
But when I check the output folder, the time flag is not shown
properly.
The file name become “CAMSiRA_.stat"
Did I setup the time flag wrong?
Best,
Shih-Wei
On Mar 25, 2020, at 10:31 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
I just wrote up a bug report for this issue:
https://github.com/NCAR/METplus/issues/407
The MET tools write log messages up to about level 10 (for the most
detailed debugging). The default is 2 which prints the input/output
files
as well as a brief description of what the tool is doing.
MET's verbosity level is controlled via the METplus wrappers using the
LOG_VERBOSITY level. Or you can set it separately for each MET tool,
as
shown below:
# Verbosity of MET output - overrides LOG_VERBOSITY for StatAnalysis
only
LOG_STAT_ANALYSIS_VERBOSITY = 2
Thanks,
John
On Wed, Mar 25, 2020 at 10:22 AM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you so much.
That’s the reason.
How many verbosity level that user can set?
I did notice that is different to what I set in the configuration
files,
but I thought it will handle by METplus properly.
Best,
Shih-We
On Mar 25, 2020, at 10:09 AM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
<mailto:met_help at ucar.edu>>
wrote:
Shih-Wei,
OK, I logged on to Cheyenne and looked in the log file you sent:
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
And I see that STAT-Analysis is retaining 0 lines. STAT-Analysis can
either run with a config file (as METplus does it) or directly on the
command line. I used the log messages to piece together the command
line
job shown below:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '(0,4,*,*)' -obs_lev
'(0,7,*,*)'
-obs_lev '(0,11,*,*)' -obtype MERRA2_plv -vx_mask FULL -line_type
SL1L2
-dump_row CAMSiRA_.stat*
This job runs and still gets 0 matching lines:
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 0*
Rerunning but at a higher verbosity level (-v 3), I see the .stat file
it's
reading:
*DEBUG 3: Processing STAT file
"/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000/grid_stat_CAMSiRA_AERMR11_vs_MERRA2_plv_SO4_000000L_20180410_000000V.stat"
... 1 of 1*
Comparing the contents of that file to the job command options,
there's a
very slight difference in the OBS_LEV column:
'0,4,*,*' != '(0,4,*,*)'
'0,7,*,*' != '(0,7,*,*)'
'0,11,*,*' != '(0,11,*,*)'
Rerunning that job without the enclosing parentheses results in 3
matching
lines:
*/glade/p/ral/jntp/MET/MET_releases/9.0/bin/stat_analysis -lookin
/glade/work/swei/MAPP/METplus_wk/output2/met_tool_wrapper/GridStat/GridStat/2018041000
-job filter -model CAMSiRA -fcst_var AERMR11 -obs_var SO4 -fcst_lev
P500
-fcst_lev P850 -fcst_lev P1000 -obs_lev '0,4,*,*' -obs_lev '0,7,*,*'
-obs_lev '0,11,*,*' -obtype MERRA2_plv -vx_mask FULL -line_type SL1L2
-dump_row CAMSiRA_.stat*
*DEBUG 2: STAT Lines read = 16DEBUG 2: STAT Lines retained = 3*
So I surmise that this is a bug in METplus in the formatting of
FCST_LEV
and OBS_LEV settings. I'll talk to George McCabe to let him know
about
this issue.
Thanks,
John
On Tue, Mar 24, 2020 at 10:13 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
I’m trying to have filter, but I did try “summary” and “aggregate”.
Both show the same message.
I’ll test in command line first.
BTW, the mask file is work, thanks for your suggestion.
Best,
Shih-Wei
On Mar 24, 2020, at 5:38 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
The critical information you didn't send is what is the job command
that
you're running? I'd recommend getting familiar with running the
STAT-Analysis tool first on the command line, and then once you
understand
the functionality, you can use METplus to automate calls to it.
John
On Tue, Mar 24, 2020 at 5:17 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
John,
Thank you, I’ll try it.
I’m still learning how to use stat_analysis.
Now I can get the stat file for “FULL” domain from grid_stat, when I
point
the file into stat_analysis the log file shows
DEBUG 2: Processing 1 STAT files.
DEBUG 2: STAT Lines read = 15
DEBUG 2: STAT Lines retained = 0
What I confused is none of lines in the stat file be retained, then I
got
empty output file of stat_analysis.
My log file is
/glade/work/swei/MAPP/METplus_wk/output2/logs/master_metplus.log.20200324155718
Thanks,
Shih-Wei
On Mar 24, 2020, at 5:07 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>> wrote:
Shih-Wei,
Yes, we typically setup the masking regions first using gen_vx_mask
and
then reference those in the Grid-Stat config file.
Sure, I can answer questions about stat_analysis.
John
On Tue, Mar 24, 2020 at 5:02 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>>
wrote:
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi John,
Thank you.
Topical region is my test run, I may need couple regions once I can
get
the statistics.
You would recommend people use gen_vx_mask if the boundary of mask
area
is
not too complicated, right?
I encountered another problem in the stat_analysis, could I also
consult
this with you?
Best,
Shih-Wei
On Mar 24, 2020, at 4:48 PM, John Halley Gotway via RT <
met_help at ucar.edu<mailto:met_help at ucar.edu><mailto:met_help at ucar.edu><mailto:met_help at ucar.edu
<mailto:met_help at ucar.edu
<mailto:met_help at ucar.edu>>
wrote:
Shih,
I'd recommend that you run the gen_vx_mask tool to create this masking
region. It looks like you really just want to include -20 to 20
latitude.
Rather than defining a polyline region to do this, I think it's
simpler
to
use the latitude masking option in gen_vx_mask. To illustrate:
(1) pull a sample global GFS file:
wget
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200323/12/gfs.t12z.pgrb2.1p00.anl
(2) run gen_vx_mask to generate the masking region:
gen_vx_mask gfs.t12z.pgrb2.1p00.anl gfs.t12z.pgrb2.1p00.anl
tropics_mask.nc
-type lat -thresh '>=-20&&<=20' -name tropics
(3) plot the result to show the mask
plot_data_plane tropics_mask.nc tropics_mask.ps 'name="tropics";
level="(*,*)";'
See the attached pdf.
When you run grid-stat you'll just setup the config file like this:
mask = {
grid = [];
poly = ["/path/to/tropics_mask.nc" ];
}
Where you replace /path/to with the actual path that file on your
machine.
Thanks,
John
On Tue, Mar 24, 2020 at 4:20 PM Wei, Shih Wei via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu>
wrote:
Tue Mar 24 16:19:47 2020: Request 94703 was acted upon.
Transaction: Ticket created by
swei at albany.edu<mailto:swei at albany.edu><mailto:swei at albany.edu>
Queue: met_help
Subject: mask specification for grid_stat
Owner: Nobody
Requestors:
swei at albany.edu<mailto:swei at albany.edu><mailto:swei at albany.edu>
Status: new
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94703 >
Hi,
I’m trying to have grid_stat for two gridded model output over certain
area, but it shows no matched pairs for the mask area in my log file.
I setup a poly file for tropical region, like this
TROP
20.0 -180.0
20.0 180.0
-20.0 180.0
-20.0 -180.0
20.0 -180.0
I’m wondering if the longitude value is matter, because one of my
output
is 0 to 360, another is -180 to 180 for longitude.
Also, I’m not sure if I use the mask in the right way.
Please give me some guidance.
Thanks,
Shih-Wei
<tropics_mask.pdf>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list