[Met_help] [rt.rap.ucar.edu #87973] History for METviewer: batch plot questions
Tatiana Burek via RT
met_help at ucar.edu
Mon Dec 17 07:53:56 MST 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello. I have several questions about making batch plots on METviewer.
Attached 4 files are
fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch script
plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script to set up
and run batch job on the above
fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of the plots
produced
test.xml: an interactive METviewer plot xml for this sort of plots
1) I'm specifying model valid times used in the plot with
<field equalize="true" name="fcst_valid_beg">
<start>%DAY1DASH% 00:00:00</start>
<end>%DAY2DASH% 23:59:59</end>
</field>
with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31) substituted in the
calling script with sed. This seems to work, but I think this includes
stats for all valid times between 2018-08-01 00:00:00 and 2018-08-31
23:59:59. In interactive METviewer setup, one can specify
fcst_valid_beg with a start time, end time and an increment (6h, 12h
etc.). How does one specify such an increment in the batch script (e.g.
if I only want valid times at 00Z/12Z, rather than 00/06/12/18Z)?
2) In the past I've been using a lot of sed substitutes and make a
single plot per mv_batch.sh call, like this:
for hscale in 024 052 100
do
for thresh in 002.0 005.0 010.0 020.0
do
cat $ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml \
| sed "s/%DAY1DASH%/$day1dash/g" \
| sed "s/%DAY2DASH%/$day2dash/g" \
| sed 's/%THRESH%/'$thresh'/g' \
| sed 's/%HSCALE%/'$hscale'/g' \
> $tmpscripts/plt_4_prodmods_${day1}-${day2}_${thresh}_${hscale}.xml
/usr1/metviewer/metviewer/bin/mv_batch.sh
$tmpscripts/plt_4_prodmods_${day1}-${day2}_${thresh}_${hscale}.xml
enddo
enddo
Follow John's examples at the Oct tutorial, I was able to replace
<field equalize="true" name="fcst_thresh">
<set name="fcst_thresh_1">
<val><%HSCALE%</val>
</set>
</field>
with
<field equalize="true" name="fcst_thresh">
<val><024</val>
<val><052</val>
<val><100</val>
</field>
and that seems to work.
But is there a way to replace this?
<dep1>
<fcst_var name="APCP/06>%THRESH%">
<stat>NBR_FSS</stat>
</fcst_var>
</dep1>
with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0? If this is
too hard/convoluted to set up then I can just use the old "sed"
substitution.
I'm trying to plot 6h FSS from our VSDBs, so the set up is a bit
peculiar: METviewer consider the forecast variable as e.g, APCP/06>002.0
(mm), and the horizontal spacial scale is the fcst_thresh (e.g., < 052km)
Also, the resulting plots from the successful use of {fcst_thresh} in
batch above have the name such as
"fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from fcst_thresh of
"<024". the "<" and ">" in file name causes some problems ... Is
there an easy way to create another variable such as Afcst_thresh=024
(fcst_thresh with the "<" removed)?
Thanks very much for looking into this -
Ying
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: METviewer: batch plot questions
From: Tatiana Burek
Time: Thu Dec 06 07:59:48 2018
Hi Ying,
#1
METviewer does not support start/end tags in the batch XML. For your
example:
<field equalize="true" name="fcst_valid_beg">
<start>%DAY1DASH% 00:00:00</start>
<end>%DAY2DASH% 23:59:59</end>
</field>
only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will be
included in plotting but nothing in between these days.
#2
Not sure what you mean by "creating another variable". Have you ever
considered using inheritance in the batch XML? It might be a better
solution than using sed.
Tatiana
On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
> Hello. I have several questions about making batch plots on
METviewer.
> Attached 4 files are
>
> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch script
>
> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script to set
up
> and run batch job on the above
>
> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of the
plots
> produced
>
> test.xml: an interactive METviewer plot xml for this sort of
plots
>
> 1) I'm specifying model valid times used in the plot with
>
> <field equalize="true" name="fcst_valid_beg">
> <start>%DAY1DASH% 00:00:00</start>
> <end>%DAY2DASH% 23:59:59</end>
> </field>
>
> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31) substituted in
the
> calling script with sed. This seems to work, but I think this
includes
> stats for all valid times between 2018-08-01 00:00:00 and 2018-08-31
> 23:59:59. In interactive METviewer setup, one can specify
> fcst_valid_beg with a start time, end time and an increment (6h, 12h
> etc.). How does one specify such an increment in the batch script
(e.g.
> if I only want valid times at 00Z/12Z, rather than 00/06/12/18Z)?
>
> 2) In the past I've been using a lot of sed substitutes and make a
> single plot per mv_batch.sh call, like this:
>
> for hscale in 024 052 100
> do
>
> for thresh in 002.0 005.0 010.0 020.0
> do
>
> cat
$ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml \
> | sed "s/%DAY1DASH%/$day1dash/g" \
> | sed "s/%DAY2DASH%/$day2dash/g" \
> | sed 's/%THRESH%/'$thresh'/g' \
> | sed 's/%HSCALE%/'$hscale'/g' \
> > $tmpscripts/plt_4_prodmods_${day1}-
${day2}_${thresh}_${hscale}.xml
>
> /usr1/metviewer/metviewer/bin/mv_batch.sh
> $tmpscripts/plt_4_prodmods_${day1}-${day2}_${thresh}_${hscale}.xml
>
> enddo
>
> enddo
>
> Follow John's examples at the Oct tutorial, I was able to replace
>
> <field equalize="true" name="fcst_thresh">
> <set name="fcst_thresh_1">
> <val><%HSCALE%</val>
> </set>
> </field>
>
> with
>
> <field equalize="true" name="fcst_thresh">
> <val><024</val>
> <val><052</val>
> <val><100</val>
> </field>
>
> and that seems to work.
>
> But is there a way to replace this?
>
> <dep1>
> <fcst_var name="APCP/06>%THRESH%">
> <stat>NBR_FSS</stat>
> </fcst_var>
> </dep1>
>
> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0? If this
is
> too hard/convoluted to set up then I can just use the old "sed"
> substitution.
>
> I'm trying to plot 6h FSS from our VSDBs, so the set up is a bit
> peculiar: METviewer consider the forecast variable as e.g,
APCP/06>002.0
> (mm), and the horizontal spacial scale is the fcst_thresh (e.g., <
052km)
>
> Also, the resulting plots from the successful use of {fcst_thresh}
in
> batch above have the name such as
> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from fcst_thresh
of
> "<024". the "<" and ">" in file name causes some problems ...
Is
> there an easy way to create another variable such as
Afcst_thresh=024
> (fcst_thresh with the "<" removed)?
>
> Thanks very much for looking into this -
>
> Ying
>
------------------------------------------------
Subject: METviewer: batch plot questions
From: Ying Lin
Time: Thu Dec 06 11:05:23 2018
Hi Tatiana,
Thank you for your reply. I didn't realize that METviewer batch
XML can't have the start/end tags - good to know. In the attached
XML
(example from John H.G. at the Oct tutorial) there is
<date_range name="fall" ><start>2011-10-01
00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
...
<plot_fix>
<field name="fcst_init_beg">
<date_range name="fall"/>
I suppose this means that the date range must be pre-defined (in the
above "date_range" tag). I've used this pre-defined set-up before (in
e.g. making seasonal plots). For a more flexible set up (e.g.
plotting
from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by setting
up
something like
<date_range
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
then just use 'sed' to do the substitution.
Questions still remain:
1) In the above pre-defined date range set up (either "fall" or
"anydaterange"), how does one specify the increment (e.g. 12h, when
the
data are 6-hourly)?
2) same as the item 2) in the original email inquiry
3) By "creating another variable", I meant that when fcst_thresh is
given the value of "<52km", and {fcst_thresh} is used in the resulting
png name, the '<' is causing trouble in unix. Is there a way to get
rid
of the '<', e.g. creating another variable that is 'fcst_thresh' minus
the '<'?
Thanks again,
Ying
On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
> Hi Ying,
>
> #1
> METviewer does not support start/end tags in the batch XML. For your
example:
> <field equalize="true" name="fcst_valid_beg">
> <start>%DAY1DASH% 00:00:00</start>
> <end>%DAY2DASH% 23:59:59</end>
> </field>
> only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will be
included in plotting but nothing in between these days.
>
> #2
> Not sure what you mean by "creating another variable". Have you ever
considered using inheritance in the batch XML? It might be a better
solution than using sed.
>
> Tatiana
>
> On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
>> Hello. I have several questions about making batch plots on
METviewer.
>> Attached 4 files are
>>
>> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch script
>>
>> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script to
set up
>> and run batch job on the above
>>
>> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of the
plots
>> produced
>>
>> test.xml: an interactive METviewer plot xml for this sort of
plots
>>
>> 1) I'm specifying model valid times used in the plot with
>>
>> <field equalize="true" name="fcst_valid_beg">
>> <start>%DAY1DASH% 00:00:00</start>
>> <end>%DAY2DASH% 23:59:59</end>
>> </field>
>>
>> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31) substituted in
the
>> calling script with sed. This seems to work, but I think this
includes
>> stats for all valid times between 2018-08-01 00:00:00 and 2018-08-
31
>> 23:59:59. In interactive METviewer setup, one can specify
>> fcst_valid_beg with a start time, end time and an increment (6h,
12h
>> etc.). How does one specify such an increment in the batch script
(e.g.
>> if I only want valid times at 00Z/12Z, rather than 00/06/12/18Z)?
>>
>> 2) In the past I've been using a lot of sed substitutes and make a
>> single plot per mv_batch.sh call, like this:
>>
>> for hscale in 024 052 100
>> do
>>
>> for thresh in 002.0 005.0 010.0 020.0
>> do
>>
>> cat
$ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml \
>> | sed "s/%DAY1DASH%/$day1dash/g" \
>> | sed "s/%DAY2DASH%/$day2dash/g" \
>> | sed 's/%THRESH%/'$thresh'/g' \
>> | sed 's/%HSCALE%/'$hscale'/g' \
>> > $tmpscripts/plt_4_prodmods_${day1}-
${day2}_${thresh}_${hscale}.xml
>>
>> /usr1/metviewer/metviewer/bin/mv_batch.sh
>> $tmpscripts/plt_4_prodmods_${day1}-${day2}_${thresh}_${hscale}.xml
>>
>> enddo
>>
>> enddo
>>
>> Follow John's examples at the Oct tutorial, I was able to replace
>>
>> <field equalize="true" name="fcst_thresh">
>> <set name="fcst_thresh_1">
>> <val><%HSCALE%</val>
>> </set>
>> </field>
>>
>> with
>>
>> <field equalize="true" name="fcst_thresh">
>> <val><024</val>
>> <val><052</val>
>> <val><100</val>
>> </field>
>>
>> and that seems to work.
>>
>> But is there a way to replace this?
>>
>> <dep1>
>> <fcst_var name="APCP/06>%THRESH%">
>> <stat>NBR_FSS</stat>
>> </fcst_var>
>> </dep1>
>>
>> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0? If
this is
>> too hard/convoluted to set up then I can just use the old "sed"
>> substitution.
>>
>> I'm trying to plot 6h FSS from our VSDBs, so the set up is a bit
>> peculiar: METviewer consider the forecast variable as e.g,
APCP/06>002.0
>> (mm), and the horizontal spacial scale is the fcst_thresh (e.g., <
052km)
>>
>> Also, the resulting plots from the successful use of {fcst_thresh}
in
>> batch above have the name such as
>> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from fcst_thresh
of
>> "<024". the "<" and ">" in file name causes some problems ...
Is
>> there an easy way to create another variable such as
Afcst_thresh=024
>> (fcst_thresh with the "<" removed)?
>>
>> Thanks very much for looking into this -
>>
>> Ying
>>
>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: METviewer: batch plot questions
From: Tatiana Burek
Time: Fri Dec 07 09:58:49 2018
Ying,
1.Predefined <date_range> from the example will work as advertised -
all dates in between including start and end will be included. There
is no increment specification in the setup.
You can specify valid_hour for example to include dates with the
specific hour.
2.What is your goal? To have individual plots for each fcst_var or
plot all fcst_var in one plot?
3.instead of using
fss06_%DAY1DASH%-%DAY2DASH%_{fcst_thresh}km_%THRESH%mm.png
you could use fss06_%DAY1DASH%-%DAY2DASH%_%THRESH_1%km_%THRESH%mm.png
and change %THRESH_1% with sed to the value you need
Tatiana
On Thu Dec 06 11:05:23 2018, ying.lin at noaa.gov wrote:
> Hi Tatiana,
>
> Thank you for your reply. I didn't realize that METviewer batch
> XML can't have the start/end tags - good to know. In the attached
> XML
> (example from John H.G. at the Oct tutorial) there is
> <date_range name="fall" ><start>2011-10-01
> 00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
> ...
> <plot_fix>
> <field name="fcst_init_beg">
> <date_range name="fall"/>
>
> I suppose this means that the date range must be pre-defined (in the
> above "date_range" tag). I've used this pre-defined set-up before
(in
> e.g. making seasonal plots). For a more flexible set up (e.g.
> plotting
> from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by setting
> up
> something like
> <date_range
>
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
> then just use 'sed' to do the substitution.
>
> Questions still remain:
> 1) In the above pre-defined date range set up (either "fall" or
> "anydaterange"), how does one specify the increment (e.g. 12h, when
> the
> data are 6-hourly)?
> 2) same as the item 2) in the original email inquiry
> 3) By "creating another variable", I meant that when fcst_thresh is
> given the value of "<52km", and {fcst_thresh} is used in the
> resulting
> png name, the '<' is causing trouble in unix. Is there a way to
get
> rid
> of the '<', e.g. creating another variable that is 'fcst_thresh'
> minus
> the '<'?
>
> Thanks again,
>
> Ying
>
>
> On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
> > Hi Ying,
> >
> > #1
> > METviewer does not support start/end tags in the batch XML. For
your
> > example:
> > <field equalize="true" name="fcst_valid_beg">
> > <start>%DAY1DASH% 00:00:00</start>
> > <end>%DAY2DASH% 23:59:59</end>
> > </field>
> > only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will be
> > included in plotting but nothing in between these days.
> >
> > #2
> > Not sure what you mean by "creating another variable". Have you
ever
> > considered using inheritance in the batch XML? It might be a
better
> > solution than using sed.
> >
> > Tatiana
> >
> > On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
> >> Hello. I have several questions about making batch plots on
> >> METviewer.
> >> Attached 4 files are
> >>
> >> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch script
> >>
> >> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script to
set
> >> up
> >> and run batch job on the above
> >>
> >> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of the
> >> plots
> >> produced
> >>
> >> test.xml: an interactive METviewer plot xml for this sort of
> >> plots
> >>
> >> 1) I'm specifying model valid times used in the plot with
> >>
> >> <field equalize="true" name="fcst_valid_beg">
> >> <start>%DAY1DASH% 00:00:00</start>
> >> <end>%DAY2DASH% 23:59:59</end>
> >> </field>
> >>
> >> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31) substituted
in
> >> the
> >> calling script with sed. This seems to work, but I think this
> >> includes
> >> stats for all valid times between 2018-08-01 00:00:00 and 2018-
08-31
> >> 23:59:59. In interactive METviewer setup, one can specify
> >> fcst_valid_beg with a start time, end time and an increment (6h,
12h
> >> etc.). How does one specify such an increment in the batch
script
> >> (e.g.
> >> if I only want valid times at 00Z/12Z, rather than 00/06/12/18Z)?
> >>
> >> 2) In the past I've been using a lot of sed substitutes and make
a
> >> single plot per mv_batch.sh call, like this:
> >>
> >> for hscale in 024 052 100
> >> do
> >>
> >> for thresh in 002.0 005.0 010.0 020.0
> >> do
> >>
> >> cat
> >> $ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml \
> >> | sed "s/%DAY1DASH%/$day1dash/g" \
> >> | sed "s/%DAY2DASH%/$day2dash/g" \
> >> | sed 's/%THRESH%/'$thresh'/g' \
> >> | sed 's/%HSCALE%/'$hscale'/g' \
> >> > $tmpscripts/plt_4_prodmods_${day1}-
> >> ${day2}_${thresh}_${hscale}.xml
> >>
> >> /usr1/metviewer/metviewer/bin/mv_batch.sh
> >> $tmpscripts/plt_4_prodmods_${day1}-
${day2}_${thresh}_${hscale}.xml
> >>
> >> enddo
> >>
> >> enddo
> >>
> >> Follow John's examples at the Oct tutorial, I was able to replace
> >>
> >> <field equalize="true" name="fcst_thresh">
> >> <set name="fcst_thresh_1">
> >> <val><%HSCALE%</val>
> >> </set>
> >> </field>
> >>
> >> with
> >>
> >> <field equalize="true" name="fcst_thresh">
> >> <val><024</val>
> >> <val><052</val>
> >> <val><100</val>
> >> </field>
> >>
> >> and that seems to work.
> >>
> >> But is there a way to replace this?
> >>
> >> <dep1>
> >> <fcst_var name="APCP/06>%THRESH%">
> >> <stat>NBR_FSS</stat>
> >> </fcst_var>
> >> </dep1>
> >>
> >> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0? If
this
> >> is
> >> too hard/convoluted to set up then I can just use the old "sed"
> >> substitution.
> >>
> >> I'm trying to plot 6h FSS from our VSDBs, so the set up is a bit
> >> peculiar: METviewer consider the forecast variable as e.g,
> >> APCP/06>002.0
> >> (mm), and the horizontal spacial scale is the fcst_thresh (e.g.,
<
> >> 052km)
> >>
> >> Also, the resulting plots from the successful use of
{fcst_thresh}
> >> in
> >> batch above have the name such as
> >> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from
fcst_thresh
> >> of
> >> "<024". the "<" and ">" in file name causes some problems ...
> >> Is
> >> there an easy way to create another variable such as
> >> Afcst_thresh=024
> >> (fcst_thresh with the "<" removed)?
> >>
> >> Thanks very much for looking into this -
> >>
> >> Ying
> >>
> >
> >
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87973] METviewer: batch plot questions
From: Ying Lin
Time: Fri Dec 07 10:12:51 2018
Thanks Tatiana. Following up:
2) My goal is to have an individual plot for each fcst_var. How to do
that?
3) Thanks, but using sed substitute is something I've already been
doing
before. The issue at hand is how to run "mv_batch.sh script.xml" once
and get a "batch" of plots done, and part of that process is to use
the
e.g. {fcst_thresh} in the resulting .png file name. When fcst_thresh
is
"<52km", how to avoid having the "<" ending up in the file name,
within
the context of the xml?
Thanks again,
Ying
On 12/7/18 11:58 AM, Tatiana Burek via RT wrote:
> Ying,
>
> 1.Predefined <date_range> from the example will work as advertised -
all dates in between including start and end will be included. There
is no increment specification in the setup.
> You can specify valid_hour for example to include dates with the
specific hour.
>
> 2.What is your goal? To have individual plots for each fcst_var or
plot all fcst_var in one plot?
>
> 3.instead of using
> fss06_%DAY1DASH%-%DAY2DASH%_{fcst_thresh}km_%THRESH%mm.png
> you could use fss06_%DAY1DASH%-
%DAY2DASH%_%THRESH_1%km_%THRESH%mm.png
> and change %THRESH_1% with sed to the value you need
>
> Tatiana
>
> On Thu Dec 06 11:05:23 2018, ying.lin at noaa.gov wrote:
>> Hi Tatiana,
>>
>> Thank you for your reply. I didn't realize that METviewer
batch
>> XML can't have the start/end tags - good to know. In the
attached
>> XML
>> (example from John H.G. at the Oct tutorial) there is
>> <date_range name="fall" ><start>2011-10-01
>> 00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
>> ...
>> <plot_fix>
>> <field name="fcst_init_beg">
>> <date_range name="fall"/>
>>
>> I suppose this means that the date range must be pre-defined (in
the
>> above "date_range" tag). I've used this pre-defined set-up before
(in
>> e.g. making seasonal plots). For a more flexible set up (e.g.
>> plotting
>> from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by
setting
>> up
>> something like
>> <date_range
>>
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
>> then just use 'sed' to do the substitution.
>>
>> Questions still remain:
>> 1) In the above pre-defined date range set up (either "fall" or
>> "anydaterange"), how does one specify the increment (e.g. 12h,
when
>> the
>> data are 6-hourly)?
>> 2) same as the item 2) in the original email inquiry
>> 3) By "creating another variable", I meant that when fcst_thresh
is
>> given the value of "<52km", and {fcst_thresh} is used in the
>> resulting
>> png name, the '<' is causing trouble in unix. Is there a way to
get
>> rid
>> of the '<', e.g. creating another variable that is 'fcst_thresh'
>> minus
>> the '<'?
>>
>> Thanks again,
>>
>> Ying
>>
>>
>> On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
>>> Hi Ying,
>>>
>>> #1
>>> METviewer does not support start/end tags in the batch XML. For
your
>>> example:
>>> <field equalize="true" name="fcst_valid_beg">
>>> <start>%DAY1DASH% 00:00:00</start>
>>> <end>%DAY2DASH% 23:59:59</end>
>>> </field>
>>> only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will be
>>> included in plotting but nothing in between these days.
>>>
>>> #2
>>> Not sure what you mean by "creating another variable". Have you
ever
>>> considered using inheritance in the batch XML? It might be a
better
>>> solution than using sed.
>>>
>>> Tatiana
>>>
>>> On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
>>>> Hello. I have several questions about making batch plots on
>>>> METviewer.
>>>> Attached 4 files are
>>>>
>>>> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch script
>>>>
>>>> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script to
set
>>>> up
>>>> and run batch job on the above
>>>>
>>>> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of the
>>>> plots
>>>> produced
>>>>
>>>> test.xml: an interactive METviewer plot xml for this sort of
>>>> plots
>>>>
>>>> 1) I'm specifying model valid times used in the plot with
>>>>
>>>> <field equalize="true" name="fcst_valid_beg">
>>>> <start>%DAY1DASH% 00:00:00</start>
>>>> <end>%DAY2DASH% 23:59:59</end>
>>>> </field>
>>>>
>>>> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31) substituted
in
>>>> the
>>>> calling script with sed. This seems to work, but I think this
>>>> includes
>>>> stats for all valid times between 2018-08-01 00:00:00 and 2018-
08-31
>>>> 23:59:59. In interactive METviewer setup, one can specify
>>>> fcst_valid_beg with a start time, end time and an increment (6h,
12h
>>>> etc.). How does one specify such an increment in the batch
script
>>>> (e.g.
>>>> if I only want valid times at 00Z/12Z, rather than 00/06/12/18Z)?
>>>>
>>>> 2) In the past I've been using a lot of sed substitutes and make
a
>>>> single plot per mv_batch.sh call, like this:
>>>>
>>>> for hscale in 024 052 100
>>>> do
>>>>
>>>> for thresh in 002.0 005.0 010.0 020.0
>>>> do
>>>>
>>>> cat
>>>> $ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml \
>>>> | sed "s/%DAY1DASH%/$day1dash/g" \
>>>> | sed "s/%DAY2DASH%/$day2dash/g" \
>>>> | sed 's/%THRESH%/'$thresh'/g' \
>>>> | sed 's/%HSCALE%/'$hscale'/g' \
>>>> > $tmpscripts/plt_4_prodmods_${day1}-
>>>> ${day2}_${thresh}_${hscale}.xml
>>>>
>>>> /usr1/metviewer/metviewer/bin/mv_batch.sh
>>>> $tmpscripts/plt_4_prodmods_${day1}-
${day2}_${thresh}_${hscale}.xml
>>>>
>>>> enddo
>>>>
>>>> enddo
>>>>
>>>> Follow John's examples at the Oct tutorial, I was able to replace
>>>>
>>>> <field equalize="true" name="fcst_thresh">
>>>> <set name="fcst_thresh_1">
>>>> <val><%HSCALE%</val>
>>>> </set>
>>>> </field>
>>>>
>>>> with
>>>>
>>>> <field equalize="true" name="fcst_thresh">
>>>> <val><024</val>
>>>> <val><052</val>
>>>> <val><100</val>
>>>> </field>
>>>>
>>>> and that seems to work.
>>>>
>>>> But is there a way to replace this?
>>>>
>>>> <dep1>
>>>> <fcst_var name="APCP/06>%THRESH%">
>>>> <stat>NBR_FSS</stat>
>>>> </fcst_var>
>>>> </dep1>
>>>>
>>>> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0? If
this
>>>> is
>>>> too hard/convoluted to set up then I can just use the old "sed"
>>>> substitution.
>>>>
>>>> I'm trying to plot 6h FSS from our VSDBs, so the set up is a bit
>>>> peculiar: METviewer consider the forecast variable as e.g,
>>>> APCP/06>002.0
>>>> (mm), and the horizontal spacial scale is the fcst_thresh (e.g.,
<
>>>> 052km)
>>>>
>>>> Also, the resulting plots from the successful use of
{fcst_thresh}
>>>> in
>>>> batch above have the name such as
>>>> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from
fcst_thresh
>>>> of
>>>> "<024". the "<" and ">" in file name causes some problems ...
>>>> Is
>>>> there an easy way to create another variable such as
>>>> Afcst_thresh=024
>>>> (fcst_thresh with the "<" removed)?
>>>>
>>>> Thanks very much for looking into this -
>>>>
>>>> Ying
>>>>
>>>
>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: METviewer: batch plot questions
From: Tatiana Burek
Time: Fri Dec 07 11:00:24 2018
Ying,
Please take a look at attache file of batch XML with inheritance. I
used your original XML as an example.
- I created two date ranges and used them in plot_fix group.
- I created an abstract job named test_thresh that describes all
common fields
in tmpl group I created two maps to "translate" fcst_thresh and
fcst_var values to filename friendly names and use them templates
- I created two executable jobs for APCP/06>002.0 and APCP/06>005.0
Tatiana
On Fri Dec 07 09:58:49 2018, tatiana wrote:
> Ying,
>
> 1.Predefined <date_range> from the example will work as advertised -
> all dates in between including start and end will be included.
There
> is no increment specification in the setup.
> You can specify valid_hour for example to include dates with the
> specific hour.
>
> 2.What is your goal? To have individual plots for each fcst_var or
> plot all fcst_var in one plot?
>
> 3.instead of using
> fss06_%DAY1DASH%-%DAY2DASH%_{fcst_thresh}km_%THRESH%mm.png
> you could use fss06_%DAY1DASH%-
%DAY2DASH%_%THRESH_1%km_%THRESH%mm.png
> and change %THRESH_1% with sed to the value you need
>
> Tatiana
>
> On Thu Dec 06 11:05:23 2018, ying.lin at noaa.gov wrote:
> > Hi Tatiana,
> >
> > Thank you for your reply. I didn't realize that METviewer
batch
> > XML can't have the start/end tags - good to know. In the
attached
> > XML
> > (example from John H.G. at the Oct tutorial) there is
> > <date_range name="fall" ><start>2011-10-01
> > 00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
> > ...
> > <plot_fix>
> > <field name="fcst_init_beg">
> > <date_range name="fall"/>
> >
> > I suppose this means that the date range must be pre-defined (in
the
> > above "date_range" tag). I've used this pre-defined set-up before
> > (in
> > e.g. making seasonal plots). For a more flexible set up (e.g.
> > plotting
> > from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by
setting
> > up
> > something like
> > <date_range
> >
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
> > then just use 'sed' to do the substitution.
> >
> > Questions still remain:
> > 1) In the above pre-defined date range set up (either "fall" or
> > "anydaterange"), how does one specify the increment (e.g. 12h,
when
> > the
> > data are 6-hourly)?
> > 2) same as the item 2) in the original email inquiry
> > 3) By "creating another variable", I meant that when fcst_thresh
is
> > given the value of "<52km", and {fcst_thresh} is used in the
> > resulting
> > png name, the '<' is causing trouble in unix. Is there a way to
get
> > rid
> > of the '<', e.g. creating another variable that is 'fcst_thresh'
> > minus
> > the '<'?
> >
> > Thanks again,
> >
> > Ying
> >
> >
> > On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
> > > Hi Ying,
> > >
> > > #1
> > > METviewer does not support start/end tags in the batch XML. For
> > > your
> > > example:
> > > <field equalize="true" name="fcst_valid_beg">
> > > <start>%DAY1DASH% 00:00:00</start>
> > > <end>%DAY2DASH% 23:59:59</end>
> > > </field>
> > > only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will be
> > > included in plotting but nothing in between these days.
> > >
> > > #2
> > > Not sure what you mean by "creating another variable". Have you
> > > ever
> > > considered using inheritance in the batch XML? It might be a
better
> > > solution than using sed.
> > >
> > > Tatiana
> > >
> > > On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
> > >> Hello. I have several questions about making batch plots on
> > >> METviewer.
> > >> Attached 4 files are
> > >>
> > >> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch script
> > >>
> > >> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script to
> > >> set
> > >> up
> > >> and run batch job on the above
> > >>
> > >> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of the
> > >> plots
> > >> produced
> > >>
> > >> test.xml: an interactive METviewer plot xml for this sort
of
> > >> plots
> > >>
> > >> 1) I'm specifying model valid times used in the plot with
> > >>
> > >> <field equalize="true" name="fcst_valid_beg">
> > >> <start>%DAY1DASH% 00:00:00</start>
> > >> <end>%DAY2DASH% 23:59:59</end>
> > >> </field>
> > >>
> > >> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31) substituted
in
> > >> the
> > >> calling script with sed. This seems to work, but I think this
> > >> includes
> > >> stats for all valid times between 2018-08-01 00:00:00 and 2018-
08-
> > >> 31
> > >> 23:59:59. In interactive METviewer setup, one can specify
> > >> fcst_valid_beg with a start time, end time and an increment
(6h,
> > >> 12h
> > >> etc.). How does one specify such an increment in the batch
script
> > >> (e.g.
> > >> if I only want valid times at 00Z/12Z, rather than
00/06/12/18Z)?
> > >>
> > >> 2) In the past I've been using a lot of sed substitutes and
make a
> > >> single plot per mv_batch.sh call, like this:
> > >>
> > >> for hscale in 024 052 100
> > >> do
> > >>
> > >> for thresh in 002.0 005.0 010.0 020.0
> > >> do
> > >>
> > >> cat
> > >> $ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml
\
> > >> | sed "s/%DAY1DASH%/$day1dash/g" \
> > >> | sed "s/%DAY2DASH%/$day2dash/g" \
> > >> | sed 's/%THRESH%/'$thresh'/g' \
> > >> | sed 's/%HSCALE%/'$hscale'/g' \
> > >> > $tmpscripts/plt_4_prodmods_${day1}-
> > >> ${day2}_${thresh}_${hscale}.xml
> > >>
> > >> /usr1/metviewer/metviewer/bin/mv_batch.sh
> > >> $tmpscripts/plt_4_prodmods_${day1}-
${day2}_${thresh}_${hscale}.xml
> > >>
> > >> enddo
> > >>
> > >> enddo
> > >>
> > >> Follow John's examples at the Oct tutorial, I was able to
replace
> > >>
> > >> <field equalize="true" name="fcst_thresh">
> > >> <set name="fcst_thresh_1">
> > >> <val><%HSCALE%</val>
> > >> </set>
> > >> </field>
> > >>
> > >> with
> > >>
> > >> <field equalize="true" name="fcst_thresh">
> > >> <val><024</val>
> > >> <val><052</val>
> > >> <val><100</val>
> > >> </field>
> > >>
> > >> and that seems to work.
> > >>
> > >> But is there a way to replace this?
> > >>
> > >> <dep1>
> > >> <fcst_var name="APCP/06>%THRESH%">
> > >> <stat>NBR_FSS</stat>
> > >> </fcst_var>
> > >> </dep1>
> > >>
> > >> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0? If
> > >> this
> > >> is
> > >> too hard/convoluted to set up then I can just use the old "sed"
> > >> substitution.
> > >>
> > >> I'm trying to plot 6h FSS from our VSDBs, so the set up is a
bit
> > >> peculiar: METviewer consider the forecast variable as e.g,
> > >> APCP/06>002.0
> > >> (mm), and the horizontal spacial scale is the fcst_thresh
(e.g., <
> > >> 052km)
> > >>
> > >> Also, the resulting plots from the successful use of
{fcst_thresh}
> > >> in
> > >> batch above have the name such as
> > >> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from
fcst_thresh
> > >> of
> > >> "<024". the "<" and ">" in file name causes some problems
> > >> ...
> > >> Is
> > >> there an easy way to create another variable such as
> > >> Afcst_thresh=024
> > >> (fcst_thresh with the "<" removed)?
> > >>
> > >> Thanks very much for looking into this -
> > >>
> > >> Ying
> > >>
> > >
> > >
------------------------------------------------
Subject: METviewer: batch plot questions
From: Ying Lin
Time: Fri Dec 07 14:11:41 2018
Thanks very much Tatiana! I think you know exactly what I was aiming
for. Thanks for taking the time to set up an example. I haven't used
'inherit' before, took a while to try to figure out the structure of
the
set-up.
I tried running your set up in /metviewer/staging/wd22yl/test.batch:
/usr1/metviewer/metviewer/bin/mv_batch.sh
fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana.xml > run.out
and now have more questions. In the above *_tatiana.xml, the only
change I made was the date (yyyy changed from 2011 to 2018).
1) there are some warning msgs in run.out:
** WARNING: template tag map_thresh not found in agg perm
** WARNING: template tag dep not found in agg perm
Does that mean that 'map_thresh' and 'dep' are not successfully
paired to 'fcst_thresh' and the fcst_var? Would you mind telling me
how
to fix this? I don't know enough - haven't been able to fix this
myself.
2) Running the xml resulted in two plots (see attached), but according
to run.out, there are total of 12 plots created, I think it was six
each
for fss06_date1 and fss06_date2, and the six are probably for
APCP/06>002.0 and APCP/06>005.0 and fcst_thresh values of 024/052/100
(2x3=6), and the png file names got "recycled" because of the "agg
perm"
error above.
BTW this is what I'm trying to do: the "6-hourly fractions skill
scores"
block
in**http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/201810/.
I _think_ that's what your sample script set up to do as well.
I really appreciate the time you already spent on this request. Looks
like with the sample script you constructed we're getting really
close.
If you don't mind looking into this further with the above follow-up
question - next week would be fine! - that'll be great. Thanks
again!
Ying
On 12/7/18 1:00 PM, Tatiana Burek via RT wrote:
> Ying,
> Please take a look at attache file of batch XML with inheritance. I
used your original XML as an example.
> - I created two date ranges and used them in plot_fix group.
> - I created an abstract job named test_thresh that describes all
common fields
> in tmpl group I created two maps to "translate" fcst_thresh and
fcst_var values to filename friendly names and use them templates
> - I created two executable jobs for APCP/06>002.0 and APCP/06>005.0
>
> Tatiana
>
> On Fri Dec 07 09:58:49 2018, tatiana wrote:
>> Ying,
>>
>> 1.Predefined <date_range> from the example will work as advertised
-
>> all dates in between including start and end will be included.
There
>> is no increment specification in the setup.
>> You can specify valid_hour for example to include dates with the
>> specific hour.
>>
>> 2.What is your goal? To have individual plots for each fcst_var or
>> plot all fcst_var in one plot?
>>
>> 3.instead of using
>> fss06_%DAY1DASH%-%DAY2DASH%_{fcst_thresh}km_%THRESH%mm.png
>> you could use fss06_%DAY1DASH%-
%DAY2DASH%_%THRESH_1%km_%THRESH%mm.png
>> and change %THRESH_1% with sed to the value you need
>>
>> Tatiana
>>
>> On Thu Dec 06 11:05:23 2018, ying.lin at noaa.gov wrote:
>>> Hi Tatiana,
>>>
>>> Thank you for your reply. I didn't realize that METviewer
batch
>>> XML can't have the start/end tags - good to know. In the
attached
>>> XML
>>> (example from John H.G. at the Oct tutorial) there is
>>> <date_range name="fall" ><start>2011-10-01
>>> 00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
>>> ...
>>> <plot_fix>
>>> <field name="fcst_init_beg">
>>> <date_range name="fall"/>
>>>
>>> I suppose this means that the date range must be pre-defined (in
the
>>> above "date_range" tag). I've used this pre-defined set-up before
>>> (in
>>> e.g. making seasonal plots). For a more flexible set up (e.g.
>>> plotting
>>> from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by
setting
>>> up
>>> something like
>>> <date_range
>>>
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
>>> then just use 'sed' to do the substitution.
>>>
>>> Questions still remain:
>>> 1) In the above pre-defined date range set up (either "fall" or
>>> "anydaterange"), how does one specify the increment (e.g. 12h,
when
>>> the
>>> data are 6-hourly)?
>>> 2) same as the item 2) in the original email inquiry
>>> 3) By "creating another variable", I meant that when fcst_thresh
is
>>> given the value of "<52km", and {fcst_thresh} is used in the
>>> resulting
>>> png name, the '<' is causing trouble in unix. Is there a way to
get
>>> rid
>>> of the '<', e.g. creating another variable that is 'fcst_thresh'
>>> minus
>>> the '<'?
>>>
>>> Thanks again,
>>>
>>> Ying
>>>
>>>
>>> On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
>>>> Hi Ying,
>>>>
>>>> #1
>>>> METviewer does not support start/end tags in the batch XML. For
>>>> your
>>>> example:
>>>> <field equalize="true" name="fcst_valid_beg">
>>>> <start>%DAY1DASH% 00:00:00</start>
>>>> <end>%DAY2DASH% 23:59:59</end>
>>>> </field>
>>>> only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will be
>>>> included in plotting but nothing in between these days.
>>>>
>>>> #2
>>>> Not sure what you mean by "creating another variable". Have you
>>>> ever
>>>> considered using inheritance in the batch XML? It might be a
better
>>>> solution than using sed.
>>>>
>>>> Tatiana
>>>>
>>>> On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
>>>>> Hello. I have several questions about making batch plots on
>>>>> METviewer.
>>>>> Attached 4 files are
>>>>>
>>>>> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch script
>>>>>
>>>>> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script to
>>>>> set
>>>>> up
>>>>> and run batch job on the above
>>>>>
>>>>> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of the
>>>>> plots
>>>>> produced
>>>>>
>>>>> test.xml: an interactive METviewer plot xml for this sort
of
>>>>> plots
>>>>>
>>>>> 1) I'm specifying model valid times used in the plot with
>>>>>
>>>>> <field equalize="true" name="fcst_valid_beg">
>>>>> <start>%DAY1DASH% 00:00:00</start>
>>>>> <end>%DAY2DASH% 23:59:59</end>
>>>>> </field>
>>>>>
>>>>> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31) substituted
in
>>>>> the
>>>>> calling script with sed. This seems to work, but I think this
>>>>> includes
>>>>> stats for all valid times between 2018-08-01 00:00:00 and 2018-
08-
>>>>> 31
>>>>> 23:59:59. In interactive METviewer setup, one can specify
>>>>> fcst_valid_beg with a start time, end time and an increment (6h,
>>>>> 12h
>>>>> etc.). How does one specify such an increment in the batch
script
>>>>> (e.g.
>>>>> if I only want valid times at 00Z/12Z, rather than
00/06/12/18Z)?
>>>>>
>>>>> 2) In the past I've been using a lot of sed substitutes and make
a
>>>>> single plot per mv_batch.sh call, like this:
>>>>>
>>>>> for hscale in 024 052 100
>>>>> do
>>>>>
>>>>> for thresh in 002.0 005.0 010.0 020.0
>>>>> do
>>>>>
>>>>> cat
>>>>> $ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml
\
>>>>> | sed "s/%DAY1DASH%/$day1dash/g" \
>>>>> | sed "s/%DAY2DASH%/$day2dash/g" \
>>>>> | sed 's/%THRESH%/'$thresh'/g' \
>>>>> | sed 's/%HSCALE%/'$hscale'/g' \
>>>>> > $tmpscripts/plt_4_prodmods_${day1}-
>>>>> ${day2}_${thresh}_${hscale}.xml
>>>>>
>>>>> /usr1/metviewer/metviewer/bin/mv_batch.sh
>>>>> $tmpscripts/plt_4_prodmods_${day1}-
${day2}_${thresh}_${hscale}.xml
>>>>>
>>>>> enddo
>>>>>
>>>>> enddo
>>>>>
>>>>> Follow John's examples at the Oct tutorial, I was able to
replace
>>>>>
>>>>> <field equalize="true" name="fcst_thresh">
>>>>> <set name="fcst_thresh_1">
>>>>> <val><%HSCALE%</val>
>>>>> </set>
>>>>> </field>
>>>>>
>>>>> with
>>>>>
>>>>> <field equalize="true" name="fcst_thresh">
>>>>> <val><024</val>
>>>>> <val><052</val>
>>>>> <val><100</val>
>>>>> </field>
>>>>>
>>>>> and that seems to work.
>>>>>
>>>>> But is there a way to replace this?
>>>>>
>>>>> <dep1>
>>>>> <fcst_var name="APCP/06>%THRESH%">
>>>>> <stat>NBR_FSS</stat>
>>>>> </fcst_var>
>>>>> </dep1>
>>>>>
>>>>> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0? If
>>>>> this
>>>>> is
>>>>> too hard/convoluted to set up then I can just use the old "sed"
>>>>> substitution.
>>>>>
>>>>> I'm trying to plot 6h FSS from our VSDBs, so the set up is a bit
>>>>> peculiar: METviewer consider the forecast variable as e.g,
>>>>> APCP/06>002.0
>>>>> (mm), and the horizontal spacial scale is the fcst_thresh (e.g.,
<
>>>>> 052km)
>>>>>
>>>>> Also, the resulting plots from the successful use of
{fcst_thresh}
>>>>> in
>>>>> batch above have the name such as
>>>>> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from
fcst_thresh
>>>>> of
>>>>> "<024". the "<" and ">" in file name causes some problems
>>>>> ...
>>>>> Is
>>>>> there an easy way to create another variable such as
>>>>> Afcst_thresh=024
>>>>> (fcst_thresh with the "<" removed)?
>>>>>
>>>>> Thanks very much for looking into this -
>>>>>
>>>>> Ying
>>>>>
>>>>
>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: METviewer: batch plot questions
From: Tatiana Burek
Time: Wed Dec 12 09:52:26 2018
Hi Ying,
I created a new XML fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana_1.xml
that returns 12 plots (2 date ranges * 3 fcst_thresh * 2 fcst_var ).
I also replaced fcst_thresh with custom values.
You can change the templates for file names and the plot title to more
human readable if needed.
I hope this is helpful.
Tatiana
On Fri Dec 07 14:11:41 2018, ying.lin at noaa.gov wrote:
> Thanks very much Tatiana! I think you know exactly what I was
aiming
> for. Thanks for taking the time to set up an example. I haven't
used
> 'inherit' before, took a while to try to figure out the structure of
> the
> set-up.
>
> I tried running your set up in /metviewer/staging/wd22yl/test.batch:
>
> /usr1/metviewer/metviewer/bin/mv_batch.sh
> fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana.xml > run.out
> and now have more questions. In the above *_tatiana.xml, the only
> change I made was the date (yyyy changed from 2011 to 2018).
>
> 1) there are some warning msgs in run.out:
> ** WARNING: template tag map_thresh not found in agg perm
> ** WARNING: template tag dep not found in agg perm
> Does that mean that 'map_thresh' and 'dep' are not
successfully
> paired to 'fcst_thresh' and the fcst_var? Would you mind telling
me
> how
> to fix this? I don't know enough - haven't been able to fix this
> myself.
>
> 2) Running the xml resulted in two plots (see attached), but
according
> to run.out, there are total of 12 plots created, I think it was six
> each
> for fss06_date1 and fss06_date2, and the six are probably for
> APCP/06>002.0 and APCP/06>005.0 and fcst_thresh values of
024/052/100
> (2x3=6), and the png file names got "recycled" because of the "agg
> perm"
> error above.
>
> BTW this is what I'm trying to do: the "6-hourly fractions skill
> scores"
> block
>
in**http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/201810/.
> I _think_ that's what your sample script set up to do as well.
>
> I really appreciate the time you already spent on this request.
Looks
> like with the sample script you constructed we're getting really
> close.
> If you don't mind looking into this further with the above follow-up
> question - next week would be fine! - that'll be great. Thanks
> again!
>
> Ying
>
> On 12/7/18 1:00 PM, Tatiana Burek via RT wrote:
> > Ying,
> > Please take a look at attache file of batch XML with inheritance.
I
> > used your original XML as an example.
> > - I created two date ranges and used them in plot_fix group.
> > - I created an abstract job named test_thresh that describes all
> > common fields
> > in tmpl group I created two maps to "translate" fcst_thresh and
> > fcst_var values to filename friendly names and use them templates
> > - I created two executable jobs for APCP/06>002.0 and
APCP/06>005.0
> >
> > Tatiana
> >
> > On Fri Dec 07 09:58:49 2018, tatiana wrote:
> >> Ying,
> >>
> >> 1.Predefined <date_range> from the example will work as
advertised -
> >> all dates in between including start and end will be included.
> >> There
> >> is no increment specification in the setup.
> >> You can specify valid_hour for example to include dates with the
> >> specific hour.
> >>
> >> 2.What is your goal? To have individual plots for each fcst_var
or
> >> plot all fcst_var in one plot?
> >>
> >> 3.instead of using
> >> fss06_%DAY1DASH%-%DAY2DASH%_{fcst_thresh}km_%THRESH%mm.png
> >> you could use fss06_%DAY1DASH%-
> >> %DAY2DASH%_%THRESH_1%km_%THRESH%mm.png
> >> and change %THRESH_1% with sed to the value you need
> >>
> >> Tatiana
> >>
> >> On Thu Dec 06 11:05:23 2018, ying.lin at noaa.gov wrote:
> >>> Hi Tatiana,
> >>>
> >>> Thank you for your reply. I didn't realize that METviewer
> >>> batch
> >>> XML can't have the start/end tags - good to know. In the
> >>> attached
> >>> XML
> >>> (example from John H.G. at the Oct tutorial) there is
> >>> <date_range name="fall" ><start>2011-10-01
> >>> 00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
> >>> ...
> >>> <plot_fix>
> >>> <field name="fcst_init_beg">
> >>> <date_range name="fall"/>
> >>>
> >>> I suppose this means that the date range must be pre-defined (in
> >>> the
> >>> above "date_range" tag). I've used this pre-defined set-up
before
> >>> (in
> >>> e.g. making seasonal plots). For a more flexible set up (e.g.
> >>> plotting
> >>> from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by
> >>> setting
> >>> up
> >>> something like
> >>> <date_range
> >>>
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
> >>> then just use 'sed' to do the substitution.
> >>>
> >>> Questions still remain:
> >>> 1) In the above pre-defined date range set up (either "fall"
or
> >>> "anydaterange"), how does one specify the increment (e.g. 12h,
> >>> when
> >>> the
> >>> data are 6-hourly)?
> >>> 2) same as the item 2) in the original email inquiry
> >>> 3) By "creating another variable", I meant that when
fcst_thresh
> >>> is
> >>> given the value of "<52km", and {fcst_thresh} is used in the
> >>> resulting
> >>> png name, the '<' is causing trouble in unix. Is there a way
to
> >>> get
> >>> rid
> >>> of the '<', e.g. creating another variable that is
'fcst_thresh'
> >>> minus
> >>> the '<'?
> >>>
> >>> Thanks again,
> >>>
> >>> Ying
> >>>
> >>>
> >>> On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
> >>>> Hi Ying,
> >>>>
> >>>> #1
> >>>> METviewer does not support start/end tags in the batch XML. For
> >>>> your
> >>>> example:
> >>>> <field equalize="true" name="fcst_valid_beg">
> >>>> <start>%DAY1DASH% 00:00:00</start>
> >>>> <end>%DAY2DASH% 23:59:59</end>
> >>>> </field>
> >>>> only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will be
> >>>> included in plotting but nothing in between these days.
> >>>>
> >>>> #2
> >>>> Not sure what you mean by "creating another variable". Have you
> >>>> ever
> >>>> considered using inheritance in the batch XML? It might be a
> >>>> better
> >>>> solution than using sed.
> >>>>
> >>>> Tatiana
> >>>>
> >>>> On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
> >>>>> Hello. I have several questions about making batch plots on
> >>>>> METviewer.
> >>>>> Attached 4 files are
> >>>>>
> >>>>> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch script
> >>>>>
> >>>>> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script
to
> >>>>> set
> >>>>> up
> >>>>> and run batch job on the above
> >>>>>
> >>>>> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of
the
> >>>>> plots
> >>>>> produced
> >>>>>
> >>>>> test.xml: an interactive METviewer plot xml for this sort
of
> >>>>> plots
> >>>>>
> >>>>> 1) I'm specifying model valid times used in the plot with
> >>>>>
> >>>>> <field equalize="true" name="fcst_valid_beg">
> >>>>> <start>%DAY1DASH% 00:00:00</start>
> >>>>> <end>%DAY2DASH% 23:59:59</end>
> >>>>> </field>
> >>>>>
> >>>>> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31)
substituted
> >>>>> in
> >>>>> the
> >>>>> calling script with sed. This seems to work, but I think this
> >>>>> includes
> >>>>> stats for all valid times between 2018-08-01 00:00:00 and
2018-
> >>>>> 08-
> >>>>> 31
> >>>>> 23:59:59. In interactive METviewer setup, one can specify
> >>>>> fcst_valid_beg with a start time, end time and an increment
(6h,
> >>>>> 12h
> >>>>> etc.). How does one specify such an increment in the batch
> >>>>> script
> >>>>> (e.g.
> >>>>> if I only want valid times at 00Z/12Z, rather than
00/06/12/18Z)?
> >>>>>
> >>>>> 2) In the past I've been using a lot of sed substitutes and
make
> >>>>> a
> >>>>> single plot per mv_batch.sh call, like this:
> >>>>>
> >>>>> for hscale in 024 052 100
> >>>>> do
> >>>>>
> >>>>> for thresh in 002.0 005.0 010.0 020.0
> >>>>> do
> >>>>>
> >>>>> cat
> >>>>>
$ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml \
> >>>>> | sed "s/%DAY1DASH%/$day1dash/g" \
> >>>>> | sed "s/%DAY2DASH%/$day2dash/g" \
> >>>>> | sed 's/%THRESH%/'$thresh'/g' \
> >>>>> | sed 's/%HSCALE%/'$hscale'/g' \
> >>>>> > $tmpscripts/plt_4_prodmods_${day1}-
> >>>>> ${day2}_${thresh}_${hscale}.xml
> >>>>>
> >>>>> /usr1/metviewer/metviewer/bin/mv_batch.sh
> >>>>> $tmpscripts/plt_4_prodmods_${day1}-
> >>>>> ${day2}_${thresh}_${hscale}.xml
> >>>>>
> >>>>> enddo
> >>>>>
> >>>>> enddo
> >>>>>
> >>>>> Follow John's examples at the Oct tutorial, I was able to
replace
> >>>>>
> >>>>> <field equalize="true" name="fcst_thresh">
> >>>>> <set name="fcst_thresh_1">
> >>>>> <val><%HSCALE%</val>
> >>>>> </set>
> >>>>> </field>
> >>>>>
> >>>>> with
> >>>>>
> >>>>> <field equalize="true" name="fcst_thresh">
> >>>>> <val><024</val>
> >>>>> <val><052</val>
> >>>>> <val><100</val>
> >>>>> </field>
> >>>>>
> >>>>> and that seems to work.
> >>>>>
> >>>>> But is there a way to replace this?
> >>>>>
> >>>>> <dep1>
> >>>>> <fcst_var name="APCP/06>%THRESH%">
> >>>>> <stat>NBR_FSS</stat>
> >>>>> </fcst_var>
> >>>>> </dep1>
> >>>>>
> >>>>> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0?
If
> >>>>> this
> >>>>> is
> >>>>> too hard/convoluted to set up then I can just use the old
"sed"
> >>>>> substitution.
> >>>>>
> >>>>> I'm trying to plot 6h FSS from our VSDBs, so the set up is a
bit
> >>>>> peculiar: METviewer consider the forecast variable as e.g,
> >>>>> APCP/06>002.0
> >>>>> (mm), and the horizontal spacial scale is the fcst_thresh
(e.g.,
> >>>>> <
> >>>>> 052km)
> >>>>>
> >>>>> Also, the resulting plots from the successful use of
> >>>>> {fcst_thresh}
> >>>>> in
> >>>>> batch above have the name such as
> >>>>> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from
> >>>>> fcst_thresh
> >>>>> of
> >>>>> "<024". the "<" and ">" in file name causes some
problems
> >>>>> ...
> >>>>> Is
> >>>>> there an easy way to create another variable such as
> >>>>> Afcst_thresh=024
> >>>>> (fcst_thresh with the "<" removed)?
> >>>>>
> >>>>> Thanks very much for looking into this -
> >>>>>
> >>>>> Ying
> >>>>>
> >>>>
> >
> >
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87973] METviewer: batch plot questions
From: Ying Lin
Time: Wed Dec 12 10:16:29 2018
Thanks Tatiana. I don't see attachment (the *_tatiana_1.xml) in your
email - would you mind sending it again? Thanks!
Ying
On 12/12/18 11:52 AM, Tatiana Burek via RT wrote:
> Hi Ying,
> I created a new XML fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana_1.xml
that returns 12 plots (2 date ranges * 3 fcst_thresh * 2 fcst_var ).
> I also replaced fcst_thresh with custom values.
> You can change the templates for file names and the plot title to
more human readable if needed.
> I hope this is helpful.
>
> Tatiana
>
> On Fri Dec 07 14:11:41 2018, ying.lin at noaa.gov wrote:
>> Thanks very much Tatiana! I think you know exactly what I was
aiming
>> for. Thanks for taking the time to set up an example. I haven't
used
>> 'inherit' before, took a while to try to figure out the structure
of
>> the
>> set-up.
>>
>> I tried running your set up in
/metviewer/staging/wd22yl/test.batch:
>>
>> /usr1/metviewer/metviewer/bin/mv_batch.sh
>> fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana.xml > run.out
>> and now have more questions. In the above *_tatiana.xml, the only
>> change I made was the date (yyyy changed from 2011 to 2018).
>>
>> 1) there are some warning msgs in run.out:
>> ** WARNING: template tag map_thresh not found in agg perm
>> ** WARNING: template tag dep not found in agg perm
>> Does that mean that 'map_thresh' and 'dep' are not
successfully
>> paired to 'fcst_thresh' and the fcst_var? Would you mind telling
me
>> how
>> to fix this? I don't know enough - haven't been able to fix this
>> myself.
>>
>> 2) Running the xml resulted in two plots (see attached), but
according
>> to run.out, there are total of 12 plots created, I think it was six
>> each
>> for fss06_date1 and fss06_date2, and the six are probably for
>> APCP/06>002.0 and APCP/06>005.0 and fcst_thresh values of
024/052/100
>> (2x3=6), and the png file names got "recycled" because of the "agg
>> perm"
>> error above.
>>
>> BTW this is what I'm trying to do: the "6-hourly fractions skill
>> scores"
>> block
>>
in**http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/201810/.
>> I _think_ that's what your sample script set up to do as well.
>>
>> I really appreciate the time you already spent on this request.
Looks
>> like with the sample script you constructed we're getting really
>> close.
>> If you don't mind looking into this further with the above follow-
up
>> question - next week would be fine! - that'll be great. Thanks
>> again!
>>
>> Ying
>>
>> On 12/7/18 1:00 PM, Tatiana Burek via RT wrote:
>>> Ying,
>>> Please take a look at attache file of batch XML with inheritance.
I
>>> used your original XML as an example.
>>> - I created two date ranges and used them in plot_fix group.
>>> - I created an abstract job named test_thresh that describes all
>>> common fields
>>> in tmpl group I created two maps to "translate" fcst_thresh and
>>> fcst_var values to filename friendly names and use them templates
>>> - I created two executable jobs for APCP/06>002.0 and
APCP/06>005.0
>>>
>>> Tatiana
>>>
>>> On Fri Dec 07 09:58:49 2018, tatiana wrote:
>>>> Ying,
>>>>
>>>> 1.Predefined <date_range> from the example will work as
advertised -
>>>> all dates in between including start and end will be included.
>>>> There
>>>> is no increment specification in the setup.
>>>> You can specify valid_hour for example to include dates with the
>>>> specific hour.
>>>>
>>>> 2.What is your goal? To have individual plots for each fcst_var
or
>>>> plot all fcst_var in one plot?
>>>>
>>>> 3.instead of using
>>>> fss06_%DAY1DASH%-%DAY2DASH%_{fcst_thresh}km_%THRESH%mm.png
>>>> you could use fss06_%DAY1DASH%-
>>>> %DAY2DASH%_%THRESH_1%km_%THRESH%mm.png
>>>> and change %THRESH_1% with sed to the value you need
>>>>
>>>> Tatiana
>>>>
>>>> On Thu Dec 06 11:05:23 2018, ying.lin at noaa.gov wrote:
>>>>> Hi Tatiana,
>>>>>
>>>>> Thank you for your reply. I didn't realize that METviewer
>>>>> batch
>>>>> XML can't have the start/end tags - good to know. In the
>>>>> attached
>>>>> XML
>>>>> (example from John H.G. at the Oct tutorial) there is
>>>>> <date_range name="fall" ><start>2011-10-01
>>>>> 00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
>>>>> ...
>>>>> <plot_fix>
>>>>> <field name="fcst_init_beg">
>>>>> <date_range name="fall"/>
>>>>>
>>>>> I suppose this means that the date range must be pre-defined (in
>>>>> the
>>>>> above "date_range" tag). I've used this pre-defined set-up
before
>>>>> (in
>>>>> e.g. making seasonal plots). For a more flexible set up (e.g.
>>>>> plotting
>>>>> from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by
>>>>> setting
>>>>> up
>>>>> something like
>>>>> <date_range
>>>>>
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
>>>>> then just use 'sed' to do the substitution.
>>>>>
>>>>> Questions still remain:
>>>>> 1) In the above pre-defined date range set up (either "fall"
or
>>>>> "anydaterange"), how does one specify the increment (e.g.
12h,
>>>>> when
>>>>> the
>>>>> data are 6-hourly)?
>>>>> 2) same as the item 2) in the original email inquiry
>>>>> 3) By "creating another variable", I meant that when
fcst_thresh
>>>>> is
>>>>> given the value of "<52km", and {fcst_thresh} is used in the
>>>>> resulting
>>>>> png name, the '<' is causing trouble in unix. Is there a way
to
>>>>> get
>>>>> rid
>>>>> of the '<', e.g. creating another variable that is
'fcst_thresh'
>>>>> minus
>>>>> the '<'?
>>>>>
>>>>> Thanks again,
>>>>>
>>>>> Ying
>>>>>
>>>>>
>>>>> On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
>>>>>> Hi Ying,
>>>>>>
>>>>>> #1
>>>>>> METviewer does not support start/end tags in the batch XML. For
>>>>>> your
>>>>>> example:
>>>>>> <field equalize="true" name="fcst_valid_beg">
>>>>>> <start>%DAY1DASH% 00:00:00</start>
>>>>>> <end>%DAY2DASH% 23:59:59</end>
>>>>>> </field>
>>>>>> only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will be
>>>>>> included in plotting but nothing in between these days.
>>>>>>
>>>>>> #2
>>>>>> Not sure what you mean by "creating another variable". Have you
>>>>>> ever
>>>>>> considered using inheritance in the batch XML? It might be a
>>>>>> better
>>>>>> solution than using sed.
>>>>>>
>>>>>> Tatiana
>>>>>>
>>>>>> On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
>>>>>>> Hello. I have several questions about making batch plots on
>>>>>>> METviewer.
>>>>>>> Attached 4 files are
>>>>>>>
>>>>>>> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch
script
>>>>>>>
>>>>>>> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script
to
>>>>>>> set
>>>>>>> up
>>>>>>> and run batch job on the above
>>>>>>>
>>>>>>> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of
the
>>>>>>> plots
>>>>>>> produced
>>>>>>>
>>>>>>> test.xml: an interactive METviewer plot xml for this sort
of
>>>>>>> plots
>>>>>>>
>>>>>>> 1) I'm specifying model valid times used in the plot with
>>>>>>>
>>>>>>> <field equalize="true" name="fcst_valid_beg">
>>>>>>> <start>%DAY1DASH% 00:00:00</start>
>>>>>>> <end>%DAY2DASH% 23:59:59</end>
>>>>>>> </field>
>>>>>>>
>>>>>>> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31)
substituted
>>>>>>> in
>>>>>>> the
>>>>>>> calling script with sed. This seems to work, but I think this
>>>>>>> includes
>>>>>>> stats for all valid times between 2018-08-01 00:00:00 and
2018-
>>>>>>> 08-
>>>>>>> 31
>>>>>>> 23:59:59. In interactive METviewer setup, one can specify
>>>>>>> fcst_valid_beg with a start time, end time and an increment
(6h,
>>>>>>> 12h
>>>>>>> etc.). How does one specify such an increment in the batch
>>>>>>> script
>>>>>>> (e.g.
>>>>>>> if I only want valid times at 00Z/12Z, rather than
00/06/12/18Z)?
>>>>>>>
>>>>>>> 2) In the past I've been using a lot of sed substitutes and
make
>>>>>>> a
>>>>>>> single plot per mv_batch.sh call, like this:
>>>>>>>
>>>>>>> for hscale in 024 052 100
>>>>>>> do
>>>>>>>
>>>>>>> for thresh in 002.0 005.0 010.0 020.0
>>>>>>> do
>>>>>>>
>>>>>>> cat
>>>>>>>
$ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml \
>>>>>>> | sed "s/%DAY1DASH%/$day1dash/g" \
>>>>>>> | sed "s/%DAY2DASH%/$day2dash/g" \
>>>>>>> | sed 's/%THRESH%/'$thresh'/g' \
>>>>>>> | sed 's/%HSCALE%/'$hscale'/g' \
>>>>>>> > $tmpscripts/plt_4_prodmods_${day1}-
>>>>>>> ${day2}_${thresh}_${hscale}.xml
>>>>>>>
>>>>>>> /usr1/metviewer/metviewer/bin/mv_batch.sh
>>>>>>> $tmpscripts/plt_4_prodmods_${day1}-
>>>>>>> ${day2}_${thresh}_${hscale}.xml
>>>>>>>
>>>>>>> enddo
>>>>>>>
>>>>>>> enddo
>>>>>>>
>>>>>>> Follow John's examples at the Oct tutorial, I was able to
replace
>>>>>>>
>>>>>>> <field equalize="true" name="fcst_thresh">
>>>>>>> <set name="fcst_thresh_1">
>>>>>>> <val><%HSCALE%</val>
>>>>>>> </set>
>>>>>>> </field>
>>>>>>>
>>>>>>> with
>>>>>>>
>>>>>>> <field equalize="true" name="fcst_thresh">
>>>>>>> <val><024</val>
>>>>>>> <val><052</val>
>>>>>>> <val><100</val>
>>>>>>> </field>
>>>>>>>
>>>>>>> and that seems to work.
>>>>>>>
>>>>>>> But is there a way to replace this?
>>>>>>>
>>>>>>> <dep1>
>>>>>>> <fcst_var name="APCP/06>%THRESH%">
>>>>>>> <stat>NBR_FSS</stat>
>>>>>>> </fcst_var>
>>>>>>> </dep1>
>>>>>>>
>>>>>>> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0?
If
>>>>>>> this
>>>>>>> is
>>>>>>> too hard/convoluted to set up then I can just use the old
"sed"
>>>>>>> substitution.
>>>>>>>
>>>>>>> I'm trying to plot 6h FSS from our VSDBs, so the set up is a
bit
>>>>>>> peculiar: METviewer consider the forecast variable as e.g,
>>>>>>> APCP/06>002.0
>>>>>>> (mm), and the horizontal spacial scale is the fcst_thresh
(e.g.,
>>>>>>> <
>>>>>>> 052km)
>>>>>>>
>>>>>>> Also, the resulting plots from the successful use of
>>>>>>> {fcst_thresh}
>>>>>>> in
>>>>>>> batch above have the name such as
>>>>>>> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from
>>>>>>> fcst_thresh
>>>>>>> of
>>>>>>> "<024". the "<" and ">" in file name causes some
problems
>>>>>>> ...
>>>>>>> Is
>>>>>>> there an easy way to create another variable such as
>>>>>>> Afcst_thresh=024
>>>>>>> (fcst_thresh with the "<" removed)?
>>>>>>>
>>>>>>> Thanks very much for looking into this -
>>>>>>>
>>>>>>> Ying
>>>>>>>
>>>
>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: METviewer: batch plot questions
From: Tatiana Burek
Time: Wed Dec 12 10:29:05 2018
The file is on 'process1' server in the same directory where your
original XML is.
Tatiana
On Wed Dec 12 10:16:29 2018, ying.lin at noaa.gov wrote:
> Thanks Tatiana. I don't see attachment (the *_tatiana_1.xml) in
your
> email - would you mind sending it again? Thanks!
>
> Ying
>
> On 12/12/18 11:52 AM, Tatiana Burek via RT wrote:
> > Hi Ying,
> > I created a new XML
fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana_1.xml
> > that returns 12 plots (2 date ranges * 3 fcst_thresh * 2 fcst_var
).
> > I also replaced fcst_thresh with custom values.
> > You can change the templates for file names and the plot title to
> > more human readable if needed.
> > I hope this is helpful.
> >
> > Tatiana
> >
> > On Fri Dec 07 14:11:41 2018, ying.lin at noaa.gov wrote:
> >> Thanks very much Tatiana! I think you know exactly what I was
> >> aiming
> >> for. Thanks for taking the time to set up an example. I haven't
> >> used
> >> 'inherit' before, took a while to try to figure out the structure
of
> >> the
> >> set-up.
> >>
> >> I tried running your set up in
/metviewer/staging/wd22yl/test.batch:
> >>
> >> /usr1/metviewer/metviewer/bin/mv_batch.sh
> >> fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana.xml > run.out
> >> and now have more questions. In the above *_tatiana.xml, the
only
> >> change I made was the date (yyyy changed from 2011 to 2018).
> >>
> >> 1) there are some warning msgs in run.out:
> >> ** WARNING: template tag map_thresh not found in agg perm
> >> ** WARNING: template tag dep not found in agg perm
> >> Does that mean that 'map_thresh' and 'dep' are not
> >> successfully
> >> paired to 'fcst_thresh' and the fcst_var? Would you mind
telling
> >> me
> >> how
> >> to fix this? I don't know enough - haven't been able to fix this
> >> myself.
> >>
> >> 2) Running the xml resulted in two plots (see attached), but
> >> according
> >> to run.out, there are total of 12 plots created, I think it was
six
> >> each
> >> for fss06_date1 and fss06_date2, and the six are probably for
> >> APCP/06>002.0 and APCP/06>005.0 and fcst_thresh values of
> >> 024/052/100
> >> (2x3=6), and the png file names got "recycled" because of the
"agg
> >> perm"
> >> error above.
> >>
> >> BTW this is what I'm trying to do: the "6-hourly fractions skill
> >> scores"
> >> block
> >>
in**http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/201810/.
> >> I _think_ that's what your sample script set up to do as well.
> >>
> >> I really appreciate the time you already spent on this request.
> >> Looks
> >> like with the sample script you constructed we're getting
really
> >> close.
> >> If you don't mind looking into this further with the above
follow-up
> >> question - next week would be fine! - that'll be great. Thanks
> >> again!
> >>
> >> Ying
> >>
> >> On 12/7/18 1:00 PM, Tatiana Burek via RT wrote:
> >>> Ying,
> >>> Please take a look at attache file of batch XML with
inheritance. I
> >>> used your original XML as an example.
> >>> - I created two date ranges and used them in plot_fix group.
> >>> - I created an abstract job named test_thresh that describes all
> >>> common fields
> >>> in tmpl group I created two maps to "translate" fcst_thresh and
> >>> fcst_var values to filename friendly names and use them
templates
> >>> - I created two executable jobs for APCP/06>002.0 and
APCP/06>005.0
> >>>
> >>> Tatiana
> >>>
> >>> On Fri Dec 07 09:58:49 2018, tatiana wrote:
> >>>> Ying,
> >>>>
> >>>> 1.Predefined <date_range> from the example will work as
advertised
> >>>> -
> >>>> all dates in between including start and end will be included.
> >>>> There
> >>>> is no increment specification in the setup.
> >>>> You can specify valid_hour for example to include dates with
the
> >>>> specific hour.
> >>>>
> >>>> 2.What is your goal? To have individual plots for each fcst_var
or
> >>>> plot all fcst_var in one plot?
> >>>>
> >>>> 3.instead of using
> >>>> fss06_%DAY1DASH%-%DAY2DASH%_{fcst_thresh}km_%THRESH%mm.png
> >>>> you could use fss06_%DAY1DASH%-
> >>>> %DAY2DASH%_%THRESH_1%km_%THRESH%mm.png
> >>>> and change %THRESH_1% with sed to the value you need
> >>>>
> >>>> Tatiana
> >>>>
> >>>> On Thu Dec 06 11:05:23 2018, ying.lin at noaa.gov wrote:
> >>>>> Hi Tatiana,
> >>>>>
> >>>>> Thank you for your reply. I didn't realize that METviewer
> >>>>> batch
> >>>>> XML can't have the start/end tags - good to know. In the
> >>>>> attached
> >>>>> XML
> >>>>> (example from John H.G. at the Oct tutorial) there is
> >>>>> <date_range name="fall" ><start>2011-10-01
> >>>>> 00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
> >>>>> ...
> >>>>> <plot_fix>
> >>>>> <field name="fcst_init_beg">
> >>>>> <date_range name="fall"/>
> >>>>>
> >>>>> I suppose this means that the date range must be pre-defined
(in
> >>>>> the
> >>>>> above "date_range" tag). I've used this pre-defined set-up
> >>>>> before
> >>>>> (in
> >>>>> e.g. making seasonal plots). For a more flexible set up (e.g.
> >>>>> plotting
> >>>>> from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by
> >>>>> setting
> >>>>> up
> >>>>> something like
> >>>>> <date_range
> >>>>>
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
> >>>>> then just use 'sed' to do the substitution.
> >>>>>
> >>>>> Questions still remain:
> >>>>> 1) In the above pre-defined date range set up (either
"fall"
> >>>>> or
> >>>>> "anydaterange"), how does one specify the increment (e.g.
12h,
> >>>>> when
> >>>>> the
> >>>>> data are 6-hourly)?
> >>>>> 2) same as the item 2) in the original email inquiry
> >>>>> 3) By "creating another variable", I meant that when
> >>>>> fcst_thresh
> >>>>> is
> >>>>> given the value of "<52km", and {fcst_thresh} is used in
the
> >>>>> resulting
> >>>>> png name, the '<' is causing trouble in unix. Is there a
way
> >>>>> to
> >>>>> get
> >>>>> rid
> >>>>> of the '<', e.g. creating another variable that is
> >>>>> 'fcst_thresh'
> >>>>> minus
> >>>>> the '<'?
> >>>>>
> >>>>> Thanks again,
> >>>>>
> >>>>> Ying
> >>>>>
> >>>>>
> >>>>> On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
> >>>>>> Hi Ying,
> >>>>>>
> >>>>>> #1
> >>>>>> METviewer does not support start/end tags in the batch XML.
For
> >>>>>> your
> >>>>>> example:
> >>>>>> <field equalize="true" name="fcst_valid_beg">
> >>>>>> <start>%DAY1DASH% 00:00:00</start>
> >>>>>> <end>%DAY2DASH% 23:59:59</end>
> >>>>>> </field>
> >>>>>> only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will
be
> >>>>>> included in plotting but nothing in between these days.
> >>>>>>
> >>>>>> #2
> >>>>>> Not sure what you mean by "creating another variable". Have
you
> >>>>>> ever
> >>>>>> considered using inheritance in the batch XML? It might be a
> >>>>>> better
> >>>>>> solution than using sed.
> >>>>>>
> >>>>>> Tatiana
> >>>>>>
> >>>>>> On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
> >>>>>>> Hello. I have several questions about making batch plots on
> >>>>>>> METviewer.
> >>>>>>> Attached 4 files are
> >>>>>>>
> >>>>>>> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch
script
> >>>>>>>
> >>>>>>> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell script
to
> >>>>>>> set
> >>>>>>> up
> >>>>>>> and run batch job on the above
> >>>>>>>
> >>>>>>> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of
the
> >>>>>>> plots
> >>>>>>> produced
> >>>>>>>
> >>>>>>> test.xml: an interactive METviewer plot xml for this
sort
> >>>>>>> of
> >>>>>>> plots
> >>>>>>>
> >>>>>>> 1) I'm specifying model valid times used in the plot with
> >>>>>>>
> >>>>>>> <field equalize="true" name="fcst_valid_beg">
> >>>>>>> <start>%DAY1DASH% 00:00:00</start>
> >>>>>>> <end>%DAY2DASH% 23:59:59</end>
> >>>>>>> </field>
> >>>>>>>
> >>>>>>> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31)
substituted
> >>>>>>> in
> >>>>>>> the
> >>>>>>> calling script with sed. This seems to work, but I think
this
> >>>>>>> includes
> >>>>>>> stats for all valid times between 2018-08-01 00:00:00 and
2018-
> >>>>>>> 08-
> >>>>>>> 31
> >>>>>>> 23:59:59. In interactive METviewer setup, one can specify
> >>>>>>> fcst_valid_beg with a start time, end time and an increment
> >>>>>>> (6h,
> >>>>>>> 12h
> >>>>>>> etc.). How does one specify such an increment in the batch
> >>>>>>> script
> >>>>>>> (e.g.
> >>>>>>> if I only want valid times at 00Z/12Z, rather than
> >>>>>>> 00/06/12/18Z)?
> >>>>>>>
> >>>>>>> 2) In the past I've been using a lot of sed substitutes and
> >>>>>>> make
> >>>>>>> a
> >>>>>>> single plot per mv_batch.sh call, like this:
> >>>>>>>
> >>>>>>> for hscale in 024 052 100
> >>>>>>> do
> >>>>>>>
> >>>>>>> for thresh in 002.0 005.0 010.0 020.0
> >>>>>>> do
> >>>>>>>
> >>>>>>> cat
> >>>>>>>
$ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml
> >>>>>>> \
> >>>>>>> | sed "s/%DAY1DASH%/$day1dash/g" \
> >>>>>>> | sed "s/%DAY2DASH%/$day2dash/g" \
> >>>>>>> | sed 's/%THRESH%/'$thresh'/g' \
> >>>>>>> | sed 's/%HSCALE%/'$hscale'/g' \
> >>>>>>> > $tmpscripts/plt_4_prodmods_${day1}-
> >>>>>>> ${day2}_${thresh}_${hscale}.xml
> >>>>>>>
> >>>>>>> /usr1/metviewer/metviewer/bin/mv_batch.sh
> >>>>>>> $tmpscripts/plt_4_prodmods_${day1}-
> >>>>>>> ${day2}_${thresh}_${hscale}.xml
> >>>>>>>
> >>>>>>> enddo
> >>>>>>>
> >>>>>>> enddo
> >>>>>>>
> >>>>>>> Follow John's examples at the Oct tutorial, I was able to
> >>>>>>> replace
> >>>>>>>
> >>>>>>> <field equalize="true" name="fcst_thresh">
> >>>>>>> <set name="fcst_thresh_1">
> >>>>>>> <val><%HSCALE%</val>
> >>>>>>> </set>
> >>>>>>> </field>
> >>>>>>>
> >>>>>>> with
> >>>>>>>
> >>>>>>> <field equalize="true" name="fcst_thresh">
> >>>>>>> <val><024</val>
> >>>>>>> <val><052</val>
> >>>>>>> <val><100</val>
> >>>>>>> </field>
> >>>>>>>
> >>>>>>> and that seems to work.
> >>>>>>>
> >>>>>>> But is there a way to replace this?
> >>>>>>>
> >>>>>>> <dep1>
> >>>>>>> <fcst_var name="APCP/06>%THRESH%">
> >>>>>>> <stat>NBR_FSS</stat>
> >>>>>>> </fcst_var>
> >>>>>>> </dep1>
> >>>>>>>
> >>>>>>> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0?
If
> >>>>>>> this
> >>>>>>> is
> >>>>>>> too hard/convoluted to set up then I can just use the old
"sed"
> >>>>>>> substitution.
> >>>>>>>
> >>>>>>> I'm trying to plot 6h FSS from our VSDBs, so the set up is a
> >>>>>>> bit
> >>>>>>> peculiar: METviewer consider the forecast variable as e.g,
> >>>>>>> APCP/06>002.0
> >>>>>>> (mm), and the horizontal spacial scale is the fcst_thresh
> >>>>>>> (e.g.,
> >>>>>>> <
> >>>>>>> 052km)
> >>>>>>>
> >>>>>>> Also, the resulting plots from the successful use of
> >>>>>>> {fcst_thresh}
> >>>>>>> in
> >>>>>>> batch above have the name such as
> >>>>>>> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from
> >>>>>>> fcst_thresh
> >>>>>>> of
> >>>>>>> "<024". the "<" and ">" in file name causes some
> >>>>>>> problems
> >>>>>>> ...
> >>>>>>> Is
> >>>>>>> there an easy way to create another variable such as
> >>>>>>> Afcst_thresh=024
> >>>>>>> (fcst_thresh with the "<" removed)?
> >>>>>>>
> >>>>>>> Thanks very much for looking into this -
> >>>>>>>
> >>>>>>> Ying
> >>>>>>>
> >>>
> >
> >
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87973] METviewer: batch plot questions
From: Ying Lin
Time: Wed Dec 12 17:02:16 2018
Hi Tatiana,
Yes it does work to produce the 12 plots now, thank you! If you don't
mind, please leave the ticket open for another day or so - I'm trying
to
tweak the script to get the ">" off the file name. Can probably do
that
by following what you did in the script, but I'm not quite there yet.
Ying
On 12/12/18 12:29 PM, Tatiana Burek via RT wrote:
> The file is on 'process1' server in the same directory where your
original XML is.
>
> Tatiana
>
> On Wed Dec 12 10:16:29 2018, ying.lin at noaa.gov wrote:
>> Thanks Tatiana. I don't see attachment (the *_tatiana_1.xml) in
your
>> email - would you mind sending it again? Thanks!
>>
>> Ying
>>
>> On 12/12/18 11:52 AM, Tatiana Burek via RT wrote:
>>> Hi Ying,
>>> I created a new XML
fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana_1.xml
>>> that returns 12 plots (2 date ranges * 3 fcst_thresh * 2 fcst_var
).
>>> I also replaced fcst_thresh with custom values.
>>> You can change the templates for file names and the plot title to
>>> more human readable if needed.
>>> I hope this is helpful.
>>>
>>> Tatiana
>>>
>>> On Fri Dec 07 14:11:41 2018, ying.lin at noaa.gov wrote:
>>>> Thanks very much Tatiana! I think you know exactly what I was
>>>> aiming
>>>> for. Thanks for taking the time to set up an example. I haven't
>>>> used
>>>> 'inherit' before, took a while to try to figure out the structure
of
>>>> the
>>>> set-up.
>>>>
>>>> I tried running your set up in
/metviewer/staging/wd22yl/test.batch:
>>>>
>>>> /usr1/metviewer/metviewer/bin/mv_batch.sh
>>>> fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana.xml > run.out
>>>> and now have more questions. In the above *_tatiana.xml, the
only
>>>> change I made was the date (yyyy changed from 2011 to 2018).
>>>>
>>>> 1) there are some warning msgs in run.out:
>>>> ** WARNING: template tag map_thresh not found in agg perm
>>>> ** WARNING: template tag dep not found in agg perm
>>>> Does that mean that 'map_thresh' and 'dep' are not
>>>> successfully
>>>> paired to 'fcst_thresh' and the fcst_var? Would you mind
telling
>>>> me
>>>> how
>>>> to fix this? I don't know enough - haven't been able to fix this
>>>> myself.
>>>>
>>>> 2) Running the xml resulted in two plots (see attached), but
>>>> according
>>>> to run.out, there are total of 12 plots created, I think it was
six
>>>> each
>>>> for fss06_date1 and fss06_date2, and the six are probably for
>>>> APCP/06>002.0 and APCP/06>005.0 and fcst_thresh values of
>>>> 024/052/100
>>>> (2x3=6), and the png file names got "recycled" because of the
"agg
>>>> perm"
>>>> error above.
>>>>
>>>> BTW this is what I'm trying to do: the "6-hourly fractions skill
>>>> scores"
>>>> block
>>>>
in**http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/201810/.
>>>> I _think_ that's what your sample script set up to do as well.
>>>>
>>>> I really appreciate the time you already spent on this request.
>>>> Looks
>>>> like with the sample script you constructed we're getting
really
>>>> close.
>>>> If you don't mind looking into this further with the above
follow-up
>>>> question - next week would be fine! - that'll be great. Thanks
>>>> again!
>>>>
>>>> Ying
>>>>
>>>> On 12/7/18 1:00 PM, Tatiana Burek via RT wrote:
>>>>> Ying,
>>>>> Please take a look at attache file of batch XML with
inheritance. I
>>>>> used your original XML as an example.
>>>>> - I created two date ranges and used them in plot_fix group.
>>>>> - I created an abstract job named test_thresh that describes all
>>>>> common fields
>>>>> in tmpl group I created two maps to "translate" fcst_thresh and
>>>>> fcst_var values to filename friendly names and use them
templates
>>>>> - I created two executable jobs for APCP/06>002.0 and
APCP/06>005.0
>>>>>
>>>>> Tatiana
>>>>>
>>>>> On Fri Dec 07 09:58:49 2018, tatiana wrote:
>>>>>> Ying,
>>>>>>
>>>>>> 1.Predefined <date_range> from the example will work as
advertised
>>>>>> -
>>>>>> all dates in between including start and end will be included.
>>>>>> There
>>>>>> is no increment specification in the setup.
>>>>>> You can specify valid_hour for example to include dates with
the
>>>>>> specific hour.
>>>>>>
>>>>>> 2.What is your goal? To have individual plots for each fcst_var
or
>>>>>> plot all fcst_var in one plot?
>>>>>>
>>>>>> 3.instead of using
>>>>>> fss06_%DAY1DASH%-%DAY2DASH%_{fcst_thresh}km_%THRESH%mm.png
>>>>>> you could use fss06_%DAY1DASH%-
>>>>>> %DAY2DASH%_%THRESH_1%km_%THRESH%mm.png
>>>>>> and change %THRESH_1% with sed to the value you need
>>>>>>
>>>>>> Tatiana
>>>>>>
>>>>>> On Thu Dec 06 11:05:23 2018, ying.lin at noaa.gov wrote:
>>>>>>> Hi Tatiana,
>>>>>>>
>>>>>>> Thank you for your reply. I didn't realize that
METviewer
>>>>>>> batch
>>>>>>> XML can't have the start/end tags - good to know. In the
>>>>>>> attached
>>>>>>> XML
>>>>>>> (example from John H.G. at the Oct tutorial) there is
>>>>>>> <date_range name="fall" ><start>2011-10-01
>>>>>>> 00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
>>>>>>> ...
>>>>>>> <plot_fix>
>>>>>>> <field name="fcst_init_beg">
>>>>>>> <date_range name="fall"/>
>>>>>>>
>>>>>>> I suppose this means that the date range must be pre-defined
(in
>>>>>>> the
>>>>>>> above "date_range" tag). I've used this pre-defined set-up
>>>>>>> before
>>>>>>> (in
>>>>>>> e.g. making seasonal plots). For a more flexible set up (e.g.
>>>>>>> plotting
>>>>>>> from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by
>>>>>>> setting
>>>>>>> up
>>>>>>> something like
>>>>>>> <date_range
>>>>>>>
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
>>>>>>> then just use 'sed' to do the substitution.
>>>>>>>
>>>>>>> Questions still remain:
>>>>>>> 1) In the above pre-defined date range set up (either
"fall"
>>>>>>> or
>>>>>>> "anydaterange"), how does one specify the increment (e.g.
12h,
>>>>>>> when
>>>>>>> the
>>>>>>> data are 6-hourly)?
>>>>>>> 2) same as the item 2) in the original email inquiry
>>>>>>> 3) By "creating another variable", I meant that when
>>>>>>> fcst_thresh
>>>>>>> is
>>>>>>> given the value of "<52km", and {fcst_thresh} is used in
the
>>>>>>> resulting
>>>>>>> png name, the '<' is causing trouble in unix. Is there a
way
>>>>>>> to
>>>>>>> get
>>>>>>> rid
>>>>>>> of the '<', e.g. creating another variable that is
>>>>>>> 'fcst_thresh'
>>>>>>> minus
>>>>>>> the '<'?
>>>>>>>
>>>>>>> Thanks again,
>>>>>>>
>>>>>>> Ying
>>>>>>>
>>>>>>>
>>>>>>> On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
>>>>>>>> Hi Ying,
>>>>>>>>
>>>>>>>> #1
>>>>>>>> METviewer does not support start/end tags in the batch XML.
For
>>>>>>>> your
>>>>>>>> example:
>>>>>>>> <field equalize="true" name="fcst_valid_beg">
>>>>>>>> <start>%DAY1DASH% 00:00:00</start>
>>>>>>>> <end>%DAY2DASH% 23:59:59</end>
>>>>>>>> </field>
>>>>>>>> only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will
be
>>>>>>>> included in plotting but nothing in between these days.
>>>>>>>>
>>>>>>>> #2
>>>>>>>> Not sure what you mean by "creating another variable". Have
you
>>>>>>>> ever
>>>>>>>> considered using inheritance in the batch XML? It might be a
>>>>>>>> better
>>>>>>>> solution than using sed.
>>>>>>>>
>>>>>>>> Tatiana
>>>>>>>>
>>>>>>>> On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
>>>>>>>>> Hello. I have several questions about making batch plots on
>>>>>>>>> METviewer.
>>>>>>>>> Attached 4 files are
>>>>>>>>>
>>>>>>>>> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch
script
>>>>>>>>>
>>>>>>>>> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell
script to
>>>>>>>>> set
>>>>>>>>> up
>>>>>>>>> and run batch job on the above
>>>>>>>>>
>>>>>>>>> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of
the
>>>>>>>>> plots
>>>>>>>>> produced
>>>>>>>>>
>>>>>>>>> test.xml: an interactive METviewer plot xml for this
sort
>>>>>>>>> of
>>>>>>>>> plots
>>>>>>>>>
>>>>>>>>> 1) I'm specifying model valid times used in the plot with
>>>>>>>>>
>>>>>>>>> <field equalize="true" name="fcst_valid_beg">
>>>>>>>>> <start>%DAY1DASH% 00:00:00</start>
>>>>>>>>> <end>%DAY2DASH% 23:59:59</end>
>>>>>>>>> </field>
>>>>>>>>>
>>>>>>>>> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31)
substituted
>>>>>>>>> in
>>>>>>>>> the
>>>>>>>>> calling script with sed. This seems to work, but I think
this
>>>>>>>>> includes
>>>>>>>>> stats for all valid times between 2018-08-01 00:00:00 and
2018-
>>>>>>>>> 08-
>>>>>>>>> 31
>>>>>>>>> 23:59:59. In interactive METviewer setup, one can specify
>>>>>>>>> fcst_valid_beg with a start time, end time and an increment
>>>>>>>>> (6h,
>>>>>>>>> 12h
>>>>>>>>> etc.). How does one specify such an increment in the batch
>>>>>>>>> script
>>>>>>>>> (e.g.
>>>>>>>>> if I only want valid times at 00Z/12Z, rather than
>>>>>>>>> 00/06/12/18Z)?
>>>>>>>>>
>>>>>>>>> 2) In the past I've been using a lot of sed substitutes and
>>>>>>>>> make
>>>>>>>>> a
>>>>>>>>> single plot per mv_batch.sh call, like this:
>>>>>>>>>
>>>>>>>>> for hscale in 024 052 100
>>>>>>>>> do
>>>>>>>>>
>>>>>>>>> for thresh in 002.0 005.0 010.0 020.0
>>>>>>>>> do
>>>>>>>>>
>>>>>>>>> cat
>>>>>>>>> $ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml
>>>>>>>>> \
>>>>>>>>> | sed "s/%DAY1DASH%/$day1dash/g" \
>>>>>>>>> | sed "s/%DAY2DASH%/$day2dash/g" \
>>>>>>>>> | sed 's/%THRESH%/'$thresh'/g' \
>>>>>>>>> | sed 's/%HSCALE%/'$hscale'/g' \
>>>>>>>>> > $tmpscripts/plt_4_prodmods_${day1}-
>>>>>>>>> ${day2}_${thresh}_${hscale}.xml
>>>>>>>>>
>>>>>>>>> /usr1/metviewer/metviewer/bin/mv_batch.sh
>>>>>>>>> $tmpscripts/plt_4_prodmods_${day1}-
>>>>>>>>> ${day2}_${thresh}_${hscale}.xml
>>>>>>>>>
>>>>>>>>> enddo
>>>>>>>>>
>>>>>>>>> enddo
>>>>>>>>>
>>>>>>>>> Follow John's examples at the Oct tutorial, I was able to
>>>>>>>>> replace
>>>>>>>>>
>>>>>>>>> <field equalize="true" name="fcst_thresh">
>>>>>>>>> <set name="fcst_thresh_1">
>>>>>>>>> <val><%HSCALE%</val>
>>>>>>>>> </set>
>>>>>>>>> </field>
>>>>>>>>>
>>>>>>>>> with
>>>>>>>>>
>>>>>>>>> <field equalize="true" name="fcst_thresh">
>>>>>>>>> <val><024</val>
>>>>>>>>> <val><052</val>
>>>>>>>>> <val><100</val>
>>>>>>>>> </field>
>>>>>>>>>
>>>>>>>>> and that seems to work.
>>>>>>>>>
>>>>>>>>> But is there a way to replace this?
>>>>>>>>>
>>>>>>>>> <dep1>
>>>>>>>>> <fcst_var name="APCP/06>%THRESH%">
>>>>>>>>> <stat>NBR_FSS</stat>
>>>>>>>>> </fcst_var>
>>>>>>>>> </dep1>
>>>>>>>>>
>>>>>>>>> with multiple THRESH values of > 002.0, 005.0, 010.0, 020.0?
If
>>>>>>>>> this
>>>>>>>>> is
>>>>>>>>> too hard/convoluted to set up then I can just use the old
"sed"
>>>>>>>>> substitution.
>>>>>>>>>
>>>>>>>>> I'm trying to plot 6h FSS from our VSDBs, so the set up is a
>>>>>>>>> bit
>>>>>>>>> peculiar: METviewer consider the forecast variable as e.g,
>>>>>>>>> APCP/06>002.0
>>>>>>>>> (mm), and the horizontal spacial scale is the fcst_thresh
>>>>>>>>> (e.g.,
>>>>>>>>> <
>>>>>>>>> 052km)
>>>>>>>>>
>>>>>>>>> Also, the resulting plots from the successful use of
>>>>>>>>> {fcst_thresh}
>>>>>>>>> in
>>>>>>>>> batch above have the name such as
>>>>>>>>> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from
>>>>>>>>> fcst_thresh
>>>>>>>>> of
>>>>>>>>> "<024". the "<" and ">" in file name causes some
>>>>>>>>> problems
>>>>>>>>> ...
>>>>>>>>> Is
>>>>>>>>> there an easy way to create another variable such as
>>>>>>>>> Afcst_thresh=024
>>>>>>>>> (fcst_thresh with the "<" removed)?
>>>>>>>>>
>>>>>>>>> Thanks very much for looking into this -
>>>>>>>>>
>>>>>>>>> Ying
>>>>>>>>>
>>>
>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87973] METviewer: batch plot questions
From: Ying Lin
Time: Sat Dec 15 18:15:13 2018
Hi Tatiana,
Thanks again for your help and the example xml, which is great.
Please close this ticket at your convenience. Still studying the xml,
in case there are further questions I'll open a new ticket.
Have a good weekend.
Ying
On 12/12/18 7:02 PM, Ying Lin wrote:
> Hi Tatiana,
>
> Yes it does work to produce the 12 plots now, thank you! If you
don't
> mind, please leave the ticket open for another day or so - I'm
trying
> to tweak the script to get the ">" off the file name. Can probably
do
> that by following what you did in the script, but I'm not quite
there
> yet.
>
> Ying
>
> On 12/12/18 12:29 PM, Tatiana Burek via RT wrote:
>> The file is on 'process1' server in the same directory where your
>> original XML is.
>>
>> Tatiana
>>
>> On Wed Dec 12 10:16:29 2018, ying.lin at noaa.gov wrote:
>>> Thanks Tatiana. I don't see attachment (the *_tatiana_1.xml) in
your
>>> email - would you mind sending it again? Thanks!
>>>
>>> Ying
>>>
>>> On 12/12/18 11:52 AM, Tatiana Burek via RT wrote:
>>>> Hi Ying,
>>>> I created a new XML
fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana_1.xml
>>>> that returns 12 plots (2 date ranges * 3 fcst_thresh * 2 fcst_var
).
>>>> I also replaced fcst_thresh with custom values.
>>>> You can change the templates for file names and the plot title to
>>>> more human readable if needed.
>>>> I hope this is helpful.
>>>>
>>>> Tatiana
>>>>
>>>> On Fri Dec 07 14:11:41 2018, ying.lin at noaa.gov wrote:
>>>>> Thanks very much Tatiana! I think you know exactly what I was
>>>>> aiming
>>>>> for. Thanks for taking the time to set up an example. I
haven't
>>>>> used
>>>>> 'inherit' before, took a while to try to figure out the
structure of
>>>>> the
>>>>> set-up.
>>>>>
>>>>> I tried running your set up in
/metviewer/staging/wd22yl/test.batch:
>>>>>
>>>>> /usr1/metviewer/metviewer/bin/mv_batch.sh
>>>>> fss06_v_fhr_gfs_nam_conusnest_hrrr_tatiana.xml > run.out
>>>>> and now have more questions. In the above *_tatiana.xml, the
only
>>>>> change I made was the date (yyyy changed from 2011 to 2018).
>>>>>
>>>>> 1) there are some warning msgs in run.out:
>>>>> ** WARNING: template tag map_thresh not found in agg perm
>>>>> ** WARNING: template tag dep not found in agg perm
>>>>> Does that mean that 'map_thresh' and 'dep' are not
>>>>> successfully
>>>>> paired to 'fcst_thresh' and the fcst_var? Would you mind
telling
>>>>> me
>>>>> how
>>>>> to fix this? I don't know enough - haven't been able to fix
this
>>>>> myself.
>>>>>
>>>>> 2) Running the xml resulted in two plots (see attached), but
>>>>> according
>>>>> to run.out, there are total of 12 plots created, I think it was
six
>>>>> each
>>>>> for fss06_date1 and fss06_date2, and the six are probably for
>>>>> APCP/06>002.0 and APCP/06>005.0 and fcst_thresh values of
>>>>> 024/052/100
>>>>> (2x3=6), and the png file names got "recycled" because of the
"agg
>>>>> perm"
>>>>> error above.
>>>>>
>>>>> BTW this is what I'm trying to do: the "6-hourly fractions skill
>>>>> scores"
>>>>> block
>>>>>
in**http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/201810/.
>>>>>
>>>>> I _think_ that's what your sample script set up to do as well.
>>>>>
>>>>> I really appreciate the time you already spent on this request.
>>>>> Looks
>>>>> like with the sample script you constructed we're getting
really
>>>>> close.
>>>>> If you don't mind looking into this further with the above
follow-up
>>>>> question - next week would be fine! - that'll be great. Thanks
>>>>> again!
>>>>>
>>>>> Ying
>>>>>
>>>>> On 12/7/18 1:00 PM, Tatiana Burek via RT wrote:
>>>>>> Ying,
>>>>>> Please take a look at attache file of batch XML with
inheritance. I
>>>>>> used your original XML as an example.
>>>>>> - I created two date ranges and used them in plot_fix group.
>>>>>> - I created an abstract job named test_thresh that describes
all
>>>>>> common fields
>>>>>> in tmpl group I created two maps to "translate" fcst_thresh and
>>>>>> fcst_var values to filename friendly names and use them
templates
>>>>>> - I created two executable jobs for APCP/06>002.0 and
APCP/06>005.0
>>>>>>
>>>>>> Tatiana
>>>>>>
>>>>>> On Fri Dec 07 09:58:49 2018, tatiana wrote:
>>>>>>> Ying,
>>>>>>>
>>>>>>> 1.Predefined <date_range> from the example will work as
advertised
>>>>>>> -
>>>>>>> all dates in between including start and end will be
included.
>>>>>>> There
>>>>>>> is no increment specification in the setup.
>>>>>>> You can specify valid_hour for example to include dates with
the
>>>>>>> specific hour.
>>>>>>>
>>>>>>> 2.What is your goal? To have individual plots for each
fcst_var or
>>>>>>> plot all fcst_var in one plot?
>>>>>>>
>>>>>>> 3.instead of using
>>>>>>> fss06_%DAY1DASH%-%DAY2DASH%_{fcst_thresh}km_%THRESH%mm.png
>>>>>>> you could use fss06_%DAY1DASH%-
>>>>>>> %DAY2DASH%_%THRESH_1%km_%THRESH%mm.png
>>>>>>> and change %THRESH_1% with sed to the value you need
>>>>>>>
>>>>>>> Tatiana
>>>>>>>
>>>>>>> On Thu Dec 06 11:05:23 2018, ying.lin at noaa.gov wrote:
>>>>>>>> Hi Tatiana,
>>>>>>>>
>>>>>>>> Thank you for your reply. I didn't realize that
METviewer
>>>>>>>> batch
>>>>>>>> XML can't have the start/end tags - good to know. In the
>>>>>>>> attached
>>>>>>>> XML
>>>>>>>> (example from John H.G. at the Oct tutorial) there is
>>>>>>>> <date_range name="fall" ><start>2011-10-01
>>>>>>>> 00:00:00</start><end>2011-12-31 12:00:00</end></date_range>
>>>>>>>> ...
>>>>>>>> <plot_fix>
>>>>>>>> <field name="fcst_init_beg">
>>>>>>>> <date_range name="fall"/>
>>>>>>>>
>>>>>>>> I suppose this means that the date range must be pre-defined
(in
>>>>>>>> the
>>>>>>>> above "date_range" tag). I've used this pre-defined set-up
>>>>>>>> before
>>>>>>>> (in
>>>>>>>> e.g. making seasonal plots). For a more flexible set up
(e.g.
>>>>>>>> plotting
>>>>>>>> from YYYY-MM-DD1 to YYYY-MM-DD2) I think this can be done by
>>>>>>>> setting
>>>>>>>> up
>>>>>>>> something like
>>>>>>>> <date_range
>>>>>>>>
name="anydaterange"><start>%DAY1%</start><end>%DAY2%</end></date_range>
>>>>>>>>
>>>>>>>> then just use 'sed' to do the substitution.
>>>>>>>>
>>>>>>>> Questions still remain:
>>>>>>>> 1) In the above pre-defined date range set up (either
"fall"
>>>>>>>> or
>>>>>>>> "anydaterange"), how does one specify the increment (e.g.
12h,
>>>>>>>> when
>>>>>>>> the
>>>>>>>> data are 6-hourly)?
>>>>>>>> 2) same as the item 2) in the original email inquiry
>>>>>>>> 3) By "creating another variable", I meant that when
>>>>>>>> fcst_thresh
>>>>>>>> is
>>>>>>>> given the value of "<52km", and {fcst_thresh} is used in
the
>>>>>>>> resulting
>>>>>>>> png name, the '<' is causing trouble in unix. Is there a
way
>>>>>>>> to
>>>>>>>> get
>>>>>>>> rid
>>>>>>>> of the '<', e.g. creating another variable that is
>>>>>>>> 'fcst_thresh'
>>>>>>>> minus
>>>>>>>> the '<'?
>>>>>>>>
>>>>>>>> Thanks again,
>>>>>>>>
>>>>>>>> Ying
>>>>>>>>
>>>>>>>>
>>>>>>>> On 12/6/18 9:59 AM, Tatiana Burek via RT wrote:
>>>>>>>>> Hi Ying,
>>>>>>>>>
>>>>>>>>> #1
>>>>>>>>> METviewer does not support start/end tags in the batch XML.
For
>>>>>>>>> your
>>>>>>>>> example:
>>>>>>>>> <field equalize="true" name="fcst_valid_beg">
>>>>>>>>> <start>%DAY1DASH% 00:00:00</start>
>>>>>>>>> <end>%DAY2DASH% 23:59:59</end>
>>>>>>>>> </field>
>>>>>>>>> only dates %DAY1DASH% 00:00:00 and %DAY2DASH% 23:59:59 will
be
>>>>>>>>> included in plotting but nothing in between these days.
>>>>>>>>>
>>>>>>>>> #2
>>>>>>>>> Not sure what you mean by "creating another variable". Have
you
>>>>>>>>> ever
>>>>>>>>> considered using inheritance in the batch XML? It might be a
>>>>>>>>> better
>>>>>>>>> solution than using sed.
>>>>>>>>>
>>>>>>>>> Tatiana
>>>>>>>>>
>>>>>>>>> On Fri Nov 30 12:28:48 2018, ying.lin at noaa.gov wrote:
>>>>>>>>>> Hello. I have several questions about making batch plots
on
>>>>>>>>>> METviewer.
>>>>>>>>>> Attached 4 files are
>>>>>>>>>>
>>>>>>>>>> fss06_v_fhr_gfs_nam_conusnest_hrrr.xml - the batch
script
>>>>>>>>>>
>>>>>>>>>> plot_fss06_fhr_gfs_nam_conusnest_hrrr.sh - shell
script to
>>>>>>>>>> set
>>>>>>>>>> up
>>>>>>>>>> and run batch job on the above
>>>>>>>>>>
>>>>>>>>>> fss06_2018-08-01-2018-08-31_<024km_002.0mm.png - one of the
>>>>>>>>>> plots
>>>>>>>>>> produced
>>>>>>>>>>
>>>>>>>>>> test.xml: an interactive METviewer plot xml for this
sort
>>>>>>>>>> of
>>>>>>>>>> plots
>>>>>>>>>>
>>>>>>>>>> 1) I'm specifying model valid times used in the plot with
>>>>>>>>>>
>>>>>>>>>> <field equalize="true" name="fcst_valid_beg">
>>>>>>>>>> <start>%DAY1DASH% 00:00:00</start>
>>>>>>>>>> <end>%DAY2DASH% 23:59:59</end>
>>>>>>>>>> </field>
>>>>>>>>>>
>>>>>>>>>> with DAY1DASH/DAY2DASH (e.g. 2018-08-01/2018-08-31)
substituted
>>>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>> calling script with sed. This seems to work, but I think
this
>>>>>>>>>> includes
>>>>>>>>>> stats for all valid times between 2018-08-01 00:00:00 and
2018-
>>>>>>>>>> 08-
>>>>>>>>>> 31
>>>>>>>>>> 23:59:59. In interactive METviewer setup, one can specify
>>>>>>>>>> fcst_valid_beg with a start time, end time and an increment
>>>>>>>>>> (6h,
>>>>>>>>>> 12h
>>>>>>>>>> etc.). How does one specify such an increment in the batch
>>>>>>>>>> script
>>>>>>>>>> (e.g.
>>>>>>>>>> if I only want valid times at 00Z/12Z, rather than
>>>>>>>>>> 00/06/12/18Z)?
>>>>>>>>>>
>>>>>>>>>> 2) In the past I've been using a lot of sed substitutes and
>>>>>>>>>> make
>>>>>>>>>> a
>>>>>>>>>> single plot per mv_batch.sh call, like this:
>>>>>>>>>>
>>>>>>>>>> for hscale in 024 052 100
>>>>>>>>>> do
>>>>>>>>>>
>>>>>>>>>> for thresh in 002.0 005.0 010.0 020.0
>>>>>>>>>> do
>>>>>>>>>>
>>>>>>>>>> cat
>>>>>>>>>> $ylscripts/fss06_v_fhr_gfs_nam_conusnest_hrrr.xml
>>>>>>>>>> \
>>>>>>>>>> | sed "s/%DAY1DASH%/$day1dash/g" \
>>>>>>>>>> | sed "s/%DAY2DASH%/$day2dash/g" \
>>>>>>>>>> | sed 's/%THRESH%/'$thresh'/g' \
>>>>>>>>>> | sed 's/%HSCALE%/'$hscale'/g' \
>>>>>>>>>> > $tmpscripts/plt_4_prodmods_${day1}-
>>>>>>>>>> ${day2}_${thresh}_${hscale}.xml
>>>>>>>>>>
>>>>>>>>>> /usr1/metviewer/metviewer/bin/mv_batch.sh
>>>>>>>>>> $tmpscripts/plt_4_prodmods_${day1}-
>>>>>>>>>> ${day2}_${thresh}_${hscale}.xml
>>>>>>>>>>
>>>>>>>>>> enddo
>>>>>>>>>>
>>>>>>>>>> enddo
>>>>>>>>>>
>>>>>>>>>> Follow John's examples at the Oct tutorial, I was able to
>>>>>>>>>> replace
>>>>>>>>>>
>>>>>>>>>> <field equalize="true" name="fcst_thresh">
>>>>>>>>>> <set name="fcst_thresh_1">
>>>>>>>>>> <val><%HSCALE%</val>
>>>>>>>>>> </set>
>>>>>>>>>> </field>
>>>>>>>>>>
>>>>>>>>>> with
>>>>>>>>>>
>>>>>>>>>> <field equalize="true" name="fcst_thresh">
>>>>>>>>>> <val><024</val>
>>>>>>>>>> <val><052</val>
>>>>>>>>>> <val><100</val>
>>>>>>>>>> </field>
>>>>>>>>>>
>>>>>>>>>> and that seems to work.
>>>>>>>>>>
>>>>>>>>>> But is there a way to replace this?
>>>>>>>>>>
>>>>>>>>>> <dep1>
>>>>>>>>>> <fcst_var name="APCP/06>%THRESH%">
>>>>>>>>>> <stat>NBR_FSS</stat>
>>>>>>>>>> </fcst_var>
>>>>>>>>>> </dep1>
>>>>>>>>>>
>>>>>>>>>> with multiple THRESH values of > 002.0, 005.0, 010.0,
020.0? If
>>>>>>>>>> this
>>>>>>>>>> is
>>>>>>>>>> too hard/convoluted to set up then I can just use the old
"sed"
>>>>>>>>>> substitution.
>>>>>>>>>>
>>>>>>>>>> I'm trying to plot 6h FSS from our VSDBs, so the set up is
a
>>>>>>>>>> bit
>>>>>>>>>> peculiar: METviewer consider the forecast variable as e.g,
>>>>>>>>>> APCP/06>002.0
>>>>>>>>>> (mm), and the horizontal spacial scale is the fcst_thresh
>>>>>>>>>> (e.g.,
>>>>>>>>>> <
>>>>>>>>>> 052km)
>>>>>>>>>>
>>>>>>>>>> Also, the resulting plots from the successful use of
>>>>>>>>>> {fcst_thresh}
>>>>>>>>>> in
>>>>>>>>>> batch above have the name such as
>>>>>>>>>> "fss06_2018-08-01-2018-08-31_<024km_002.0mm.png", from
>>>>>>>>>> fcst_thresh
>>>>>>>>>> of
>>>>>>>>>> "<024". the "<" and ">" in file name causes some
>>>>>>>>>> problems
>>>>>>>>>> ...
>>>>>>>>>> Is
>>>>>>>>>> there an easy way to create another variable such as
>>>>>>>>>> Afcst_thresh=024
>>>>>>>>>> (fcst_thresh with the "<" removed)?
>>>>>>>>>>
>>>>>>>>>> Thanks very much for looking into this -
>>>>>>>>>>
>>>>>>>>>> Ying
>>>>>>>>>>
>>>>
>>
>>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list