[Met_help] [rt.rap.ucar.edu #97283] History for plots no longer being generated
John Halley Gotway via RT
met_help at ucar.edu
Tue Nov 17 10:48:46 MST 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning,
I was preparing to carry over some plots to my local machine when I
realized that plots have not been generated for days. I decided to
manually copy the contents from the file that I use to process data and
generate plots found here: *
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
*
After copying the contents over from the file and pasting it to the command
line, I see the expected printout begin. However, instead of proceeding,
it hangs up and does not go past the line shown in red below:
*0:00', '2020-10-26 03:00:00', '2020-10-26 04:00:00', '2020-10-26
05:00:00', '2020-10-26 06:00:00', '2020-10-26 07:00:00', '2020-10-26
08:00:00', '2020-10-26 09:00:00', '2020-10-26 10:00:00', '2020-10-26
11:00:00', '2020-10-26 12:00:00', '2020-10-26 13:00:00', '2020-10-26
14:00:00', '2020-10-26 15:00:00', '2020-10-26 16:00:00', '2020-10-26
17:00:00', '2020-10-26 18:00:00', '2020-10-26 19:00:00', '2020-10-26
20:00:00', '2020-10-26 21:00:00', '2020-10-26 22:00:00', '2020-10-26
23:00:00') AND BINARY h.model IN ('PARA6', 'PARA6A', 'PARA6A_BC',
'PARA6_BC', 'PROD', 'PROD_BC') AND BINARY if( (select fcst_lead_offset
FROM model_fcst_lead_offset WHERE model = h.model) is NULL , ld.fcst_lead ,
ld.fcst_lead + (select fcst_lead_offset FROM model_fcst_lead_offset WHERE
model = h.model) ) IN ('0', '10000', '20000', '30000', '40000', '50000',
'60000', '70000', '80000', '90000', '100000', '110000', '120000', '130000',
'140000', '150000', '160000', '170000', '180000', '190000', '200000',
'210000', '220000', '230000', '240000', '250000', '260000', '270000',
'280000', '290000', '300000', '310000', '320000', '330000', '340000',
'350000', '360000', '370000', '380000', '390000', '400000', '410000',
'420000', '430000', '440000', '450000', '460000', '470000', '480000') AND
BINARY h.fcst_var = 'PMTF' AND ld.stat_header_id = h.stat_header_id;*
The xml file looks fine. It's the same xml files that I've been using for
a while. Looking at the err file when this step is run using a cron, I
find that the above printout is not included (
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/err.Time_Series_Aggregate*).
Overall I don't see anything suspect in the error file. The out file (
*out.Time_Series_Aggregate*) suggests everything is fine. I see the above
printout. I just don't know why my plots are not being generated and have
not been generated for days.
I can elaborate with my detail if desired.
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: plots no longer being generated
From: John Halley Gotway
Time: Sun Nov 01 20:49:52 2020
Hi Ed,
Wanted to let you know that I saw this email and will take a look
tomorrow.
Thanks,
John
On Fri, Oct 30, 2020 at 6:41 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Fri Oct 30 06:40:44 2020: Request 97283 was acted upon.
> Transaction: Ticket created by edward.strobach at noaa.gov
> Queue: met_help
> Subject: plots no longer being generated
> Owner: Nobody
> Requestors: edward.strobach at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>
>
> Good morning,
>
> I was preparing to carry over some plots to my local machine when I
> realized that plots have not been generated for days. I decided to
> manually copy the contents from the file that I use to process data
and
> generate plots found here: *
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
> *
>
> After copying the contents over from the file and pasting it to the
command
> line, I see the expected printout begin. However, instead of
proceeding,
> it hangs up and does not go past the line shown in red below:
>
>
>
>
>
> *0:00', '2020-10-26 03:00:00', '2020-10-26 04:00:00', '2020-10-26
> 05:00:00', '2020-10-26 06:00:00', '2020-10-26 07:00:00', '2020-10-26
> 08:00:00', '2020-10-26 09:00:00', '2020-10-26 10:00:00', '2020-10-26
> 11:00:00', '2020-10-26 12:00:00', '2020-10-26 13:00:00', '2020-10-26
> 14:00:00', '2020-10-26 15:00:00', '2020-10-26 16:00:00', '2020-10-26
> 17:00:00', '2020-10-26 18:00:00', '2020-10-26 19:00:00', '2020-10-26
> 20:00:00', '2020-10-26 21:00:00', '2020-10-26 22:00:00', '2020-10-26
> 23:00:00') AND BINARY h.model IN ('PARA6', 'PARA6A', 'PARA6A_BC',
> 'PARA6_BC', 'PROD', 'PROD_BC') AND BINARY if( (select
fcst_lead_offset
> FROM model_fcst_lead_offset WHERE model = h.model) is NULL ,
ld.fcst_lead ,
> ld.fcst_lead + (select fcst_lead_offset FROM model_fcst_lead_offset
WHERE
> model = h.model) ) IN ('0', '10000', '20000', '30000', '40000',
'50000',
> '60000', '70000', '80000', '90000', '100000', '110000', '120000',
'130000',
> '140000', '150000', '160000', '170000', '180000', '190000',
'200000',
> '210000', '220000', '230000', '240000', '250000', '260000',
'270000',
> '280000', '290000', '300000', '310000', '320000', '330000',
'340000',
> '350000', '360000', '370000', '380000', '390000', '400000',
'410000',
> '420000', '430000', '440000', '450000', '460000', '470000',
'480000') AND
> BINARY h.fcst_var = 'PMTF' AND ld.stat_header_id =
h.stat_header_id;*
>
> The xml file looks fine. It's the same xml files that I've been
using for
> a while. Looking at the err file when this step is run using a
cron, I
> find that the above printout is not included (
>
>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/err.Time_Series_Aggregate*).
> Overall I don't see anything suspect in the error file. The out
file (
> *out.Time_Series_Aggregate*) suggests everything is fine. I see the
above
> printout. I just don't know why my plots are not being generated
and have
> not been generated for days.
>
> I can elaborate with my detail if desired.
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: plots no longer being generated
From: John Halley Gotway
Time: Tue Nov 03 17:27:57 2020
Hi Ed,
Sorry for the delay. I took a look on wcoss today. You describe that
your
plotting script has stopped working but you're not sure why.
I ran across a rather large directory:
ls /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
| wc
-w
That has 71,398 xml files in there. But I see timestamps in those
filenames
from 20200921 to 20200925.
Is it the case that this stopped working on 20200925? Or are those
dated
filenames remnants from older work?
So it hasn't been working for over a month. Is that right?
John
On Sun, Nov 1, 2020 at 8:50 PM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Sun Nov 01 20:49:52 2020: Request 97283 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: plots no longer being generated
> Owner: johnhg
> Requestors: edward.strobach at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Wed Nov 04 06:21:35 2020
Hi John,
No problem.
I'm including my responses below:
Sorry for the delay. I took a look on wcoss today. You describe that
your
plotting script has stopped working but you're not sure why.
It seems more intermittent. Take a look at the example below. You
can see
that for some reason there are gaps in the plots being generated.
*-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4 14:52
PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1 Edward.Strobach
emcmodel 41781 Oct 5 15:15
PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1 Edward.Strobach
emcmodel 56691 Oct 19 15:02
PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1 Edward.Strobach
emcmodel 55548 Oct 20 16:08
PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1 Edward.Strobach
emcmodel 55598 Oct 21 16:08
PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1 Edward.Strobach
emcmodel 55599 Oct 22 16:09
PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1 Edward.Strobach
emcmodel 55352 Oct 24 16:34
PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1 Edward.Strobach
emcmodel 26175 Oct 31 08:20
PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1 Edward.Strobach
emcmodel 26178 Nov 1 08:20
PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1 Edward.Strobach
emcmodel 26160 Nov 3 08:22
PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1 Edward.Strobach
emcmodel 25354 Nov 3 08:22 TMP_FULL_102020_2020-10-01_2020-10-31.png*
You can tell when a plot is successful by looking at the file size. I
highlighted an example of that in red. However, the files showing
about
25000 are the ones where no line plots are being produced. You can
also
see that stat files are generated everyday, so as long as I'm loading
the
data, which I am, I should be able to use the xml files to generate
the
plots. I'm attaching a directory list from wcoss that shows the dates
where data is produced.
*20190801 20190817 20200624 20200710 20200726 20200812 20200828
20201003 2020101920190802 20190818 20200625 20200711 20200727
20200813 20200829 20201004 2020102020190803 20190819 20200626
20200712 20200728 20200814 20200830 20201005 2020102120190804
20190820 20200627 20200713 20200729 20200815 20200831 20201006
2020102220190805 20190821 20200628 20200714 20200730 20200816
20200921 20201007 2020102320190806 20190822 20200629 20200715
20200731 20200817 20200922 20201008 2020102420190807 20190823
20200630 20200716 20200801 20200818 20200923 20201009
2020102520190808 20190824 20200701 20200717 20200802 20200819
20200924 20201010 2020102620190809 20190825 20200702 20200718
20200803 20200820 20200925 20201011 2020102720190810 20190826
20200703 20200719 20200804 20200821 20200926 20201012
2020102820190811 20190827 20200704 20200720 20200805 20200822
20200927 20201013 2020102920190812 20190828 20200705 20200721
20200806 20200823 20200928 20201014 2020103020190813 20190829
20200706 20200722 20200807 20200824 20200929 20201015
2020103120190814 20190830 20200707 20200723 20200808 20200825
20200930 20201016 2020110120190815 20200622 20200708 20200724
20200810 20200826 20201001 20201017 2020110220190816 20200623
20200709 20200725 20200811 20200827 20201002 20201018*
Furthermore, what's interesting is that I have mixed success with
other
cases. Using the same field - PMTF - for the same cycle - the 06z
cycle -
you can see that some of the dates where data was plotted for EAST did
not
necessarily plot for FULL. This surprises me because EAST is a subset
of
FULL. See below:
*-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4 14:54
PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1 Edward.Strobach
emcmodel 49039 Oct 5 15:17
PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1 Edward.Strobach
emcmodel 56791 Oct 20 16:09
PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1 Edward.Strobach
emcmodel 56834 Oct 21 16:09
PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1 Edward.Strobach
emcmodel 56921 Oct 22 16:11
PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1 Edward.Strobach
emcmodel 56518 Oct 24 16:36
PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1 Edward.Strobach
emcmodel 61187 Oct 31 08:21
PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1 Edward.Strobach
emcmodel 26475 Nov 2 08:22
PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1 Edward.Strobach
emcmodel 26455 Nov 3 08:23 PMTF_EAST_102020_2020-10-01_2020-10-
31.png*
What I find equally concerning as that sometimes plots are incorrectly
stored in the wrong directory list as highlighted in green above. I've
consulted the plot xml file for PMTF, for instance, and find nothing
problematic. The only setting that might pose an issue when it comes
to
generating line plots is when I set event_equal to true. However, all
the
stat files are populated for all experiments. In the case of ozone you
see
something different. I find that ozone tends to result in populated
plots
but with intermittent reporting different than PMTF - see below:
*[Edward.Strobach at m72a1 202010]$ ls -ltrtotal 6144-rw-r--r-- 1
Edward.Strobach emcmodel 37861 Oct 4 10:36
OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
emcmodel 54318 Oct 6 10:51
OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
Edward.Strobach
emcmodel 54305 Oct 7 10:49
OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
Edward.Strobach
emcmodel 62179 Oct 13 13:25
OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
Edward.Strobach
emcmodel 63811 Oct 19 12:18
OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
Edward.Strobach
emcmodel 60112 Oct 20 12:21
OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
emcmodel 60165 Oct 21 12:33
OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
emcmodel 60160 Oct 22 12:21
OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
emcmodel 60056 Oct 23 12:40
OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
Edward.Strobach
emcmodel 58517 Oct 24 13:31
OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
emcmodel 58062 Oct 31 14:17
OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
emcmodel 58119 Nov 1 14:19 OZCON_FULL_102020_2020-10-01_2020-10-
29.png*
Similar issues have been found for other fields, cycles, and regions
including verification being done for meteorology.
I can't seem to reason this intermittency and inconsistency between
reporting since the xml files are correct and I'm using the same
procedure
that I've always used. I do realize that I need to start creating a
better
way to manage all these files that are being produced, both in terms
of
data/plots and xml files.
I ran across a rather large directory:
ls /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
| wc
-w
That has 71,398 xml files in there. But I see timestamps in those
filenames
from 20200921 to 20200925.
Is it the case that this stopped working on 20200925? Or are those
dated
filenames remnants from older work?
So it hasn't been working for over a month. Is that right?
There are a lot of files and I need to begin managing that better. I
think
during that time there was a machine switch so it stopped on that day.
I've been able to produce plots even now, but it's not consistent in
reporting while some regions generate plots and others do not. I've
reduced the level processing in hopes that I don't hit a walk clock
issue
since I'm submitting batch jobs. However, I've run this step outside
of
batch and it always seems to work. This issue seems to happen with
batch
and it is not at all clear why. Let me know what else you need from
me to
better coordinate what I would need to do next. Thanks.
On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Hi Ed,
>
> Sorry for the delay. I took a look on wcoss today. You describe that
your
> plotting script has stopped working but you're not sure why.
>
> I ran across a rather large directory:
> ls
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls |
wc
> -w
>
> That has 71,398 xml files in there. But I see timestamps in those
filenames
> from 20200921 to 20200925.
>
> Is it the case that this stopped working on 20200925? Or are those
dated
> filenames remnants from older work?
>
> So it hasn't been working for over a month. Is that right?
>
> John
>
> On Sun, Nov 1, 2020 at 8:50 PM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Sun Nov 01 20:49:52 2020: Request 97283 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by RT_System
> > Queue: met_help
> > Subject: plots no longer being generated
> > Owner: johnhg
> > Requestors: edward.strobach at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >
> >
> > This transaction appears to have no content
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: John Halley Gotway
Time: Wed Nov 04 08:56:52 2020
Ed,
It's really difficult for me to see an obvious problem that's causing
the
behavior you describe.
Unfortunately, the red and green highlighting you describe doesn't
show up
in the Gmail web client. There must be some mail system
incompatibility
there.
Looking at your bsub script:
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
I am struck by the number of individual calls you're making to
METviewer:
2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot types = 656
calls to
METviewer
When making plots via the METviewer GUI, it's true that 1 XML = 1
plot.
However, when making plots via the batch engine, the plot XML can be
set up
to create many, many plots. While I have no direct proof of this, it
might
be the case that executing 600+ batch jobs for METviewer is leading to
trouble on the system. One thing to try would be setting up the XML to
make
many, many more plots in each call. Then check to see if running
METviewer
in this way is more stable.
I don't think I fully appreciate all of the logic that lives in
Time_Series_file.bsub, Plot_XML_builder.py, and
test_Time_Series_AGG.sh.
However, it *might* be possible to construct a single XML file which
produces all 656 plots.
Let me know what you think and how you'd like to proceed.
Thanks,
John
On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>
> Hi John,
>
> No problem.
>
> I'm including my responses below:
> Sorry for the delay. I took a look on wcoss today. You describe that
your
> plotting script has stopped working but you're not sure why.
> It seems more intermittent. Take a look at the example below. You
can see
> that for some reason there are gaps in the plots being generated.
>
>
>
>
>
>
>
>
>
>
> *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4 14:52
> PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 41781 Oct 5 15:15
> PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 56691 Oct 19 15:02
> PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 55548 Oct 20 16:08
> PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 55598 Oct 21 16:08
> PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 55599 Oct 22 16:09
> PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 55352 Oct 24 16:34
> PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 26175 Oct 31 08:20
> PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 26178 Nov 1 08:20
> PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 26160 Nov 3 08:22
> PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 25354 Nov 3 08:22 TMP_FULL_102020_2020-10-01_2020-10-
31.png*
> You can tell when a plot is successful by looking at the file size.
I
> highlighted an example of that in red. However, the files showing
about
> 25000 are the ones where no line plots are being produced. You can
also
> see that stat files are generated everyday, so as long as I'm
loading the
> data, which I am, I should be able to use the xml files to generate
the
> plots. I'm attaching a directory list from wcoss that shows the
dates
> where data is produced.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *20190801 20190817 20200624 20200710 20200726 20200812
20200828
> 20201003 2020101920190802 20190818 20200625 20200711 20200727
> 20200813 20200829 20201004 2020102020190803 20190819 20200626
> 20200712 20200728 20200814 20200830 20201005 2020102120190804
> 20190820 20200627 20200713 20200729 20200815 20200831
20201006
> 2020102220190805 20190821 20200628 20200714 20200730 20200816
> 20200921 20201007 2020102320190806 20190822 20200629 20200715
> 20200731 20200817 20200922 20201008 2020102420190807 20190823
> 20200630 20200716 20200801 20200818 20200923 20201009
> 2020102520190808 20190824 20200701 20200717 20200802 20200819
> 20200924 20201010 2020102620190809 20190825 20200702 20200718
> 20200803 20200820 20200925 20201011 2020102720190810 20190826
> 20200703 20200719 20200804 20200821 20200926 20201012
> 2020102820190811 20190827 20200704 20200720 20200805 20200822
> 20200927 20201013 2020102920190812 20190828 20200705 20200721
> 20200806 20200823 20200928 20201014 2020103020190813 20190829
> 20200706 20200722 20200807 20200824 20200929 20201015
> 2020103120190814 20190830 20200707 20200723 20200808 20200825
> 20200930 20201016 2020110120190815 20200622 20200708 20200724
> 20200810 20200826 20201001 20201017 2020110220190816 20200623
> 20200709 20200725 20200811 20200827 20201002 20201018*
>
> Furthermore, what's interesting is that I have mixed success with
other
> cases. Using the same field - PMTF - for the same cycle - the 06z
cycle -
> you can see that some of the dates where data was plotted for EAST
did not
> necessarily plot for FULL. This surprises me because EAST is a
subset of
> FULL. See below:
>
>
>
>
>
>
>
>
> *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4 14:54
> PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 49039 Oct 5 15:17
> PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 56791 Oct 20 16:09
> PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 56834 Oct 21 16:09
> PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 56921 Oct 22 16:11
> PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 56518 Oct 24 16:36
> PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 61187 Oct 31 08:21
> PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 26475 Nov 2 08:22
> PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 26455 Nov 3 08:23 PMTF_EAST_102020_2020-10-01_2020-10-
31.png*
>
> What I find equally concerning as that sometimes plots are
incorrectly
> stored in the wrong directory list as highlighted in green above.
I've
> consulted the plot xml file for PMTF, for instance, and find nothing
> problematic. The only setting that might pose an issue when it
comes to
> generating line plots is when I set event_equal to true. However,
all the
> stat files are populated for all experiments. In the case of ozone
you see
> something different. I find that ozone tends to result in populated
plots
> but with intermittent reporting different than PMTF - see below:
>
>
>
>
>
>
>
>
>
>
>
>
>
> *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal 6144-rw-r--r-- 1
> Edward.Strobach emcmodel 37861 Oct 4 10:36
> OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 54318 Oct 6 10:51
> OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 54305 Oct 7 10:49
> OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 62179 Oct 13 13:25
> OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 63811 Oct 19 12:18
> OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 60112 Oct 20 12:21
> OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 60165 Oct 21 12:33
> OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 60160 Oct 22 12:21
> OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 60056 Oct 23 12:40
> OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 58517 Oct 24 13:31
> OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 58062 Oct 31 14:17
> OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
> emcmodel 58119 Nov 1 14:19 OZCON_FULL_102020_2020-10-01_2020-10-
29.png*
>
> Similar issues have been found for other fields, cycles, and regions
> including verification being done for meteorology.
> I can't seem to reason this intermittency and inconsistency between
> reporting since the xml files are correct and I'm using the same
procedure
> that I've always used. I do realize that I need to start creating a
better
> way to manage all these files that are being produced, both in terms
of
> data/plots and xml files.
>
> I ran across a rather large directory:
> ls
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls |
wc
> -w
>
> That has 71,398 xml files in there. But I see timestamps in those
filenames
> from 20200921 to 20200925.
>
> Is it the case that this stopped working on 20200925? Or are those
dated
> filenames remnants from older work?
>
> So it hasn't been working for over a month. Is that right?
>
> There are a lot of files and I need to begin managing that better.
I think
> during that time there was a machine switch so it stopped on that
day.
> I've been able to produce plots even now, but it's not consistent in
> reporting while some regions generate plots and others do not. I've
> reduced the level processing in hopes that I don't hit a walk clock
issue
> since I'm submitting batch jobs. However, I've run this step
outside of
> batch and it always seems to work. This issue seems to happen with
batch
> and it is not at all clear why. Let me know what else you need from
me to
> better coordinate what I would need to do next. Thanks.
>
>
>
>
> On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hi Ed,
> >
> > Sorry for the delay. I took a look on wcoss today. You describe
that your
> > plotting script has stopped working but you're not sure why.
> >
> > I ran across a rather large directory:
> > ls
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls |
> wc
> > -w
> >
> > That has 71,398 xml files in there. But I see timestamps in those
> filenames
> > from 20200921 to 20200925.
> >
> > Is it the case that this stopped working on 20200925? Or are those
dated
> > filenames remnants from older work?
> >
> > So it hasn't been working for over a month. Is that right?
> >
> > John
> >
> > On Sun, Nov 1, 2020 at 8:50 PM The RT System itself via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Sun Nov 01 20:49:52 2020: Request 97283 was acted upon.
> > > Transaction: Given to johnhg (John Halley Gotway) by RT_System
> > > Queue: met_help
> > > Subject: plots no longer being generated
> > > Owner: johnhg
> > > Requestors: edward.strobach at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Wed Nov 04 09:16:36 2020
Hi John,
I know there are a lot of calls and speculate that this could be part
of
the problem. I don't think that I'm following best practice by
approaching
it in this way. Generating plots usually works well if I process a
subset
of the computations outside of batch scripting. The python script was
given to me by Logan, but I have since made significant changes to it
in
order to dynamically fill in the xml file based on parameters set.
The
test*.sh script was also given to me. I created the Time_Series*bsub
as a
way to process a series of plots based on cycle time, region, etc. I
think
the first two scripts are solid and do well for what I need to get
done.
It's the Time_Series*bsub script that bothers me a bit.
I can't see how a single XML file can process all these plots in a
single
setting based on what I've learned thus far. How would one go about
that?
I would be open going in that direction if you can provide some
guidance.
Unfortunately, I haven't gotten a lot of feedback or dialogue exchange
about what I've done so far at EMC with this new set-up. This has
also
been part of the problem. I can't seem to get them to focus so that
we can
prioritize. I know that I'm doing the best I can. If I can learn how
to
go about this more efficiently it would be of great help. I can then
expand this to other verification as venture to other projects in the
future. I know you're busy with many things, so I'm willing to go
along
with your schedule as you find an opening.
Thanks
On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Ed,
>
> It's really difficult for me to see an obvious problem that's
causing the
> behavior you describe.
>
> Unfortunately, the red and green highlighting you describe doesn't
show up
> in the Gmail web client. There must be some mail system
incompatibility
> there.
>
> Looking at your bsub script:
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
>
> I am struck by the number of individual calls you're making to
METviewer:
> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot types = 656
calls to
> METviewer
>
> When making plots via the METviewer GUI, it's true that 1 XML = 1
plot.
> However, when making plots via the batch engine, the plot XML can be
set up
> to create many, many plots. While I have no direct proof of this, it
might
> be the case that executing 600+ batch jobs for METviewer is leading
to
> trouble on the system. One thing to try would be setting up the XML
to make
> many, many more plots in each call. Then check to see if running
METviewer
> in this way is more stable.
>
> I don't think I fully appreciate all of the logic that lives in
> Time_Series_file.bsub, Plot_XML_builder.py, and
test_Time_Series_AGG.sh.
> However, it *might* be possible to construct a single XML file which
> produces all 656 plots.
>
> Let me know what you think and how you'd like to proceed.
>
> Thanks,
> John
>
> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >
> > Hi John,
> >
> > No problem.
> >
> > I'm including my responses below:
> > Sorry for the delay. I took a look on wcoss today. You describe
that your
> > plotting script has stopped working but you're not sure why.
> > It seems more intermittent. Take a look at the example below.
You can
> see
> > that for some reason there are gaps in the plots being generated.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4 14:52
> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 41781 Oct 5 15:15
> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 56691 Oct 19 15:02
> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 55548 Oct 20 16:08
> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 55598 Oct 21 16:08
> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 55599 Oct 22 16:09
> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 55352 Oct 24 16:34
> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 26175 Oct 31 08:20
> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 26178 Nov 1 08:20
> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 26160 Nov 3 08:22
> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 25354 Nov 3 08:22 TMP_FULL_102020_2020-10-01_2020-10-
31.png*
> > You can tell when a plot is successful by looking at the file
size. I
> > highlighted an example of that in red. However, the files showing
about
> > 25000 are the ones where no line plots are being produced. You
can also
> > see that stat files are generated everyday, so as long as I'm
loading the
> > data, which I am, I should be able to use the xml files to
generate the
> > plots. I'm attaching a directory list from wcoss that shows the
dates
> > where data is produced.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *20190801 20190817 20200624 20200710 20200726 20200812
20200828
> > 20201003 2020101920190802 20190818 20200625 20200711
20200727
> > 20200813 20200829 20201004 2020102020190803 20190819
20200626
> > 20200712 20200728 20200814 20200830 20201005
2020102120190804
> > 20190820 20200627 20200713 20200729 20200815 20200831
20201006
> > 2020102220190805 20190821 20200628 20200714 20200730
20200816
> > 20200921 20201007 2020102320190806 20190822 20200629
20200715
> > 20200731 20200817 20200922 20201008 2020102420190807
20190823
> > 20200630 20200716 20200801 20200818 20200923 20201009
> > 2020102520190808 20190824 20200701 20200717 20200802
20200819
> > 20200924 20201010 2020102620190809 20190825 20200702
20200718
> > 20200803 20200820 20200925 20201011 2020102720190810
20190826
> > 20200703 20200719 20200804 20200821 20200926 20201012
> > 2020102820190811 20190827 20200704 20200720 20200805
20200822
> > 20200927 20201013 2020102920190812 20190828 20200705
20200721
> > 20200806 20200823 20200928 20201014 2020103020190813
20190829
> > 20200706 20200722 20200807 20200824 20200929 20201015
> > 2020103120190814 20190830 20200707 20200723 20200808
20200825
> > 20200930 20201016 2020110120190815 20200622 20200708
20200724
> > 20200810 20200826 20201001 20201017 2020110220190816
20200623
> > 20200709 20200725 20200811 20200827 20201002 20201018*
> >
> > Furthermore, what's interesting is that I have mixed success with
other
> > cases. Using the same field - PMTF - for the same cycle - the 06z
cycle
> -
> > you can see that some of the dates where data was plotted for EAST
did
> not
> > necessarily plot for FULL. This surprises me because EAST is a
subset of
> > FULL. See below:
> >
> >
> >
> >
> >
> >
> >
> >
> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4 14:54
> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 49039 Oct 5 15:17
> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 56791 Oct 20 16:09
> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 56834 Oct 21 16:09
> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 56921 Oct 22 16:11
> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 56518 Oct 24 16:36
> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 61187 Oct 31 08:21
> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 26475 Nov 2 08:22
> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 26455 Nov 3 08:23 PMTF_EAST_102020_2020-10-01_2020-10-
31.png*
> >
> > What I find equally concerning as that sometimes plots are
incorrectly
> > stored in the wrong directory list as highlighted in green above.
I've
> > consulted the plot xml file for PMTF, for instance, and find
nothing
> > problematic. The only setting that might pose an issue when it
comes to
> > generating line plots is when I set event_equal to true. However,
all
> the
> > stat files are populated for all experiments. In the case of ozone
you
> see
> > something different. I find that ozone tends to result in
populated plots
> > but with intermittent reporting different than PMTF - see below:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal 6144-rw-r--r-- 1
> > Edward.Strobach emcmodel 37861 Oct 4 10:36
> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 54318 Oct 6 10:51
> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 54305 Oct 7 10:49
> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 62179 Oct 13 13:25
> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 63811 Oct 19 12:18
> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 60112 Oct 20 12:21
> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 60165 Oct 21 12:33
> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 60160 Oct 22 12:21
> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 60056 Oct 23 12:40
> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 58517 Oct 24 13:31
> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 58062 Oct 31 14:17
> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
> > emcmodel 58119 Nov 1 14:19 OZCON_FULL_102020_2020-10-01_2020-10-
29.png*
> >
> > Similar issues have been found for other fields, cycles, and
regions
> > including verification being done for meteorology.
> > I can't seem to reason this intermittency and inconsistency
between
> > reporting since the xml files are correct and I'm using the same
> procedure
> > that I've always used. I do realize that I need to start creating
a
> better
> > way to manage all these files that are being produced, both in
terms of
> > data/plots and xml files.
> >
> > I ran across a rather large directory:
> > ls
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls |
> wc
> > -w
> >
> > That has 71,398 xml files in there. But I see timestamps in those
> filenames
> > from 20200921 to 20200925.
> >
> > Is it the case that this stopped working on 20200925? Or are those
dated
> > filenames remnants from older work?
> >
> > So it hasn't been working for over a month. Is that right?
> >
> > There are a lot of files and I need to begin managing that better.
I
> think
> > during that time there was a machine switch so it stopped on that
day.
> > I've been able to produce plots even now, but it's not consistent
in
> > reporting while some regions generate plots and others do not.
I've
> > reduced the level processing in hopes that I don't hit a walk
clock issue
> > since I'm submitting batch jobs. However, I've run this step
outside of
> > batch and it always seems to work. This issue seems to happen with
batch
> > and it is not at all clear why. Let me know what else you need
from me
> to
> > better coordinate what I would need to do next. Thanks.
> >
> >
> >
> >
> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Hi Ed,
> > >
> > > Sorry for the delay. I took a look on wcoss today. You describe
that
> your
> > > plotting script has stopped working but you're not sure why.
> > >
> > > I ran across a rather large directory:
> > > ls
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls |
> > wc
> > > -w
> > >
> > > That has 71,398 xml files in there. But I see timestamps in
those
> > filenames
> > > from 20200921 to 20200925.
> > >
> > > Is it the case that this stopped working on 20200925? Or are
those
> dated
> > > filenames remnants from older work?
> > >
> > > So it hasn't been working for over a month. Is that right?
> > >
> > > John
> > >
> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System itself via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Sun Nov 01 20:49:52 2020: Request 97283 was acted upon.
> > > > Transaction: Given to johnhg (John Halley Gotway) by RT_System
> > > > Queue: met_help
> > > > Subject: plots no longer being generated
> > > > Owner: johnhg
> > > > Requestors: edward.strobach at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > >
> > > >
> > > >
> > > > This transaction appears to have no content
> > > >
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Wed Nov 04 09:22:26 2020
Oh, and one more thing..
The green text that didn't show up was meant to highlight the fact
that
sometimes plots that shouldn't be stored in certain directories end up
there. The pasted example below shows one instance where OZMAX* was
stored
where TMP* should have been stored.
-rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5 10:52
TMP_FULL_102020_2020-10-01_2020-10-02.png
-rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6 10:52
TMP_FULL_102020_2020-10-01_2020-10-03.png
-rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7 10:00
TMP_FULL_102020_2020-10-01_2020-10-04.png
-rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13 13:49
TMP_FULL_102020_2020-10-01_2020-10-10.png
-rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14 11:31
TMP_FULL_102020_2020-10-01_2020-10-11.png
-rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19 11:37
TMP_FULL_102020_2020-10-01_2020-10-12.png
-rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20 11:35
OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
-rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21 11:45
TMP_FULL_102020_2020-10-01_2020-10-18.png
-rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22 11:42
TMP_FULL_102020_2020-10-01_2020-10-19.png
-rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23 11:50
TMP_FULL_102020_2020-10-01_2020-10-20.png
-rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24 12:57
TMP_FULL_102020_2020-10-01_2020-10-21.png
-rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31 08:47
TMP_FULL_102020_2020-10-01_2020-10-28.png
-rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2 08:50
TMP_FULL_102020_2020-10-01_2020-10-30.png
-rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3 08:52
TMP_FULL_102020_2020-10-01_2020-10-31.png
On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Hi John,
>
> I know there are a lot of calls and speculate that this could be
part of
> the problem. I don't think that I'm following best practice by
approaching
> it in this way. Generating plots usually works well if I process a
subset
> of the computations outside of batch scripting. The python script
was
> given to me by Logan, but I have since made significant changes to
it in
> order to dynamically fill in the xml file based on parameters set.
The
> test*.sh script was also given to me. I created the
Time_Series*bsub as a
> way to process a series of plots based on cycle time, region, etc. I
think
> the first two scripts are solid and do well for what I need to get
done.
> It's the Time_Series*bsub script that bothers me a bit.
>
> I can't see how a single XML file can process all these plots in a
single
> setting based on what I've learned thus far. How would one go about
that?
> I would be open going in that direction if you can provide some
guidance.
>
> Unfortunately, I haven't gotten a lot of feedback or dialogue
exchange
> about what I've done so far at EMC with this new set-up. This has
also
> been part of the problem. I can't seem to get them to focus so that
we can
> prioritize. I know that I'm doing the best I can. If I can learn
how to
> go about this more efficiently it would be of great help. I can
then
> expand this to other verification as venture to other projects in
the
> future. I know you're busy with many things, so I'm willing to go
along
> with your schedule as you find an opening.
>
> Thanks
>
> On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Ed,
>>
>> It's really difficult for me to see an obvious problem that's
causing the
>> behavior you describe.
>>
>> Unfortunately, the red and green highlighting you describe doesn't
show up
>> in the Gmail web client. There must be some mail system
incompatibility
>> there.
>>
>> Looking at your bsub script:
>>
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
>>
>> I am struck by the number of individual calls you're making to
METviewer:
>> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot types = 656
calls to
>> METviewer
>>
>> When making plots via the METviewer GUI, it's true that 1 XML = 1
plot.
>> However, when making plots via the batch engine, the plot XML can
be set
>> up
>> to create many, many plots. While I have no direct proof of this,
it might
>> be the case that executing 600+ batch jobs for METviewer is leading
to
>> trouble on the system. One thing to try would be setting up the XML
to
>> make
>> many, many more plots in each call. Then check to see if running
METviewer
>> in this way is more stable.
>>
>> I don't think I fully appreciate all of the logic that lives in
>> Time_Series_file.bsub, Plot_XML_builder.py, and
test_Time_Series_AGG.sh.
>> However, it *might* be possible to construct a single XML file
which
>> produces all 656 plots.
>>
>> Let me know what you think and how you'd like to proceed.
>>
>> Thanks,
>> John
>>
>> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>> >
>> > Hi John,
>> >
>> > No problem.
>> >
>> > I'm including my responses below:
>> > Sorry for the delay. I took a look on wcoss today. You describe
that
>> your
>> > plotting script has stopped working but you're not sure why.
>> > It seems more intermittent. Take a look at the example below.
You can
>> see
>> > that for some reason there are gaps in the plots being generated.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4 14:52
>> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 41781 Oct 5 15:15
>> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 56691 Oct 19 15:02
>> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 55548 Oct 20 16:08
>> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 55598 Oct 21 16:08
>> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 55599 Oct 22 16:09
>> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 55352 Oct 24 16:34
>> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 26175 Oct 31 08:20
>> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 26178 Nov 1 08:20
>> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 26160 Nov 3 08:22
>> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 25354 Nov 3 08:22 TMP_FULL_102020_2020-10-01_2020-10-
31.png*
>> > You can tell when a plot is successful by looking at the file
size. I
>> > highlighted an example of that in red. However, the files
showing about
>> > 25000 are the ones where no line plots are being produced. You
can also
>> > see that stat files are generated everyday, so as long as I'm
loading
>> the
>> > data, which I am, I should be able to use the xml files to
generate the
>> > plots. I'm attaching a directory list from wcoss that shows the
dates
>> > where data is produced.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *20190801 20190817 20200624 20200710 20200726 20200812
20200828
>> > 20201003 2020101920190802 20190818 20200625 20200711
20200727
>> > 20200813 20200829 20201004 2020102020190803 20190819
20200626
>> > 20200712 20200728 20200814 20200830 20201005
2020102120190804
>> > 20190820 20200627 20200713 20200729 20200815 20200831
20201006
>> > 2020102220190805 20190821 20200628 20200714 20200730
20200816
>> > 20200921 20201007 2020102320190806 20190822 20200629
20200715
>> > 20200731 20200817 20200922 20201008 2020102420190807
20190823
>> > 20200630 20200716 20200801 20200818 20200923 20201009
>> > 2020102520190808 20190824 20200701 20200717 20200802
20200819
>> > 20200924 20201010 2020102620190809 20190825 20200702
20200718
>> > 20200803 20200820 20200925 20201011 2020102720190810
20190826
>> > 20200703 20200719 20200804 20200821 20200926 20201012
>> > 2020102820190811 20190827 20200704 20200720 20200805
20200822
>> > 20200927 20201013 2020102920190812 20190828 20200705
20200721
>> > 20200806 20200823 20200928 20201014 2020103020190813
20190829
>> > 20200706 20200722 20200807 20200824 20200929 20201015
>> > 2020103120190814 20190830 20200707 20200723 20200808
20200825
>> > 20200930 20201016 2020110120190815 20200622 20200708
20200724
>> > 20200810 20200826 20201001 20201017 2020110220190816
20200623
>> > 20200709 20200725 20200811 20200827 20201002 20201018*
>> >
>> > Furthermore, what's interesting is that I have mixed success with
other
>> > cases. Using the same field - PMTF - for the same cycle - the
06z
>> cycle -
>> > you can see that some of the dates where data was plotted for
EAST did
>> not
>> > necessarily plot for FULL. This surprises me because EAST is a
subset
>> of
>> > FULL. See below:
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4 14:54
>> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 49039 Oct 5 15:17
>> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 56791 Oct 20 16:09
>> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 56834 Oct 21 16:09
>> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 56921 Oct 22 16:11
>> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 56518 Oct 24 16:36
>> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 61187 Oct 31 08:21
>> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 26475 Nov 2 08:22
>> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 26455 Nov 3 08:23 PMTF_EAST_102020_2020-10-01_2020-10-
31.png*
>> >
>> > What I find equally concerning as that sometimes plots are
incorrectly
>> > stored in the wrong directory list as highlighted in green above.
I've
>> > consulted the plot xml file for PMTF, for instance, and find
nothing
>> > problematic. The only setting that might pose an issue when it
comes to
>> > generating line plots is when I set event_equal to true.
However, all
>> the
>> > stat files are populated for all experiments. In the case of
ozone you
>> see
>> > something different. I find that ozone tends to result in
populated
>> plots
>> > but with intermittent reporting different than PMTF - see below:
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal 6144-rw-r--r-- 1
>> > Edward.Strobach emcmodel 37861 Oct 4 10:36
>> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 54318 Oct 6 10:51
>> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 54305 Oct 7 10:49
>> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 62179 Oct 13 13:25
>> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 63811 Oct 19 12:18
>> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 60112 Oct 20 12:21
>> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 60165 Oct 21 12:33
>> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 60160 Oct 22 12:21
>> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 60056 Oct 23 12:40
>> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 58517 Oct 24 13:31
>> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 58062 Oct 31 14:17
>> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
>> > emcmodel 58119 Nov 1 14:19 OZCON_FULL_102020_2020-10-01_2020-10-
29.png*
>> >
>> > Similar issues have been found for other fields, cycles, and
regions
>> > including verification being done for meteorology.
>> > I can't seem to reason this intermittency and inconsistency
between
>> > reporting since the xml files are correct and I'm using the same
>> procedure
>> > that I've always used. I do realize that I need to start
creating a
>> better
>> > way to manage all these files that are being produced, both in
terms of
>> > data/plots and xml files.
>> >
>> > I ran across a rather large directory:
>> > ls
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls |
>> wc
>> > -w
>> >
>> > That has 71,398 xml files in there. But I see timestamps in those
>> filenames
>> > from 20200921 to 20200925.
>> >
>> > Is it the case that this stopped working on 20200925? Or are
those dated
>> > filenames remnants from older work?
>> >
>> > So it hasn't been working for over a month. Is that right?
>> >
>> > There are a lot of files and I need to begin managing that
better. I
>> think
>> > during that time there was a machine switch so it stopped on that
day.
>> > I've been able to produce plots even now, but it's not consistent
in
>> > reporting while some regions generate plots and others do not.
I've
>> > reduced the level processing in hopes that I don't hit a walk
clock
>> issue
>> > since I'm submitting batch jobs. However, I've run this step
outside of
>> > batch and it always seems to work. This issue seems to happen
with batch
>> > and it is not at all clear why. Let me know what else you need
from me
>> to
>> > better coordinate what I would need to do next. Thanks.
>> >
>> >
>> >
>> >
>> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > Hi Ed,
>> > >
>> > > Sorry for the delay. I took a look on wcoss today. You describe
that
>> your
>> > > plotting script has stopped working but you're not sure why.
>> > >
>> > > I ran across a rather large directory:
>> > > ls
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>> |
>> > wc
>> > > -w
>> > >
>> > > That has 71,398 xml files in there. But I see timestamps in
those
>> > filenames
>> > > from 20200921 to 20200925.
>> > >
>> > > Is it the case that this stopped working on 20200925? Or are
those
>> dated
>> > > filenames remnants from older work?
>> > >
>> > > So it hasn't been working for over a month. Is that right?
>> > >
>> > > John
>> > >
>> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System itself via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > Sun Nov 01 20:49:52 2020: Request 97283 was acted upon.
>> > > > Transaction: Given to johnhg (John Halley Gotway) by
RT_System
>> > > > Queue: met_help
>> > > > Subject: plots no longer being generated
>> > > > Owner: johnhg
>> > > > Requestors: edward.strobach at noaa.gov
>> > > > Status: new
>> > > > Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>> > >
>> > > >
>> > > >
>> > > > This transaction appears to have no content
>> > > >
>> > >
>> > >
>> >
>> > --
>> > Edward Strobach
>> > EMC/NCEP/NWS/
>> > IMSG Contractor
>> > Cubicle#: 2029
>> > 301-683-3717
>> >
>> >
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: John Halley Gotway
Time: Wed Nov 04 10:14:39 2020
Ed,
Image files showing up in an unexpected output directory likely point
to a
problem in the logic of the processing scripts. It's pretty unlikely
that
this behavior is caused by the METviewer code itself. But I've
definitely
been surprised before!
So what we really need here is an example of taking a plot from the
METviewer GUI and adapting it to make it generate multiple plots. I
did go
through examples of doing this in previous tutorials, but we didn't
start
recording them until last summer. I checked the July 2019 NRL tutorial
videos and don't see it there.
We've begun recording training videos for common topics like this. But
we
only have one for METviewer so far:
https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
But this definitely seems like a good candidate for a new training
video.
For now though, how about this...
- You use METviewer to manually make a single plot, formatting it how
you'd
like.
- You send me your sample plot and corresponding XML file.
- I'll tweak it to generate many plots via batch instead of just one.
- And then you can continue tweaking to add more permutations.
John
On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>
> Oh, and one more thing..
>
> The green text that didn't show up was meant to highlight the fact
that
> sometimes plots that shouldn't be stored in certain directories end
up
> there. The pasted example below shows one instance where OZMAX* was
stored
> where TMP* should have been stored.
>
> -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5 10:52
> TMP_FULL_102020_2020-10-01_2020-10-02.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6 10:52
> TMP_FULL_102020_2020-10-01_2020-10-03.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7 10:00
> TMP_FULL_102020_2020-10-01_2020-10-04.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13 13:49
> TMP_FULL_102020_2020-10-01_2020-10-10.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14 11:31
> TMP_FULL_102020_2020-10-01_2020-10-11.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19 11:37
> TMP_FULL_102020_2020-10-01_2020-10-12.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20 11:35
> OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21 11:45
> TMP_FULL_102020_2020-10-01_2020-10-18.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22 11:42
> TMP_FULL_102020_2020-10-01_2020-10-19.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23 11:50
> TMP_FULL_102020_2020-10-01_2020-10-20.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24 12:57
> TMP_FULL_102020_2020-10-01_2020-10-21.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31 08:47
> TMP_FULL_102020_2020-10-01_2020-10-28.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2 08:50
> TMP_FULL_102020_2020-10-01_2020-10-30.png
> -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3 08:52
> TMP_FULL_102020_2020-10-01_2020-10-31.png
>
> On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Hi John,
> >
> > I know there are a lot of calls and speculate that this could be
part of
> > the problem. I don't think that I'm following best practice by
> approaching
> > it in this way. Generating plots usually works well if I process
a
> subset
> > of the computations outside of batch scripting. The python script
was
> > given to me by Logan, but I have since made significant changes to
it in
> > order to dynamically fill in the xml file based on parameters set.
The
> > test*.sh script was also given to me. I created the
Time_Series*bsub as
> a
> > way to process a series of plots based on cycle time, region, etc.
I
> think
> > the first two scripts are solid and do well for what I need to get
done.
> > It's the Time_Series*bsub script that bothers me a bit.
> >
> > I can't see how a single XML file can process all these plots in a
single
> > setting based on what I've learned thus far. How would one go
about
> that?
> > I would be open going in that direction if you can provide some
guidance.
> >
> > Unfortunately, I haven't gotten a lot of feedback or dialogue
exchange
> > about what I've done so far at EMC with this new set-up. This has
also
> > been part of the problem. I can't seem to get them to focus so
that we
> can
> > prioritize. I know that I'm doing the best I can. If I can learn
how to
> > go about this more efficiently it would be of great help. I can
then
> > expand this to other verification as venture to other projects in
the
> > future. I know you're busy with many things, so I'm willing to go
along
> > with your schedule as you find an opening.
> >
> > Thanks
> >
> > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Ed,
> >>
> >> It's really difficult for me to see an obvious problem that's
causing
> the
> >> behavior you describe.
> >>
> >> Unfortunately, the red and green highlighting you describe
doesn't show
> up
> >> in the Gmail web client. There must be some mail system
incompatibility
> >> there.
> >>
> >> Looking at your bsub script:
> >>
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
> >>
> >> I am struck by the number of individual calls you're making to
> METviewer:
> >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot types = 656
calls
> to
> >> METviewer
> >>
> >> When making plots via the METviewer GUI, it's true that 1 XML = 1
plot.
> >> However, when making plots via the batch engine, the plot XML can
be set
> >> up
> >> to create many, many plots. While I have no direct proof of this,
it
> might
> >> be the case that executing 600+ batch jobs for METviewer is
leading to
> >> trouble on the system. One thing to try would be setting up the
XML to
> >> make
> >> many, many more plots in each call. Then check to see if running
> METviewer
> >> in this way is more stable.
> >>
> >> I don't think I fully appreciate all of the logic that lives in
> >> Time_Series_file.bsub, Plot_XML_builder.py, and
test_Time_Series_AGG.sh.
> >> However, it *might* be possible to construct a single XML file
which
> >> produces all 656 plots.
> >>
> >> Let me know what you think and how you'd like to proceed.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA Affiliate
via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >> >
> >> > Hi John,
> >> >
> >> > No problem.
> >> >
> >> > I'm including my responses below:
> >> > Sorry for the delay. I took a look on wcoss today. You describe
that
> >> your
> >> > plotting script has stopped working but you're not sure why.
> >> > It seems more intermittent. Take a look at the example below.
You
> can
> >> see
> >> > that for some reason there are gaps in the plots being
generated.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4 14:52
> >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 41781 Oct 5 15:15
> >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 56691 Oct 19 15:02
> >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 55548 Oct 20 16:08
> >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 55598 Oct 21 16:08
> >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 55599 Oct 22 16:09
> >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 55352 Oct 24 16:34
> >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 26175 Oct 31 08:20
> >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 26178 Nov 1 08:20
> >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 26160 Nov 3 08:22
> >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 25354 Nov 3 08:22 TMP_FULL_102020_2020-10-01_2020-10-
31.png*
> >> > You can tell when a plot is successful by looking at the file
size. I
> >> > highlighted an example of that in red. However, the files
showing
> about
> >> > 25000 are the ones where no line plots are being produced. You
can
> also
> >> > see that stat files are generated everyday, so as long as I'm
loading
> >> the
> >> > data, which I am, I should be able to use the xml files to
generate
> the
> >> > plots. I'm attaching a directory list from wcoss that shows
the dates
> >> > where data is produced.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > *20190801 20190817 20200624 20200710 20200726 20200812
20200828
> >> > 20201003 2020101920190802 20190818 20200625 20200711
20200727
> >> > 20200813 20200829 20201004 2020102020190803 20190819
20200626
> >> > 20200712 20200728 20200814 20200830 20201005
2020102120190804
> >> > 20190820 20200627 20200713 20200729 20200815 20200831
20201006
> >> > 2020102220190805 20190821 20200628 20200714 20200730
20200816
> >> > 20200921 20201007 2020102320190806 20190822 20200629
20200715
> >> > 20200731 20200817 20200922 20201008 2020102420190807
20190823
> >> > 20200630 20200716 20200801 20200818 20200923 20201009
> >> > 2020102520190808 20190824 20200701 20200717 20200802
20200819
> >> > 20200924 20201010 2020102620190809 20190825 20200702
20200718
> >> > 20200803 20200820 20200925 20201011 2020102720190810
20190826
> >> > 20200703 20200719 20200804 20200821 20200926 20201012
> >> > 2020102820190811 20190827 20200704 20200720 20200805
20200822
> >> > 20200927 20201013 2020102920190812 20190828 20200705
20200721
> >> > 20200806 20200823 20200928 20201014 2020103020190813
20190829
> >> > 20200706 20200722 20200807 20200824 20200929 20201015
> >> > 2020103120190814 20190830 20200707 20200723 20200808
20200825
> >> > 20200930 20201016 2020110120190815 20200622 20200708
20200724
> >> > 20200810 20200826 20201001 20201017 2020110220190816
20200623
> >> > 20200709 20200725 20200811 20200827 20201002 20201018*
> >> >
> >> > Furthermore, what's interesting is that I have mixed success
with
> other
> >> > cases. Using the same field - PMTF - for the same cycle - the
06z
> >> cycle -
> >> > you can see that some of the dates where data was plotted for
EAST did
> >> not
> >> > necessarily plot for FULL. This surprises me because EAST is a
subset
> >> of
> >> > FULL. See below:
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4 14:54
> >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 49039 Oct 5 15:17
> >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 56791 Oct 20 16:09
> >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 56834 Oct 21 16:09
> >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 56921 Oct 22 16:11
> >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 56518 Oct 24 16:36
> >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 61187 Oct 31 08:21
> >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 26475 Nov 2 08:22
> >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1
Edward.Strobach
> >> > emcmodel 26455 Nov 3 08:23
> PMTF_EAST_102020_2020-10-01_2020-10-31.png*
> >> >
> >> > What I find equally concerning as that sometimes plots are
incorrectly
> >> > stored in the wrong directory list as highlighted in green
above. I've
> >> > consulted the plot xml file for PMTF, for instance, and find
nothing
> >> > problematic. The only setting that might pose an issue when it
comes
> to
> >> > generating line plots is when I set event_equal to true.
However, all
> >> the
> >> > stat files are populated for all experiments. In the case of
ozone you
> >> see
> >> > something different. I find that ozone tends to result in
populated
> >> plots
> >> > but with intermittent reporting different than PMTF - see
below:
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal 6144-rw-r--r-- 1
> >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
> >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 54318 Oct 6 10:51
> >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 54305 Oct 7 10:49
> >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 62179 Oct 13 13:25
> >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 63811 Oct 19 12:18
> >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 60112 Oct 20 12:21
> >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 60165 Oct 21 12:33
> >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 60160 Oct 22 12:21
> >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 60056 Oct 23 12:40
> >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 58517 Oct 24 13:31
> >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 58062 Oct 31 14:17
> >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> Edward.Strobach
> >> > emcmodel 58119 Nov 1 14:19
> OZCON_FULL_102020_2020-10-01_2020-10-29.png*
> >> >
> >> > Similar issues have been found for other fields, cycles, and
regions
> >> > including verification being done for meteorology.
> >> > I can't seem to reason this intermittency and inconsistency
between
> >> > reporting since the xml files are correct and I'm using the
same
> >> procedure
> >> > that I've always used. I do realize that I need to start
creating a
> >> better
> >> > way to manage all these files that are being produced, both in
terms
> of
> >> > data/plots and xml files.
> >> >
> >> > I ran across a rather large directory:
> >> > ls
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> |
> >> wc
> >> > -w
> >> >
> >> > That has 71,398 xml files in there. But I see timestamps in
those
> >> filenames
> >> > from 20200921 to 20200925.
> >> >
> >> > Is it the case that this stopped working on 20200925? Or are
those
> dated
> >> > filenames remnants from older work?
> >> >
> >> > So it hasn't been working for over a month. Is that right?
> >> >
> >> > There are a lot of files and I need to begin managing that
better. I
> >> think
> >> > during that time there was a machine switch so it stopped on
that day.
> >> > I've been able to produce plots even now, but it's not
consistent in
> >> > reporting while some regions generate plots and others do not.
I've
> >> > reduced the level processing in hopes that I don't hit a walk
clock
> >> issue
> >> > since I'm submitting batch jobs. However, I've run this step
outside
> of
> >> > batch and it always seems to work. This issue seems to happen
with
> batch
> >> > and it is not at all clear why. Let me know what else you need
from
> me
> >> to
> >> > better coordinate what I would need to do next. Thanks.
> >> >
> >> >
> >> >
> >> >
> >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via RT <
> >> > met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Hi Ed,
> >> > >
> >> > > Sorry for the delay. I took a look on wcoss today. You
describe that
> >> your
> >> > > plotting script has stopped working but you're not sure why.
> >> > >
> >> > > I ran across a rather large directory:
> >> > > ls
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> >> |
> >> > wc
> >> > > -w
> >> > >
> >> > > That has 71,398 xml files in there. But I see timestamps in
those
> >> > filenames
> >> > > from 20200921 to 20200925.
> >> > >
> >> > > Is it the case that this stopped working on 20200925? Or are
those
> >> dated
> >> > > filenames remnants from older work?
> >> > >
> >> > > So it hasn't been working for over a month. Is that right?
> >> > >
> >> > > John
> >> > >
> >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System itself via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was acted upon.
> >> > > > Transaction: Given to johnhg (John Halley Gotway) by
RT_System
> >> > > > Queue: met_help
> >> > > > Subject: plots no longer being generated
> >> > > > Owner: johnhg
> >> > > > Requestors: edward.strobach at noaa.gov
> >> > > > Status: new
> >> > > > Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> >> > >
> >> > > >
> >> > > >
> >> > > > This transaction appears to have no content
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > Edward Strobach
> >> > EMC/NCEP/NWS/
> >> > IMSG Contractor
> >> > Cubicle#: 2029
> >> > 301-683-3717
> >> >
> >> >
> >>
> >>
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Wed Nov 04 11:07:09 2020
Ed,
Image files showing up in an unexpected output directory likely point
to a
problem in the logic of the processing scripts. It's pretty unlikely
that
this behavior is caused by the METviewer code itself. But I've
definitely
been surprised before!
I tested the logic outside of a batch job and it works fine. I
suspect
that the issuance of another batch job while the other one is getting
processed is causing some confusion in the processing. These issues
don't
happen everyday and appear more sporadic than anything.
So what we really need here is an example of taking a plot from the
METviewer GUI and adapting it to make it generate multiple plots. I
did go
through examples of doing this in previous tutorials, but we didn't
start
recording them until last summer. I checked the July 2019 NRL tutorial
videos and don't see it there.
Is there something you need from me to help on this? Just let me know
We've begun recording training videos for common topics like this. But
we
only have one for METviewer so far:
https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
But this definitely seems like a good candidate for a new training
video.
For now though, how about this...
- You use METviewer to manually make a single plot, formatting it how
you'd
like.
- You send me your sample plot and corresponding XML file.
- I'll tweak it to generate many plots via batch instead of just one.
- And then you can continue tweaking to add more permutations.
Okay, I'll work on that now. Thanks.
John
On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Ed,
>
> Image files showing up in an unexpected output directory likely
point to a
> problem in the logic of the processing scripts. It's pretty unlikely
that
> this behavior is caused by the METviewer code itself. But I've
definitely
> been surprised before!
>
> So what we really need here is an example of taking a plot from the
> METviewer GUI and adapting it to make it generate multiple plots. I
did go
> through examples of doing this in previous tutorials, but we didn't
start
> recording them until last summer. I checked the July 2019 NRL
tutorial
> videos and don't see it there.
>
> We've begun recording training videos for common topics like this.
But we
> only have one for METviewer so far:
> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>
> But this definitely seems like a good candidate for a new training
video.
>
> For now though, how about this...
> - You use METviewer to manually make a single plot, formatting it
how you'd
> like.
> - You send me your sample plot and corresponding XML file.
> - I'll tweak it to generate many plots via batch instead of just
one.
> - And then you can continue tweaking to add more permutations.
>
> John
>
> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >
> > Oh, and one more thing..
> >
> > The green text that didn't show up was meant to highlight the fact
that
> > sometimes plots that shouldn't be stored in certain directories
end up
> > there. The pasted example below shows one instance where OZMAX*
was
> stored
> > where TMP* should have been stored.
> >
> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5 10:52
> > TMP_FULL_102020_2020-10-01_2020-10-02.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6 10:52
> > TMP_FULL_102020_2020-10-01_2020-10-03.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7 10:00
> > TMP_FULL_102020_2020-10-01_2020-10-04.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13 13:49
> > TMP_FULL_102020_2020-10-01_2020-10-10.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14 11:31
> > TMP_FULL_102020_2020-10-01_2020-10-11.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19 11:37
> > TMP_FULL_102020_2020-10-01_2020-10-12.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20 11:35
> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21 11:45
> > TMP_FULL_102020_2020-10-01_2020-10-18.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22 11:42
> > TMP_FULL_102020_2020-10-01_2020-10-19.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23 11:50
> > TMP_FULL_102020_2020-10-01_2020-10-20.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24 12:57
> > TMP_FULL_102020_2020-10-01_2020-10-21.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31 08:47
> > TMP_FULL_102020_2020-10-01_2020-10-28.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2 08:50
> > TMP_FULL_102020_2020-10-01_2020-10-30.png
> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3 08:52
> > TMP_FULL_102020_2020-10-01_2020-10-31.png
> >
> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > Hi John,
> > >
> > > I know there are a lot of calls and speculate that this could be
part
> of
> > > the problem. I don't think that I'm following best practice by
> > approaching
> > > it in this way. Generating plots usually works well if I
process a
> > subset
> > > of the computations outside of batch scripting. The python
script was
> > > given to me by Logan, but I have since made significant changes
to it
> in
> > > order to dynamically fill in the xml file based on parameters
set. The
> > > test*.sh script was also given to me. I created the
Time_Series*bsub
> as
> > a
> > > way to process a series of plots based on cycle time, region,
etc. I
> > think
> > > the first two scripts are solid and do well for what I need to
get
> done.
> > > It's the Time_Series*bsub script that bothers me a bit.
> > >
> > > I can't see how a single XML file can process all these plots in
a
> single
> > > setting based on what I've learned thus far. How would one go
about
> > that?
> > > I would be open going in that direction if you can provide some
> guidance.
> > >
> > > Unfortunately, I haven't gotten a lot of feedback or dialogue
exchange
> > > about what I've done so far at EMC with this new set-up. This
has also
> > > been part of the problem. I can't seem to get them to focus so
that we
> > can
> > > prioritize. I know that I'm doing the best I can. If I can
learn how
> to
> > > go about this more efficiently it would be of great help. I can
then
> > > expand this to other verification as venture to other projects
in the
> > > future. I know you're busy with many things, so I'm willing to
go along
> > > with your schedule as you find an opening.
> > >
> > > Thanks
> > >
> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Ed,
> > >>
> > >> It's really difficult for me to see an obvious problem that's
causing
> > the
> > >> behavior you describe.
> > >>
> > >> Unfortunately, the red and green highlighting you describe
doesn't
> show
> > up
> > >> in the Gmail web client. There must be some mail system
> incompatibility
> > >> there.
> > >>
> > >> Looking at your bsub script:
> > >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
> > >>
> > >> I am struck by the number of individual calls you're making to
> > METviewer:
> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot types =
656
> calls
> > to
> > >> METviewer
> > >>
> > >> When making plots via the METviewer GUI, it's true that 1 XML =
1
> plot.
> > >> However, when making plots via the batch engine, the plot XML
can be
> set
> > >> up
> > >> to create many, many plots. While I have no direct proof of
this, it
> > might
> > >> be the case that executing 600+ batch jobs for METviewer is
leading to
> > >> trouble on the system. One thing to try would be setting up the
XML to
> > >> make
> > >> many, many more plots in each call. Then check to see if
running
> > METviewer
> > >> in this way is more stable.
> > >>
> > >> I don't think I fully appreciate all of the logic that lives in
> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
> test_Time_Series_AGG.sh.
> > >> However, it *might* be possible to construct a single XML file
which
> > >> produces all 656 plots.
> > >>
> > >> Let me know what you think and how you'd like to proceed.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>
> > >> >
> > >> > Hi John,
> > >> >
> > >> > No problem.
> > >> >
> > >> > I'm including my responses below:
> > >> > Sorry for the delay. I took a look on wcoss today. You
describe that
> > >> your
> > >> > plotting script has stopped working but you're not sure why.
> > >> > It seems more intermittent. Take a look at the example
below. You
> > can
> > >> see
> > >> > that for some reason there are gaps in the plots being
generated.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4 14:52
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 41781 Oct 5 15:15
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 56691 Oct 19 15:02
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 55548 Oct 20 16:08
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 55598 Oct 21 16:08
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 55599 Oct 22 16:09
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 55352 Oct 24 16:34
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 26175 Oct 31 08:20
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 26178 Nov 1 08:20
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 26160 Nov 3 08:22
> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 25354 Nov 3 08:22
> TMP_FULL_102020_2020-10-01_2020-10-31.png*
> > >> > You can tell when a plot is successful by looking at the file
> size. I
> > >> > highlighted an example of that in red. However, the files
showing
> > about
> > >> > 25000 are the ones where no line plots are being produced.
You can
> > also
> > >> > see that stat files are generated everyday, so as long as I'm
> loading
> > >> the
> > >> > data, which I am, I should be able to use the xml files to
generate
> > the
> > >> > plots. I'm attaching a directory list from wcoss that shows
the
> dates
> > >> > where data is produced.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > *20190801 20190817 20200624 20200710 20200726 20200812
> 20200828
> > >> > 20201003 2020101920190802 20190818 20200625 20200711
20200727
> > >> > 20200813 20200829 20201004 2020102020190803 20190819
20200626
> > >> > 20200712 20200728 20200814 20200830 20201005
2020102120190804
> > >> > 20190820 20200627 20200713 20200729 20200815 20200831
> 20201006
> > >> > 2020102220190805 20190821 20200628 20200714 20200730
20200816
> > >> > 20200921 20201007 2020102320190806 20190822 20200629
20200715
> > >> > 20200731 20200817 20200922 20201008 2020102420190807
20190823
> > >> > 20200630 20200716 20200801 20200818 20200923 20201009
> > >> > 2020102520190808 20190824 20200701 20200717 20200802
20200819
> > >> > 20200924 20201010 2020102620190809 20190825 20200702
20200718
> > >> > 20200803 20200820 20200925 20201011 2020102720190810
20190826
> > >> > 20200703 20200719 20200804 20200821 20200926 20201012
> > >> > 2020102820190811 20190827 20200704 20200720 20200805
20200822
> > >> > 20200927 20201013 2020102920190812 20190828 20200705
20200721
> > >> > 20200806 20200823 20200928 20201014 2020103020190813
20190829
> > >> > 20200706 20200722 20200807 20200824 20200929 20201015
> > >> > 2020103120190814 20190830 20200707 20200723 20200808
20200825
> > >> > 20200930 20201016 2020110120190815 20200622 20200708
20200724
> > >> > 20200810 20200826 20201001 20201017 2020110220190816
20200623
> > >> > 20200709 20200725 20200811 20200827 20201002 20201018*
> > >> >
> > >> > Furthermore, what's interesting is that I have mixed success
with
> > other
> > >> > cases. Using the same field - PMTF - for the same cycle -
the 06z
> > >> cycle -
> > >> > you can see that some of the dates where data was plotted for
EAST
> did
> > >> not
> > >> > necessarily plot for FULL. This surprises me because EAST is
a
> subset
> > >> of
> > >> > FULL. See below:
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4 14:54
> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 49039 Oct 5 15:17
> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 56791 Oct 20 16:09
> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 56834 Oct 21 16:09
> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 56921 Oct 22 16:11
> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 56518 Oct 24 16:36
> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 61187 Oct 31 08:21
> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 26475 Nov 2 08:22
> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1
> Edward.Strobach
> > >> > emcmodel 26455 Nov 3 08:23
> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
> > >> >
> > >> > What I find equally concerning as that sometimes plots are
> incorrectly
> > >> > stored in the wrong directory list as highlighted in green
above.
> I've
> > >> > consulted the plot xml file for PMTF, for instance, and find
nothing
> > >> > problematic. The only setting that might pose an issue when
it
> comes
> > to
> > >> > generating line plots is when I set event_equal to true.
However,
> all
> > >> the
> > >> > stat files are populated for all experiments. In the case of
ozone
> you
> > >> see
> > >> > something different. I find that ozone tends to result in
populated
> > >> plots
> > >> > but with intermittent reporting different than PMTF - see
below:
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal 6144-rw-r--r--
1
> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 54318 Oct 6 10:51
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 54305 Oct 7 10:49
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 62179 Oct 13 13:25
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 63811 Oct 19 12:18
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 60112 Oct 20 12:21
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 60165 Oct 21 12:33
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 60160 Oct 22 12:21
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 60056 Oct 23 12:40
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 58517 Oct 24 13:31
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 58062 Oct 31 14:17
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> > Edward.Strobach
> > >> > emcmodel 58119 Nov 1 14:19
> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
> > >> >
> > >> > Similar issues have been found for other fields, cycles, and
regions
> > >> > including verification being done for meteorology.
> > >> > I can't seem to reason this intermittency and inconsistency
between
> > >> > reporting since the xml files are correct and I'm using the
same
> > >> procedure
> > >> > that I've always used. I do realize that I need to start
creating a
> > >> better
> > >> > way to manage all these files that are being produced, both
in terms
> > of
> > >> > data/plots and xml files.
> > >> >
> > >> > I ran across a rather large directory:
> > >> > ls
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > |
> > >> wc
> > >> > -w
> > >> >
> > >> > That has 71,398 xml files in there. But I see timestamps in
those
> > >> filenames
> > >> > from 20200921 to 20200925.
> > >> >
> > >> > Is it the case that this stopped working on 20200925? Or are
those
> > dated
> > >> > filenames remnants from older work?
> > >> >
> > >> > So it hasn't been working for over a month. Is that right?
> > >> >
> > >> > There are a lot of files and I need to begin managing that
better.
> I
> > >> think
> > >> > during that time there was a machine switch so it stopped on
that
> day.
> > >> > I've been able to produce plots even now, but it's not
consistent in
> > >> > reporting while some regions generate plots and others do
not. I've
> > >> > reduced the level processing in hopes that I don't hit a walk
clock
> > >> issue
> > >> > since I'm submitting batch jobs. However, I've run this step
> outside
> > of
> > >> > batch and it always seems to work. This issue seems to happen
with
> > batch
> > >> > and it is not at all clear why. Let me know what else you
need from
> > me
> > >> to
> > >> > better coordinate what I would need to do next. Thanks.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via RT <
> > >> > met_help at ucar.edu>
> > >> > wrote:
> > >> >
> > >> > > Hi Ed,
> > >> > >
> > >> > > Sorry for the delay. I took a look on wcoss today. You
describe
> that
> > >> your
> > >> > > plotting script has stopped working but you're not sure
why.
> > >> > >
> > >> > > I ran across a rather large directory:
> > >> > > ls
> > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > >> |
> > >> > wc
> > >> > > -w
> > >> > >
> > >> > > That has 71,398 xml files in there. But I see timestamps in
those
> > >> > filenames
> > >> > > from 20200921 to 20200925.
> > >> > >
> > >> > > Is it the case that this stopped working on 20200925? Or
are those
> > >> dated
> > >> > > filenames remnants from older work?
> > >> > >
> > >> > > So it hasn't been working for over a month. Is that right?
> > >> > >
> > >> > > John
> > >> > >
> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System itself via RT
<
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > > >
> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was acted upon.
> > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
RT_System
> > >> > > > Queue: met_help
> > >> > > > Subject: plots no longer being generated
> > >> > > > Owner: johnhg
> > >> > > > Requestors: edward.strobach at noaa.gov
> > >> > > > Status: new
> > >> > > > Ticket <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > >> > >
> > >> > > >
> > >> > > >
> > >> > > > This transaction appears to have no content
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > Edward Strobach
> > >> > EMC/NCEP/NWS/
> > >> > IMSG Contractor
> > >> > Cubicle#: 2029
> > >> > 301-683-3717
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Wed Nov 04 11:32:35 2020
Hi John,
I think I was able to cover your request okay. Below are 4
attachments
that include the test script that was used to generate the plots, a
sample
XML file, and the two plots that were generated using the test run
script.
These plots were not created previously using the batch job but uses
the
same exact logic as the original run script. Thanks again for working
with
me on this. Please let me know if you need anything else or would
like me
to test something.
On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Ed,
>
> Image files showing up in an unexpected output directory likely
point to a
> problem in the logic of the processing scripts. It's pretty unlikely
that
> this behavior is caused by the METviewer code itself. But I've
definitely
> been surprised before!
>
> I tested the logic outside of a batch job and it works fine. I
suspect
> that the issuance of another batch job while the other one is
getting
> processed is causing some confusion in the processing. These issues
don't
> happen everyday and appear more sporadic than anything.
>
> So what we really need here is an example of taking a plot from the
> METviewer GUI and adapting it to make it generate multiple plots. I
did go
> through examples of doing this in previous tutorials, but we didn't
start
> recording them until last summer. I checked the July 2019 NRL
tutorial
> videos and don't see it there.
>
> Is there something you need from me to help on this? Just let me
know
>
> We've begun recording training videos for common topics like this.
But we
> only have one for METviewer so far:
> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>
> But this definitely seems like a good candidate for a new training
video.
>
> For now though, how about this...
> - You use METviewer to manually make a single plot, formatting it
how you'd
> like.
> - You send me your sample plot and corresponding XML file.
> - I'll tweak it to generate many plots via batch instead of just
one.
> - And then you can continue tweaking to add more permutations.
>
> Okay, I'll work on that now. Thanks.
>
> John
>
> On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Ed,
>>
>> Image files showing up in an unexpected output directory likely
point to a
>> problem in the logic of the processing scripts. It's pretty
unlikely that
>> this behavior is caused by the METviewer code itself. But I've
definitely
>> been surprised before!
>>
>> So what we really need here is an example of taking a plot from the
>> METviewer GUI and adapting it to make it generate multiple plots. I
did go
>> through examples of doing this in previous tutorials, but we didn't
start
>> recording them until last summer. I checked the July 2019 NRL
tutorial
>> videos and don't see it there.
>>
>> We've begun recording training videos for common topics like this.
But we
>> only have one for METviewer so far:
>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>>
>> But this definitely seems like a good candidate for a new training
video.
>>
>> For now though, how about this...
>> - You use METviewer to manually make a single plot, formatting it
how
>> you'd
>> like.
>> - You send me your sample plot and corresponding XML file.
>> - I'll tweak it to generate many plots via batch instead of just
one.
>> - And then you can continue tweaking to add more permutations.
>>
>> John
>>
>> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>> >
>> > Oh, and one more thing..
>> >
>> > The green text that didn't show up was meant to highlight the
fact that
>> > sometimes plots that shouldn't be stored in certain directories
end up
>> > there. The pasted example below shows one instance where OZMAX*
was
>> stored
>> > where TMP* should have been stored.
>> >
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5 10:52
>> > TMP_FULL_102020_2020-10-01_2020-10-02.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6 10:52
>> > TMP_FULL_102020_2020-10-01_2020-10-03.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7 10:00
>> > TMP_FULL_102020_2020-10-01_2020-10-04.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13 13:49
>> > TMP_FULL_102020_2020-10-01_2020-10-10.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14 11:31
>> > TMP_FULL_102020_2020-10-01_2020-10-11.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19 11:37
>> > TMP_FULL_102020_2020-10-01_2020-10-12.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20 11:35
>> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21 11:45
>> > TMP_FULL_102020_2020-10-01_2020-10-18.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22 11:42
>> > TMP_FULL_102020_2020-10-01_2020-10-19.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23 11:50
>> > TMP_FULL_102020_2020-10-01_2020-10-20.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24 12:57
>> > TMP_FULL_102020_2020-10-01_2020-10-21.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31 08:47
>> > TMP_FULL_102020_2020-10-01_2020-10-28.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2 08:50
>> > TMP_FULL_102020_2020-10-01_2020-10-30.png
>> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3 08:52
>> > TMP_FULL_102020_2020-10-01_2020-10-31.png
>> >
>> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA Affiliate
<
>> > edward.strobach at noaa.gov> wrote:
>> >
>> > > Hi John,
>> > >
>> > > I know there are a lot of calls and speculate that this could
be part
>> of
>> > > the problem. I don't think that I'm following best practice by
>> > approaching
>> > > it in this way. Generating plots usually works well if I
process a
>> > subset
>> > > of the computations outside of batch scripting. The python
script was
>> > > given to me by Logan, but I have since made significant changes
to it
>> in
>> > > order to dynamically fill in the xml file based on parameters
set.
>> The
>> > > test*.sh script was also given to me. I created the
Time_Series*bsub
>> as
>> > a
>> > > way to process a series of plots based on cycle time, region,
etc. I
>> > think
>> > > the first two scripts are solid and do well for what I need to
get
>> done.
>> > > It's the Time_Series*bsub script that bothers me a bit.
>> > >
>> > > I can't see how a single XML file can process all these plots
in a
>> single
>> > > setting based on what I've learned thus far. How would one go
about
>> > that?
>> > > I would be open going in that direction if you can provide some
>> guidance.
>> > >
>> > > Unfortunately, I haven't gotten a lot of feedback or dialogue
exchange
>> > > about what I've done so far at EMC with this new set-up. This
has
>> also
>> > > been part of the problem. I can't seem to get them to focus so
that
>> we
>> > can
>> > > prioritize. I know that I'm doing the best I can. If I can
learn
>> how to
>> > > go about this more efficiently it would be of great help. I
can then
>> > > expand this to other verification as venture to other projects
in the
>> > > future. I know you're busy with many things, so I'm willing to
go
>> along
>> > > with your schedule as you find an opening.
>> > >
>> > > Thanks
>> > >
>> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > >> Ed,
>> > >>
>> > >> It's really difficult for me to see an obvious problem that's
causing
>> > the
>> > >> behavior you describe.
>> > >>
>> > >> Unfortunately, the red and green highlighting you describe
doesn't
>> show
>> > up
>> > >> in the Gmail web client. There must be some mail system
>> incompatibility
>> > >> there.
>> > >>
>> > >> Looking at your bsub script:
>> > >>
>> > >>
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
>> > >>
>> > >> I am struck by the number of individual calls you're making to
>> > METviewer:
>> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot types =
656
>> calls
>> > to
>> > >> METviewer
>> > >>
>> > >> When making plots via the METviewer GUI, it's true that 1 XML
= 1
>> plot.
>> > >> However, when making plots via the batch engine, the plot XML
can be
>> set
>> > >> up
>> > >> to create many, many plots. While I have no direct proof of
this, it
>> > might
>> > >> be the case that executing 600+ batch jobs for METviewer is
leading
>> to
>> > >> trouble on the system. One thing to try would be setting up
the XML
>> to
>> > >> make
>> > >> many, many more plots in each call. Then check to see if
running
>> > METviewer
>> > >> in this way is more stable.
>> > >>
>> > >> I don't think I fully appreciate all of the logic that lives
in
>> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
>> test_Time_Series_AGG.sh.
>> > >> However, it *might* be possible to construct a single XML file
which
>> > >> produces all 656 plots.
>> > >>
>> > >> Let me know what you think and how you'd like to proceed.
>> > >>
>> > >> Thanks,
>> > >> John
>> > >>
>> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA
Affiliate via
>> RT <
>> > >> met_help at ucar.edu> wrote:
>> > >>
>> > >> >
>> > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>> > >> >
>> > >> > Hi John,
>> > >> >
>> > >> > No problem.
>> > >> >
>> > >> > I'm including my responses below:
>> > >> > Sorry for the delay. I took a look on wcoss today. You
describe
>> that
>> > >> your
>> > >> > plotting script has stopped working but you're not sure why.
>> > >> > It seems more intermittent. Take a look at the example
below. You
>> > can
>> > >> see
>> > >> > that for some reason there are gaps in the plots being
generated.
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4 14:52
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 41781 Oct 5 15:15
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 56691 Oct 19 15:02
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 55548 Oct 20 16:08
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 55598 Oct 21 16:08
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 55599 Oct 22 16:09
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 55352 Oct 24 16:34
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 26175 Oct 31 08:20
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 26178 Nov 1 08:20
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 26160 Nov 3 08:22
>> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 25354 Nov 3 08:22
>> TMP_FULL_102020_2020-10-01_2020-10-31.png*
>> > >> > You can tell when a plot is successful by looking at the
file
>> size. I
>> > >> > highlighted an example of that in red. However, the files
showing
>> > about
>> > >> > 25000 are the ones where no line plots are being produced.
You can
>> > also
>> > >> > see that stat files are generated everyday, so as long as
I'm
>> loading
>> > >> the
>> > >> > data, which I am, I should be able to use the xml files to
generate
>> > the
>> > >> > plots. I'm attaching a directory list from wcoss that shows
the
>> dates
>> > >> > where data is produced.
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > *20190801 20190817 20200624 20200710 20200726 20200812
>> 20200828
>> > >> > 20201003 2020101920190802 20190818 20200625 20200711
20200727
>> > >> > 20200813 20200829 20201004 2020102020190803 20190819
20200626
>> > >> > 20200712 20200728 20200814 20200830 20201005
2020102120190804
>> > >> > 20190820 20200627 20200713 20200729 20200815 20200831
>> 20201006
>> > >> > 2020102220190805 20190821 20200628 20200714 20200730
20200816
>> > >> > 20200921 20201007 2020102320190806 20190822 20200629
20200715
>> > >> > 20200731 20200817 20200922 20201008 2020102420190807
20190823
>> > >> > 20200630 20200716 20200801 20200818 20200923 20201009
>> > >> > 2020102520190808 20190824 20200701 20200717 20200802
20200819
>> > >> > 20200924 20201010 2020102620190809 20190825 20200702
20200718
>> > >> > 20200803 20200820 20200925 20201011 2020102720190810
20190826
>> > >> > 20200703 20200719 20200804 20200821 20200926 20201012
>> > >> > 2020102820190811 20190827 20200704 20200720 20200805
20200822
>> > >> > 20200927 20201013 2020102920190812 20190828 20200705
20200721
>> > >> > 20200806 20200823 20200928 20201014 2020103020190813
20190829
>> > >> > 20200706 20200722 20200807 20200824 20200929 20201015
>> > >> > 2020103120190814 20190830 20200707 20200723 20200808
20200825
>> > >> > 20200930 20201016 2020110120190815 20200622 20200708
20200724
>> > >> > 20200810 20200826 20201001 20201017 2020110220190816
20200623
>> > >> > 20200709 20200725 20200811 20200827 20201002 20201018*
>> > >> >
>> > >> > Furthermore, what's interesting is that I have mixed success
with
>> > other
>> > >> > cases. Using the same field - PMTF - for the same cycle -
the 06z
>> > >> cycle -
>> > >> > you can see that some of the dates where data was plotted
for EAST
>> did
>> > >> not
>> > >> > necessarily plot for FULL. This surprises me because EAST
is a
>> subset
>> > >> of
>> > >> > FULL. See below:
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4 14:54
>> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 49039 Oct 5 15:17
>> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 56791 Oct 20 16:09
>> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 56834 Oct 21 16:09
>> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 56921 Oct 22 16:11
>> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 56518 Oct 24 16:36
>> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 61187 Oct 31 08:21
>> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 26475 Nov 2 08:22
>> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1
>> Edward.Strobach
>> > >> > emcmodel 26455 Nov 3 08:23
>> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
>> > >> >
>> > >> > What I find equally concerning as that sometimes plots are
>> incorrectly
>> > >> > stored in the wrong directory list as highlighted in green
above.
>> I've
>> > >> > consulted the plot xml file for PMTF, for instance, and find
>> nothing
>> > >> > problematic. The only setting that might pose an issue when
it
>> comes
>> > to
>> > >> > generating line plots is when I set event_equal to true.
However,
>> all
>> > >> the
>> > >> > stat files are populated for all experiments. In the case of
ozone
>> you
>> > >> see
>> > >> > something different. I find that ozone tends to result in
populated
>> > >> plots
>> > >> > but with intermittent reporting different than PMTF - see
below:
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal 6144-rw-r--r--
1
>> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 54318 Oct 6 10:51
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 54305 Oct 7 10:49
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 62179 Oct 13 13:25
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 63811 Oct 19 12:18
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 60112 Oct 20 12:21
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 60165 Oct 21 12:33
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 60160 Oct 22 12:21
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 60056 Oct 23 12:40
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 58517 Oct 24 13:31
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 58062 Oct 31 14:17
>> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
>> > Edward.Strobach
>> > >> > emcmodel 58119 Nov 1 14:19
>> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
>> > >> >
>> > >> > Similar issues have been found for other fields, cycles, and
>> regions
>> > >> > including verification being done for meteorology.
>> > >> > I can't seem to reason this intermittency and inconsistency
between
>> > >> > reporting since the xml files are correct and I'm using the
same
>> > >> procedure
>> > >> > that I've always used. I do realize that I need to start
creating
>> a
>> > >> better
>> > >> > way to manage all these files that are being produced, both
in
>> terms
>> > of
>> > >> > data/plots and xml files.
>> > >> >
>> > >> > I ran across a rather large directory:
>> > >> > ls
>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>> > |
>> > >> wc
>> > >> > -w
>> > >> >
>> > >> > That has 71,398 xml files in there. But I see timestamps in
those
>> > >> filenames
>> > >> > from 20200921 to 20200925.
>> > >> >
>> > >> > Is it the case that this stopped working on 20200925? Or are
those
>> > dated
>> > >> > filenames remnants from older work?
>> > >> >
>> > >> > So it hasn't been working for over a month. Is that right?
>> > >> >
>> > >> > There are a lot of files and I need to begin managing that
>> better. I
>> > >> think
>> > >> > during that time there was a machine switch so it stopped on
that
>> day.
>> > >> > I've been able to produce plots even now, but it's not
consistent
>> in
>> > >> > reporting while some regions generate plots and others do
not.
>> I've
>> > >> > reduced the level processing in hopes that I don't hit a
walk clock
>> > >> issue
>> > >> > since I'm submitting batch jobs. However, I've run this
step
>> outside
>> > of
>> > >> > batch and it always seems to work. This issue seems to
happen with
>> > batch
>> > >> > and it is not at all clear why. Let me know what else you
need
>> from
>> > me
>> > >> to
>> > >> > better coordinate what I would need to do next. Thanks.
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via RT <
>> > >> > met_help at ucar.edu>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Ed,
>> > >> > >
>> > >> > > Sorry for the delay. I took a look on wcoss today. You
describe
>> that
>> > >> your
>> > >> > > plotting script has stopped working but you're not sure
why.
>> > >> > >
>> > >> > > I ran across a rather large directory:
>> > >> > > ls
>> >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>> > >> |
>> > >> > wc
>> > >> > > -w
>> > >> > >
>> > >> > > That has 71,398 xml files in there. But I see timestamps
in those
>> > >> > filenames
>> > >> > > from 20200921 to 20200925.
>> > >> > >
>> > >> > > Is it the case that this stopped working on 20200925? Or
are
>> those
>> > >> dated
>> > >> > > filenames remnants from older work?
>> > >> > >
>> > >> > > So it hasn't been working for over a month. Is that right?
>> > >> > >
>> > >> > > John
>> > >> > >
>> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System itself via RT
<
>> > >> > > met_help at ucar.edu> wrote:
>> > >> > >
>> > >> > > >
>> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was acted upon.
>> > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
RT_System
>> > >> > > > Queue: met_help
>> > >> > > > Subject: plots no longer being generated
>> > >> > > > Owner: johnhg
>> > >> > > > Requestors: edward.strobach at noaa.gov
>> > >> > > > Status: new
>> > >> > > > Ticket <URL:
>> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>> > >> > >
>> > >> > > >
>> > >> > > >
>> > >> > > > This transaction appears to have no content
>> > >> > > >
>> > >> > >
>> > >> > >
>> > >> >
>> > >> > --
>> > >> > Edward Strobach
>> > >> > EMC/NCEP/NWS/
>> > >> > IMSG Contractor
>> > >> > Cubicle#: 2029
>> > >> > 301-683-3717
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >
>> > > --
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/
>> > > IMSG Contractor
>> > > Cubicle#: 2029
>> > > 301-683-3717
>> > >
>> >
>> >
>> > --
>> > Edward Strobach
>> > EMC/NCEP/NWS/
>> > IMSG Contractor
>> > Cubicle#: 2029
>> > 301-683-3717
>> >
>> >
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: John Halley Gotway
Time: Thu Nov 05 12:01:32 2020
Ed,
Thanks for sending examples. My goal was to use METviewer's XML upload
functionality but I'm not able to upload a pdf version of the xml
file.
I did spend some time trying to replicate this exact plot with
METviewer at:
https://metviewer.nws.noaa.gov/metviewer1.jsp
But I was never quite successful. So I made just a similar version
instead.
See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
This plots FBAR and OBAR for 6 different models for the month of
October
for all 06Z initializations over the FULL model domain.
I wanted to show you how to modify this for batch mode... and then
adapt it
to make multiple plots. But I'm not able to make batch plots on wcoss
using
the AWS instance. I don't think I have the right permissions to do so.
So let me just say this. Generally, you just update the list of items
in
plot_fix section.
Original from METviewer with 1 init hour and 1 mask:
<field equalize="false" name="vx_mask">
<set name="vx_mask_0">
<val>FULL</val>
</set>
</field>
<field equalize="false" name="init_hour">
<set name="init_hour_1">
<val>06</val>
</set>
</field>
Change to 2 init hours and 4 masks:
<field equalize="false" name="vx_mask">
<val>FULL</val>
<val>CONUS</val>
<val>EAST</val>
<val>WEST</val>
</field>
<field equalize="false" name="init_hour">
<val>06</val>
<val>12</val>
</field>
And then reference those values in the naming of the titles and output
files:
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
<title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
Domain)</title>
So that's the general idea. And when you submit this to mv_batch,
it'll
make 8 plots instead of 1.
And you can see how you'd easily expand this to many plots.
But I worry that this may not be sufficient information to get you
going.
John
On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>
> Hi John,
>
> I think I was able to cover your request okay. Below are 4
attachments
> that include the test script that was used to generate the plots, a
sample
> XML file, and the two plots that were generated using the test run
script.
> These plots were not created previously using the batch job but uses
the
> same exact logic as the original run script. Thanks again for
working with
> me on this. Please let me know if you need anything else or would
like me
> to test something.
>
> On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Ed,
> >
> > Image files showing up in an unexpected output directory likely
point to
> a
> > problem in the logic of the processing scripts. It's pretty
unlikely that
> > this behavior is caused by the METviewer code itself. But I've
definitely
> > been surprised before!
> >
> > I tested the logic outside of a batch job and it works fine. I
suspect
> > that the issuance of another batch job while the other one is
getting
> > processed is causing some confusion in the processing. These
issues
> don't
> > happen everyday and appear more sporadic than anything.
> >
> > So what we really need here is an example of taking a plot from
the
> > METviewer GUI and adapting it to make it generate multiple plots.
I did
> go
> > through examples of doing this in previous tutorials, but we
didn't start
> > recording them until last summer. I checked the July 2019 NRL
tutorial
> > videos and don't see it there.
> >
> > Is there something you need from me to help on this? Just let me
know
> >
> > We've begun recording training videos for common topics like this.
But we
> > only have one for METviewer so far:
> > https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> >
> > But this definitely seems like a good candidate for a new training
video.
> >
> > For now though, how about this...
> > - You use METviewer to manually make a single plot, formatting it
how
> you'd
> > like.
> > - You send me your sample plot and corresponding XML file.
> > - I'll tweak it to generate many plots via batch instead of just
one.
> > - And then you can continue tweaking to add more permutations.
> >
> > Okay, I'll work on that now. Thanks.
> >
> > John
> >
> > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Ed,
> >>
> >> Image files showing up in an unexpected output directory likely
point
> to a
> >> problem in the logic of the processing scripts. It's pretty
unlikely
> that
> >> this behavior is caused by the METviewer code itself. But I've
> definitely
> >> been surprised before!
> >>
> >> So what we really need here is an example of taking a plot from
the
> >> METviewer GUI and adapting it to make it generate multiple plots.
I did
> go
> >> through examples of doing this in previous tutorials, but we
didn't
> start
> >> recording them until last summer. I checked the July 2019 NRL
tutorial
> >> videos and don't see it there.
> >>
> >> We've begun recording training videos for common topics like
this. But
> we
> >> only have one for METviewer so far:
> >>
> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> >>
> >> But this definitely seems like a good candidate for a new
training
> video.
> >>
> >> For now though, how about this...
> >> - You use METviewer to manually make a single plot, formatting it
how
> >> you'd
> >> like.
> >> - You send me your sample plot and corresponding XML file.
> >> - I'll tweak it to generate many plots via batch instead of just
one.
> >> - And then you can continue tweaking to add more permutations.
> >>
> >> John
> >>
> >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA Affiliate
via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >> >
> >> > Oh, and one more thing..
> >> >
> >> > The green text that didn't show up was meant to highlight the
fact
> that
> >> > sometimes plots that shouldn't be stored in certain directories
end up
> >> > there. The pasted example below shows one instance where
OZMAX* was
> >> stored
> >> > where TMP* should have been stored.
> >> >
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5 10:52
> >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6 10:52
> >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7 10:00
> >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13 13:49
> >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14 11:31
> >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19 11:37
> >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20 11:35
> >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21 11:45
> >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22 11:42
> >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23 11:50
> >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24 12:57
> >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31 08:47
> >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2 08:50
> >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
> >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3 08:52
> >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
> >> >
> >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA
Affiliate <
> >> > edward.strobach at noaa.gov> wrote:
> >> >
> >> > > Hi John,
> >> > >
> >> > > I know there are a lot of calls and speculate that this could
be
> part
> >> of
> >> > > the problem. I don't think that I'm following best practice
by
> >> > approaching
> >> > > it in this way. Generating plots usually works well if I
process a
> >> > subset
> >> > > of the computations outside of batch scripting. The python
script
> was
> >> > > given to me by Logan, but I have since made significant
changes to
> it
> >> in
> >> > > order to dynamically fill in the xml file based on parameters
set.
> >> The
> >> > > test*.sh script was also given to me. I created the
> Time_Series*bsub
> >> as
> >> > a
> >> > > way to process a series of plots based on cycle time, region,
etc. I
> >> > think
> >> > > the first two scripts are solid and do well for what I need
to get
> >> done.
> >> > > It's the Time_Series*bsub script that bothers me a bit.
> >> > >
> >> > > I can't see how a single XML file can process all these plots
in a
> >> single
> >> > > setting based on what I've learned thus far. How would one
go about
> >> > that?
> >> > > I would be open going in that direction if you can provide
some
> >> guidance.
> >> > >
> >> > > Unfortunately, I haven't gotten a lot of feedback or dialogue
> exchange
> >> > > about what I've done so far at EMC with this new set-up.
This has
> >> also
> >> > > been part of the problem. I can't seem to get them to focus
so that
> >> we
> >> > can
> >> > > prioritize. I know that I'm doing the best I can. If I can
learn
> >> how to
> >> > > go about this more efficiently it would be of great help. I
can
> then
> >> > > expand this to other verification as venture to other
projects in
> the
> >> > > future. I know you're busy with many things, so I'm willing
to go
> >> along
> >> > > with your schedule as you find an opening.
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > >> Ed,
> >> > >>
> >> > >> It's really difficult for me to see an obvious problem
that's
> causing
> >> > the
> >> > >> behavior you describe.
> >> > >>
> >> > >> Unfortunately, the red and green highlighting you describe
doesn't
> >> show
> >> > up
> >> > >> in the Gmail web client. There must be some mail system
> >> incompatibility
> >> > >> there.
> >> > >>
> >> > >> Looking at your bsub script:
> >> > >>
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
> >> > >>
> >> > >> I am struck by the number of individual calls you're making
to
> >> > METviewer:
> >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot types
= 656
> >> calls
> >> > to
> >> > >> METviewer
> >> > >>
> >> > >> When making plots via the METviewer GUI, it's true that 1
XML = 1
> >> plot.
> >> > >> However, when making plots via the batch engine, the plot
XML can
> be
> >> set
> >> > >> up
> >> > >> to create many, many plots. While I have no direct proof of
this,
> it
> >> > might
> >> > >> be the case that executing 600+ batch jobs for METviewer is
leading
> >> to
> >> > >> trouble on the system. One thing to try would be setting up
the XML
> >> to
> >> > >> make
> >> > >> many, many more plots in each call. Then check to see if
running
> >> > METviewer
> >> > >> in this way is more stable.
> >> > >>
> >> > >> I don't think I fully appreciate all of the logic that lives
in
> >> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
> >> test_Time_Series_AGG.sh.
> >> > >> However, it *might* be possible to construct a single XML
file
> which
> >> > >> produces all 656 plots.
> >> > >>
> >> > >> Let me know what you think and how you'd like to proceed.
> >> > >>
> >> > >> Thanks,
> >> > >> John
> >> > >>
> >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA
Affiliate via
> >> RT <
> >> > >> met_help at ucar.edu> wrote:
> >> > >>
> >> > >> >
> >> > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >> > >> >
> >> > >> > Hi John,
> >> > >> >
> >> > >> > No problem.
> >> > >> >
> >> > >> > I'm including my responses below:
> >> > >> > Sorry for the delay. I took a look on wcoss today. You
describe
> >> that
> >> > >> your
> >> > >> > plotting script has stopped working but you're not sure
why.
> >> > >> > It seems more intermittent. Take a look at the example
below.
> You
> >> > can
> >> > >> see
> >> > >> > that for some reason there are gaps in the plots being
generated.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4 14:52
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 41781 Oct 5 15:15
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 56691 Oct 19 15:02
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 55548 Oct 20 16:08
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 55598 Oct 21 16:08
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 55599 Oct 22 16:09
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 55352 Oct 24 16:34
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 26175 Oct 31 08:20
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 26178 Nov 1 08:20
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 26160 Nov 3 08:22
> >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 25354 Nov 3 08:22
> >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
> >> > >> > You can tell when a plot is successful by looking at the
file
> >> size. I
> >> > >> > highlighted an example of that in red. However, the files
> showing
> >> > about
> >> > >> > 25000 are the ones where no line plots are being produced.
You
> can
> >> > also
> >> > >> > see that stat files are generated everyday, so as long as
I'm
> >> loading
> >> > >> the
> >> > >> > data, which I am, I should be able to use the xml files to
> generate
> >> > the
> >> > >> > plots. I'm attaching a directory list from wcoss that
shows the
> >> dates
> >> > >> > where data is produced.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > *20190801 20190817 20200624 20200710 20200726
20200812
> >> 20200828
> >> > >> > 20201003 2020101920190802 20190818 20200625 20200711
> 20200727
> >> > >> > 20200813 20200829 20201004 2020102020190803 20190819
> 20200626
> >> > >> > 20200712 20200728 20200814 20200830 20201005
> 2020102120190804
> >> > >> > 20190820 20200627 20200713 20200729 20200815
20200831
> >> 20201006
> >> > >> > 2020102220190805 20190821 20200628 20200714 20200730
> 20200816
> >> > >> > 20200921 20201007 2020102320190806 20190822 20200629
> 20200715
> >> > >> > 20200731 20200817 20200922 20201008 2020102420190807
> 20190823
> >> > >> > 20200630 20200716 20200801 20200818 20200923
20201009
> >> > >> > 2020102520190808 20190824 20200701 20200717 20200802
> 20200819
> >> > >> > 20200924 20201010 2020102620190809 20190825 20200702
> 20200718
> >> > >> > 20200803 20200820 20200925 20201011 2020102720190810
> 20190826
> >> > >> > 20200703 20200719 20200804 20200821 20200926
20201012
> >> > >> > 2020102820190811 20190827 20200704 20200720 20200805
> 20200822
> >> > >> > 20200927 20201013 2020102920190812 20190828 20200705
> 20200721
> >> > >> > 20200806 20200823 20200928 20201014 2020103020190813
> 20190829
> >> > >> > 20200706 20200722 20200807 20200824 20200929
20201015
> >> > >> > 2020103120190814 20190830 20200707 20200723 20200808
> 20200825
> >> > >> > 20200930 20201016 2020110120190815 20200622 20200708
> 20200724
> >> > >> > 20200810 20200826 20201001 20201017 2020110220190816
> 20200623
> >> > >> > 20200709 20200725 20200811 20200827 20201002
20201018*
> >> > >> >
> >> > >> > Furthermore, what's interesting is that I have mixed
success with
> >> > other
> >> > >> > cases. Using the same field - PMTF - for the same cycle -
the
> 06z
> >> > >> cycle -
> >> > >> > you can see that some of the dates where data was plotted
for
> EAST
> >> did
> >> > >> not
> >> > >> > necessarily plot for FULL. This surprises me because EAST
is a
> >> subset
> >> > >> of
> >> > >> > FULL. See below:
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4 14:54
> >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 49039 Oct 5 15:17
> >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 56791 Oct 20 16:09
> >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 56834 Oct 21 16:09
> >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 56921 Oct 22 16:11
> >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 56518 Oct 24 16:36
> >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 61187 Oct 31 08:21
> >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 26475 Nov 2 08:22
> >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1
> >> Edward.Strobach
> >> > >> > emcmodel 26455 Nov 3 08:23
> >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
> >> > >> >
> >> > >> > What I find equally concerning as that sometimes plots are
> >> incorrectly
> >> > >> > stored in the wrong directory list as highlighted in green
above.
> >> I've
> >> > >> > consulted the plot xml file for PMTF, for instance, and
find
> >> nothing
> >> > >> > problematic. The only setting that might pose an issue
when it
> >> comes
> >> > to
> >> > >> > generating line plots is when I set event_equal to true.
> However,
> >> all
> >> > >> the
> >> > >> > stat files are populated for all experiments. In the case
of
> ozone
> >> you
> >> > >> see
> >> > >> > something different. I find that ozone tends to result in
> populated
> >> > >> plots
> >> > >> > but with intermittent reporting different than PMTF - see
below:
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal 6144-rw-r--
r-- 1
> >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 54318 Oct 6 10:51
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 54305 Oct 7 10:49
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 62179 Oct 13 13:25
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 63811 Oct 19 12:18
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 60112 Oct 20 12:21
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 60165 Oct 21 12:33
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 60160 Oct 22 12:21
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 60056 Oct 23 12:40
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 58517 Oct 24 13:31
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 58062 Oct 31 14:17
> >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> >> > Edward.Strobach
> >> > >> > emcmodel 58119 Nov 1 14:19
> >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
> >> > >> >
> >> > >> > Similar issues have been found for other fields, cycles,
and
> >> regions
> >> > >> > including verification being done for meteorology.
> >> > >> > I can't seem to reason this intermittency and
inconsistency
> between
> >> > >> > reporting since the xml files are correct and I'm using
the same
> >> > >> procedure
> >> > >> > that I've always used. I do realize that I need to start
> creating
> >> a
> >> > >> better
> >> > >> > way to manage all these files that are being produced,
both in
> >> terms
> >> > of
> >> > >> > data/plots and xml files.
> >> > >> >
> >> > >> > I ran across a rather large directory:
> >> > >> > ls
> >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> >> > |
> >> > >> wc
> >> > >> > -w
> >> > >> >
> >> > >> > That has 71,398 xml files in there. But I see timestamps
in those
> >> > >> filenames
> >> > >> > from 20200921 to 20200925.
> >> > >> >
> >> > >> > Is it the case that this stopped working on 20200925? Or
are
> those
> >> > dated
> >> > >> > filenames remnants from older work?
> >> > >> >
> >> > >> > So it hasn't been working for over a month. Is that right?
> >> > >> >
> >> > >> > There are a lot of files and I need to begin managing that
> >> better. I
> >> > >> think
> >> > >> > during that time there was a machine switch so it stopped
on that
> >> day.
> >> > >> > I've been able to produce plots even now, but it's not
consistent
> >> in
> >> > >> > reporting while some regions generate plots and others do
not.
> >> I've
> >> > >> > reduced the level processing in hopes that I don't hit a
walk
> clock
> >> > >> issue
> >> > >> > since I'm submitting batch jobs. However, I've run this
step
> >> outside
> >> > of
> >> > >> > batch and it always seems to work. This issue seems to
happen
> with
> >> > batch
> >> > >> > and it is not at all clear why. Let me know what else you
need
> >> from
> >> > me
> >> > >> to
> >> > >> > better coordinate what I would need to do next. Thanks.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via RT <
> >> > >> > met_help at ucar.edu>
> >> > >> > wrote:
> >> > >> >
> >> > >> > > Hi Ed,
> >> > >> > >
> >> > >> > > Sorry for the delay. I took a look on wcoss today. You
describe
> >> that
> >> > >> your
> >> > >> > > plotting script has stopped working but you're not sure
why.
> >> > >> > >
> >> > >> > > I ran across a rather large directory:
> >> > >> > > ls
> >> >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> >> > >> |
> >> > >> > wc
> >> > >> > > -w
> >> > >> > >
> >> > >> > > That has 71,398 xml files in there. But I see timestamps
in
> those
> >> > >> > filenames
> >> > >> > > from 20200921 to 20200925.
> >> > >> > >
> >> > >> > > Is it the case that this stopped working on 20200925? Or
are
> >> those
> >> > >> dated
> >> > >> > > filenames remnants from older work?
> >> > >> > >
> >> > >> > > So it hasn't been working for over a month. Is that
right?
> >> > >> > >
> >> > >> > > John
> >> > >> > >
> >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System itself via
RT <
> >> > >> > > met_help at ucar.edu> wrote:
> >> > >> > >
> >> > >> > > >
> >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was acted
upon.
> >> > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
> RT_System
> >> > >> > > > Queue: met_help
> >> > >> > > > Subject: plots no longer being generated
> >> > >> > > > Owner: johnhg
> >> > >> > > > Requestors: edward.strobach at noaa.gov
> >> > >> > > > Status: new
> >> > >> > > > Ticket <URL:
> >> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> >> > >> > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > This transaction appears to have no content
> >> > >> > > >
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >> > --
> >> > >> > Edward Strobach
> >> > >> > EMC/NCEP/NWS/
> >> > >> > IMSG Contractor
> >> > >> > Cubicle#: 2029
> >> > >> > 301-683-3717
> >> > >> >
> >> > >> >
> >> > >>
> >> > >>
> >> > >
> >> > > --
> >> > > Edward Strobach
> >> > > EMC/NCEP/NWS/
> >> > > IMSG Contractor
> >> > > Cubicle#: 2029
> >> > > 301-683-3717
> >> > >
> >> >
> >> >
> >> > --
> >> > Edward Strobach
> >> > EMC/NCEP/NWS/
> >> > IMSG Contractor
> >> > Cubicle#: 2029
> >> > 301-683-3717
> >> >
> >> >
> >>
> >>
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Fri Nov 06 03:48:45 2020
Hi John,
This definitely helps. I never thought about it this way because I
assumed, based on my experience with the interactive MET GUI, that I
could
only generate a single plot regardless of how I specified my settings
in
the XML. For example, if I listed EAST and WEST under vx_mask, then
my
thinking would be that both those regions would be grouped and a
single
plot created. I guess I was wrong on that. The concern I still have,
however, is that the plot title and plot name - the png file - is also
set
in the XML file. Even after adding the list you gave, I can't see how
unique file names will be created. Is that true?
On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Ed,
>
> Thanks for sending examples. My goal was to use METviewer's XML
upload
> functionality but I'm not able to upload a pdf version of the xml
file.
>
> I did spend some time trying to replicate this exact plot with
METviewer
> at:
> https://metviewer.nws.noaa.gov/metviewer1.jsp
>
> But I was never quite successful. So I made just a similar version
instead.
> See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
>
> This plots FBAR and OBAR for 6 different models for the month of
October
> for all 06Z initializations over the FULL model domain.
>
> I wanted to show you how to modify this for batch mode... and then
adapt it
> to make multiple plots. But I'm not able to make batch plots on
wcoss using
> the AWS instance. I don't think I have the right permissions to do
so.
>
> So let me just say this. Generally, you just update the list of
items in
> plot_fix section.
>
> Original from METviewer with 1 init hour and 1 mask:
>
> <field equalize="false" name="vx_mask">
> <set name="vx_mask_0">
> <val>FULL</val>
> </set>
> </field>
> <field equalize="false" name="init_hour">
> <set name="init_hour_1">
> <val>06</val>
> </set>
> </field>
>
> Change to 2 init hours and 4 masks:
>
> <field equalize="false" name="vx_mask">
> <val>FULL</val>
> <val>CONUS</val>
> <val>EAST</val>
> <val>WEST</val>
> </field>
> <field equalize="false" name="init_hour">
> <val>06</val>
> <val>12</val>
> </field>
>
> And then reference those values in the naming of the titles and
output
> files:
>
>
>
>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>
>
>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>
>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>
> <title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
> Domain)</title>
>
> So that's the general idea. And when you submit this to mv_batch,
it'll
> make 8 plots instead of 1.
> And you can see how you'd easily expand this to many plots.
>
> But I worry that this may not be sufficient information to get you
going.
>
> John
>
> On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >
> > Hi John,
> >
> > I think I was able to cover your request okay. Below are 4
attachments
> > that include the test script that was used to generate the plots,
a
> sample
> > XML file, and the two plots that were generated using the test run
> script.
> > These plots were not created previously using the batch job but
uses the
> > same exact logic as the original run script. Thanks again for
working
> with
> > me on this. Please let me know if you need anything else or would
like
> me
> > to test something.
> >
> > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > Ed,
> > >
> > > Image files showing up in an unexpected output directory likely
point
> to
> > a
> > > problem in the logic of the processing scripts. It's pretty
unlikely
> that
> > > this behavior is caused by the METviewer code itself. But I've
> definitely
> > > been surprised before!
> > >
> > > I tested the logic outside of a batch job and it works fine. I
suspect
> > > that the issuance of another batch job while the other one is
getting
> > > processed is causing some confusion in the processing. These
issues
> > don't
> > > happen everyday and appear more sporadic than anything.
> > >
> > > So what we really need here is an example of taking a plot from
the
> > > METviewer GUI and adapting it to make it generate multiple
plots. I did
> > go
> > > through examples of doing this in previous tutorials, but we
didn't
> start
> > > recording them until last summer. I checked the July 2019 NRL
tutorial
> > > videos and don't see it there.
> > >
> > > Is there something you need from me to help on this? Just let
me know
> > >
> > > We've begun recording training videos for common topics like
this. But
> we
> > > only have one for METviewer so far:
> > >
> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> > >
> > > But this definitely seems like a good candidate for a new
training
> video.
> > >
> > > For now though, how about this...
> > > - You use METviewer to manually make a single plot, formatting
it how
> > you'd
> > > like.
> > > - You send me your sample plot and corresponding XML file.
> > > - I'll tweak it to generate many plots via batch instead of just
one.
> > > - And then you can continue tweaking to add more permutations.
> > >
> > > Okay, I'll work on that now. Thanks.
> > >
> > > John
> > >
> > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Ed,
> > >>
> > >> Image files showing up in an unexpected output directory likely
point
> > to a
> > >> problem in the logic of the processing scripts. It's pretty
unlikely
> > that
> > >> this behavior is caused by the METviewer code itself. But I've
> > definitely
> > >> been surprised before!
> > >>
> > >> So what we really need here is an example of taking a plot from
the
> > >> METviewer GUI and adapting it to make it generate multiple
plots. I
> did
> > go
> > >> through examples of doing this in previous tutorials, but we
didn't
> > start
> > >> recording them until last summer. I checked the July 2019 NRL
tutorial
> > >> videos and don't see it there.
> > >>
> > >> We've begun recording training videos for common topics like
this. But
> > we
> > >> only have one for METviewer so far:
> > >>
> > https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> > >>
> > >> But this definitely seems like a good candidate for a new
training
> > video.
> > >>
> > >> For now though, how about this...
> > >> - You use METviewer to manually make a single plot, formatting
it how
> > >> you'd
> > >> like.
> > >> - You send me your sample plot and corresponding XML file.
> > >> - I'll tweak it to generate many plots via batch instead of
just one.
> > >> - And then you can continue tweaking to add more permutations.
> > >>
> > >> John
> > >>
> > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>
> > >> >
> > >> > Oh, and one more thing..
> > >> >
> > >> > The green text that didn't show up was meant to highlight the
fact
> > that
> > >> > sometimes plots that shouldn't be stored in certain
directories end
> up
> > >> > there. The pasted example below shows one instance where
OZMAX* was
> > >> stored
> > >> > where TMP* should have been stored.
> > >> >
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5 10:52
> > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6 10:52
> > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7 10:00
> > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13 13:49
> > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14 11:31
> > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19 11:37
> > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20 11:35
> > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21 11:45
> > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22 11:42
> > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23 11:50
> > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24 12:57
> > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31 08:47
> > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2 08:50
> > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
> > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3 08:52
> > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
> > >> >
> > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA
Affiliate <
> > >> > edward.strobach at noaa.gov> wrote:
> > >> >
> > >> > > Hi John,
> > >> > >
> > >> > > I know there are a lot of calls and speculate that this
could be
> > part
> > >> of
> > >> > > the problem. I don't think that I'm following best
practice by
> > >> > approaching
> > >> > > it in this way. Generating plots usually works well if I
process
> a
> > >> > subset
> > >> > > of the computations outside of batch scripting. The python
script
> > was
> > >> > > given to me by Logan, but I have since made significant
changes to
> > it
> > >> in
> > >> > > order to dynamically fill in the xml file based on
parameters set.
> > >> The
> > >> > > test*.sh script was also given to me. I created the
> > Time_Series*bsub
> > >> as
> > >> > a
> > >> > > way to process a series of plots based on cycle time,
region,
> etc. I
> > >> > think
> > >> > > the first two scripts are solid and do well for what I need
to get
> > >> done.
> > >> > > It's the Time_Series*bsub script that bothers me a bit.
> > >> > >
> > >> > > I can't see how a single XML file can process all these
plots in a
> > >> single
> > >> > > setting based on what I've learned thus far. How would one
go
> about
> > >> > that?
> > >> > > I would be open going in that direction if you can provide
some
> > >> guidance.
> > >> > >
> > >> > > Unfortunately, I haven't gotten a lot of feedback or
dialogue
> > exchange
> > >> > > about what I've done so far at EMC with this new set-up.
This has
> > >> also
> > >> > > been part of the problem. I can't seem to get them to
focus so
> that
> > >> we
> > >> > can
> > >> > > prioritize. I know that I'm doing the best I can. If I
can learn
> > >> how to
> > >> > > go about this more efficiently it would be of great help.
I can
> > then
> > >> > > expand this to other verification as venture to other
projects in
> > the
> > >> > > future. I know you're busy with many things, so I'm willing
to go
> > >> along
> > >> > > with your schedule as you find an opening.
> > >> > >
> > >> > > Thanks
> > >> > >
> > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via RT <
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > >> Ed,
> > >> > >>
> > >> > >> It's really difficult for me to see an obvious problem
that's
> > causing
> > >> > the
> > >> > >> behavior you describe.
> > >> > >>
> > >> > >> Unfortunately, the red and green highlighting you describe
> doesn't
> > >> show
> > >> > up
> > >> > >> in the Gmail web client. There must be some mail system
> > >> incompatibility
> > >> > >> there.
> > >> > >>
> > >> > >> Looking at your bsub script:
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
> > >> > >>
> > >> > >> I am struck by the number of individual calls you're
making to
> > >> > METviewer:
> > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot
types = 656
> > >> calls
> > >> > to
> > >> > >> METviewer
> > >> > >>
> > >> > >> When making plots via the METviewer GUI, it's true that 1
XML = 1
> > >> plot.
> > >> > >> However, when making plots via the batch engine, the plot
XML can
> > be
> > >> set
> > >> > >> up
> > >> > >> to create many, many plots. While I have no direct proof
of this,
> > it
> > >> > might
> > >> > >> be the case that executing 600+ batch jobs for METviewer
is
> leading
> > >> to
> > >> > >> trouble on the system. One thing to try would be setting
up the
> XML
> > >> to
> > >> > >> make
> > >> > >> many, many more plots in each call. Then check to see if
running
> > >> > METviewer
> > >> > >> in this way is more stable.
> > >> > >>
> > >> > >> I don't think I fully appreciate all of the logic that
lives in
> > >> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
> > >> test_Time_Series_AGG.sh.
> > >> > >> However, it *might* be possible to construct a single XML
file
> > which
> > >> > >> produces all 656 plots.
> > >> > >>
> > >> > >> Let me know what you think and how you'd like to proceed.
> > >> > >>
> > >> > >> Thanks,
> > >> > >> John
> > >> > >>
> > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA
Affiliate
> via
> > >> RT <
> > >> > >> met_help at ucar.edu> wrote:
> > >> > >>
> > >> > >> >
> > >> > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> >
> > >> > >> >
> > >> > >> > Hi John,
> > >> > >> >
> > >> > >> > No problem.
> > >> > >> >
> > >> > >> > I'm including my responses below:
> > >> > >> > Sorry for the delay. I took a look on wcoss today. You
describe
> > >> that
> > >> > >> your
> > >> > >> > plotting script has stopped working but you're not sure
why.
> > >> > >> > It seems more intermittent. Take a look at the example
below.
> > You
> > >> > can
> > >> > >> see
> > >> > >> > that for some reason there are gaps in the plots being
> generated.
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4
14:52
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 41781 Oct 5 15:15
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 56691 Oct 19 15:02
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 55548 Oct 20 16:08
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 55598 Oct 21 16:08
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 55599 Oct 22 16:09
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 55352 Oct 24 16:34
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 26175 Oct 31 08:20
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 26178 Nov 1 08:20
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 26160 Nov 3 08:22
> > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 25354 Nov 3 08:22
> > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
> > >> > >> > You can tell when a plot is successful by looking at the
file
> > >> size. I
> > >> > >> > highlighted an example of that in red. However, the
files
> > showing
> > >> > about
> > >> > >> > 25000 are the ones where no line plots are being
produced. You
> > can
> > >> > also
> > >> > >> > see that stat files are generated everyday, so as long
as I'm
> > >> loading
> > >> > >> the
> > >> > >> > data, which I am, I should be able to use the xml files
to
> > generate
> > >> > the
> > >> > >> > plots. I'm attaching a directory list from wcoss that
shows
> the
> > >> dates
> > >> > >> > where data is produced.
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > *20190801 20190817 20200624 20200710 20200726
20200812
> > >> 20200828
> > >> > >> > 20201003 2020101920190802 20190818 20200625
20200711
> > 20200727
> > >> > >> > 20200813 20200829 20201004 2020102020190803
20190819
> > 20200626
> > >> > >> > 20200712 20200728 20200814 20200830 20201005
> > 2020102120190804
> > >> > >> > 20190820 20200627 20200713 20200729 20200815
20200831
> > >> 20201006
> > >> > >> > 2020102220190805 20190821 20200628 20200714
20200730
> > 20200816
> > >> > >> > 20200921 20201007 2020102320190806 20190822
20200629
> > 20200715
> > >> > >> > 20200731 20200817 20200922 20201008
2020102420190807
> > 20190823
> > >> > >> > 20200630 20200716 20200801 20200818 20200923
20201009
> > >> > >> > 2020102520190808 20190824 20200701 20200717
20200802
> > 20200819
> > >> > >> > 20200924 20201010 2020102620190809 20190825
20200702
> > 20200718
> > >> > >> > 20200803 20200820 20200925 20201011
2020102720190810
> > 20190826
> > >> > >> > 20200703 20200719 20200804 20200821 20200926
20201012
> > >> > >> > 2020102820190811 20190827 20200704 20200720
20200805
> > 20200822
> > >> > >> > 20200927 20201013 2020102920190812 20190828
20200705
> > 20200721
> > >> > >> > 20200806 20200823 20200928 20201014
2020103020190813
> > 20190829
> > >> > >> > 20200706 20200722 20200807 20200824 20200929
20201015
> > >> > >> > 2020103120190814 20190830 20200707 20200723
20200808
> > 20200825
> > >> > >> > 20200930 20201016 2020110120190815 20200622
20200708
> > 20200724
> > >> > >> > 20200810 20200826 20201001 20201017
2020110220190816
> > 20200623
> > >> > >> > 20200709 20200725 20200811 20200827 20201002
20201018*
> > >> > >> >
> > >> > >> > Furthermore, what's interesting is that I have mixed
success
> with
> > >> > other
> > >> > >> > cases. Using the same field - PMTF - for the same cycle
- the
> > 06z
> > >> > >> cycle -
> > >> > >> > you can see that some of the dates where data was
plotted for
> > EAST
> > >> did
> > >> > >> not
> > >> > >> > necessarily plot for FULL. This surprises me because
EAST is a
> > >> subset
> > >> > >> of
> > >> > >> > FULL. See below:
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4
14:54
> > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 49039 Oct 5 15:17
> > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 56791 Oct 20 16:09
> > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 56834 Oct 21 16:09
> > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 56921 Oct 22 16:11
> > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 56518 Oct 24 16:36
> > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 61187 Oct 31 08:21
> > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 26475 Nov 2 08:22
> > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1
> > >> Edward.Strobach
> > >> > >> > emcmodel 26455 Nov 3 08:23
> > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
> > >> > >> >
> > >> > >> > What I find equally concerning as that sometimes plots
are
> > >> incorrectly
> > >> > >> > stored in the wrong directory list as highlighted in
green
> above.
> > >> I've
> > >> > >> > consulted the plot xml file for PMTF, for instance, and
find
> > >> nothing
> > >> > >> > problematic. The only setting that might pose an issue
when it
> > >> comes
> > >> > to
> > >> > >> > generating line plots is when I set event_equal to true.
> > However,
> > >> all
> > >> > >> the
> > >> > >> > stat files are populated for all experiments. In the
case of
> > ozone
> > >> you
> > >> > >> see
> > >> > >> > something different. I find that ozone tends to result
in
> > populated
> > >> > >> plots
> > >> > >> > but with intermittent reporting different than PMTF -
see
> below:
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal 6144-rw-
r--r-- 1
> > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 54318 Oct 6 10:51
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 54305 Oct 7 10:49
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 62179 Oct 13 13:25
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 63811 Oct 19 12:18
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 60112 Oct 20 12:21
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 60165 Oct 21 12:33
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 60160 Oct 22 12:21
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 60056 Oct 23 12:40
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 58517 Oct 24 13:31
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 58062 Oct 31 14:17
> > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> > >> > Edward.Strobach
> > >> > >> > emcmodel 58119 Nov 1 14:19
> > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
> > >> > >> >
> > >> > >> > Similar issues have been found for other fields, cycles,
and
> > >> regions
> > >> > >> > including verification being done for meteorology.
> > >> > >> > I can't seem to reason this intermittency and
inconsistency
> > between
> > >> > >> > reporting since the xml files are correct and I'm using
the
> same
> > >> > >> procedure
> > >> > >> > that I've always used. I do realize that I need to
start
> > creating
> > >> a
> > >> > >> better
> > >> > >> > way to manage all these files that are being produced,
both in
> > >> terms
> > >> > of
> > >> > >> > data/plots and xml files.
> > >> > >> >
> > >> > >> > I ran across a rather large directory:
> > >> > >> > ls
> > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > >> > |
> > >> > >> wc
> > >> > >> > -w
> > >> > >> >
> > >> > >> > That has 71,398 xml files in there. But I see timestamps
in
> those
> > >> > >> filenames
> > >> > >> > from 20200921 to 20200925.
> > >> > >> >
> > >> > >> > Is it the case that this stopped working on 20200925? Or
are
> > those
> > >> > dated
> > >> > >> > filenames remnants from older work?
> > >> > >> >
> > >> > >> > So it hasn't been working for over a month. Is that
right?
> > >> > >> >
> > >> > >> > There are a lot of files and I need to begin managing
that
> > >> better. I
> > >> > >> think
> > >> > >> > during that time there was a machine switch so it
stopped on
> that
> > >> day.
> > >> > >> > I've been able to produce plots even now, but it's not
> consistent
> > >> in
> > >> > >> > reporting while some regions generate plots and others
do not.
> > >> I've
> > >> > >> > reduced the level processing in hopes that I don't hit a
walk
> > clock
> > >> > >> issue
> > >> > >> > since I'm submitting batch jobs. However, I've run this
step
> > >> outside
> > >> > of
> > >> > >> > batch and it always seems to work. This issue seems to
happen
> > with
> > >> > batch
> > >> > >> > and it is not at all clear why. Let me know what else
you need
> > >> from
> > >> > me
> > >> > >> to
> > >> > >> > better coordinate what I would need to do next. Thanks.
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via RT
<
> > >> > >> > met_help at ucar.edu>
> > >> > >> > wrote:
> > >> > >> >
> > >> > >> > > Hi Ed,
> > >> > >> > >
> > >> > >> > > Sorry for the delay. I took a look on wcoss today. You
> describe
> > >> that
> > >> > >> your
> > >> > >> > > plotting script has stopped working but you're not
sure why.
> > >> > >> > >
> > >> > >> > > I ran across a rather large directory:
> > >> > >> > > ls
> > >> >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > >> > >> |
> > >> > >> > wc
> > >> > >> > > -w
> > >> > >> > >
> > >> > >> > > That has 71,398 xml files in there. But I see
timestamps in
> > those
> > >> > >> > filenames
> > >> > >> > > from 20200921 to 20200925.
> > >> > >> > >
> > >> > >> > > Is it the case that this stopped working on 20200925?
Or are
> > >> those
> > >> > >> dated
> > >> > >> > > filenames remnants from older work?
> > >> > >> > >
> > >> > >> > > So it hasn't been working for over a month. Is that
right?
> > >> > >> > >
> > >> > >> > > John
> > >> > >> > >
> > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System itself
via RT <
> > >> > >> > > met_help at ucar.edu> wrote:
> > >> > >> > >
> > >> > >> > > >
> > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was acted
upon.
> > >> > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
> > RT_System
> > >> > >> > > > Queue: met_help
> > >> > >> > > > Subject: plots no longer being generated
> > >> > >> > > > Owner: johnhg
> > >> > >> > > > Requestors: edward.strobach at noaa.gov
> > >> > >> > > > Status: new
> > >> > >> > > > Ticket <URL:
> > >> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > >> > >> > >
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > This transaction appears to have no content
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> >
> > >> > >> > --
> > >> > >> > Edward Strobach
> > >> > >> > EMC/NCEP/NWS/
> > >> > >> > IMSG Contractor
> > >> > >> > Cubicle#: 2029
> > >> > >> > 301-683-3717
> > >> > >> >
> > >> > >> >
> > >> > >>
> > >> > >>
> > >> > >
> > >> > > --
> > >> > > Edward Strobach
> > >> > > EMC/NCEP/NWS/
> > >> > > IMSG Contractor
> > >> > > Cubicle#: 2029
> > >> > > 301-683-3717
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > Edward Strobach
> > >> > EMC/NCEP/NWS/
> > >> > IMSG Contractor
> > >> > Cubicle#: 2029
> > >> > 301-683-3717
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: John Halley Gotway
Time: Fri Nov 06 08:53:34 2020
Ed,
You're right about the METviewer GUI grouping multiple selections
together.
That looks like this in the XML using the <set> tag:
<field equalize="false" name="vx_mask">
<set name="vx_mask_0">
<val>FULL</val>
</set>
</field>
But in the updates I sent, I remove the <set> tag:
<field equalize="false" name="vx_mask">
<val>FULL</val>
<val>CONUS</val>
<val>EAST</val>
<val>WEST</val>
</field>
So instead of processing this as 1 group of 4 values, it'll process
the 4
values separately.
The current vx_mask value is referenced using {vx_mask} syntax. And I
included examples of using that in the output file names and titles in
my
previous email:
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
<title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
Domain)</title>
John
On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>
> Hi John,
>
> This definitely helps. I never thought about it this way because I
> assumed, based on my experience with the interactive MET GUI, that I
could
> only generate a single plot regardless of how I specified my
settings in
> the XML. For example, if I listed EAST and WEST under vx_mask, then
my
> thinking would be that both those regions would be grouped and a
single
> plot created. I guess I was wrong on that. The concern I still
have,
> however, is that the plot title and plot name - the png file - is
also set
> in the XML file. Even after adding the list you gave, I can't see
how
> unique file names will be created. Is that true?
>
> On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Ed,
> >
> > Thanks for sending examples. My goal was to use METviewer's XML
upload
> > functionality but I'm not able to upload a pdf version of the xml
file.
> >
> > I did spend some time trying to replicate this exact plot with
METviewer
> > at:
> > https://metviewer.nws.noaa.gov/metviewer1.jsp
> >
> > But I was never quite successful. So I made just a similar version
> instead.
> > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
> >
> > This plots FBAR and OBAR for 6 different models for the month of
October
> > for all 06Z initializations over the FULL model domain.
> >
> > I wanted to show you how to modify this for batch mode... and then
adapt
> it
> > to make multiple plots. But I'm not able to make batch plots on
wcoss
> using
> > the AWS instance. I don't think I have the right permissions to do
so.
> >
> > So let me just say this. Generally, you just update the list of
items in
> > plot_fix section.
> >
> > Original from METviewer with 1 init hour and 1 mask:
> >
> > <field equalize="false" name="vx_mask">
> > <set name="vx_mask_0">
> > <val>FULL</val>
> > </set>
> > </field>
> > <field equalize="false" name="init_hour">
> > <set name="init_hour_1">
> > <val>06</val>
> > </set>
> > </field>
> >
> > Change to 2 init hours and 4 masks:
> >
> > <field equalize="false" name="vx_mask">
> > <val>FULL</val>
> > <val>CONUS</val>
> > <val>EAST</val>
> > <val>WEST</val>
> > </field>
> > <field equalize="false" name="init_hour">
> > <val>06</val>
> > <val>12</val>
> > </field>
> >
> > And then reference those values in the naming of the titles and
output
> > files:
> >
> >
> >
> >
>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
> >
> >
> >
>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
> >
> >
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
> >
> > <title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
> > Domain)</title>
> >
> > So that's the general idea. And when you submit this to mv_batch,
it'll
> > make 8 plots instead of 1.
> > And you can see how you'd easily expand this to many plots.
> >
> > But I worry that this may not be sufficient information to get you
going.
> >
> > John
> >
> > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> > >
> > > Hi John,
> > >
> > > I think I was able to cover your request okay. Below are 4
attachments
> > > that include the test script that was used to generate the
plots, a
> > sample
> > > XML file, and the two plots that were generated using the test
run
> > script.
> > > These plots were not created previously using the batch job but
uses
> the
> > > same exact logic as the original run script. Thanks again for
working
> > with
> > > me on this. Please let me know if you need anything else or
would like
> > me
> > > to test something.
> > >
> > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA Affiliate
<
> > > edward.strobach at noaa.gov> wrote:
> > >
> > > > Ed,
> > > >
> > > > Image files showing up in an unexpected output directory
likely point
> > to
> > > a
> > > > problem in the logic of the processing scripts. It's pretty
unlikely
> > that
> > > > this behavior is caused by the METviewer code itself. But I've
> > definitely
> > > > been surprised before!
> > > >
> > > > I tested the logic outside of a batch job and it works fine.
I
> suspect
> > > > that the issuance of another batch job while the other one is
getting
> > > > processed is causing some confusion in the processing. These
issues
> > > don't
> > > > happen everyday and appear more sporadic than anything.
> > > >
> > > > So what we really need here is an example of taking a plot
from the
> > > > METviewer GUI and adapting it to make it generate multiple
plots. I
> did
> > > go
> > > > through examples of doing this in previous tutorials, but we
didn't
> > start
> > > > recording them until last summer. I checked the July 2019 NRL
> tutorial
> > > > videos and don't see it there.
> > > >
> > > > Is there something you need from me to help on this? Just let
me
> know
> > > >
> > > > We've begun recording training videos for common topics like
this.
> But
> > we
> > > > only have one for METviewer so far:
> > > >
> > https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> > > >
> > > > But this definitely seems like a good candidate for a new
training
> > video.
> > > >
> > > > For now though, how about this...
> > > > - You use METviewer to manually make a single plot, formatting
it how
> > > you'd
> > > > like.
> > > > - You send me your sample plot and corresponding XML file.
> > > > - I'll tweak it to generate many plots via batch instead of
just one.
> > > > - And then you can continue tweaking to add more permutations.
> > > >
> > > > Okay, I'll work on that now. Thanks.
> > > >
> > > > John
> > > >
> > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Ed,
> > > >>
> > > >> Image files showing up in an unexpected output directory
likely
> point
> > > to a
> > > >> problem in the logic of the processing scripts. It's pretty
unlikely
> > > that
> > > >> this behavior is caused by the METviewer code itself. But
I've
> > > definitely
> > > >> been surprised before!
> > > >>
> > > >> So what we really need here is an example of taking a plot
from the
> > > >> METviewer GUI and adapting it to make it generate multiple
plots. I
> > did
> > > go
> > > >> through examples of doing this in previous tutorials, but we
didn't
> > > start
> > > >> recording them until last summer. I checked the July 2019 NRL
> tutorial
> > > >> videos and don't see it there.
> > > >>
> > > >> We've begun recording training videos for common topics like
this.
> But
> > > we
> > > >> only have one for METviewer so far:
> > > >>
> > >
> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> > > >>
> > > >> But this definitely seems like a good candidate for a new
training
> > > video.
> > > >>
> > > >> For now though, how about this...
> > > >> - You use METviewer to manually make a single plot,
formatting it
> how
> > > >> you'd
> > > >> like.
> > > >> - You send me your sample plot and corresponding XML file.
> > > >> - I'll tweak it to generate many plots via batch instead of
just
> one.
> > > >> - And then you can continue tweaking to add more
permutations.
> > > >>
> > > >> John
> > > >>
> > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA
Affiliate via
> > RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> > > >> >
> > > >> > Oh, and one more thing..
> > > >> >
> > > >> > The green text that didn't show up was meant to highlight
the fact
> > > that
> > > >> > sometimes plots that shouldn't be stored in certain
directories
> end
> > up
> > > >> > there. The pasted example below shows one instance where
OZMAX*
> was
> > > >> stored
> > > >> > where TMP* should have been stored.
> > > >> >
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5 10:52
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6 10:52
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7 10:00
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13 13:49
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14 11:31
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19 11:37
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20 11:35
> > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21 11:45
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22 11:42
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23 11:50
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24 12:57
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31 08:47
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2 08:50
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
> > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3 08:52
> > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
> > > >> >
> > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA
Affiliate <
> > > >> > edward.strobach at noaa.gov> wrote:
> > > >> >
> > > >> > > Hi John,
> > > >> > >
> > > >> > > I know there are a lot of calls and speculate that this
could be
> > > part
> > > >> of
> > > >> > > the problem. I don't think that I'm following best
practice by
> > > >> > approaching
> > > >> > > it in this way. Generating plots usually works well if I
> process
> > a
> > > >> > subset
> > > >> > > of the computations outside of batch scripting. The
python
> script
> > > was
> > > >> > > given to me by Logan, but I have since made significant
changes
> to
> > > it
> > > >> in
> > > >> > > order to dynamically fill in the xml file based on
parameters
> set.
> > > >> The
> > > >> > > test*.sh script was also given to me. I created the
> > > Time_Series*bsub
> > > >> as
> > > >> > a
> > > >> > > way to process a series of plots based on cycle time,
region,
> > etc. I
> > > >> > think
> > > >> > > the first two scripts are solid and do well for what I
need to
> get
> > > >> done.
> > > >> > > It's the Time_Series*bsub script that bothers me a bit.
> > > >> > >
> > > >> > > I can't see how a single XML file can process all these
plots
> in a
> > > >> single
> > > >> > > setting based on what I've learned thus far. How would
one go
> > about
> > > >> > that?
> > > >> > > I would be open going in that direction if you can
provide some
> > > >> guidance.
> > > >> > >
> > > >> > > Unfortunately, I haven't gotten a lot of feedback or
dialogue
> > > exchange
> > > >> > > about what I've done so far at EMC with this new set-up.
This
> has
> > > >> also
> > > >> > > been part of the problem. I can't seem to get them to
focus so
> > that
> > > >> we
> > > >> > can
> > > >> > > prioritize. I know that I'm doing the best I can. If I
can
> learn
> > > >> how to
> > > >> > > go about this more efficiently it would be of great help.
I can
> > > then
> > > >> > > expand this to other verification as venture to other
projects
> in
> > > the
> > > >> > > future. I know you're busy with many things, so I'm
willing to
> go
> > > >> along
> > > >> > > with your schedule as you find an opening.
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via RT
<
> > > >> > > met_help at ucar.edu> wrote:
> > > >> > >
> > > >> > >> Ed,
> > > >> > >>
> > > >> > >> It's really difficult for me to see an obvious problem
that's
> > > causing
> > > >> > the
> > > >> > >> behavior you describe.
> > > >> > >>
> > > >> > >> Unfortunately, the red and green highlighting you
describe
> > doesn't
> > > >> show
> > > >> > up
> > > >> > >> in the Gmail web client. There must be some mail system
> > > >> incompatibility
> > > >> > >> there.
> > > >> > >>
> > > >> > >> Looking at your bsub script:
> > > >> > >>
> > > >> > >>
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
> > > >> > >>
> > > >> > >> I am struck by the number of individual calls you're
making to
> > > >> > METviewer:
> > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot
types =
> 656
> > > >> calls
> > > >> > to
> > > >> > >> METviewer
> > > >> > >>
> > > >> > >> When making plots via the METviewer GUI, it's true that
1 XML
> = 1
> > > >> plot.
> > > >> > >> However, when making plots via the batch engine, the
plot XML
> can
> > > be
> > > >> set
> > > >> > >> up
> > > >> > >> to create many, many plots. While I have no direct proof
of
> this,
> > > it
> > > >> > might
> > > >> > >> be the case that executing 600+ batch jobs for METviewer
is
> > leading
> > > >> to
> > > >> > >> trouble on the system. One thing to try would be setting
up the
> > XML
> > > >> to
> > > >> > >> make
> > > >> > >> many, many more plots in each call. Then check to see if
> running
> > > >> > METviewer
> > > >> > >> in this way is more stable.
> > > >> > >>
> > > >> > >> I don't think I fully appreciate all of the logic that
lives in
> > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
> > > >> test_Time_Series_AGG.sh.
> > > >> > >> However, it *might* be possible to construct a single
XML file
> > > which
> > > >> > >> produces all 656 plots.
> > > >> > >>
> > > >> > >> Let me know what you think and how you'd like to
proceed.
> > > >> > >>
> > > >> > >> Thanks,
> > > >> > >> John
> > > >> > >>
> > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA
Affiliate
> > via
> > > >> RT <
> > > >> > >> met_help at ucar.edu> wrote:
> > > >> > >>
> > > >> > >> >
> > > >> > >> > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > >
> > > >> > >> >
> > > >> > >> > Hi John,
> > > >> > >> >
> > > >> > >> > No problem.
> > > >> > >> >
> > > >> > >> > I'm including my responses below:
> > > >> > >> > Sorry for the delay. I took a look on wcoss today. You
> describe
> > > >> that
> > > >> > >> your
> > > >> > >> > plotting script has stopped working but you're not
sure why.
> > > >> > >> > It seems more intermittent. Take a look at the
example
> below.
> > > You
> > > >> > can
> > > >> > >> see
> > > >> > >> > that for some reason there are gaps in the plots being
> > generated.
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4
14:52
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 41781 Oct 5 15:15
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 56691 Oct 19 15:02
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 55548 Oct 20 16:08
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 55598 Oct 21 16:08
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 55599 Oct 22 16:09
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 55352 Oct 24 16:34
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 26175 Oct 31 08:20
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 26178 Nov 1 08:20
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 26160 Nov 3 08:22
> > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 25354 Nov 3 08:22
> > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
> > > >> > >> > You can tell when a plot is successful by looking at
the file
> > > >> size. I
> > > >> > >> > highlighted an example of that in red. However, the
files
> > > showing
> > > >> > about
> > > >> > >> > 25000 are the ones where no line plots are being
produced.
> You
> > > can
> > > >> > also
> > > >> > >> > see that stat files are generated everyday, so as long
as I'm
> > > >> loading
> > > >> > >> the
> > > >> > >> > data, which I am, I should be able to use the xml
files to
> > > generate
> > > >> > the
> > > >> > >> > plots. I'm attaching a directory list from wcoss that
shows
> > the
> > > >> dates
> > > >> > >> > where data is produced.
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > *20190801 20190817 20200624 20200710 20200726
20200812
> > > >> 20200828
> > > >> > >> > 20201003 2020101920190802 20190818 20200625
20200711
> > > 20200727
> > > >> > >> > 20200813 20200829 20201004 2020102020190803
20190819
> > > 20200626
> > > >> > >> > 20200712 20200728 20200814 20200830 20201005
> > > 2020102120190804
> > > >> > >> > 20190820 20200627 20200713 20200729 20200815
20200831
> > > >> 20201006
> > > >> > >> > 2020102220190805 20190821 20200628 20200714
20200730
> > > 20200816
> > > >> > >> > 20200921 20201007 2020102320190806 20190822
20200629
> > > 20200715
> > > >> > >> > 20200731 20200817 20200922 20201008
2020102420190807
> > > 20190823
> > > >> > >> > 20200630 20200716 20200801 20200818 20200923
20201009
> > > >> > >> > 2020102520190808 20190824 20200701 20200717
20200802
> > > 20200819
> > > >> > >> > 20200924 20201010 2020102620190809 20190825
20200702
> > > 20200718
> > > >> > >> > 20200803 20200820 20200925 20201011
2020102720190810
> > > 20190826
> > > >> > >> > 20200703 20200719 20200804 20200821 20200926
20201012
> > > >> > >> > 2020102820190811 20190827 20200704 20200720
20200805
> > > 20200822
> > > >> > >> > 20200927 20201013 2020102920190812 20190828
20200705
> > > 20200721
> > > >> > >> > 20200806 20200823 20200928 20201014
2020103020190813
> > > 20190829
> > > >> > >> > 20200706 20200722 20200807 20200824 20200929
20201015
> > > >> > >> > 2020103120190814 20190830 20200707 20200723
20200808
> > > 20200825
> > > >> > >> > 20200930 20201016 2020110120190815 20200622
20200708
> > > 20200724
> > > >> > >> > 20200810 20200826 20201001 20201017
2020110220190816
> > > 20200623
> > > >> > >> > 20200709 20200725 20200811 20200827 20201002
20201018*
> > > >> > >> >
> > > >> > >> > Furthermore, what's interesting is that I have mixed
success
> > with
> > > >> > other
> > > >> > >> > cases. Using the same field - PMTF - for the same
cycle -
> the
> > > 06z
> > > >> > >> cycle -
> > > >> > >> > you can see that some of the dates where data was
plotted for
> > > EAST
> > > >> did
> > > >> > >> not
> > > >> > >> > necessarily plot for FULL. This surprises me because
EAST
> is a
> > > >> subset
> > > >> > >> of
> > > >> > >> > FULL. See below:
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4
14:54
> > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 49039 Oct 5 15:17
> > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 56791 Oct 20 16:09
> > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 56834 Oct 21 16:09
> > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 56921 Oct 22 16:11
> > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 56518 Oct 24 16:36
> > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 61187 Oct 31 08:21
> > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 26475 Nov 2 08:22
> > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r-- 1
> > > >> Edward.Strobach
> > > >> > >> > emcmodel 26455 Nov 3 08:23
> > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
> > > >> > >> >
> > > >> > >> > What I find equally concerning as that sometimes plots
are
> > > >> incorrectly
> > > >> > >> > stored in the wrong directory list as highlighted in
green
> > above.
> > > >> I've
> > > >> > >> > consulted the plot xml file for PMTF, for instance,
and find
> > > >> nothing
> > > >> > >> > problematic. The only setting that might pose an
issue when
> it
> > > >> comes
> > > >> > to
> > > >> > >> > generating line plots is when I set event_equal to
true.
> > > However,
> > > >> all
> > > >> > >> the
> > > >> > >> > stat files are populated for all experiments. In the
case of
> > > ozone
> > > >> you
> > > >> > >> see
> > > >> > >> > something different. I find that ozone tends to result
in
> > > populated
> > > >> > >> plots
> > > >> > >> > but with intermittent reporting different than PMTF -
see
> > below:
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal
> 6144-rw-r--r-- 1
> > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 54318 Oct 6 10:51
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 54305 Oct 7 10:49
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 62179 Oct 13 13:25
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 63811 Oct 19 12:18
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 60112 Oct 20 12:21
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 60165 Oct 21 12:33
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 60160 Oct 22 12:21
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 60056 Oct 23 12:40
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 58517 Oct 24 13:31
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 58062 Oct 31 14:17
> > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r--
1
> > > >> > Edward.Strobach
> > > >> > >> > emcmodel 58119 Nov 1 14:19
> > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
> > > >> > >> >
> > > >> > >> > Similar issues have been found for other fields,
cycles, and
> > > >> regions
> > > >> > >> > including verification being done for meteorology.
> > > >> > >> > I can't seem to reason this intermittency and
inconsistency
> > > between
> > > >> > >> > reporting since the xml files are correct and I'm
using the
> > same
> > > >> > >> procedure
> > > >> > >> > that I've always used. I do realize that I need to
start
> > > creating
> > > >> a
> > > >> > >> better
> > > >> > >> > way to manage all these files that are being produced,
both
> in
> > > >> terms
> > > >> > of
> > > >> > >> > data/plots and xml files.
> > > >> > >> >
> > > >> > >> > I ran across a rather large directory:
> > > >> > >> > ls
> > > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > > >> > |
> > > >> > >> wc
> > > >> > >> > -w
> > > >> > >> >
> > > >> > >> > That has 71,398 xml files in there. But I see
timestamps in
> > those
> > > >> > >> filenames
> > > >> > >> > from 20200921 to 20200925.
> > > >> > >> >
> > > >> > >> > Is it the case that this stopped working on 20200925?
Or are
> > > those
> > > >> > dated
> > > >> > >> > filenames remnants from older work?
> > > >> > >> >
> > > >> > >> > So it hasn't been working for over a month. Is that
right?
> > > >> > >> >
> > > >> > >> > There are a lot of files and I need to begin managing
that
> > > >> better. I
> > > >> > >> think
> > > >> > >> > during that time there was a machine switch so it
stopped on
> > that
> > > >> day.
> > > >> > >> > I've been able to produce plots even now, but it's not
> > consistent
> > > >> in
> > > >> > >> > reporting while some regions generate plots and others
do
> not.
> > > >> I've
> > > >> > >> > reduced the level processing in hopes that I don't hit
a walk
> > > clock
> > > >> > >> issue
> > > >> > >> > since I'm submitting batch jobs. However, I've run
this step
> > > >> outside
> > > >> > of
> > > >> > >> > batch and it always seems to work. This issue seems to
happen
> > > with
> > > >> > batch
> > > >> > >> > and it is not at all clear why. Let me know what else
you
> need
> > > >> from
> > > >> > me
> > > >> > >> to
> > > >> > >> > better coordinate what I would need to do next.
Thanks.
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway via
RT <
> > > >> > >> > met_help at ucar.edu>
> > > >> > >> > wrote:
> > > >> > >> >
> > > >> > >> > > Hi Ed,
> > > >> > >> > >
> > > >> > >> > > Sorry for the delay. I took a look on wcoss today.
You
> > describe
> > > >> that
> > > >> > >> your
> > > >> > >> > > plotting script has stopped working but you're not
sure
> why.
> > > >> > >> > >
> > > >> > >> > > I ran across a rather large directory:
> > > >> > >> > > ls
> > > >> >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > > >> > >> |
> > > >> > >> > wc
> > > >> > >> > > -w
> > > >> > >> > >
> > > >> > >> > > That has 71,398 xml files in there. But I see
timestamps in
> > > those
> > > >> > >> > filenames
> > > >> > >> > > from 20200921 to 20200925.
> > > >> > >> > >
> > > >> > >> > > Is it the case that this stopped working on
20200925? Or
> are
> > > >> those
> > > >> > >> dated
> > > >> > >> > > filenames remnants from older work?
> > > >> > >> > >
> > > >> > >> > > So it hasn't been working for over a month. Is that
right?
> > > >> > >> > >
> > > >> > >> > > John
> > > >> > >> > >
> > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System itself
via RT
> <
> > > >> > >> > > met_help at ucar.edu> wrote:
> > > >> > >> > >
> > > >> > >> > > >
> > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was acted
upon.
> > > >> > >> > > > Transaction: Given to johnhg (John Halley Gotway)
by
> > > RT_System
> > > >> > >> > > > Queue: met_help
> > > >> > >> > > > Subject: plots no longer being generated
> > > >> > >> > > > Owner: johnhg
> > > >> > >> > > > Requestors: edward.strobach at noaa.gov
> > > >> > >> > > > Status: new
> > > >> > >> > > > Ticket <URL:
> > > >> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > > >> > >> > >
> > > >> > >> > > >
> > > >> > >> > > >
> > > >> > >> > > > This transaction appears to have no content
> > > >> > >> > > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> >
> > > >> > >> > --
> > > >> > >> > Edward Strobach
> > > >> > >> > EMC/NCEP/NWS/
> > > >> > >> > IMSG Contractor
> > > >> > >> > Cubicle#: 2029
> > > >> > >> > 301-683-3717
> > > >> > >> >
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> > > --
> > > >> > > Edward Strobach
> > > >> > > EMC/NCEP/NWS/
> > > >> > > IMSG Contractor
> > > >> > > Cubicle#: 2029
> > > >> > > 301-683-3717
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Edward Strobach
> > > >> > EMC/NCEP/NWS/
> > > >> > IMSG Contractor
> > > >> > Cubicle#: 2029
> > > >> > 301-683-3717
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Mon Nov 09 05:42:18 2020
Thanks John. Let me try that and I'll let you know how it goes.
On Fri, Nov 6, 2020 at 10:53 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Ed,
>
> You're right about the METviewer GUI grouping multiple selections
together.
> That looks like this in the XML using the <set> tag:
> <field equalize="false" name="vx_mask">
> <set name="vx_mask_0">
> <val>FULL</val>
> </set>
> </field>
>
> But in the updates I sent, I remove the <set> tag:
> <field equalize="false" name="vx_mask">
> <val>FULL</val>
> <val>CONUS</val>
> <val>EAST</val>
> <val>WEST</val>
> </field>
> So instead of processing this as 1 group of 4 values, it'll process
the 4
> values separately.
>
> The current vx_mask value is referenced using {vx_mask} syntax. And
I
> included examples of using that in the output file names and titles
in my
> previous email:
>
>
>
>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>
>
>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>
>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>
> <title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
> Domain)</title>
>
> John
>
> On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >
> > Hi John,
> >
> > This definitely helps. I never thought about it this way because
I
> > assumed, based on my experience with the interactive MET GUI, that
I
> could
> > only generate a single plot regardless of how I specified my
settings in
> > the XML. For example, if I listed EAST and WEST under vx_mask,
then my
> > thinking would be that both those regions would be grouped and a
single
> > plot created. I guess I was wrong on that. The concern I still
have,
> > however, is that the plot title and plot name - the png file - is
also
> set
> > in the XML file. Even after adding the list you gave, I can't see
how
> > unique file names will be created. Is that true?
> >
> > On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Ed,
> > >
> > > Thanks for sending examples. My goal was to use METviewer's XML
upload
> > > functionality but I'm not able to upload a pdf version of the
xml file.
> > >
> > > I did spend some time trying to replicate this exact plot with
> METviewer
> > > at:
> > > https://metviewer.nws.noaa.gov/metviewer1.jsp
> > >
> > > But I was never quite successful. So I made just a similar
version
> > instead.
> > > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
> > >
> > > This plots FBAR and OBAR for 6 different models for the month of
> October
> > > for all 06Z initializations over the FULL model domain.
> > >
> > > I wanted to show you how to modify this for batch mode... and
then
> adapt
> > it
> > > to make multiple plots. But I'm not able to make batch plots on
wcoss
> > using
> > > the AWS instance. I don't think I have the right permissions to
do so.
> > >
> > > So let me just say this. Generally, you just update the list of
items
> in
> > > plot_fix section.
> > >
> > > Original from METviewer with 1 init hour and 1 mask:
> > >
> > > <field equalize="false" name="vx_mask">
> > > <set name="vx_mask_0">
> > > <val>FULL</val>
> > > </set>
> > > </field>
> > > <field equalize="false" name="init_hour">
> > > <set name="init_hour_1">
> > > <val>06</val>
> > > </set>
> > > </field>
> > >
> > > Change to 2 init hours and 4 masks:
> > >
> > > <field equalize="false" name="vx_mask">
> > > <val>FULL</val>
> > > <val>CONUS</val>
> > > <val>EAST</val>
> > > <val>WEST</val>
> > > </field>
> > > <field equalize="false" name="init_hour">
> > > <val>06</val>
> > > <val>12</val>
> > > </field>
> > >
> > > And then reference those values in the naming of the titles and
output
> > > files:
> > >
> > >
> > >
> > >
> >
>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
> > >
> > >
> > >
> >
>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
> > >
> > >
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
> > >
> > > <title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
> > > Domain)</title>
> > >
> > > So that's the general idea. And when you submit this to
mv_batch, it'll
> > > make 8 plots instead of 1.
> > > And you can see how you'd easily expand this to many plots.
> > >
> > > But I worry that this may not be sufficient information to get
you
> going.
> > >
> > > John
> > >
> > > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>
> > > >
> > > > Hi John,
> > > >
> > > > I think I was able to cover your request okay. Below are 4
> attachments
> > > > that include the test script that was used to generate the
plots, a
> > > sample
> > > > XML file, and the two plots that were generated using the test
run
> > > script.
> > > > These plots were not created previously using the batch job
but uses
> > the
> > > > same exact logic as the original run script. Thanks again for
> working
> > > with
> > > > me on this. Please let me know if you need anything else or
would
> like
> > > me
> > > > to test something.
> > > >
> > > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA
Affiliate <
> > > > edward.strobach at noaa.gov> wrote:
> > > >
> > > > > Ed,
> > > > >
> > > > > Image files showing up in an unexpected output directory
likely
> point
> > > to
> > > > a
> > > > > problem in the logic of the processing scripts. It's pretty
> unlikely
> > > that
> > > > > this behavior is caused by the METviewer code itself. But
I've
> > > definitely
> > > > > been surprised before!
> > > > >
> > > > > I tested the logic outside of a batch job and it works fine.
I
> > suspect
> > > > > that the issuance of another batch job while the other one
is
> getting
> > > > > processed is causing some confusion in the processing.
These
> issues
> > > > don't
> > > > > happen everyday and appear more sporadic than anything.
> > > > >
> > > > > So what we really need here is an example of taking a plot
from the
> > > > > METviewer GUI and adapting it to make it generate multiple
plots. I
> > did
> > > > go
> > > > > through examples of doing this in previous tutorials, but we
didn't
> > > start
> > > > > recording them until last summer. I checked the July 2019
NRL
> > tutorial
> > > > > videos and don't see it there.
> > > > >
> > > > > Is there something you need from me to help on this? Just
let me
> > know
> > > > >
> > > > > We've begun recording training videos for common topics like
this.
> > But
> > > we
> > > > > only have one for METviewer so far:
> > > > >
> > >
> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> > > > >
> > > > > But this definitely seems like a good candidate for a new
training
> > > video.
> > > > >
> > > > > For now though, how about this...
> > > > > - You use METviewer to manually make a single plot,
formatting it
> how
> > > > you'd
> > > > > like.
> > > > > - You send me your sample plot and corresponding XML file.
> > > > > - I'll tweak it to generate many plots via batch instead of
just
> one.
> > > > > - And then you can continue tweaking to add more
permutations.
> > > > >
> > > > > Okay, I'll work on that now. Thanks.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >> Ed,
> > > > >>
> > > > >> Image files showing up in an unexpected output directory
likely
> > point
> > > > to a
> > > > >> problem in the logic of the processing scripts. It's pretty
> unlikely
> > > > that
> > > > >> this behavior is caused by the METviewer code itself. But
I've
> > > > definitely
> > > > >> been surprised before!
> > > > >>
> > > > >> So what we really need here is an example of taking a plot
from
> the
> > > > >> METviewer GUI and adapting it to make it generate multiple
plots.
> I
> > > did
> > > > go
> > > > >> through examples of doing this in previous tutorials, but
we
> didn't
> > > > start
> > > > >> recording them until last summer. I checked the July 2019
NRL
> > tutorial
> > > > >> videos and don't see it there.
> > > > >>
> > > > >> We've begun recording training videos for common topics
like this.
> > But
> > > > we
> > > > >> only have one for METviewer so far:
> > > > >>
> > > >
> > https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> > > > >>
> > > > >> But this definitely seems like a good candidate for a new
training
> > > > video.
> > > > >>
> > > > >> For now though, how about this...
> > > > >> - You use METviewer to manually make a single plot,
formatting it
> > how
> > > > >> you'd
> > > > >> like.
> > > > >> - You send me your sample plot and corresponding XML file.
> > > > >> - I'll tweak it to generate many plots via batch instead of
just
> > one.
> > > > >> - And then you can continue tweaking to add more
permutations.
> > > > >>
> > > > >> John
> > > > >>
> > > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA
Affiliate
> via
> > > RT <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> > > > >> >
> > > > >> > Oh, and one more thing..
> > > > >> >
> > > > >> > The green text that didn't show up was meant to highlight
the
> fact
> > > > that
> > > > >> > sometimes plots that shouldn't be stored in certain
directories
> > end
> > > up
> > > > >> > there. The pasted example below shows one instance where
OZMAX*
> > was
> > > > >> stored
> > > > >> > where TMP* should have been stored.
> > > > >> >
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5 10:52
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6 10:52
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7 10:00
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13 13:49
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14 11:31
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19 11:37
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20 11:35
> > > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21 11:45
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22 11:42
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23 11:50
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24 12:57
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31 08:47
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2 08:50
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3 08:52
> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
> > > > >> >
> > > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA
> Affiliate <
> > > > >> > edward.strobach at noaa.gov> wrote:
> > > > >> >
> > > > >> > > Hi John,
> > > > >> > >
> > > > >> > > I know there are a lot of calls and speculate that this
could
> be
> > > > part
> > > > >> of
> > > > >> > > the problem. I don't think that I'm following best
practice
> by
> > > > >> > approaching
> > > > >> > > it in this way. Generating plots usually works well if
I
> > process
> > > a
> > > > >> > subset
> > > > >> > > of the computations outside of batch scripting. The
python
> > script
> > > > was
> > > > >> > > given to me by Logan, but I have since made significant
> changes
> > to
> > > > it
> > > > >> in
> > > > >> > > order to dynamically fill in the xml file based on
parameters
> > set.
> > > > >> The
> > > > >> > > test*.sh script was also given to me. I created the
> > > > Time_Series*bsub
> > > > >> as
> > > > >> > a
> > > > >> > > way to process a series of plots based on cycle time,
region,
> > > etc. I
> > > > >> > think
> > > > >> > > the first two scripts are solid and do well for what I
need to
> > get
> > > > >> done.
> > > > >> > > It's the Time_Series*bsub script that bothers me a bit.
> > > > >> > >
> > > > >> > > I can't see how a single XML file can process all these
plots
> > in a
> > > > >> single
> > > > >> > > setting based on what I've learned thus far. How would
one go
> > > about
> > > > >> > that?
> > > > >> > > I would be open going in that direction if you can
provide
> some
> > > > >> guidance.
> > > > >> > >
> > > > >> > > Unfortunately, I haven't gotten a lot of feedback or
dialogue
> > > > exchange
> > > > >> > > about what I've done so far at EMC with this new set-
up. This
> > has
> > > > >> also
> > > > >> > > been part of the problem. I can't seem to get them to
focus
> so
> > > that
> > > > >> we
> > > > >> > can
> > > > >> > > prioritize. I know that I'm doing the best I can. If
I can
> > learn
> > > > >> how to
> > > > >> > > go about this more efficiently it would be of great
help. I
> can
> > > > then
> > > > >> > > expand this to other verification as venture to other
projects
> > in
> > > > the
> > > > >> > > future. I know you're busy with many things, so I'm
willing to
> > go
> > > > >> along
> > > > >> > > with your schedule as you find an opening.
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > >
> > > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via
RT <
> > > > >> > > met_help at ucar.edu> wrote:
> > > > >> > >
> > > > >> > >> Ed,
> > > > >> > >>
> > > > >> > >> It's really difficult for me to see an obvious problem
that's
> > > > causing
> > > > >> > the
> > > > >> > >> behavior you describe.
> > > > >> > >>
> > > > >> > >> Unfortunately, the red and green highlighting you
describe
> > > doesn't
> > > > >> show
> > > > >> > up
> > > > >> > >> in the Gmail web client. There must be some mail
system
> > > > >> incompatibility
> > > > >> > >> there.
> > > > >> > >>
> > > > >> > >> Looking at your bsub script:
> > > > >> > >>
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
> > > > >> > >>
> > > > >> > >> I am struck by the number of individual calls you're
making
> to
> > > > >> > METviewer:
> > > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot
types =
> > 656
> > > > >> calls
> > > > >> > to
> > > > >> > >> METviewer
> > > > >> > >>
> > > > >> > >> When making plots via the METviewer GUI, it's true
that 1 XML
> > = 1
> > > > >> plot.
> > > > >> > >> However, when making plots via the batch engine, the
plot XML
> > can
> > > > be
> > > > >> set
> > > > >> > >> up
> > > > >> > >> to create many, many plots. While I have no direct
proof of
> > this,
> > > > it
> > > > >> > might
> > > > >> > >> be the case that executing 600+ batch jobs for
METviewer is
> > > leading
> > > > >> to
> > > > >> > >> trouble on the system. One thing to try would be
setting up
> the
> > > XML
> > > > >> to
> > > > >> > >> make
> > > > >> > >> many, many more plots in each call. Then check to see
if
> > running
> > > > >> > METviewer
> > > > >> > >> in this way is more stable.
> > > > >> > >>
> > > > >> > >> I don't think I fully appreciate all of the logic that
lives
> in
> > > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
> > > > >> test_Time_Series_AGG.sh.
> > > > >> > >> However, it *might* be possible to construct a single
XML
> file
> > > > which
> > > > >> > >> produces all 656 plots.
> > > > >> > >>
> > > > >> > >> Let me know what you think and how you'd like to
proceed.
> > > > >> > >>
> > > > >> > >> Thanks,
> > > > >> > >> John
> > > > >> > >>
> > > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA
> Affiliate
> > > via
> > > > >> RT <
> > > > >> > >> met_help at ucar.edu> wrote:
> > > > >> > >>
> > > > >> > >> >
> > > > >> > >> > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > > >
> > > > >> > >> >
> > > > >> > >> > Hi John,
> > > > >> > >> >
> > > > >> > >> > No problem.
> > > > >> > >> >
> > > > >> > >> > I'm including my responses below:
> > > > >> > >> > Sorry for the delay. I took a look on wcoss today.
You
> > describe
> > > > >> that
> > > > >> > >> your
> > > > >> > >> > plotting script has stopped working but you're not
sure
> why.
> > > > >> > >> > It seems more intermittent. Take a look at the
example
> > below.
> > > > You
> > > > >> > can
> > > > >> > >> see
> > > > >> > >> > that for some reason there are gaps in the plots
being
> > > generated.
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4
14:52
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 41781 Oct 5 15:15
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 56691 Oct 19 15:02
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 55548 Oct 20 16:08
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 55598 Oct 21 16:08
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 55599 Oct 22 16:09
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 55352 Oct 24 16:34
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 26175 Oct 31 08:20
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 26178 Nov 1 08:20
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 26160 Nov 3 08:22
> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 25354 Nov 3 08:22
> > > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
> > > > >> > >> > You can tell when a plot is successful by looking at
the
> file
> > > > >> size. I
> > > > >> > >> > highlighted an example of that in red. However, the
files
> > > > showing
> > > > >> > about
> > > > >> > >> > 25000 are the ones where no line plots are being
produced.
> > You
> > > > can
> > > > >> > also
> > > > >> > >> > see that stat files are generated everyday, so as
long as
> I'm
> > > > >> loading
> > > > >> > >> the
> > > > >> > >> > data, which I am, I should be able to use the xml
files to
> > > > generate
> > > > >> > the
> > > > >> > >> > plots. I'm attaching a directory list from wcoss
that
> shows
> > > the
> > > > >> dates
> > > > >> > >> > where data is produced.
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > *20190801 20190817 20200624 20200710 20200726
20200812
> > > > >> 20200828
> > > > >> > >> > 20201003 2020101920190802 20190818 20200625
20200711
> > > > 20200727
> > > > >> > >> > 20200813 20200829 20201004 2020102020190803
20190819
> > > > 20200626
> > > > >> > >> > 20200712 20200728 20200814 20200830 20201005
> > > > 2020102120190804
> > > > >> > >> > 20190820 20200627 20200713 20200729 20200815
20200831
> > > > >> 20201006
> > > > >> > >> > 2020102220190805 20190821 20200628 20200714
20200730
> > > > 20200816
> > > > >> > >> > 20200921 20201007 2020102320190806 20190822
20200629
> > > > 20200715
> > > > >> > >> > 20200731 20200817 20200922 20201008
2020102420190807
> > > > 20190823
> > > > >> > >> > 20200630 20200716 20200801 20200818 20200923
20201009
> > > > >> > >> > 2020102520190808 20190824 20200701 20200717
20200802
> > > > 20200819
> > > > >> > >> > 20200924 20201010 2020102620190809 20190825
20200702
> > > > 20200718
> > > > >> > >> > 20200803 20200820 20200925 20201011
2020102720190810
> > > > 20190826
> > > > >> > >> > 20200703 20200719 20200804 20200821 20200926
20201012
> > > > >> > >> > 2020102820190811 20190827 20200704 20200720
20200805
> > > > 20200822
> > > > >> > >> > 20200927 20201013 2020102920190812 20190828
20200705
> > > > 20200721
> > > > >> > >> > 20200806 20200823 20200928 20201014
2020103020190813
> > > > 20190829
> > > > >> > >> > 20200706 20200722 20200807 20200824 20200929
20201015
> > > > >> > >> > 2020103120190814 20190830 20200707 20200723
20200808
> > > > 20200825
> > > > >> > >> > 20200930 20201016 2020110120190815 20200622
20200708
> > > > 20200724
> > > > >> > >> > 20200810 20200826 20201001 20201017
2020110220190816
> > > > 20200623
> > > > >> > >> > 20200709 20200725 20200811 20200827 20201002
> 20201018*
> > > > >> > >> >
> > > > >> > >> > Furthermore, what's interesting is that I have mixed
> success
> > > with
> > > > >> > other
> > > > >> > >> > cases. Using the same field - PMTF - for the same
cycle -
> > the
> > > > 06z
> > > > >> > >> cycle -
> > > > >> > >> > you can see that some of the dates where data was
plotted
> for
> > > > EAST
> > > > >> did
> > > > >> > >> not
> > > > >> > >> > necessarily plot for FULL. This surprises me
because EAST
> > is a
> > > > >> subset
> > > > >> > >> of
> > > > >> > >> > FULL. See below:
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4
14:54
> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 49039 Oct 5 15:17
> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 56791 Oct 20 16:09
> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 56834 Oct 21 16:09
> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 56921 Oct 22 16:11
> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 56518 Oct 24 16:36
> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 61187 Oct 31 08:21
> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 26475 Nov 2 08:22
> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r--
1
> > > > >> Edward.Strobach
> > > > >> > >> > emcmodel 26455 Nov 3 08:23
> > > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
> > > > >> > >> >
> > > > >> > >> > What I find equally concerning as that sometimes
plots are
> > > > >> incorrectly
> > > > >> > >> > stored in the wrong directory list as highlighted in
green
> > > above.
> > > > >> I've
> > > > >> > >> > consulted the plot xml file for PMTF, for instance,
and
> find
> > > > >> nothing
> > > > >> > >> > problematic. The only setting that might pose an
issue
> when
> > it
> > > > >> comes
> > > > >> > to
> > > > >> > >> > generating line plots is when I set event_equal to
true.
> > > > However,
> > > > >> all
> > > > >> > >> the
> > > > >> > >> > stat files are populated for all experiments. In the
case
> of
> > > > ozone
> > > > >> you
> > > > >> > >> see
> > > > >> > >> > something different. I find that ozone tends to
result in
> > > > populated
> > > > >> > >> plots
> > > > >> > >> > but with intermittent reporting different than PMTF
- see
> > > below:
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal
> > 6144-rw-r--r-- 1
> > > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 54318 Oct 6 10:51
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 54305 Oct 7 10:49
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 62179 Oct 13 13:25
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 63811 Oct 19 12:18
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 60112 Oct 20 12:21
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 60165 Oct 21 12:33
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 60160 Oct 22 12:21
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 60056 Oct 23 12:40
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 58517 Oct 24 13:31
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 58062 Oct 31 14:17
> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--
r-- 1
> > > > >> > Edward.Strobach
> > > > >> > >> > emcmodel 58119 Nov 1 14:19
> > > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
> > > > >> > >> >
> > > > >> > >> > Similar issues have been found for other fields,
cycles,
> and
> > > > >> regions
> > > > >> > >> > including verification being done for meteorology.
> > > > >> > >> > I can't seem to reason this intermittency and
inconsistency
> > > > between
> > > > >> > >> > reporting since the xml files are correct and I'm
using the
> > > same
> > > > >> > >> procedure
> > > > >> > >> > that I've always used. I do realize that I need to
start
> > > > creating
> > > > >> a
> > > > >> > >> better
> > > > >> > >> > way to manage all these files that are being
produced, both
> > in
> > > > >> terms
> > > > >> > of
> > > > >> > >> > data/plots and xml files.
> > > > >> > >> >
> > > > >> > >> > I ran across a rather large directory:
> > > > >> > >> > ls
> > > > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > > > >> > |
> > > > >> > >> wc
> > > > >> > >> > -w
> > > > >> > >> >
> > > > >> > >> > That has 71,398 xml files in there. But I see
timestamps in
> > > those
> > > > >> > >> filenames
> > > > >> > >> > from 20200921 to 20200925.
> > > > >> > >> >
> > > > >> > >> > Is it the case that this stopped working on
20200925? Or
> are
> > > > those
> > > > >> > dated
> > > > >> > >> > filenames remnants from older work?
> > > > >> > >> >
> > > > >> > >> > So it hasn't been working for over a month. Is that
right?
> > > > >> > >> >
> > > > >> > >> > There are a lot of files and I need to begin
managing that
> > > > >> better. I
> > > > >> > >> think
> > > > >> > >> > during that time there was a machine switch so it
stopped
> on
> > > that
> > > > >> day.
> > > > >> > >> > I've been able to produce plots even now, but it's
not
> > > consistent
> > > > >> in
> > > > >> > >> > reporting while some regions generate plots and
others do
> > not.
> > > > >> I've
> > > > >> > >> > reduced the level processing in hopes that I don't
hit a
> walk
> > > > clock
> > > > >> > >> issue
> > > > >> > >> > since I'm submitting batch jobs. However, I've run
this
> step
> > > > >> outside
> > > > >> > of
> > > > >> > >> > batch and it always seems to work. This issue seems
to
> happen
> > > > with
> > > > >> > batch
> > > > >> > >> > and it is not at all clear why. Let me know what
else you
> > need
> > > > >> from
> > > > >> > me
> > > > >> > >> to
> > > > >> > >> > better coordinate what I would need to do next.
Thanks.
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway
via RT <
> > > > >> > >> > met_help at ucar.edu>
> > > > >> > >> > wrote:
> > > > >> > >> >
> > > > >> > >> > > Hi Ed,
> > > > >> > >> > >
> > > > >> > >> > > Sorry for the delay. I took a look on wcoss today.
You
> > > describe
> > > > >> that
> > > > >> > >> your
> > > > >> > >> > > plotting script has stopped working but you're not
sure
> > why.
> > > > >> > >> > >
> > > > >> > >> > > I ran across a rather large directory:
> > > > >> > >> > > ls
> > > > >> >
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > > > >> > >> |
> > > > >> > >> > wc
> > > > >> > >> > > -w
> > > > >> > >> > >
> > > > >> > >> > > That has 71,398 xml files in there. But I see
timestamps
> in
> > > > those
> > > > >> > >> > filenames
> > > > >> > >> > > from 20200921 to 20200925.
> > > > >> > >> > >
> > > > >> > >> > > Is it the case that this stopped working on
20200925? Or
> > are
> > > > >> those
> > > > >> > >> dated
> > > > >> > >> > > filenames remnants from older work?
> > > > >> > >> > >
> > > > >> > >> > > So it hasn't been working for over a month. Is
that
> right?
> > > > >> > >> > >
> > > > >> > >> > > John
> > > > >> > >> > >
> > > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System
itself via
> RT
> > <
> > > > >> > >> > > met_help at ucar.edu> wrote:
> > > > >> > >> > >
> > > > >> > >> > > >
> > > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was
acted upon.
> > > > >> > >> > > > Transaction: Given to johnhg (John Halley
Gotway) by
> > > > RT_System
> > > > >> > >> > > > Queue: met_help
> > > > >> > >> > > > Subject: plots no longer being generated
> > > > >> > >> > > > Owner: johnhg
> > > > >> > >> > > > Requestors: edward.strobach at noaa.gov
> > > > >> > >> > > > Status: new
> > > > >> > >> > > > Ticket <URL:
> > > > >> > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > > > >> > >> > >
> > > > >> > >> > > >
> > > > >> > >> > > >
> > > > >> > >> > > > This transaction appears to have no content
> > > > >> > >> > > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >> > --
> > > > >> > >> > Edward Strobach
> > > > >> > >> > EMC/NCEP/NWS/
> > > > >> > >> > IMSG Contractor
> > > > >> > >> > Cubicle#: 2029
> > > > >> > >> > 301-683-3717
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >
> > > > >> > > --
> > > > >> > > Edward Strobach
> > > > >> > > EMC/NCEP/NWS/
> > > > >> > > IMSG Contractor
> > > > >> > > Cubicle#: 2029
> > > > >> > > 301-683-3717
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Edward Strobach
> > > > >> > EMC/NCEP/NWS/
> > > > >> > IMSG Contractor
> > > > >> > Cubicle#: 2029
> > > > >> > 301-683-3717
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Mon Nov 09 06:20:30 2020
Just a follow-up on one other item..
Can also list multiple stat types in the same way that I specify the
init_hour and vx_mask? In other words, can I do the following:
<dep1>
<fcst_var name="PMTF">
<stat>FBAR_OBAR</stat>
<stat>RMSE</stat>
</fcst_var>
</dep1>
I know that the "set" block is not being used here.
On Mon, Nov 9, 2020 at 7:41 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Thanks John. Let me try that and I'll let you know how it goes.
>
> On Fri, Nov 6, 2020 at 10:53 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Ed,
>>
>> You're right about the METviewer GUI grouping multiple selections
>> together.
>> That looks like this in the XML using the <set> tag:
>> <field equalize="false" name="vx_mask">
>> <set name="vx_mask_0">
>> <val>FULL</val>
>> </set>
>> </field>
>>
>> But in the updates I sent, I remove the <set> tag:
>> <field equalize="false" name="vx_mask">
>> <val>FULL</val>
>> <val>CONUS</val>
>> <val>EAST</val>
>> <val>WEST</val>
>> </field>
>> So instead of processing this as 1 group of 4 values, it'll process
the 4
>> values separately.
>>
>> The current vx_mask value is referenced using {vx_mask} syntax. And
I
>> included examples of using that in the output file names and titles
in my
>> previous email:
>>
>>
>>
>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>>
>>
>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>>
>>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>>
>> <title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
>> Domain)</title>
>>
>> John
>>
>> On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>> >
>> > Hi John,
>> >
>> > This definitely helps. I never thought about it this way because
I
>> > assumed, based on my experience with the interactive MET GUI,
that I
>> could
>> > only generate a single plot regardless of how I specified my
settings in
>> > the XML. For example, if I listed EAST and WEST under vx_mask,
then my
>> > thinking would be that both those regions would be grouped and a
single
>> > plot created. I guess I was wrong on that. The concern I still
have,
>> > however, is that the plot title and plot name - the png file - is
also
>> set
>> > in the XML file. Even after adding the list you gave, I can't
see how
>> > unique file names will be created. Is that true?
>> >
>> > On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > Ed,
>> > >
>> > > Thanks for sending examples. My goal was to use METviewer's XML
upload
>> > > functionality but I'm not able to upload a pdf version of the
xml
>> file.
>> > >
>> > > I did spend some time trying to replicate this exact plot with
>> METviewer
>> > > at:
>> > > https://metviewer.nws.noaa.gov/metviewer1.jsp
>> > >
>> > > But I was never quite successful. So I made just a similar
version
>> > instead.
>> > > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
>> > >
>> > > This plots FBAR and OBAR for 6 different models for the month
of
>> October
>> > > for all 06Z initializations over the FULL model domain.
>> > >
>> > > I wanted to show you how to modify this for batch mode... and
then
>> adapt
>> > it
>> > > to make multiple plots. But I'm not able to make batch plots on
wcoss
>> > using
>> > > the AWS instance. I don't think I have the right permissions to
do so.
>> > >
>> > > So let me just say this. Generally, you just update the list of
items
>> in
>> > > plot_fix section.
>> > >
>> > > Original from METviewer with 1 init hour and 1 mask:
>> > >
>> > > <field equalize="false" name="vx_mask">
>> > > <set name="vx_mask_0">
>> > > <val>FULL</val>
>> > > </set>
>> > > </field>
>> > > <field equalize="false" name="init_hour">
>> > > <set name="init_hour_1">
>> > > <val>06</val>
>> > > </set>
>> > > </field>
>> > >
>> > > Change to 2 init hours and 4 masks:
>> > >
>> > > <field equalize="false" name="vx_mask">
>> > > <val>FULL</val>
>> > > <val>CONUS</val>
>> > > <val>EAST</val>
>> > > <val>WEST</val>
>> > > </field>
>> > > <field equalize="false" name="init_hour">
>> > > <val>06</val>
>> > > <val>12</val>
>> > > </field>
>> > >
>> > > And then reference those values in the naming of the titles and
output
>> > > files:
>> > >
>> > >
>> > >
>> > >
>> >
>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>> > >
>> > >
>> > >
>> >
>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>> > >
>> > >
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>> > >
>> > > <title>Average PMTF (October 2020; {init_hour}z cycle;
{vx_mask}
>> > > Domain)</title>
>> > >
>> > > So that's the general idea. And when you submit this to
mv_batch,
>> it'll
>> > > make 8 plots instead of 1.
>> > > And you can see how you'd easily expand this to many plots.
>> > >
>> > > But I worry that this may not be sufficient information to get
you
>> going.
>> > >
>> > > John
>> > >
>> > > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA
Affiliate via
>> RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>
>> > > >
>> > > > Hi John,
>> > > >
>> > > > I think I was able to cover your request okay. Below are 4
>> attachments
>> > > > that include the test script that was used to generate the
plots, a
>> > > sample
>> > > > XML file, and the two plots that were generated using the
test run
>> > > script.
>> > > > These plots were not created previously using the batch job
but uses
>> > the
>> > > > same exact logic as the original run script. Thanks again
for
>> working
>> > > with
>> > > > me on this. Please let me know if you need anything else or
would
>> like
>> > > me
>> > > > to test something.
>> > > >
>> > > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA
Affiliate <
>> > > > edward.strobach at noaa.gov> wrote:
>> > > >
>> > > > > Ed,
>> > > > >
>> > > > > Image files showing up in an unexpected output directory
likely
>> point
>> > > to
>> > > > a
>> > > > > problem in the logic of the processing scripts. It's pretty
>> unlikely
>> > > that
>> > > > > this behavior is caused by the METviewer code itself. But
I've
>> > > definitely
>> > > > > been surprised before!
>> > > > >
>> > > > > I tested the logic outside of a batch job and it works
fine. I
>> > suspect
>> > > > > that the issuance of another batch job while the other one
is
>> getting
>> > > > > processed is causing some confusion in the processing.
These
>> issues
>> > > > don't
>> > > > > happen everyday and appear more sporadic than anything.
>> > > > >
>> > > > > So what we really need here is an example of taking a plot
from
>> the
>> > > > > METviewer GUI and adapting it to make it generate multiple
plots.
>> I
>> > did
>> > > > go
>> > > > > through examples of doing this in previous tutorials, but
we
>> didn't
>> > > start
>> > > > > recording them until last summer. I checked the July 2019
NRL
>> > tutorial
>> > > > > videos and don't see it there.
>> > > > >
>> > > > > Is there something you need from me to help on this? Just
let me
>> > know
>> > > > >
>> > > > > We've begun recording training videos for common topics
like this.
>> > But
>> > > we
>> > > > > only have one for METviewer so far:
>> > > > >
>> > >
>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>> > > > >
>> > > > > But this definitely seems like a good candidate for a new
training
>> > > video.
>> > > > >
>> > > > > For now though, how about this...
>> > > > > - You use METviewer to manually make a single plot,
formatting it
>> how
>> > > > you'd
>> > > > > like.
>> > > > > - You send me your sample plot and corresponding XML file.
>> > > > > - I'll tweak it to generate many plots via batch instead of
just
>> one.
>> > > > > - And then you can continue tweaking to add more
permutations.
>> > > > >
>> > > > > Okay, I'll work on that now. Thanks.
>> > > > >
>> > > > > John
>> > > > >
>> > > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via RT <
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > >> Ed,
>> > > > >>
>> > > > >> Image files showing up in an unexpected output directory
likely
>> > point
>> > > > to a
>> > > > >> problem in the logic of the processing scripts. It's
pretty
>> unlikely
>> > > > that
>> > > > >> this behavior is caused by the METviewer code itself. But
I've
>> > > > definitely
>> > > > >> been surprised before!
>> > > > >>
>> > > > >> So what we really need here is an example of taking a plot
from
>> the
>> > > > >> METviewer GUI and adapting it to make it generate multiple
>> plots. I
>> > > did
>> > > > go
>> > > > >> through examples of doing this in previous tutorials, but
we
>> didn't
>> > > > start
>> > > > >> recording them until last summer. I checked the July 2019
NRL
>> > tutorial
>> > > > >> videos and don't see it there.
>> > > > >>
>> > > > >> We've begun recording training videos for common topics
like
>> this.
>> > But
>> > > > we
>> > > > >> only have one for METviewer so far:
>> > > > >>
>> > > >
>> >
>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>> > > > >>
>> > > > >> But this definitely seems like a good candidate for a new
>> training
>> > > > video.
>> > > > >>
>> > > > >> For now though, how about this...
>> > > > >> - You use METviewer to manually make a single plot,
formatting it
>> > how
>> > > > >> you'd
>> > > > >> like.
>> > > > >> - You send me your sample plot and corresponding XML file.
>> > > > >> - I'll tweak it to generate many plots via batch instead
of just
>> > one.
>> > > > >> - And then you can continue tweaking to add more
permutations.
>> > > > >>
>> > > > >> John
>> > > > >>
>> > > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA
Affiliate
>> via
>> > > RT <
>> > > > >> met_help at ucar.edu> wrote:
>> > > > >>
>> > > > >> >
>> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>> >
>> > > > >> >
>> > > > >> > Oh, and one more thing..
>> > > > >> >
>> > > > >> > The green text that didn't show up was meant to
highlight the
>> fact
>> > > > that
>> > > > >> > sometimes plots that shouldn't be stored in certain
directories
>> > end
>> > > up
>> > > > >> > there. The pasted example below shows one instance
where
>> OZMAX*
>> > was
>> > > > >> stored
>> > > > >> > where TMP* should have been stored.
>> > > > >> >
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5 10:52
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6 10:52
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7 10:00
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13 13:49
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14 11:31
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19 11:37
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20 11:35
>> > > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21 11:45
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22 11:42
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23 11:50
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24 12:57
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31 08:47
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2 08:50
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3 08:52
>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
>> > > > >> >
>> > > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA
>> Affiliate <
>> > > > >> > edward.strobach at noaa.gov> wrote:
>> > > > >> >
>> > > > >> > > Hi John,
>> > > > >> > >
>> > > > >> > > I know there are a lot of calls and speculate that
this
>> could be
>> > > > part
>> > > > >> of
>> > > > >> > > the problem. I don't think that I'm following best
practice
>> by
>> > > > >> > approaching
>> > > > >> > > it in this way. Generating plots usually works well
if I
>> > process
>> > > a
>> > > > >> > subset
>> > > > >> > > of the computations outside of batch scripting. The
python
>> > script
>> > > > was
>> > > > >> > > given to me by Logan, but I have since made
significant
>> changes
>> > to
>> > > > it
>> > > > >> in
>> > > > >> > > order to dynamically fill in the xml file based on
parameters
>> > set.
>> > > > >> The
>> > > > >> > > test*.sh script was also given to me. I created the
>> > > > Time_Series*bsub
>> > > > >> as
>> > > > >> > a
>> > > > >> > > way to process a series of plots based on cycle time,
region,
>> > > etc. I
>> > > > >> > think
>> > > > >> > > the first two scripts are solid and do well for what I
need
>> to
>> > get
>> > > > >> done.
>> > > > >> > > It's the Time_Series*bsub script that bothers me a
bit.
>> > > > >> > >
>> > > > >> > > I can't see how a single XML file can process all
these plots
>> > in a
>> > > > >> single
>> > > > >> > > setting based on what I've learned thus far. How
would one
>> go
>> > > about
>> > > > >> > that?
>> > > > >> > > I would be open going in that direction if you can
provide
>> some
>> > > > >> guidance.
>> > > > >> > >
>> > > > >> > > Unfortunately, I haven't gotten a lot of feedback or
dialogue
>> > > > exchange
>> > > > >> > > about what I've done so far at EMC with this new set-
up.
>> This
>> > has
>> > > > >> also
>> > > > >> > > been part of the problem. I can't seem to get them to
focus
>> so
>> > > that
>> > > > >> we
>> > > > >> > can
>> > > > >> > > prioritize. I know that I'm doing the best I can. If
I can
>> > learn
>> > > > >> how to
>> > > > >> > > go about this more efficiently it would be of great
help. I
>> can
>> > > > then
>> > > > >> > > expand this to other verification as venture to other
>> projects
>> > in
>> > > > the
>> > > > >> > > future. I know you're busy with many things, so I'm
willing
>> to
>> > go
>> > > > >> along
>> > > > >> > > with your schedule as you find an opening.
>> > > > >> > >
>> > > > >> > > Thanks
>> > > > >> > >
>> > > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway via
RT <
>> > > > >> > > met_help at ucar.edu> wrote:
>> > > > >> > >
>> > > > >> > >> Ed,
>> > > > >> > >>
>> > > > >> > >> It's really difficult for me to see an obvious
problem
>> that's
>> > > > causing
>> > > > >> > the
>> > > > >> > >> behavior you describe.
>> > > > >> > >>
>> > > > >> > >> Unfortunately, the red and green highlighting you
describe
>> > > doesn't
>> > > > >> show
>> > > > >> > up
>> > > > >> > >> in the Gmail web client. There must be some mail
system
>> > > > >> incompatibility
>> > > > >> > >> there.
>> > > > >> > >>
>> > > > >> > >> Looking at your bsub script:
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
>> > > > >> > >>
>> > > > >> > >> I am struck by the number of individual calls you're
making
>> to
>> > > > >> > METviewer:
>> > > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2 plot
types
>> =
>> > 656
>> > > > >> calls
>> > > > >> > to
>> > > > >> > >> METviewer
>> > > > >> > >>
>> > > > >> > >> When making plots via the METviewer GUI, it's true
that 1
>> XML
>> > = 1
>> > > > >> plot.
>> > > > >> > >> However, when making plots via the batch engine, the
plot
>> XML
>> > can
>> > > > be
>> > > > >> set
>> > > > >> > >> up
>> > > > >> > >> to create many, many plots. While I have no direct
proof of
>> > this,
>> > > > it
>> > > > >> > might
>> > > > >> > >> be the case that executing 600+ batch jobs for
METviewer is
>> > > leading
>> > > > >> to
>> > > > >> > >> trouble on the system. One thing to try would be
setting up
>> the
>> > > XML
>> > > > >> to
>> > > > >> > >> make
>> > > > >> > >> many, many more plots in each call. Then check to see
if
>> > running
>> > > > >> > METviewer
>> > > > >> > >> in this way is more stable.
>> > > > >> > >>
>> > > > >> > >> I don't think I fully appreciate all of the logic
that
>> lives in
>> > > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
>> > > > >> test_Time_Series_AGG.sh.
>> > > > >> > >> However, it *might* be possible to construct a single
XML
>> file
>> > > > which
>> > > > >> > >> produces all 656 plots.
>> > > > >> > >>
>> > > > >> > >> Let me know what you think and how you'd like to
proceed.
>> > > > >> > >>
>> > > > >> > >> Thanks,
>> > > > >> > >> John
>> > > > >> > >>
>> > > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach - NOAA
>> Affiliate
>> > > via
>> > > > >> RT <
>> > > > >> > >> met_help at ucar.edu> wrote:
>> > > > >> > >>
>> > > > >> > >> >
>> > > > >> > >> > <URL:
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>> > > >
>> > > > >> > >> >
>> > > > >> > >> > Hi John,
>> > > > >> > >> >
>> > > > >> > >> > No problem.
>> > > > >> > >> >
>> > > > >> > >> > I'm including my responses below:
>> > > > >> > >> > Sorry for the delay. I took a look on wcoss today.
You
>> > describe
>> > > > >> that
>> > > > >> > >> your
>> > > > >> > >> > plotting script has stopped working but you're not
sure
>> why.
>> > > > >> > >> > It seems more intermittent. Take a look at the
example
>> > below.
>> > > > You
>> > > > >> > can
>> > > > >> > >> see
>> > > > >> > >> > that for some reason there are gaps in the plots
being
>> > > generated.
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct 4
14:52
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 41781 Oct 5 15:15
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 56691 Oct 19 15:02
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 55548 Oct 20 16:08
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 55598 Oct 21 16:08
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 55599 Oct 22 16:09
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 55352 Oct 24 16:34
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 26175 Oct 31 08:20
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 26178 Nov 1 08:20
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 26160 Nov 3 08:22
>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 25354 Nov 3 08:22
>> > > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
>> > > > >> > >> > You can tell when a plot is successful by looking
at the
>> file
>> > > > >> size. I
>> > > > >> > >> > highlighted an example of that in red. However,
the files
>> > > > showing
>> > > > >> > about
>> > > > >> > >> > 25000 are the ones where no line plots are being
produced.
>> > You
>> > > > can
>> > > > >> > also
>> > > > >> > >> > see that stat files are generated everyday, so as
long as
>> I'm
>> > > > >> loading
>> > > > >> > >> the
>> > > > >> > >> > data, which I am, I should be able to use the xml
files to
>> > > > generate
>> > > > >> > the
>> > > > >> > >> > plots. I'm attaching a directory list from wcoss
that
>> shows
>> > > the
>> > > > >> dates
>> > > > >> > >> > where data is produced.
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> > *20190801 20190817 20200624 20200710 20200726
>> 20200812
>> > > > >> 20200828
>> > > > >> > >> > 20201003 2020101920190802 20190818 20200625
20200711
>> > > > 20200727
>> > > > >> > >> > 20200813 20200829 20201004 2020102020190803
20190819
>> > > > 20200626
>> > > > >> > >> > 20200712 20200728 20200814 20200830 20201005
>> > > > 2020102120190804
>> > > > >> > >> > 20190820 20200627 20200713 20200729 20200815
>> 20200831
>> > > > >> 20201006
>> > > > >> > >> > 2020102220190805 20190821 20200628 20200714
20200730
>> > > > 20200816
>> > > > >> > >> > 20200921 20201007 2020102320190806 20190822
20200629
>> > > > 20200715
>> > > > >> > >> > 20200731 20200817 20200922 20201008
2020102420190807
>> > > > 20190823
>> > > > >> > >> > 20200630 20200716 20200801 20200818 20200923
>> 20201009
>> > > > >> > >> > 2020102520190808 20190824 20200701 20200717
20200802
>> > > > 20200819
>> > > > >> > >> > 20200924 20201010 2020102620190809 20190825
20200702
>> > > > 20200718
>> > > > >> > >> > 20200803 20200820 20200925 20201011
2020102720190810
>> > > > 20190826
>> > > > >> > >> > 20200703 20200719 20200804 20200821 20200926
>> 20201012
>> > > > >> > >> > 2020102820190811 20190827 20200704 20200720
20200805
>> > > > 20200822
>> > > > >> > >> > 20200927 20201013 2020102920190812 20190828
20200705
>> > > > 20200721
>> > > > >> > >> > 20200806 20200823 20200928 20201014
2020103020190813
>> > > > 20190829
>> > > > >> > >> > 20200706 20200722 20200807 20200824 20200929
>> 20201015
>> > > > >> > >> > 2020103120190814 20190830 20200707 20200723
20200808
>> > > > 20200825
>> > > > >> > >> > 20200930 20201016 2020110120190815 20200622
20200708
>> > > > 20200724
>> > > > >> > >> > 20200810 20200826 20201001 20201017
2020110220190816
>> > > > 20200623
>> > > > >> > >> > 20200709 20200725 20200811 20200827 20201002
>> 20201018*
>> > > > >> > >> >
>> > > > >> > >> > Furthermore, what's interesting is that I have
mixed
>> success
>> > > with
>> > > > >> > other
>> > > > >> > >> > cases. Using the same field - PMTF - for the same
cycle -
>> > the
>> > > > 06z
>> > > > >> > >> cycle -
>> > > > >> > >> > you can see that some of the dates where data was
plotted
>> for
>> > > > EAST
>> > > > >> did
>> > > > >> > >> not
>> > > > >> > >> > necessarily plot for FULL. This surprises me
because EAST
>> > is a
>> > > > >> subset
>> > > > >> > >> of
>> > > > >> > >> > FULL. See below:
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct 4
14:54
>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 49039 Oct 5 15:17
>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 56791 Oct 20 16:09
>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 56834 Oct 21 16:09
>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 56921 Oct 22 16:11
>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 56518 Oct 24 16:36
>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 61187 Oct 31 08:21
>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 26475 Nov 2 08:22
>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--
r-- 1
>> > > > >> Edward.Strobach
>> > > > >> > >> > emcmodel 26455 Nov 3 08:23
>> > > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
>> > > > >> > >> >
>> > > > >> > >> > What I find equally concerning as that sometimes
plots are
>> > > > >> incorrectly
>> > > > >> > >> > stored in the wrong directory list as highlighted
in green
>> > > above.
>> > > > >> I've
>> > > > >> > >> > consulted the plot xml file for PMTF, for instance,
and
>> find
>> > > > >> nothing
>> > > > >> > >> > problematic. The only setting that might pose an
issue
>> when
>> > it
>> > > > >> comes
>> > > > >> > to
>> > > > >> > >> > generating line plots is when I set event_equal to
true.
>> > > > However,
>> > > > >> all
>> > > > >> > >> the
>> > > > >> > >> > stat files are populated for all experiments. In
the case
>> of
>> > > > ozone
>> > > > >> you
>> > > > >> > >> see
>> > > > >> > >> > something different. I find that ozone tends to
result in
>> > > > populated
>> > > > >> > >> plots
>> > > > >> > >> > but with intermittent reporting different than PMTF
- see
>> > > below:
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal
>> > 6144-rw-r--r-- 1
>> > > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 54318 Oct 6 10:51
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 54305 Oct 7 10:49
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 62179 Oct 13 13:25
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 63811 Oct 19 12:18
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 60112 Oct 20 12:21
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 60165 Oct 21 12:33
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 60160 Oct 22 12:21
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 60056 Oct 23 12:40
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 58517 Oct 24 13:31
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 58062 Oct 31 14:17
>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--
r-- 1
>> > > > >> > Edward.Strobach
>> > > > >> > >> > emcmodel 58119 Nov 1 14:19
>> > > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
>> > > > >> > >> >
>> > > > >> > >> > Similar issues have been found for other fields,
cycles,
>> and
>> > > > >> regions
>> > > > >> > >> > including verification being done for meteorology.
>> > > > >> > >> > I can't seem to reason this intermittency and
>> inconsistency
>> > > > between
>> > > > >> > >> > reporting since the xml files are correct and I'm
using
>> the
>> > > same
>> > > > >> > >> procedure
>> > > > >> > >> > that I've always used. I do realize that I need to
start
>> > > > creating
>> > > > >> a
>> > > > >> > >> better
>> > > > >> > >> > way to manage all these files that are being
produced,
>> both
>> > in
>> > > > >> terms
>> > > > >> > of
>> > > > >> > >> > data/plots and xml files.
>> > > > >> > >> >
>> > > > >> > >> > I ran across a rather large directory:
>> > > > >> > >> > ls
>> > > > >>
>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>> > > > >> > |
>> > > > >> > >> wc
>> > > > >> > >> > -w
>> > > > >> > >> >
>> > > > >> > >> > That has 71,398 xml files in there. But I see
timestamps
>> in
>> > > those
>> > > > >> > >> filenames
>> > > > >> > >> > from 20200921 to 20200925.
>> > > > >> > >> >
>> > > > >> > >> > Is it the case that this stopped working on
20200925? Or
>> are
>> > > > those
>> > > > >> > dated
>> > > > >> > >> > filenames remnants from older work?
>> > > > >> > >> >
>> > > > >> > >> > So it hasn't been working for over a month. Is that
right?
>> > > > >> > >> >
>> > > > >> > >> > There are a lot of files and I need to begin
managing that
>> > > > >> better. I
>> > > > >> > >> think
>> > > > >> > >> > during that time there was a machine switch so it
stopped
>> on
>> > > that
>> > > > >> day.
>> > > > >> > >> > I've been able to produce plots even now, but it's
not
>> > > consistent
>> > > > >> in
>> > > > >> > >> > reporting while some regions generate plots and
others do
>> > not.
>> > > > >> I've
>> > > > >> > >> > reduced the level processing in hopes that I don't
hit a
>> walk
>> > > > clock
>> > > > >> > >> issue
>> > > > >> > >> > since I'm submitting batch jobs. However, I've run
this
>> step
>> > > > >> outside
>> > > > >> > of
>> > > > >> > >> > batch and it always seems to work. This issue seems
to
>> happen
>> > > > with
>> > > > >> > batch
>> > > > >> > >> > and it is not at all clear why. Let me know what
else you
>> > need
>> > > > >> from
>> > > > >> > me
>> > > > >> > >> to
>> > > > >> > >> > better coordinate what I would need to do next.
Thanks.
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway
via RT <
>> > > > >> > >> > met_help at ucar.edu>
>> > > > >> > >> > wrote:
>> > > > >> > >> >
>> > > > >> > >> > > Hi Ed,
>> > > > >> > >> > >
>> > > > >> > >> > > Sorry for the delay. I took a look on wcoss
today. You
>> > > describe
>> > > > >> that
>> > > > >> > >> your
>> > > > >> > >> > > plotting script has stopped working but you're
not sure
>> > why.
>> > > > >> > >> > >
>> > > > >> > >> > > I ran across a rather large directory:
>> > > > >> > >> > > ls
>> > > > >> >
>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>> > > > >> > >> |
>> > > > >> > >> > wc
>> > > > >> > >> > > -w
>> > > > >> > >> > >
>> > > > >> > >> > > That has 71,398 xml files in there. But I see
>> timestamps in
>> > > > those
>> > > > >> > >> > filenames
>> > > > >> > >> > > from 20200921 to 20200925.
>> > > > >> > >> > >
>> > > > >> > >> > > Is it the case that this stopped working on
20200925? Or
>> > are
>> > > > >> those
>> > > > >> > >> dated
>> > > > >> > >> > > filenames remnants from older work?
>> > > > >> > >> > >
>> > > > >> > >> > > So it hasn't been working for over a month. Is
that
>> right?
>> > > > >> > >> > >
>> > > > >> > >> > > John
>> > > > >> > >> > >
>> > > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System
itself via
>> RT
>> > <
>> > > > >> > >> > > met_help at ucar.edu> wrote:
>> > > > >> > >> > >
>> > > > >> > >> > > >
>> > > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was
acted
>> upon.
>> > > > >> > >> > > > Transaction: Given to johnhg (John Halley
Gotway) by
>> > > > RT_System
>> > > > >> > >> > > > Queue: met_help
>> > > > >> > >> > > > Subject: plots no longer being generated
>> > > > >> > >> > > > Owner: johnhg
>> > > > >> > >> > > > Requestors: edward.strobach at noaa.gov
>> > > > >> > >> > > > Status: new
>> > > > >> > >> > > > Ticket <URL:
>> > > > >> > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>> > > > >> > >> > >
>> > > > >> > >> > > >
>> > > > >> > >> > > >
>> > > > >> > >> > > > This transaction appears to have no content
>> > > > >> > >> > > >
>> > > > >> > >> > >
>> > > > >> > >> > >
>> > > > >> > >> >
>> > > > >> > >> > --
>> > > > >> > >> > Edward Strobach
>> > > > >> > >> > EMC/NCEP/NWS/
>> > > > >> > >> > IMSG Contractor
>> > > > >> > >> > Cubicle#: 2029
>> > > > >> > >> > 301-683-3717
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > Edward Strobach
>> > > > >> > > EMC/NCEP/NWS/
>> > > > >> > > IMSG Contractor
>> > > > >> > > Cubicle#: 2029
>> > > > >> > > 301-683-3717
>> > > > >> > >
>> > > > >> >
>> > > > >> >
>> > > > >> > --
>> > > > >> > Edward Strobach
>> > > > >> > EMC/NCEP/NWS/
>> > > > >> > IMSG Contractor
>> > > > >> > Cubicle#: 2029
>> > > > >> > 301-683-3717
>> > > > >> >
>> > > > >> >
>> > > > >>
>> > > > >>
>> > > > >
>> > > > > --
>> > > > > Edward Strobach
>> > > > > EMC/NCEP/NWS/
>> > > > > IMSG Contractor
>> > > > > Cubicle#: 2029
>> > > > > 301-683-3717
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Edward Strobach
>> > > > EMC/NCEP/NWS/
>> > > > IMSG Contractor
>> > > > Cubicle#: 2029
>> > > > 301-683-3717
>> > > >
>> > > >
>> > >
>> > >
>> >
>> > --
>> > Edward Strobach
>> > EMC/NCEP/NWS/
>> > IMSG Contractor
>> > Cubicle#: 2029
>> > 301-683-3717
>> >
>> >
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Mon Nov 09 07:27:38 2020
Actually, I don't think I can do that. FBAR and OBAR are split and
adding
RMSE as a third line might confuse the process.
I was able to run a test with mixed success. I was successful at
generating 8 plots (4 regions for 2 initial hours). The regions were
Mixed_Forest, FULL, EAST, and WEST while the initial hours were 06 and
12.
The test ran well with indications that all plots were generated and
stored
here: Created plot
/data/mv_data/edward.strobach/plots/PMTF_Mixed_Forest_112020_2020-11-
01_2020-11-06_t06z.png
However, I was not successful at saving the plots on Dell. Only the
last
plot was saved here:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011.
It seems that in my bash script the line that produces the plots that
are
then stored in the above directory must be done in a loop otherwise it
will
only move the last plot over as suggested by the fact that only the
plot
featuring WEST at 12z was moved over - the last plot of the series of
plots
generated. I suppose I could create logic that extracts the
region/initial
hour information and construct a two part loop to move over all plots?
On Mon, Nov 9, 2020 at 8:19 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Just a follow-up on one other item..
>
> Can also list multiple stat types in the same way that I specify the
> init_hour and vx_mask? In other words, can I do the following:
>
> <dep1>
> <fcst_var name="PMTF">
> <stat>FBAR_OBAR</stat>
> <stat>RMSE</stat>
> </fcst_var>
> </dep1>
>
> I know that the "set" block is not being used here.
>
> On Mon, Nov 9, 2020 at 7:41 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> Thanks John. Let me try that and I'll let you know how it goes.
>>
>> On Fri, Nov 6, 2020 at 10:53 AM John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Ed,
>>>
>>> You're right about the METviewer GUI grouping multiple selections
>>> together.
>>> That looks like this in the XML using the <set> tag:
>>> <field equalize="false" name="vx_mask">
>>> <set name="vx_mask_0">
>>> <val>FULL</val>
>>> </set>
>>> </field>
>>>
>>> But in the updates I sent, I remove the <set> tag:
>>> <field equalize="false" name="vx_mask">
>>> <val>FULL</val>
>>> <val>CONUS</val>
>>> <val>EAST</val>
>>> <val>WEST</val>
>>> </field>
>>> So instead of processing this as 1 group of 4 values, it'll
process the 4
>>> values separately.
>>>
>>> The current vx_mask value is referenced using {vx_mask} syntax.
And I
>>> included examples of using that in the output file names and
titles in my
>>> previous email:
>>>
>>>
>>>
>>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>>>
>>>
>>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>>>
>>>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>>>
>>> <title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
>>> Domain)</title>
>>>
>>> John
>>>
>>> On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA Affiliate
via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>> >
>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>>> >
>>> > Hi John,
>>> >
>>> > This definitely helps. I never thought about it this way
because I
>>> > assumed, based on my experience with the interactive MET GUI,
that I
>>> could
>>> > only generate a single plot regardless of how I specified my
settings
>>> in
>>> > the XML. For example, if I listed EAST and WEST under vx_mask,
then my
>>> > thinking would be that both those regions would be grouped and a
single
>>> > plot created. I guess I was wrong on that. The concern I still
have,
>>> > however, is that the plot title and plot name - the png file -
is also
>>> set
>>> > in the XML file. Even after adding the list you gave, I can't
see how
>>> > unique file names will be created. Is that true?
>>> >
>>> > On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT <
>>> > met_help at ucar.edu>
>>> > wrote:
>>> >
>>> > > Ed,
>>> > >
>>> > > Thanks for sending examples. My goal was to use METviewer's
XML
>>> upload
>>> > > functionality but I'm not able to upload a pdf version of the
xml
>>> file.
>>> > >
>>> > > I did spend some time trying to replicate this exact plot with
>>> METviewer
>>> > > at:
>>> > > https://metviewer.nws.noaa.gov/metviewer1.jsp
>>> > >
>>> > > But I was never quite successful. So I made just a similar
version
>>> > instead.
>>> > > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
>>> > >
>>> > > This plots FBAR and OBAR for 6 different models for the month
of
>>> October
>>> > > for all 06Z initializations over the FULL model domain.
>>> > >
>>> > > I wanted to show you how to modify this for batch mode... and
then
>>> adapt
>>> > it
>>> > > to make multiple plots. But I'm not able to make batch plots
on wcoss
>>> > using
>>> > > the AWS instance. I don't think I have the right permissions
to do
>>> so.
>>> > >
>>> > > So let me just say this. Generally, you just update the list
of
>>> items in
>>> > > plot_fix section.
>>> > >
>>> > > Original from METviewer with 1 init hour and 1 mask:
>>> > >
>>> > > <field equalize="false" name="vx_mask">
>>> > > <set name="vx_mask_0">
>>> > > <val>FULL</val>
>>> > > </set>
>>> > > </field>
>>> > > <field equalize="false" name="init_hour">
>>> > > <set name="init_hour_1">
>>> > > <val>06</val>
>>> > > </set>
>>> > > </field>
>>> > >
>>> > > Change to 2 init hours and 4 masks:
>>> > >
>>> > > <field equalize="false" name="vx_mask">
>>> > > <val>FULL</val>
>>> > > <val>CONUS</val>
>>> > > <val>EAST</val>
>>> > > <val>WEST</val>
>>> > > </field>
>>> > > <field equalize="false" name="init_hour">
>>> > > <val>06</val>
>>> > > <val>12</val>
>>> > > </field>
>>> > >
>>> > > And then reference those values in the naming of the titles
and
>>> output
>>> > > files:
>>> > >
>>> > >
>>> > >
>>> > >
>>> >
>>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>>> > >
>>> > >
>>> > >
>>> >
>>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>>> > >
>>> > >
>>>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>>> > >
>>> > > <title>Average PMTF (October 2020; {init_hour}z cycle;
{vx_mask}
>>> > > Domain)</title>
>>> > >
>>> > > So that's the general idea. And when you submit this to
mv_batch,
>>> it'll
>>> > > make 8 plots instead of 1.
>>> > > And you can see how you'd easily expand this to many plots.
>>> > >
>>> > > But I worry that this may not be sufficient information to get
you
>>> going.
>>> > >
>>> > > John
>>> > >
>>> > > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA
Affiliate via
>>> RT <
>>> > > met_help at ucar.edu> wrote:
>>> > >
>>> > > >
>>> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>>> > > >
>>> > > > Hi John,
>>> > > >
>>> > > > I think I was able to cover your request okay. Below are 4
>>> attachments
>>> > > > that include the test script that was used to generate the
plots, a
>>> > > sample
>>> > > > XML file, and the two plots that were generated using the
test run
>>> > > script.
>>> > > > These plots were not created previously using the batch job
but
>>> uses
>>> > the
>>> > > > same exact logic as the original run script. Thanks again
for
>>> working
>>> > > with
>>> > > > me on this. Please let me know if you need anything else or
would
>>> like
>>> > > me
>>> > > > to test something.
>>> > > >
>>> > > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA
Affiliate <
>>> > > > edward.strobach at noaa.gov> wrote:
>>> > > >
>>> > > > > Ed,
>>> > > > >
>>> > > > > Image files showing up in an unexpected output directory
likely
>>> point
>>> > > to
>>> > > > a
>>> > > > > problem in the logic of the processing scripts. It's
pretty
>>> unlikely
>>> > > that
>>> > > > > this behavior is caused by the METviewer code itself. But
I've
>>> > > definitely
>>> > > > > been surprised before!
>>> > > > >
>>> > > > > I tested the logic outside of a batch job and it works
fine. I
>>> > suspect
>>> > > > > that the issuance of another batch job while the other one
is
>>> getting
>>> > > > > processed is causing some confusion in the processing.
These
>>> issues
>>> > > > don't
>>> > > > > happen everyday and appear more sporadic than anything.
>>> > > > >
>>> > > > > So what we really need here is an example of taking a plot
from
>>> the
>>> > > > > METviewer GUI and adapting it to make it generate multiple
>>> plots. I
>>> > did
>>> > > > go
>>> > > > > through examples of doing this in previous tutorials, but
we
>>> didn't
>>> > > start
>>> > > > > recording them until last summer. I checked the July 2019
NRL
>>> > tutorial
>>> > > > > videos and don't see it there.
>>> > > > >
>>> > > > > Is there something you need from me to help on this? Just
let me
>>> > know
>>> > > > >
>>> > > > > We've begun recording training videos for common topics
like
>>> this.
>>> > But
>>> > > we
>>> > > > > only have one for METviewer so far:
>>> > > > >
>>> > >
>>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>>> > > > >
>>> > > > > But this definitely seems like a good candidate for a new
>>> training
>>> > > video.
>>> > > > >
>>> > > > > For now though, how about this...
>>> > > > > - You use METviewer to manually make a single plot,
formatting
>>> it how
>>> > > > you'd
>>> > > > > like.
>>> > > > > - You send me your sample plot and corresponding XML file.
>>> > > > > - I'll tweak it to generate many plots via batch instead
of just
>>> one.
>>> > > > > - And then you can continue tweaking to add more
permutations.
>>> > > > >
>>> > > > > Okay, I'll work on that now. Thanks.
>>> > > > >
>>> > > > > John
>>> > > > >
>>> > > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via RT
<
>>> > > > > met_help at ucar.edu> wrote:
>>> > > > >
>>> > > > >> Ed,
>>> > > > >>
>>> > > > >> Image files showing up in an unexpected output directory
likely
>>> > point
>>> > > > to a
>>> > > > >> problem in the logic of the processing scripts. It's
pretty
>>> unlikely
>>> > > > that
>>> > > > >> this behavior is caused by the METviewer code itself. But
I've
>>> > > > definitely
>>> > > > >> been surprised before!
>>> > > > >>
>>> > > > >> So what we really need here is an example of taking a
plot from
>>> the
>>> > > > >> METviewer GUI and adapting it to make it generate
multiple
>>> plots. I
>>> > > did
>>> > > > go
>>> > > > >> through examples of doing this in previous tutorials, but
we
>>> didn't
>>> > > > start
>>> > > > >> recording them until last summer. I checked the July 2019
NRL
>>> > tutorial
>>> > > > >> videos and don't see it there.
>>> > > > >>
>>> > > > >> We've begun recording training videos for common topics
like
>>> this.
>>> > But
>>> > > > we
>>> > > > >> only have one for METviewer so far:
>>> > > > >>
>>> > > >
>>> >
>>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>>> > > > >>
>>> > > > >> But this definitely seems like a good candidate for a new
>>> training
>>> > > > video.
>>> > > > >>
>>> > > > >> For now though, how about this...
>>> > > > >> - You use METviewer to manually make a single plot,
formatting
>>> it
>>> > how
>>> > > > >> you'd
>>> > > > >> like.
>>> > > > >> - You send me your sample plot and corresponding XML
file.
>>> > > > >> - I'll tweak it to generate many plots via batch instead
of just
>>> > one.
>>> > > > >> - And then you can continue tweaking to add more
permutations.
>>> > > > >>
>>> > > > >> John
>>> > > > >>
>>> > > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA
Affiliate
>>> via
>>> > > RT <
>>> > > > >> met_help at ucar.edu> wrote:
>>> > > > >>
>>> > > > >> >
>>> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>> >
>>> > > > >> >
>>> > > > >> > Oh, and one more thing..
>>> > > > >> >
>>> > > > >> > The green text that didn't show up was meant to
highlight the
>>> fact
>>> > > > that
>>> > > > >> > sometimes plots that shouldn't be stored in certain
>>> directories
>>> > end
>>> > > up
>>> > > > >> > there. The pasted example below shows one instance
where
>>> OZMAX*
>>> > was
>>> > > > >> stored
>>> > > > >> > where TMP* should have been stored.
>>> > > > >> >
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5
10:52
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6
10:52
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7
10:00
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13
13:49
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14
11:31
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19
11:37
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20
11:35
>>> > > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21
11:45
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22
11:42
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23
11:50
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24
12:57
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31
08:47
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2
08:50
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3
08:52
>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
>>> > > > >> >
>>> > > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA
>>> Affiliate <
>>> > > > >> > edward.strobach at noaa.gov> wrote:
>>> > > > >> >
>>> > > > >> > > Hi John,
>>> > > > >> > >
>>> > > > >> > > I know there are a lot of calls and speculate that
this
>>> could be
>>> > > > part
>>> > > > >> of
>>> > > > >> > > the problem. I don't think that I'm following best
>>> practice by
>>> > > > >> > approaching
>>> > > > >> > > it in this way. Generating plots usually works well
if I
>>> > process
>>> > > a
>>> > > > >> > subset
>>> > > > >> > > of the computations outside of batch scripting. The
python
>>> > script
>>> > > > was
>>> > > > >> > > given to me by Logan, but I have since made
significant
>>> changes
>>> > to
>>> > > > it
>>> > > > >> in
>>> > > > >> > > order to dynamically fill in the xml file based on
>>> parameters
>>> > set.
>>> > > > >> The
>>> > > > >> > > test*.sh script was also given to me. I created the
>>> > > > Time_Series*bsub
>>> > > > >> as
>>> > > > >> > a
>>> > > > >> > > way to process a series of plots based on cycle time,
>>> region,
>>> > > etc. I
>>> > > > >> > think
>>> > > > >> > > the first two scripts are solid and do well for what
I need
>>> to
>>> > get
>>> > > > >> done.
>>> > > > >> > > It's the Time_Series*bsub script that bothers me a
bit.
>>> > > > >> > >
>>> > > > >> > > I can't see how a single XML file can process all
these
>>> plots
>>> > in a
>>> > > > >> single
>>> > > > >> > > setting based on what I've learned thus far. How
would one
>>> go
>>> > > about
>>> > > > >> > that?
>>> > > > >> > > I would be open going in that direction if you can
provide
>>> some
>>> > > > >> guidance.
>>> > > > >> > >
>>> > > > >> > > Unfortunately, I haven't gotten a lot of feedback or
>>> dialogue
>>> > > > exchange
>>> > > > >> > > about what I've done so far at EMC with this new set-
up.
>>> This
>>> > has
>>> > > > >> also
>>> > > > >> > > been part of the problem. I can't seem to get them
to
>>> focus so
>>> > > that
>>> > > > >> we
>>> > > > >> > can
>>> > > > >> > > prioritize. I know that I'm doing the best I can.
If I can
>>> > learn
>>> > > > >> how to
>>> > > > >> > > go about this more efficiently it would be of great
help.
>>> I can
>>> > > > then
>>> > > > >> > > expand this to other verification as venture to other
>>> projects
>>> > in
>>> > > > the
>>> > > > >> > > future. I know you're busy with many things, so I'm
willing
>>> to
>>> > go
>>> > > > >> along
>>> > > > >> > > with your schedule as you find an opening.
>>> > > > >> > >
>>> > > > >> > > Thanks
>>> > > > >> > >
>>> > > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway
via RT <
>>> > > > >> > > met_help at ucar.edu> wrote:
>>> > > > >> > >
>>> > > > >> > >> Ed,
>>> > > > >> > >>
>>> > > > >> > >> It's really difficult for me to see an obvious
problem
>>> that's
>>> > > > causing
>>> > > > >> > the
>>> > > > >> > >> behavior you describe.
>>> > > > >> > >>
>>> > > > >> > >> Unfortunately, the red and green highlighting you
describe
>>> > > doesn't
>>> > > > >> show
>>> > > > >> > up
>>> > > > >> > >> in the Gmail web client. There must be some mail
system
>>> > > > >> incompatibility
>>> > > > >> > >> there.
>>> > > > >> > >>
>>> > > > >> > >> Looking at your bsub script:
>>> > > > >> > >>
>>> > > > >> > >>
>>> > > > >> >
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
>>> > > > >> > >>
>>> > > > >> > >> I am struck by the number of individual calls you're
>>> making to
>>> > > > >> > METviewer:
>>> > > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2
plot
>>> types =
>>> > 656
>>> > > > >> calls
>>> > > > >> > to
>>> > > > >> > >> METviewer
>>> > > > >> > >>
>>> > > > >> > >> When making plots via the METviewer GUI, it's true
that 1
>>> XML
>>> > = 1
>>> > > > >> plot.
>>> > > > >> > >> However, when making plots via the batch engine, the
plot
>>> XML
>>> > can
>>> > > > be
>>> > > > >> set
>>> > > > >> > >> up
>>> > > > >> > >> to create many, many plots. While I have no direct
proof of
>>> > this,
>>> > > > it
>>> > > > >> > might
>>> > > > >> > >> be the case that executing 600+ batch jobs for
METviewer is
>>> > > leading
>>> > > > >> to
>>> > > > >> > >> trouble on the system. One thing to try would be
setting
>>> up the
>>> > > XML
>>> > > > >> to
>>> > > > >> > >> make
>>> > > > >> > >> many, many more plots in each call. Then check to
see if
>>> > running
>>> > > > >> > METviewer
>>> > > > >> > >> in this way is more stable.
>>> > > > >> > >>
>>> > > > >> > >> I don't think I fully appreciate all of the logic
that
>>> lives in
>>> > > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
>>> > > > >> test_Time_Series_AGG.sh.
>>> > > > >> > >> However, it *might* be possible to construct a
single XML
>>> file
>>> > > > which
>>> > > > >> > >> produces all 656 plots.
>>> > > > >> > >>
>>> > > > >> > >> Let me know what you think and how you'd like to
proceed.
>>> > > > >> > >>
>>> > > > >> > >> Thanks,
>>> > > > >> > >> John
>>> > > > >> > >>
>>> > > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach -
NOAA
>>> Affiliate
>>> > > via
>>> > > > >> RT <
>>> > > > >> > >> met_help at ucar.edu> wrote:
>>> > > > >> > >>
>>> > > > >> > >> >
>>> > > > >> > >> > <URL:
>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>> > > >
>>> > > > >> > >> >
>>> > > > >> > >> > Hi John,
>>> > > > >> > >> >
>>> > > > >> > >> > No problem.
>>> > > > >> > >> >
>>> > > > >> > >> > I'm including my responses below:
>>> > > > >> > >> > Sorry for the delay. I took a look on wcoss today.
You
>>> > describe
>>> > > > >> that
>>> > > > >> > >> your
>>> > > > >> > >> > plotting script has stopped working but you're not
sure
>>> why.
>>> > > > >> > >> > It seems more intermittent. Take a look at the
example
>>> > below.
>>> > > > You
>>> > > > >> > can
>>> > > > >> > >> see
>>> > > > >> > >> > that for some reason there are gaps in the plots
being
>>> > > generated.
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct
4 14:52
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 41781 Oct 5 15:15
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 56691 Oct 19 15:02
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 55548 Oct 20 16:08
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 55598 Oct 21 16:08
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 55599 Oct 22 16:09
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 55352 Oct 24 16:34
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 26175 Oct 31 08:20
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 26178 Nov 1 08:20
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 26160 Nov 3 08:22
>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 25354 Nov 3 08:22
>>> > > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
>>> > > > >> > >> > You can tell when a plot is successful by looking
at the
>>> file
>>> > > > >> size. I
>>> > > > >> > >> > highlighted an example of that in red. However,
the
>>> files
>>> > > > showing
>>> > > > >> > about
>>> > > > >> > >> > 25000 are the ones where no line plots are being
>>> produced.
>>> > You
>>> > > > can
>>> > > > >> > also
>>> > > > >> > >> > see that stat files are generated everyday, so as
long
>>> as I'm
>>> > > > >> loading
>>> > > > >> > >> the
>>> > > > >> > >> > data, which I am, I should be able to use the xml
files
>>> to
>>> > > > generate
>>> > > > >> > the
>>> > > > >> > >> > plots. I'm attaching a directory list from wcoss
that
>>> shows
>>> > > the
>>> > > > >> dates
>>> > > > >> > >> > where data is produced.
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> > *20190801 20190817 20200624 20200710 20200726
>>> 20200812
>>> > > > >> 20200828
>>> > > > >> > >> > 20201003 2020101920190802 20190818 20200625
20200711
>>> > > > 20200727
>>> > > > >> > >> > 20200813 20200829 20201004 2020102020190803
20190819
>>> > > > 20200626
>>> > > > >> > >> > 20200712 20200728 20200814 20200830 20201005
>>> > > > 2020102120190804
>>> > > > >> > >> > 20190820 20200627 20200713 20200729 20200815
>>> 20200831
>>> > > > >> 20201006
>>> > > > >> > >> > 2020102220190805 20190821 20200628 20200714
20200730
>>> > > > 20200816
>>> > > > >> > >> > 20200921 20201007 2020102320190806 20190822
20200629
>>> > > > 20200715
>>> > > > >> > >> > 20200731 20200817 20200922 20201008
2020102420190807
>>> > > > 20190823
>>> > > > >> > >> > 20200630 20200716 20200801 20200818 20200923
>>> 20201009
>>> > > > >> > >> > 2020102520190808 20190824 20200701 20200717
20200802
>>> > > > 20200819
>>> > > > >> > >> > 20200924 20201010 2020102620190809 20190825
20200702
>>> > > > 20200718
>>> > > > >> > >> > 20200803 20200820 20200925 20201011
2020102720190810
>>> > > > 20190826
>>> > > > >> > >> > 20200703 20200719 20200804 20200821 20200926
>>> 20201012
>>> > > > >> > >> > 2020102820190811 20190827 20200704 20200720
20200805
>>> > > > 20200822
>>> > > > >> > >> > 20200927 20201013 2020102920190812 20190828
20200705
>>> > > > 20200721
>>> > > > >> > >> > 20200806 20200823 20200928 20201014
2020103020190813
>>> > > > 20190829
>>> > > > >> > >> > 20200706 20200722 20200807 20200824 20200929
>>> 20201015
>>> > > > >> > >> > 2020103120190814 20190830 20200707 20200723
20200808
>>> > > > 20200825
>>> > > > >> > >> > 20200930 20201016 2020110120190815 20200622
20200708
>>> > > > 20200724
>>> > > > >> > >> > 20200810 20200826 20201001 20201017
2020110220190816
>>> > > > 20200623
>>> > > > >> > >> > 20200709 20200725 20200811 20200827 20201002
>>> 20201018*
>>> > > > >> > >> >
>>> > > > >> > >> > Furthermore, what's interesting is that I have
mixed
>>> success
>>> > > with
>>> > > > >> > other
>>> > > > >> > >> > cases. Using the same field - PMTF - for the same
cycle
>>> -
>>> > the
>>> > > > 06z
>>> > > > >> > >> cycle -
>>> > > > >> > >> > you can see that some of the dates where data was
>>> plotted for
>>> > > > EAST
>>> > > > >> did
>>> > > > >> > >> not
>>> > > > >> > >> > necessarily plot for FULL. This surprises me
because
>>> EAST
>>> > is a
>>> > > > >> subset
>>> > > > >> > >> of
>>> > > > >> > >> > FULL. See below:
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct
4 14:54
>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 49039 Oct 5 15:17
>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 56791 Oct 20 16:09
>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 56834 Oct 21 16:09
>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 56921 Oct 22 16:11
>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 56518 Oct 24 16:36
>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 61187 Oct 31 08:21
>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 26475 Nov 2 08:22
>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--
r-- 1
>>> > > > >> Edward.Strobach
>>> > > > >> > >> > emcmodel 26455 Nov 3 08:23
>>> > > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
>>> > > > >> > >> >
>>> > > > >> > >> > What I find equally concerning as that sometimes
plots
>>> are
>>> > > > >> incorrectly
>>> > > > >> > >> > stored in the wrong directory list as highlighted
in
>>> green
>>> > > above.
>>> > > > >> I've
>>> > > > >> > >> > consulted the plot xml file for PMTF, for
instance, and
>>> find
>>> > > > >> nothing
>>> > > > >> > >> > problematic. The only setting that might pose an
issue
>>> when
>>> > it
>>> > > > >> comes
>>> > > > >> > to
>>> > > > >> > >> > generating line plots is when I set event_equal to
true.
>>> > > > However,
>>> > > > >> all
>>> > > > >> > >> the
>>> > > > >> > >> > stat files are populated for all experiments. In
the
>>> case of
>>> > > > ozone
>>> > > > >> you
>>> > > > >> > >> see
>>> > > > >> > >> > something different. I find that ozone tends to
result in
>>> > > > populated
>>> > > > >> > >> plots
>>> > > > >> > >> > but with intermittent reporting different than
PMTF - see
>>> > > below:
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal
>>> > 6144-rw-r--r-- 1
>>> > > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 54318 Oct 6 10:51
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 54305 Oct 7 10:49
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 62179 Oct 13 13:25
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 63811 Oct 19 12:18
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 60112 Oct 20 12:21
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 60165 Oct 21 12:33
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 60160 Oct 22 12:21
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 60056 Oct 23 12:40
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 58517 Oct 24 13:31
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 58062 Oct 31 14:17
>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--
r-- 1
>>> > > > >> > Edward.Strobach
>>> > > > >> > >> > emcmodel 58119 Nov 1 14:19
>>> > > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
>>> > > > >> > >> >
>>> > > > >> > >> > Similar issues have been found for other fields,
cycles,
>>> and
>>> > > > >> regions
>>> > > > >> > >> > including verification being done for meteorology.
>>> > > > >> > >> > I can't seem to reason this intermittency and
>>> inconsistency
>>> > > > between
>>> > > > >> > >> > reporting since the xml files are correct and I'm
using
>>> the
>>> > > same
>>> > > > >> > >> procedure
>>> > > > >> > >> > that I've always used. I do realize that I need
to start
>>> > > > creating
>>> > > > >> a
>>> > > > >> > >> better
>>> > > > >> > >> > way to manage all these files that are being
produced,
>>> both
>>> > in
>>> > > > >> terms
>>> > > > >> > of
>>> > > > >> > >> > data/plots and xml files.
>>> > > > >> > >> >
>>> > > > >> > >> > I ran across a rather large directory:
>>> > > > >> > >> > ls
>>> > > > >>
>>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>>> > > > >> > |
>>> > > > >> > >> wc
>>> > > > >> > >> > -w
>>> > > > >> > >> >
>>> > > > >> > >> > That has 71,398 xml files in there. But I see
timestamps
>>> in
>>> > > those
>>> > > > >> > >> filenames
>>> > > > >> > >> > from 20200921 to 20200925.
>>> > > > >> > >> >
>>> > > > >> > >> > Is it the case that this stopped working on
20200925? Or
>>> are
>>> > > > those
>>> > > > >> > dated
>>> > > > >> > >> > filenames remnants from older work?
>>> > > > >> > >> >
>>> > > > >> > >> > So it hasn't been working for over a month. Is
that
>>> right?
>>> > > > >> > >> >
>>> > > > >> > >> > There are a lot of files and I need to begin
managing
>>> that
>>> > > > >> better. I
>>> > > > >> > >> think
>>> > > > >> > >> > during that time there was a machine switch so it
>>> stopped on
>>> > > that
>>> > > > >> day.
>>> > > > >> > >> > I've been able to produce plots even now, but it's
not
>>> > > consistent
>>> > > > >> in
>>> > > > >> > >> > reporting while some regions generate plots and
others do
>>> > not.
>>> > > > >> I've
>>> > > > >> > >> > reduced the level processing in hopes that I don't
hit a
>>> walk
>>> > > > clock
>>> > > > >> > >> issue
>>> > > > >> > >> > since I'm submitting batch jobs. However, I've
run this
>>> step
>>> > > > >> outside
>>> > > > >> > of
>>> > > > >> > >> > batch and it always seems to work. This issue
seems to
>>> happen
>>> > > > with
>>> > > > >> > batch
>>> > > > >> > >> > and it is not at all clear why. Let me know what
else
>>> you
>>> > need
>>> > > > >> from
>>> > > > >> > me
>>> > > > >> > >> to
>>> > > > >> > >> > better coordinate what I would need to do next.
Thanks.
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway
via RT
>>> <
>>> > > > >> > >> > met_help at ucar.edu>
>>> > > > >> > >> > wrote:
>>> > > > >> > >> >
>>> > > > >> > >> > > Hi Ed,
>>> > > > >> > >> > >
>>> > > > >> > >> > > Sorry for the delay. I took a look on wcoss
today. You
>>> > > describe
>>> > > > >> that
>>> > > > >> > >> your
>>> > > > >> > >> > > plotting script has stopped working but you're
not sure
>>> > why.
>>> > > > >> > >> > >
>>> > > > >> > >> > > I ran across a rather large directory:
>>> > > > >> > >> > > ls
>>> > > > >> >
>>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>>> > > > >> > >> |
>>> > > > >> > >> > wc
>>> > > > >> > >> > > -w
>>> > > > >> > >> > >
>>> > > > >> > >> > > That has 71,398 xml files in there. But I see
>>> timestamps in
>>> > > > those
>>> > > > >> > >> > filenames
>>> > > > >> > >> > > from 20200921 to 20200925.
>>> > > > >> > >> > >
>>> > > > >> > >> > > Is it the case that this stopped working on
20200925?
>>> Or
>>> > are
>>> > > > >> those
>>> > > > >> > >> dated
>>> > > > >> > >> > > filenames remnants from older work?
>>> > > > >> > >> > >
>>> > > > >> > >> > > So it hasn't been working for over a month. Is
that
>>> right?
>>> > > > >> > >> > >
>>> > > > >> > >> > > John
>>> > > > >> > >> > >
>>> > > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System
itself
>>> via RT
>>> > <
>>> > > > >> > >> > > met_help at ucar.edu> wrote:
>>> > > > >> > >> > >
>>> > > > >> > >> > > >
>>> > > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was
acted
>>> upon.
>>> > > > >> > >> > > > Transaction: Given to johnhg (John Halley
Gotway) by
>>> > > > RT_System
>>> > > > >> > >> > > > Queue: met_help
>>> > > > >> > >> > > > Subject: plots no longer being generated
>>> > > > >> > >> > > > Owner: johnhg
>>> > > > >> > >> > > > Requestors: edward.strobach at noaa.gov
>>> > > > >> > >> > > > Status: new
>>> > > > >> > >> > > > Ticket <URL:
>>> > > > >> > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>> > > > >> > >> > >
>>> > > > >> > >> > > >
>>> > > > >> > >> > > >
>>> > > > >> > >> > > > This transaction appears to have no content
>>> > > > >> > >> > > >
>>> > > > >> > >> > >
>>> > > > >> > >> > >
>>> > > > >> > >> >
>>> > > > >> > >> > --
>>> > > > >> > >> > Edward Strobach
>>> > > > >> > >> > EMC/NCEP/NWS/
>>> > > > >> > >> > IMSG Contractor
>>> > > > >> > >> > Cubicle#: 2029
>>> > > > >> > >> > 301-683-3717
>>> > > > >> > >> >
>>> > > > >> > >> >
>>> > > > >> > >>
>>> > > > >> > >>
>>> > > > >> > >
>>> > > > >> > > --
>>> > > > >> > > Edward Strobach
>>> > > > >> > > EMC/NCEP/NWS/
>>> > > > >> > > IMSG Contractor
>>> > > > >> > > Cubicle#: 2029
>>> > > > >> > > 301-683-3717
>>> > > > >> > >
>>> > > > >> >
>>> > > > >> >
>>> > > > >> > --
>>> > > > >> > Edward Strobach
>>> > > > >> > EMC/NCEP/NWS/
>>> > > > >> > IMSG Contractor
>>> > > > >> > Cubicle#: 2029
>>> > > > >> > 301-683-3717
>>> > > > >> >
>>> > > > >> >
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > > > > --
>>> > > > > Edward Strobach
>>> > > > > EMC/NCEP/NWS/
>>> > > > > IMSG Contractor
>>> > > > > Cubicle#: 2029
>>> > > > > 301-683-3717
>>> > > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Edward Strobach
>>> > > > EMC/NCEP/NWS/
>>> > > > IMSG Contractor
>>> > > > Cubicle#: 2029
>>> > > > 301-683-3717
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> >
>>> > --
>>> > Edward Strobach
>>> > EMC/NCEP/NWS/
>>> > IMSG Contractor
>>> > Cubicle#: 2029
>>> > 301-683-3717
>>> >
>>> >
>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Mon Nov 09 08:00:59 2020
I did run some additional tests outside the main test*sh script.
Specifically I ran the following line several times to see what plots
would
be copied over to WCOSS:
bash
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
edward.strobach
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls/Test.xml
The first time around 3 plots were copied over:
PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
100% 51KB 18.6MB/s 00:00
PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
100% 58KB 12.0MB/s 00:00
PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
100% 57KB 2.8MB/s 00:00
The second time 2 plots were copied over:
PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
100% 58KB 2.6MB/s 00:00
PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
100% 57KB 6.2MB/s 00:00
The third time 3 plots were copied over:
PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
100% 51KB
2.4MB/s 00:00
PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
100% 58KB
6.1MB/s 00:00
PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
100% 57KB
11.1MB/s 00:00
The order of init_hour and vx_mask is as follows:
<06>
<12>
and
<Mixed_Forest>
<FULL>
<EAST>
<WEST>
It does appear that the latest plots are copied over in the process
(i.e.
if two, then the last two; if three, then the last three).
I don't really have a feel for why two files are copied over for some
instances and 3 for other instances. Lastly, I ran test_v2.sh again,
and
this time it copied over 4 plots. See below:
PMTF_EAST_112020_2020-11-01_2020-11-06_t06z.png
100% 53KB
12.7MB/s 00:00
PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
100% 51KB
9.7MB/s 00:00
PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
100% 58KB
11.3MB/s 00:00
PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
100% 57KB
11.4MB/s 00:00
This is very bizarre to me.
On Mon, Nov 9, 2020 at 9:26 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Actually, I don't think I can do that. FBAR and OBAR are split and
adding
> RMSE as a third line might confuse the process.
>
> I was able to run a test with mixed success. I was successful at
> generating 8 plots (4 regions for 2 initial hours). The regions
were
> Mixed_Forest, FULL, EAST, and WEST while the initial hours were 06
and 12.
> The test ran well with indications that all plots were generated and
stored
> here: Created plot
> /data/mv_data/edward.strobach/plots/PMTF_Mixed_Forest_112020_2020-
11-01_2020-11-06_t06z.png
>
> However, I was not successful at saving the plots on Dell. Only the
last
> plot was saved here:
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011.
>
> It seems that in my bash script the line that produces the plots
that are
> then stored in the above directory must be done in a loop otherwise
it will
> only move the last plot over as suggested by the fact that only the
plot
> featuring WEST at 12z was moved over - the last plot of the series
of plots
> generated. I suppose I could create logic that extracts the
region/initial
> hour information and construct a two part loop to move over all
plots?
>
> On Mon, Nov 9, 2020 at 8:19 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> Just a follow-up on one other item..
>>
>> Can also list multiple stat types in the same way that I specify
the
>> init_hour and vx_mask? In other words, can I do the following:
>>
>> <dep1>
>> <fcst_var name="PMTF">
>> <stat>FBAR_OBAR</stat>
>> <stat>RMSE</stat>
>> </fcst_var>
>> </dep1>
>>
>> I know that the "set" block is not being used here.
>>
>> On Mon, Nov 9, 2020 at 7:41 AM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> Thanks John. Let me try that and I'll let you know how it goes.
>>>
>>> On Fri, Nov 6, 2020 at 10:53 AM John Halley Gotway via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>>> Ed,
>>>>
>>>> You're right about the METviewer GUI grouping multiple selections
>>>> together.
>>>> That looks like this in the XML using the <set> tag:
>>>> <field equalize="false" name="vx_mask">
>>>> <set name="vx_mask_0">
>>>> <val>FULL</val>
>>>> </set>
>>>> </field>
>>>>
>>>> But in the updates I sent, I remove the <set> tag:
>>>> <field equalize="false" name="vx_mask">
>>>> <val>FULL</val>
>>>> <val>CONUS</val>
>>>> <val>EAST</val>
>>>> <val>WEST</val>
>>>> </field>
>>>> So instead of processing this as 1 group of 4 values, it'll
process the
>>>> 4
>>>> values separately.
>>>>
>>>> The current vx_mask value is referenced using {vx_mask} syntax.
And I
>>>> included examples of using that in the output file names and
titles in
>>>> my
>>>> previous email:
>>>>
>>>>
>>>>
>>>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>>>>
>>>>
>>>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>>>>
>>>>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>>>>
>>>> <title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
>>>> Domain)</title>
>>>>
>>>> John
>>>>
>>>> On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA Affiliate
via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>> >
>>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>>>> >
>>>> > Hi John,
>>>> >
>>>> > This definitely helps. I never thought about it this way
because I
>>>> > assumed, based on my experience with the interactive MET GUI,
that I
>>>> could
>>>> > only generate a single plot regardless of how I specified my
settings
>>>> in
>>>> > the XML. For example, if I listed EAST and WEST under vx_mask,
then
>>>> my
>>>> > thinking would be that both those regions would be grouped and
a
>>>> single
>>>> > plot created. I guess I was wrong on that. The concern I
still have,
>>>> > however, is that the plot title and plot name - the png file -
is
>>>> also set
>>>> > in the XML file. Even after adding the list you gave, I can't
see how
>>>> > unique file names will be created. Is that true?
>>>> >
>>>> > On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT <
>>>> > met_help at ucar.edu>
>>>> > wrote:
>>>> >
>>>> > > Ed,
>>>> > >
>>>> > > Thanks for sending examples. My goal was to use METviewer's
XML
>>>> upload
>>>> > > functionality but I'm not able to upload a pdf version of the
xml
>>>> file.
>>>> > >
>>>> > > I did spend some time trying to replicate this exact plot
with
>>>> METviewer
>>>> > > at:
>>>> > > https://metviewer.nws.noaa.gov/metviewer1.jsp
>>>> > >
>>>> > > But I was never quite successful. So I made just a similar
version
>>>> > instead.
>>>> > > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
>>>> > >
>>>> > > This plots FBAR and OBAR for 6 different models for the month
of
>>>> October
>>>> > > for all 06Z initializations over the FULL model domain.
>>>> > >
>>>> > > I wanted to show you how to modify this for batch mode... and
then
>>>> adapt
>>>> > it
>>>> > > to make multiple plots. But I'm not able to make batch plots
on
>>>> wcoss
>>>> > using
>>>> > > the AWS instance. I don't think I have the right permissions
to do
>>>> so.
>>>> > >
>>>> > > So let me just say this. Generally, you just update the list
of
>>>> items in
>>>> > > plot_fix section.
>>>> > >
>>>> > > Original from METviewer with 1 init hour and 1 mask:
>>>> > >
>>>> > > <field equalize="false" name="vx_mask">
>>>> > > <set name="vx_mask_0">
>>>> > > <val>FULL</val>
>>>> > > </set>
>>>> > > </field>
>>>> > > <field equalize="false" name="init_hour">
>>>> > > <set name="init_hour_1">
>>>> > > <val>06</val>
>>>> > > </set>
>>>> > > </field>
>>>> > >
>>>> > > Change to 2 init hours and 4 masks:
>>>> > >
>>>> > > <field equalize="false" name="vx_mask">
>>>> > > <val>FULL</val>
>>>> > > <val>CONUS</val>
>>>> > > <val>EAST</val>
>>>> > > <val>WEST</val>
>>>> > > </field>
>>>> > > <field equalize="false" name="init_hour">
>>>> > > <val>06</val>
>>>> > > <val>12</val>
>>>> > > </field>
>>>> > >
>>>> > > And then reference those values in the naming of the titles
and
>>>> output
>>>> > > files:
>>>> > >
>>>> > >
>>>> > >
>>>> > >
>>>> >
>>>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>>>> > >
>>>> > >
>>>> > >
>>>> >
>>>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>>>> > >
>>>> > >
>>>>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>>>> > >
>>>> > > <title>Average PMTF (October 2020; {init_hour}z cycle;
{vx_mask}
>>>> > > Domain)</title>
>>>> > >
>>>> > > So that's the general idea. And when you submit this to
mv_batch,
>>>> it'll
>>>> > > make 8 plots instead of 1.
>>>> > > And you can see how you'd easily expand this to many plots.
>>>> > >
>>>> > > But I worry that this may not be sufficient information to
get you
>>>> going.
>>>> > >
>>>> > > John
>>>> > >
>>>> > > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA
Affiliate
>>>> via RT <
>>>> > > met_help at ucar.edu> wrote:
>>>> > >
>>>> > > >
>>>> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>>>> > > >
>>>> > > > Hi John,
>>>> > > >
>>>> > > > I think I was able to cover your request okay. Below are 4
>>>> attachments
>>>> > > > that include the test script that was used to generate the
plots,
>>>> a
>>>> > > sample
>>>> > > > XML file, and the two plots that were generated using the
test run
>>>> > > script.
>>>> > > > These plots were not created previously using the batch job
but
>>>> uses
>>>> > the
>>>> > > > same exact logic as the original run script. Thanks again
for
>>>> working
>>>> > > with
>>>> > > > me on this. Please let me know if you need anything else
or
>>>> would like
>>>> > > me
>>>> > > > to test something.
>>>> > > >
>>>> > > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA
Affiliate <
>>>> > > > edward.strobach at noaa.gov> wrote:
>>>> > > >
>>>> > > > > Ed,
>>>> > > > >
>>>> > > > > Image files showing up in an unexpected output directory
likely
>>>> point
>>>> > > to
>>>> > > > a
>>>> > > > > problem in the logic of the processing scripts. It's
pretty
>>>> unlikely
>>>> > > that
>>>> > > > > this behavior is caused by the METviewer code itself. But
I've
>>>> > > definitely
>>>> > > > > been surprised before!
>>>> > > > >
>>>> > > > > I tested the logic outside of a batch job and it works
fine. I
>>>> > suspect
>>>> > > > > that the issuance of another batch job while the other
one is
>>>> getting
>>>> > > > > processed is causing some confusion in the processing.
These
>>>> issues
>>>> > > > don't
>>>> > > > > happen everyday and appear more sporadic than anything.
>>>> > > > >
>>>> > > > > So what we really need here is an example of taking a
plot from
>>>> the
>>>> > > > > METviewer GUI and adapting it to make it generate
multiple
>>>> plots. I
>>>> > did
>>>> > > > go
>>>> > > > > through examples of doing this in previous tutorials, but
we
>>>> didn't
>>>> > > start
>>>> > > > > recording them until last summer. I checked the July 2019
NRL
>>>> > tutorial
>>>> > > > > videos and don't see it there.
>>>> > > > >
>>>> > > > > Is there something you need from me to help on this?
Just let
>>>> me
>>>> > know
>>>> > > > >
>>>> > > > > We've begun recording training videos for common topics
like
>>>> this.
>>>> > But
>>>> > > we
>>>> > > > > only have one for METviewer so far:
>>>> > > > >
>>>> > >
>>>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>>>> > > > >
>>>> > > > > But this definitely seems like a good candidate for a new
>>>> training
>>>> > > video.
>>>> > > > >
>>>> > > > > For now though, how about this...
>>>> > > > > - You use METviewer to manually make a single plot,
formatting
>>>> it how
>>>> > > > you'd
>>>> > > > > like.
>>>> > > > > - You send me your sample plot and corresponding XML
file.
>>>> > > > > - I'll tweak it to generate many plots via batch instead
of
>>>> just one.
>>>> > > > > - And then you can continue tweaking to add more
permutations.
>>>> > > > >
>>>> > > > > Okay, I'll work on that now. Thanks.
>>>> > > > >
>>>> > > > > John
>>>> > > > >
>>>> > > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via RT
<
>>>> > > > > met_help at ucar.edu> wrote:
>>>> > > > >
>>>> > > > >> Ed,
>>>> > > > >>
>>>> > > > >> Image files showing up in an unexpected output directory
likely
>>>> > point
>>>> > > > to a
>>>> > > > >> problem in the logic of the processing scripts. It's
pretty
>>>> unlikely
>>>> > > > that
>>>> > > > >> this behavior is caused by the METviewer code itself.
But I've
>>>> > > > definitely
>>>> > > > >> been surprised before!
>>>> > > > >>
>>>> > > > >> So what we really need here is an example of taking a
plot
>>>> from the
>>>> > > > >> METviewer GUI and adapting it to make it generate
multiple
>>>> plots. I
>>>> > > did
>>>> > > > go
>>>> > > > >> through examples of doing this in previous tutorials,
but we
>>>> didn't
>>>> > > > start
>>>> > > > >> recording them until last summer. I checked the July
2019 NRL
>>>> > tutorial
>>>> > > > >> videos and don't see it there.
>>>> > > > >>
>>>> > > > >> We've begun recording training videos for common topics
like
>>>> this.
>>>> > But
>>>> > > > we
>>>> > > > >> only have one for METviewer so far:
>>>> > > > >>
>>>> > > >
>>>> >
>>>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>>>> > > > >>
>>>> > > > >> But this definitely seems like a good candidate for a
new
>>>> training
>>>> > > > video.
>>>> > > > >>
>>>> > > > >> For now though, how about this...
>>>> > > > >> - You use METviewer to manually make a single plot,
formatting
>>>> it
>>>> > how
>>>> > > > >> you'd
>>>> > > > >> like.
>>>> > > > >> - You send me your sample plot and corresponding XML
file.
>>>> > > > >> - I'll tweak it to generate many plots via batch instead
of
>>>> just
>>>> > one.
>>>> > > > >> - And then you can continue tweaking to add more
permutations.
>>>> > > > >>
>>>> > > > >> John
>>>> > > > >>
>>>> > > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA
>>>> Affiliate via
>>>> > > RT <
>>>> > > > >> met_help at ucar.edu> wrote:
>>>> > > > >>
>>>> > > > >> >
>>>> > > > >> > <URL:
>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>>>> > > > >> >
>>>> > > > >> > Oh, and one more thing..
>>>> > > > >> >
>>>> > > > >> > The green text that didn't show up was meant to
highlight
>>>> the fact
>>>> > > > that
>>>> > > > >> > sometimes plots that shouldn't be stored in certain
>>>> directories
>>>> > end
>>>> > > up
>>>> > > > >> > there. The pasted example below shows one instance
where
>>>> OZMAX*
>>>> > was
>>>> > > > >> stored
>>>> > > > >> > where TMP* should have been stored.
>>>> > > > >> >
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5
10:52
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6
10:52
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7
10:00
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13
13:49
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14
11:31
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19
11:37
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20
11:35
>>>> > > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21
11:45
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22
11:42
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23
11:50
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24
12:57
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31
08:47
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2
08:50
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3
08:52
>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
>>>> > > > >> >
>>>> > > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach - NOAA
>>>> Affiliate <
>>>> > > > >> > edward.strobach at noaa.gov> wrote:
>>>> > > > >> >
>>>> > > > >> > > Hi John,
>>>> > > > >> > >
>>>> > > > >> > > I know there are a lot of calls and speculate that
this
>>>> could be
>>>> > > > part
>>>> > > > >> of
>>>> > > > >> > > the problem. I don't think that I'm following best
>>>> practice by
>>>> > > > >> > approaching
>>>> > > > >> > > it in this way. Generating plots usually works well
if I
>>>> > process
>>>> > > a
>>>> > > > >> > subset
>>>> > > > >> > > of the computations outside of batch scripting. The
python
>>>> > script
>>>> > > > was
>>>> > > > >> > > given to me by Logan, but I have since made
significant
>>>> changes
>>>> > to
>>>> > > > it
>>>> > > > >> in
>>>> > > > >> > > order to dynamically fill in the xml file based on
>>>> parameters
>>>> > set.
>>>> > > > >> The
>>>> > > > >> > > test*.sh script was also given to me. I created the
>>>> > > > Time_Series*bsub
>>>> > > > >> as
>>>> > > > >> > a
>>>> > > > >> > > way to process a series of plots based on cycle
time,
>>>> region,
>>>> > > etc. I
>>>> > > > >> > think
>>>> > > > >> > > the first two scripts are solid and do well for what
I
>>>> need to
>>>> > get
>>>> > > > >> done.
>>>> > > > >> > > It's the Time_Series*bsub script that bothers me a
bit.
>>>> > > > >> > >
>>>> > > > >> > > I can't see how a single XML file can process all
these
>>>> plots
>>>> > in a
>>>> > > > >> single
>>>> > > > >> > > setting based on what I've learned thus far. How
would
>>>> one go
>>>> > > about
>>>> > > > >> > that?
>>>> > > > >> > > I would be open going in that direction if you can
provide
>>>> some
>>>> > > > >> guidance.
>>>> > > > >> > >
>>>> > > > >> > > Unfortunately, I haven't gotten a lot of feedback or
>>>> dialogue
>>>> > > > exchange
>>>> > > > >> > > about what I've done so far at EMC with this new
set-up.
>>>> This
>>>> > has
>>>> > > > >> also
>>>> > > > >> > > been part of the problem. I can't seem to get them
to
>>>> focus so
>>>> > > that
>>>> > > > >> we
>>>> > > > >> > can
>>>> > > > >> > > prioritize. I know that I'm doing the best I can.
If I
>>>> can
>>>> > learn
>>>> > > > >> how to
>>>> > > > >> > > go about this more efficiently it would be of great
help.
>>>> I can
>>>> > > > then
>>>> > > > >> > > expand this to other verification as venture to
other
>>>> projects
>>>> > in
>>>> > > > the
>>>> > > > >> > > future. I know you're busy with many things, so I'm
>>>> willing to
>>>> > go
>>>> > > > >> along
>>>> > > > >> > > with your schedule as you find an opening.
>>>> > > > >> > >
>>>> > > > >> > > Thanks
>>>> > > > >> > >
>>>> > > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway
via RT <
>>>> > > > >> > > met_help at ucar.edu> wrote:
>>>> > > > >> > >
>>>> > > > >> > >> Ed,
>>>> > > > >> > >>
>>>> > > > >> > >> It's really difficult for me to see an obvious
problem
>>>> that's
>>>> > > > causing
>>>> > > > >> > the
>>>> > > > >> > >> behavior you describe.
>>>> > > > >> > >>
>>>> > > > >> > >> Unfortunately, the red and green highlighting you
describe
>>>> > > doesn't
>>>> > > > >> show
>>>> > > > >> > up
>>>> > > > >> > >> in the Gmail web client. There must be some mail
system
>>>> > > > >> incompatibility
>>>> > > > >> > >> there.
>>>> > > > >> > >>
>>>> > > > >> > >> Looking at your bsub script:
>>>> > > > >> > >>
>>>> > > > >> > >>
>>>> > > > >> >
>>>> > > > >>
>>>> > > >
>>>> > >
>>>> >
>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
>>>> > > > >> > >>
>>>> > > > >> > >> I am struck by the number of individual calls
you're
>>>> making to
>>>> > > > >> > METviewer:
>>>> > > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2
plot
>>>> types =
>>>> > 656
>>>> > > > >> calls
>>>> > > > >> > to
>>>> > > > >> > >> METviewer
>>>> > > > >> > >>
>>>> > > > >> > >> When making plots via the METviewer GUI, it's true
that 1
>>>> XML
>>>> > = 1
>>>> > > > >> plot.
>>>> > > > >> > >> However, when making plots via the batch engine,
the plot
>>>> XML
>>>> > can
>>>> > > > be
>>>> > > > >> set
>>>> > > > >> > >> up
>>>> > > > >> > >> to create many, many plots. While I have no direct
proof
>>>> of
>>>> > this,
>>>> > > > it
>>>> > > > >> > might
>>>> > > > >> > >> be the case that executing 600+ batch jobs for
METviewer
>>>> is
>>>> > > leading
>>>> > > > >> to
>>>> > > > >> > >> trouble on the system. One thing to try would be
setting
>>>> up the
>>>> > > XML
>>>> > > > >> to
>>>> > > > >> > >> make
>>>> > > > >> > >> many, many more plots in each call. Then check to
see if
>>>> > running
>>>> > > > >> > METviewer
>>>> > > > >> > >> in this way is more stable.
>>>> > > > >> > >>
>>>> > > > >> > >> I don't think I fully appreciate all of the logic
that
>>>> lives in
>>>> > > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
>>>> > > > >> test_Time_Series_AGG.sh.
>>>> > > > >> > >> However, it *might* be possible to construct a
single XML
>>>> file
>>>> > > > which
>>>> > > > >> > >> produces all 656 plots.
>>>> > > > >> > >>
>>>> > > > >> > >> Let me know what you think and how you'd like to
proceed.
>>>> > > > >> > >>
>>>> > > > >> > >> Thanks,
>>>> > > > >> > >> John
>>>> > > > >> > >>
>>>> > > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach -
NOAA
>>>> Affiliate
>>>> > > via
>>>> > > > >> RT <
>>>> > > > >> > >> met_help at ucar.edu> wrote:
>>>> > > > >> > >>
>>>> > > > >> > >> >
>>>> > > > >> > >> > <URL:
>>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>>> > > >
>>>> > > > >> > >> >
>>>> > > > >> > >> > Hi John,
>>>> > > > >> > >> >
>>>> > > > >> > >> > No problem.
>>>> > > > >> > >> >
>>>> > > > >> > >> > I'm including my responses below:
>>>> > > > >> > >> > Sorry for the delay. I took a look on wcoss
today. You
>>>> > describe
>>>> > > > >> that
>>>> > > > >> > >> your
>>>> > > > >> > >> > plotting script has stopped working but you're
not sure
>>>> why.
>>>> > > > >> > >> > It seems more intermittent. Take a look at the
example
>>>> > below.
>>>> > > > You
>>>> > > > >> > can
>>>> > > > >> > >> see
>>>> > > > >> > >> > that for some reason there are gaps in the plots
being
>>>> > > generated.
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct
4
>>>> 14:52
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 41781 Oct 5 15:15
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 56691 Oct 19 15:02
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 55548 Oct 20 16:08
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 55598 Oct 21 16:08
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 55599 Oct 22 16:09
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 55352 Oct 24 16:34
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 26175 Oct 31 08:20
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 26178 Nov 1 08:20
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 26160 Nov 3 08:22
>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 25354 Nov 3 08:22
>>>> > > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
>>>> > > > >> > >> > You can tell when a plot is successful by looking
at
>>>> the file
>>>> > > > >> size. I
>>>> > > > >> > >> > highlighted an example of that in red. However,
the
>>>> files
>>>> > > > showing
>>>> > > > >> > about
>>>> > > > >> > >> > 25000 are the ones where no line plots are being
>>>> produced.
>>>> > You
>>>> > > > can
>>>> > > > >> > also
>>>> > > > >> > >> > see that stat files are generated everyday, so as
long
>>>> as I'm
>>>> > > > >> loading
>>>> > > > >> > >> the
>>>> > > > >> > >> > data, which I am, I should be able to use the xml
files
>>>> to
>>>> > > > generate
>>>> > > > >> > the
>>>> > > > >> > >> > plots. I'm attaching a directory list from wcoss
that
>>>> shows
>>>> > > the
>>>> > > > >> dates
>>>> > > > >> > >> > where data is produced.
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> > *20190801 20190817 20200624 20200710 20200726
>>>> 20200812
>>>> > > > >> 20200828
>>>> > > > >> > >> > 20201003 2020101920190802 20190818 20200625
>>>> 20200711
>>>> > > > 20200727
>>>> > > > >> > >> > 20200813 20200829 20201004 2020102020190803
>>>> 20190819
>>>> > > > 20200626
>>>> > > > >> > >> > 20200712 20200728 20200814 20200830 20201005
>>>> > > > 2020102120190804
>>>> > > > >> > >> > 20190820 20200627 20200713 20200729 20200815
>>>> 20200831
>>>> > > > >> 20201006
>>>> > > > >> > >> > 2020102220190805 20190821 20200628 20200714
>>>> 20200730
>>>> > > > 20200816
>>>> > > > >> > >> > 20200921 20201007 2020102320190806 20190822
>>>> 20200629
>>>> > > > 20200715
>>>> > > > >> > >> > 20200731 20200817 20200922 20201008
>>>> 2020102420190807
>>>> > > > 20190823
>>>> > > > >> > >> > 20200630 20200716 20200801 20200818 20200923
>>>> 20201009
>>>> > > > >> > >> > 2020102520190808 20190824 20200701 20200717
>>>> 20200802
>>>> > > > 20200819
>>>> > > > >> > >> > 20200924 20201010 2020102620190809 20190825
>>>> 20200702
>>>> > > > 20200718
>>>> > > > >> > >> > 20200803 20200820 20200925 20201011
>>>> 2020102720190810
>>>> > > > 20190826
>>>> > > > >> > >> > 20200703 20200719 20200804 20200821 20200926
>>>> 20201012
>>>> > > > >> > >> > 2020102820190811 20190827 20200704 20200720
>>>> 20200805
>>>> > > > 20200822
>>>> > > > >> > >> > 20200927 20201013 2020102920190812 20190828
>>>> 20200705
>>>> > > > 20200721
>>>> > > > >> > >> > 20200806 20200823 20200928 20201014
>>>> 2020103020190813
>>>> > > > 20190829
>>>> > > > >> > >> > 20200706 20200722 20200807 20200824 20200929
>>>> 20201015
>>>> > > > >> > >> > 2020103120190814 20190830 20200707 20200723
>>>> 20200808
>>>> > > > 20200825
>>>> > > > >> > >> > 20200930 20201016 2020110120190815 20200622
>>>> 20200708
>>>> > > > 20200724
>>>> > > > >> > >> > 20200810 20200826 20201001 20201017
>>>> 2020110220190816
>>>> > > > 20200623
>>>> > > > >> > >> > 20200709 20200725 20200811 20200827 20201002
>>>> 20201018*
>>>> > > > >> > >> >
>>>> > > > >> > >> > Furthermore, what's interesting is that I have
mixed
>>>> success
>>>> > > with
>>>> > > > >> > other
>>>> > > > >> > >> > cases. Using the same field - PMTF - for the
same
>>>> cycle -
>>>> > the
>>>> > > > 06z
>>>> > > > >> > >> cycle -
>>>> > > > >> > >> > you can see that some of the dates where data was
>>>> plotted for
>>>> > > > EAST
>>>> > > > >> did
>>>> > > > >> > >> not
>>>> > > > >> > >> > necessarily plot for FULL. This surprises me
because
>>>> EAST
>>>> > is a
>>>> > > > >> subset
>>>> > > > >> > >> of
>>>> > > > >> > >> > FULL. See below:
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct
4
>>>> 14:54
>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 49039 Oct 5 15:17
>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 56791 Oct 20 16:09
>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 56834 Oct 21 16:09
>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 56921 Oct 22 16:11
>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 56518 Oct 24 16:36
>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 61187 Oct 31 08:21
>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 26475 Nov 2 08:22
>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--
r-- 1
>>>> > > > >> Edward.Strobach
>>>> > > > >> > >> > emcmodel 26455 Nov 3 08:23
>>>> > > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
>>>> > > > >> > >> >
>>>> > > > >> > >> > What I find equally concerning as that sometimes
plots
>>>> are
>>>> > > > >> incorrectly
>>>> > > > >> > >> > stored in the wrong directory list as highlighted
in
>>>> green
>>>> > > above.
>>>> > > > >> I've
>>>> > > > >> > >> > consulted the plot xml file for PMTF, for
instance, and
>>>> find
>>>> > > > >> nothing
>>>> > > > >> > >> > problematic. The only setting that might pose an
issue
>>>> when
>>>> > it
>>>> > > > >> comes
>>>> > > > >> > to
>>>> > > > >> > >> > generating line plots is when I set event_equal
to true.
>>>> > > > However,
>>>> > > > >> all
>>>> > > > >> > >> the
>>>> > > > >> > >> > stat files are populated for all experiments. In
the
>>>> case of
>>>> > > > ozone
>>>> > > > >> you
>>>> > > > >> > >> see
>>>> > > > >> > >> > something different. I find that ozone tends to
result
>>>> in
>>>> > > > populated
>>>> > > > >> > >> plots
>>>> > > > >> > >> > but with intermittent reporting different than
PMTF -
>>>> see
>>>> > > below:
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal
>>>> > 6144-rw-r--r-- 1
>>>> > > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 54318 Oct 6 10:51
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 54305 Oct 7 10:49
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 62179 Oct 13 13:25
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 63811 Oct 19 12:18
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 60112 Oct 20 12:21
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 60165 Oct 21 12:33
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 60160 Oct 22 12:21
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 60056 Oct 23 12:40
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 58517 Oct 24 13:31
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 58062 Oct 31 14:17
>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-
r--r-- 1
>>>> > > > >> > Edward.Strobach
>>>> > > > >> > >> > emcmodel 58119 Nov 1 14:19
>>>> > > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
>>>> > > > >> > >> >
>>>> > > > >> > >> > Similar issues have been found for other fields,
>>>> cycles, and
>>>> > > > >> regions
>>>> > > > >> > >> > including verification being done for
meteorology.
>>>> > > > >> > >> > I can't seem to reason this intermittency and
>>>> inconsistency
>>>> > > > between
>>>> > > > >> > >> > reporting since the xml files are correct and I'm
using
>>>> the
>>>> > > same
>>>> > > > >> > >> procedure
>>>> > > > >> > >> > that I've always used. I do realize that I need
to
>>>> start
>>>> > > > creating
>>>> > > > >> a
>>>> > > > >> > >> better
>>>> > > > >> > >> > way to manage all these files that are being
produced,
>>>> both
>>>> > in
>>>> > > > >> terms
>>>> > > > >> > of
>>>> > > > >> > >> > data/plots and xml files.
>>>> > > > >> > >> >
>>>> > > > >> > >> > I ran across a rather large directory:
>>>> > > > >> > >> > ls
>>>> > > > >>
>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>>>> > > > >> > |
>>>> > > > >> > >> wc
>>>> > > > >> > >> > -w
>>>> > > > >> > >> >
>>>> > > > >> > >> > That has 71,398 xml files in there. But I see
>>>> timestamps in
>>>> > > those
>>>> > > > >> > >> filenames
>>>> > > > >> > >> > from 20200921 to 20200925.
>>>> > > > >> > >> >
>>>> > > > >> > >> > Is it the case that this stopped working on
20200925?
>>>> Or are
>>>> > > > those
>>>> > > > >> > dated
>>>> > > > >> > >> > filenames remnants from older work?
>>>> > > > >> > >> >
>>>> > > > >> > >> > So it hasn't been working for over a month. Is
that
>>>> right?
>>>> > > > >> > >> >
>>>> > > > >> > >> > There are a lot of files and I need to begin
managing
>>>> that
>>>> > > > >> better. I
>>>> > > > >> > >> think
>>>> > > > >> > >> > during that time there was a machine switch so it
>>>> stopped on
>>>> > > that
>>>> > > > >> day.
>>>> > > > >> > >> > I've been able to produce plots even now, but
it's not
>>>> > > consistent
>>>> > > > >> in
>>>> > > > >> > >> > reporting while some regions generate plots and
others
>>>> do
>>>> > not.
>>>> > > > >> I've
>>>> > > > >> > >> > reduced the level processing in hopes that I
don't hit
>>>> a walk
>>>> > > > clock
>>>> > > > >> > >> issue
>>>> > > > >> > >> > since I'm submitting batch jobs. However, I've
run
>>>> this step
>>>> > > > >> outside
>>>> > > > >> > of
>>>> > > > >> > >> > batch and it always seems to work. This issue
seems to
>>>> happen
>>>> > > > with
>>>> > > > >> > batch
>>>> > > > >> > >> > and it is not at all clear why. Let me know what
else
>>>> you
>>>> > need
>>>> > > > >> from
>>>> > > > >> > me
>>>> > > > >> > >> to
>>>> > > > >> > >> > better coordinate what I would need to do next.
Thanks.
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley Gotway
via
>>>> RT <
>>>> > > > >> > >> > met_help at ucar.edu>
>>>> > > > >> > >> > wrote:
>>>> > > > >> > >> >
>>>> > > > >> > >> > > Hi Ed,
>>>> > > > >> > >> > >
>>>> > > > >> > >> > > Sorry for the delay. I took a look on wcoss
today. You
>>>> > > describe
>>>> > > > >> that
>>>> > > > >> > >> your
>>>> > > > >> > >> > > plotting script has stopped working but you're
not
>>>> sure
>>>> > why.
>>>> > > > >> > >> > >
>>>> > > > >> > >> > > I ran across a rather large directory:
>>>> > > > >> > >> > > ls
>>>> > > > >> >
>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>>>> > > > >> > >> |
>>>> > > > >> > >> > wc
>>>> > > > >> > >> > > -w
>>>> > > > >> > >> > >
>>>> > > > >> > >> > > That has 71,398 xml files in there. But I see
>>>> timestamps in
>>>> > > > those
>>>> > > > >> > >> > filenames
>>>> > > > >> > >> > > from 20200921 to 20200925.
>>>> > > > >> > >> > >
>>>> > > > >> > >> > > Is it the case that this stopped working on
20200925?
>>>> Or
>>>> > are
>>>> > > > >> those
>>>> > > > >> > >> dated
>>>> > > > >> > >> > > filenames remnants from older work?
>>>> > > > >> > >> > >
>>>> > > > >> > >> > > So it hasn't been working for over a month. Is
that
>>>> right?
>>>> > > > >> > >> > >
>>>> > > > >> > >> > > John
>>>> > > > >> > >> > >
>>>> > > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System
itself
>>>> via RT
>>>> > <
>>>> > > > >> > >> > > met_help at ucar.edu> wrote:
>>>> > > > >> > >> > >
>>>> > > > >> > >> > > >
>>>> > > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was
acted
>>>> upon.
>>>> > > > >> > >> > > > Transaction: Given to johnhg (John Halley
Gotway) by
>>>> > > > RT_System
>>>> > > > >> > >> > > > Queue: met_help
>>>> > > > >> > >> > > > Subject: plots no longer being generated
>>>> > > > >> > >> > > > Owner: johnhg
>>>> > > > >> > >> > > > Requestors: edward.strobach at noaa.gov
>>>> > > > >> > >> > > > Status: new
>>>> > > > >> > >> > > > Ticket <URL:
>>>> > > > >> > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>>> > > > >> > >> > >
>>>> > > > >> > >> > > >
>>>> > > > >> > >> > > >
>>>> > > > >> > >> > > > This transaction appears to have no content
>>>> > > > >> > >> > > >
>>>> > > > >> > >> > >
>>>> > > > >> > >> > >
>>>> > > > >> > >> >
>>>> > > > >> > >> > --
>>>> > > > >> > >> > Edward Strobach
>>>> > > > >> > >> > EMC/NCEP/NWS/
>>>> > > > >> > >> > IMSG Contractor
>>>> > > > >> > >> > Cubicle#: 2029
>>>> > > > >> > >> > 301-683-3717
>>>> > > > >> > >> >
>>>> > > > >> > >> >
>>>> > > > >> > >>
>>>> > > > >> > >>
>>>> > > > >> > >
>>>> > > > >> > > --
>>>> > > > >> > > Edward Strobach
>>>> > > > >> > > EMC/NCEP/NWS/
>>>> > > > >> > > IMSG Contractor
>>>> > > > >> > > Cubicle#: 2029
>>>> > > > >> > > 301-683-3717
>>>> > > > >> > >
>>>> > > > >> >
>>>> > > > >> >
>>>> > > > >> > --
>>>> > > > >> > Edward Strobach
>>>> > > > >> > EMC/NCEP/NWS/
>>>> > > > >> > IMSG Contractor
>>>> > > > >> > Cubicle#: 2029
>>>> > > > >> > 301-683-3717
>>>> > > > >> >
>>>> > > > >> >
>>>> > > > >>
>>>> > > > >>
>>>> > > > >
>>>> > > > > --
>>>> > > > > Edward Strobach
>>>> > > > > EMC/NCEP/NWS/
>>>> > > > > IMSG Contractor
>>>> > > > > Cubicle#: 2029
>>>> > > > > 301-683-3717
>>>> > > > >
>>>> > > >
>>>> > > >
>>>> > > > --
>>>> > > > Edward Strobach
>>>> > > > EMC/NCEP/NWS/
>>>> > > > IMSG Contractor
>>>> > > > Cubicle#: 2029
>>>> > > > 301-683-3717
>>>> > > >
>>>> > > >
>>>> > >
>>>> > >
>>>> >
>>>> > --
>>>> > Edward Strobach
>>>> > EMC/NCEP/NWS/
>>>> > IMSG Contractor
>>>> > Cubicle#: 2029
>>>> > 301-683-3717
>>>> >
>>>> >
>>>>
>>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Mon Nov 09 08:02:26 2020
FYI, when running test_v2.sh again after running it the first time
only one
plot was copied over:
PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
100% 57KB
19.9MB/s 00:00
On Mon, Nov 9, 2020 at 10:00 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> I did run some additional tests outside the main test*sh script.
> Specifically I ran the following line several times to see what
plots would
> be copied over to WCOSS:
> bash
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> edward.strobach
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls/Test.xml
>
> The first time around 3 plots were copied over:
> PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
> 100% 51KB 18.6MB/s 00:00
> PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> 100% 58KB 12.0MB/s 00:00
> PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> 100% 57KB 2.8MB/s 00:00
>
> The second time 2 plots were copied over:
> PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> 100% 58KB 2.6MB/s 00:00
> PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> 100% 57KB 6.2MB/s 00:00
>
> The third time 3 plots were copied over:
> PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
> 100%
51KB
> 2.4MB/s 00:00
> PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> 100%
58KB
> 6.1MB/s 00:00
> PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> 100%
57KB
> 11.1MB/s 00:00
>
> The order of init_hour and vx_mask is as follows:
> <06>
> <12>
> and
> <Mixed_Forest>
> <FULL>
> <EAST>
> <WEST>
>
> It does appear that the latest plots are copied over in the process
(i.e.
> if two, then the last two; if three, then the last three).
> I don't really have a feel for why two files are copied over for
some
> instances and 3 for other instances. Lastly, I ran test_v2.sh
again, and
> this time it copied over 4 plots. See below:
>
> PMTF_EAST_112020_2020-11-01_2020-11-06_t06z.png
> 100%
53KB
> 12.7MB/s 00:00
> PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
> 100%
51KB
> 9.7MB/s 00:00
> PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> 100%
58KB
> 11.3MB/s 00:00
> PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> 100%
57KB
> 11.4MB/s 00:00
>
> This is very bizarre to me.
>
>
>
> On Mon, Nov 9, 2020 at 9:26 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> Actually, I don't think I can do that. FBAR and OBAR are split and
>> adding RMSE as a third line might confuse the process.
>>
>> I was able to run a test with mixed success. I was successful at
>> generating 8 plots (4 regions for 2 initial hours). The regions
were
>> Mixed_Forest, FULL, EAST, and WEST while the initial hours were 06
and 12.
>> The test ran well with indications that all plots were generated
and stored
>> here: Created plot
>> /data/mv_data/edward.strobach/plots/PMTF_Mixed_Forest_112020_2020-
11-01_2020-11-06_t06z.png
>>
>> However, I was not successful at saving the plots on Dell. Only the
last
>> plot was saved here:
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011.
>>
>> It seems that in my bash script the line that produces the plots
that are
>> then stored in the above directory must be done in a loop otherwise
it will
>> only move the last plot over as suggested by the fact that only the
plot
>> featuring WEST at 12z was moved over - the last plot of the series
of plots
>> generated. I suppose I could create logic that extracts the
region/initial
>> hour information and construct a two part loop to move over all
plots?
>>
>> On Mon, Nov 9, 2020 at 8:19 AM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> Just a follow-up on one other item..
>>>
>>> Can also list multiple stat types in the same way that I specify
the
>>> init_hour and vx_mask? In other words, can I do the following:
>>>
>>> <dep1>
>>> <fcst_var name="PMTF">
>>> <stat>FBAR_OBAR</stat>
>>> <stat>RMSE</stat>
>>> </fcst_var>
>>> </dep1>
>>>
>>> I know that the "set" block is not being used here.
>>>
>>> On Mon, Nov 9, 2020 at 7:41 AM Edward Strobach - NOAA Affiliate <
>>> edward.strobach at noaa.gov> wrote:
>>>
>>>> Thanks John. Let me try that and I'll let you know how it goes.
>>>>
>>>> On Fri, Nov 6, 2020 at 10:53 AM John Halley Gotway via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>> Ed,
>>>>>
>>>>> You're right about the METviewer GUI grouping multiple
selections
>>>>> together.
>>>>> That looks like this in the XML using the <set> tag:
>>>>> <field equalize="false" name="vx_mask">
>>>>> <set name="vx_mask_0">
>>>>> <val>FULL</val>
>>>>> </set>
>>>>> </field>
>>>>>
>>>>> But in the updates I sent, I remove the <set> tag:
>>>>> <field equalize="false" name="vx_mask">
>>>>> <val>FULL</val>
>>>>> <val>CONUS</val>
>>>>> <val>EAST</val>
>>>>> <val>WEST</val>
>>>>> </field>
>>>>> So instead of processing this as 1 group of 4 values, it'll
process
>>>>> the 4
>>>>> values separately.
>>>>>
>>>>> The current vx_mask value is referenced using {vx_mask} syntax.
And I
>>>>> included examples of using that in the output file names and
titles in
>>>>> my
>>>>> previous email:
>>>>>
>>>>>
>>>>>
>>>>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>>>>>
>>>>>
>>>>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>>>>>
>>>>>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>>>>>
>>>>> <title>Average PMTF (October 2020; {init_hour}z cycle; {vx_mask}
>>>>> Domain)</title>
>>>>>
>>>>> John
>>>>>
>>>>> On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA Affiliate
via RT
>>>>> <
>>>>> met_help at ucar.edu> wrote:
>>>>>
>>>>> >
>>>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>
>>>>> >
>>>>> > Hi John,
>>>>> >
>>>>> > This definitely helps. I never thought about it this way
because I
>>>>> > assumed, based on my experience with the interactive MET GUI,
that I
>>>>> could
>>>>> > only generate a single plot regardless of how I specified my
>>>>> settings in
>>>>> > the XML. For example, if I listed EAST and WEST under
vx_mask, then
>>>>> my
>>>>> > thinking would be that both those regions would be grouped and
a
>>>>> single
>>>>> > plot created. I guess I was wrong on that. The concern I
still
>>>>> have,
>>>>> > however, is that the plot title and plot name - the png file -
is
>>>>> also set
>>>>> > in the XML file. Even after adding the list you gave, I can't
see
>>>>> how
>>>>> > unique file names will be created. Is that true?
>>>>> >
>>>>> > On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT <
>>>>> > met_help at ucar.edu>
>>>>> > wrote:
>>>>> >
>>>>> > > Ed,
>>>>> > >
>>>>> > > Thanks for sending examples. My goal was to use METviewer's
XML
>>>>> upload
>>>>> > > functionality but I'm not able to upload a pdf version of
the xml
>>>>> file.
>>>>> > >
>>>>> > > I did spend some time trying to replicate this exact plot
with
>>>>> METviewer
>>>>> > > at:
>>>>> > > https://metviewer.nws.noaa.gov/metviewer1.jsp
>>>>> > >
>>>>> > > But I was never quite successful. So I made just a similar
version
>>>>> > instead.
>>>>> > > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
>>>>> > >
>>>>> > > This plots FBAR and OBAR for 6 different models for the
month of
>>>>> October
>>>>> > > for all 06Z initializations over the FULL model domain.
>>>>> > >
>>>>> > > I wanted to show you how to modify this for batch mode...
and then
>>>>> adapt
>>>>> > it
>>>>> > > to make multiple plots. But I'm not able to make batch plots
on
>>>>> wcoss
>>>>> > using
>>>>> > > the AWS instance. I don't think I have the right permissions
to do
>>>>> so.
>>>>> > >
>>>>> > > So let me just say this. Generally, you just update the list
of
>>>>> items in
>>>>> > > plot_fix section.
>>>>> > >
>>>>> > > Original from METviewer with 1 init hour and 1 mask:
>>>>> > >
>>>>> > > <field equalize="false" name="vx_mask">
>>>>> > > <set name="vx_mask_0">
>>>>> > > <val>FULL</val>
>>>>> > > </set>
>>>>> > > </field>
>>>>> > > <field equalize="false" name="init_hour">
>>>>> > > <set name="init_hour_1">
>>>>> > > <val>06</val>
>>>>> > > </set>
>>>>> > > </field>
>>>>> > >
>>>>> > > Change to 2 init hours and 4 masks:
>>>>> > >
>>>>> > > <field equalize="false" name="vx_mask">
>>>>> > > <val>FULL</val>
>>>>> > > <val>CONUS</val>
>>>>> > > <val>EAST</val>
>>>>> > > <val>WEST</val>
>>>>> > > </field>
>>>>> > > <field equalize="false" name="init_hour">
>>>>> > > <val>06</val>
>>>>> > > <val>12</val>
>>>>> > > </field>
>>>>> > >
>>>>> > > And then reference those values in the naming of the titles
and
>>>>> output
>>>>> > > files:
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>>>>> > >
>>>>> > >
>>>>>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>>>>> > >
>>>>> > > <title>Average PMTF (October 2020; {init_hour}z cycle;
{vx_mask}
>>>>> > > Domain)</title>
>>>>> > >
>>>>> > > So that's the general idea. And when you submit this to
mv_batch,
>>>>> it'll
>>>>> > > make 8 plots instead of 1.
>>>>> > > And you can see how you'd easily expand this to many plots.
>>>>> > >
>>>>> > > But I worry that this may not be sufficient information to
get you
>>>>> going.
>>>>> > >
>>>>> > > John
>>>>> > >
>>>>> > > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA
Affiliate
>>>>> via RT <
>>>>> > > met_help at ucar.edu> wrote:
>>>>> > >
>>>>> > > >
>>>>> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>>>>> > > >
>>>>> > > > Hi John,
>>>>> > > >
>>>>> > > > I think I was able to cover your request okay. Below are
4
>>>>> attachments
>>>>> > > > that include the test script that was used to generate the
>>>>> plots, a
>>>>> > > sample
>>>>> > > > XML file, and the two plots that were generated using the
test
>>>>> run
>>>>> > > script.
>>>>> > > > These plots were not created previously using the batch
job but
>>>>> uses
>>>>> > the
>>>>> > > > same exact logic as the original run script. Thanks again
for
>>>>> working
>>>>> > > with
>>>>> > > > me on this. Please let me know if you need anything else
or
>>>>> would like
>>>>> > > me
>>>>> > > > to test something.
>>>>> > > >
>>>>> > > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA
Affiliate <
>>>>> > > > edward.strobach at noaa.gov> wrote:
>>>>> > > >
>>>>> > > > > Ed,
>>>>> > > > >
>>>>> > > > > Image files showing up in an unexpected output directory
>>>>> likely point
>>>>> > > to
>>>>> > > > a
>>>>> > > > > problem in the logic of the processing scripts. It's
pretty
>>>>> unlikely
>>>>> > > that
>>>>> > > > > this behavior is caused by the METviewer code itself.
But I've
>>>>> > > definitely
>>>>> > > > > been surprised before!
>>>>> > > > >
>>>>> > > > > I tested the logic outside of a batch job and it works
fine. I
>>>>> > suspect
>>>>> > > > > that the issuance of another batch job while the other
one is
>>>>> getting
>>>>> > > > > processed is causing some confusion in the processing.
These
>>>>> issues
>>>>> > > > don't
>>>>> > > > > happen everyday and appear more sporadic than anything.
>>>>> > > > >
>>>>> > > > > So what we really need here is an example of taking a
plot
>>>>> from the
>>>>> > > > > METviewer GUI and adapting it to make it generate
multiple
>>>>> plots. I
>>>>> > did
>>>>> > > > go
>>>>> > > > > through examples of doing this in previous tutorials,
but we
>>>>> didn't
>>>>> > > start
>>>>> > > > > recording them until last summer. I checked the July
2019 NRL
>>>>> > tutorial
>>>>> > > > > videos and don't see it there.
>>>>> > > > >
>>>>> > > > > Is there something you need from me to help on this?
Just let
>>>>> me
>>>>> > know
>>>>> > > > >
>>>>> > > > > We've begun recording training videos for common topics
like
>>>>> this.
>>>>> > But
>>>>> > > we
>>>>> > > > > only have one for METviewer so far:
>>>>> > > > >
>>>>> > >
>>>>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>>>>> > > > >
>>>>> > > > > But this definitely seems like a good candidate for a
new
>>>>> training
>>>>> > > video.
>>>>> > > > >
>>>>> > > > > For now though, how about this...
>>>>> > > > > - You use METviewer to manually make a single plot,
formatting
>>>>> it how
>>>>> > > > you'd
>>>>> > > > > like.
>>>>> > > > > - You send me your sample plot and corresponding XML
file.
>>>>> > > > > - I'll tweak it to generate many plots via batch instead
of
>>>>> just one.
>>>>> > > > > - And then you can continue tweaking to add more
permutations.
>>>>> > > > >
>>>>> > > > > Okay, I'll work on that now. Thanks.
>>>>> > > > >
>>>>> > > > > John
>>>>> > > > >
>>>>> > > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via
RT <
>>>>> > > > > met_help at ucar.edu> wrote:
>>>>> > > > >
>>>>> > > > >> Ed,
>>>>> > > > >>
>>>>> > > > >> Image files showing up in an unexpected output
directory
>>>>> likely
>>>>> > point
>>>>> > > > to a
>>>>> > > > >> problem in the logic of the processing scripts. It's
pretty
>>>>> unlikely
>>>>> > > > that
>>>>> > > > >> this behavior is caused by the METviewer code itself.
But I've
>>>>> > > > definitely
>>>>> > > > >> been surprised before!
>>>>> > > > >>
>>>>> > > > >> So what we really need here is an example of taking a
plot
>>>>> from the
>>>>> > > > >> METviewer GUI and adapting it to make it generate
multiple
>>>>> plots. I
>>>>> > > did
>>>>> > > > go
>>>>> > > > >> through examples of doing this in previous tutorials,
but we
>>>>> didn't
>>>>> > > > start
>>>>> > > > >> recording them until last summer. I checked the July
2019 NRL
>>>>> > tutorial
>>>>> > > > >> videos and don't see it there.
>>>>> > > > >>
>>>>> > > > >> We've begun recording training videos for common topics
like
>>>>> this.
>>>>> > But
>>>>> > > > we
>>>>> > > > >> only have one for METviewer so far:
>>>>> > > > >>
>>>>> > > >
>>>>> >
>>>>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>>>>> > > > >>
>>>>> > > > >> But this definitely seems like a good candidate for a
new
>>>>> training
>>>>> > > > video.
>>>>> > > > >>
>>>>> > > > >> For now though, how about this...
>>>>> > > > >> - You use METviewer to manually make a single plot,
>>>>> formatting it
>>>>> > how
>>>>> > > > >> you'd
>>>>> > > > >> like.
>>>>> > > > >> - You send me your sample plot and corresponding XML
file.
>>>>> > > > >> - I'll tweak it to generate many plots via batch
instead of
>>>>> just
>>>>> > one.
>>>>> > > > >> - And then you can continue tweaking to add more
permutations.
>>>>> > > > >>
>>>>> > > > >> John
>>>>> > > > >>
>>>>> > > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA
>>>>> Affiliate via
>>>>> > > RT <
>>>>> > > > >> met_help at ucar.edu> wrote:
>>>>> > > > >>
>>>>> > > > >> >
>>>>> > > > >> > <URL:
>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>>>>> > > > >> >
>>>>> > > > >> > Oh, and one more thing..
>>>>> > > > >> >
>>>>> > > > >> > The green text that didn't show up was meant to
highlight
>>>>> the fact
>>>>> > > > that
>>>>> > > > >> > sometimes plots that shouldn't be stored in certain
>>>>> directories
>>>>> > end
>>>>> > > up
>>>>> > > > >> > there. The pasted example below shows one instance
where
>>>>> OZMAX*
>>>>> > was
>>>>> > > > >> stored
>>>>> > > > >> > where TMP* should have been stored.
>>>>> > > > >> >
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5
10:52
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6
10:52
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7
10:00
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13
13:49
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14
11:31
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19
11:37
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20
11:35
>>>>> > > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21
11:45
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22
11:42
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23
11:50
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24
12:57
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31
08:47
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2
08:50
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
>>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3
08:52
>>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
>>>>> > > > >> >
>>>>> > > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach -
NOAA
>>>>> Affiliate <
>>>>> > > > >> > edward.strobach at noaa.gov> wrote:
>>>>> > > > >> >
>>>>> > > > >> > > Hi John,
>>>>> > > > >> > >
>>>>> > > > >> > > I know there are a lot of calls and speculate that
this
>>>>> could be
>>>>> > > > part
>>>>> > > > >> of
>>>>> > > > >> > > the problem. I don't think that I'm following best
>>>>> practice by
>>>>> > > > >> > approaching
>>>>> > > > >> > > it in this way. Generating plots usually works
well if I
>>>>> > process
>>>>> > > a
>>>>> > > > >> > subset
>>>>> > > > >> > > of the computations outside of batch scripting.
The
>>>>> python
>>>>> > script
>>>>> > > > was
>>>>> > > > >> > > given to me by Logan, but I have since made
significant
>>>>> changes
>>>>> > to
>>>>> > > > it
>>>>> > > > >> in
>>>>> > > > >> > > order to dynamically fill in the xml file based on
>>>>> parameters
>>>>> > set.
>>>>> > > > >> The
>>>>> > > > >> > > test*.sh script was also given to me. I created
the
>>>>> > > > Time_Series*bsub
>>>>> > > > >> as
>>>>> > > > >> > a
>>>>> > > > >> > > way to process a series of plots based on cycle
time,
>>>>> region,
>>>>> > > etc. I
>>>>> > > > >> > think
>>>>> > > > >> > > the first two scripts are solid and do well for
what I
>>>>> need to
>>>>> > get
>>>>> > > > >> done.
>>>>> > > > >> > > It's the Time_Series*bsub script that bothers me a
bit.
>>>>> > > > >> > >
>>>>> > > > >> > > I can't see how a single XML file can process all
these
>>>>> plots
>>>>> > in a
>>>>> > > > >> single
>>>>> > > > >> > > setting based on what I've learned thus far. How
would
>>>>> one go
>>>>> > > about
>>>>> > > > >> > that?
>>>>> > > > >> > > I would be open going in that direction if you can
>>>>> provide some
>>>>> > > > >> guidance.
>>>>> > > > >> > >
>>>>> > > > >> > > Unfortunately, I haven't gotten a lot of feedback
or
>>>>> dialogue
>>>>> > > > exchange
>>>>> > > > >> > > about what I've done so far at EMC with this new
set-up.
>>>>> This
>>>>> > has
>>>>> > > > >> also
>>>>> > > > >> > > been part of the problem. I can't seem to get them
to
>>>>> focus so
>>>>> > > that
>>>>> > > > >> we
>>>>> > > > >> > can
>>>>> > > > >> > > prioritize. I know that I'm doing the best I can.
If I
>>>>> can
>>>>> > learn
>>>>> > > > >> how to
>>>>> > > > >> > > go about this more efficiently it would be of great
>>>>> help. I can
>>>>> > > > then
>>>>> > > > >> > > expand this to other verification as venture to
other
>>>>> projects
>>>>> > in
>>>>> > > > the
>>>>> > > > >> > > future. I know you're busy with many things, so I'm
>>>>> willing to
>>>>> > go
>>>>> > > > >> along
>>>>> > > > >> > > with your schedule as you find an opening.
>>>>> > > > >> > >
>>>>> > > > >> > > Thanks
>>>>> > > > >> > >
>>>>> > > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley Gotway
via RT
>>>>> <
>>>>> > > > >> > > met_help at ucar.edu> wrote:
>>>>> > > > >> > >
>>>>> > > > >> > >> Ed,
>>>>> > > > >> > >>
>>>>> > > > >> > >> It's really difficult for me to see an obvious
problem
>>>>> that's
>>>>> > > > causing
>>>>> > > > >> > the
>>>>> > > > >> > >> behavior you describe.
>>>>> > > > >> > >>
>>>>> > > > >> > >> Unfortunately, the red and green highlighting you
>>>>> describe
>>>>> > > doesn't
>>>>> > > > >> show
>>>>> > > > >> > up
>>>>> > > > >> > >> in the Gmail web client. There must be some mail
system
>>>>> > > > >> incompatibility
>>>>> > > > >> > >> there.
>>>>> > > > >> > >>
>>>>> > > > >> > >> Looking at your bsub script:
>>>>> > > > >> > >>
>>>>> > > > >> > >>
>>>>> > > > >> >
>>>>> > > > >>
>>>>> > > >
>>>>> > >
>>>>> >
>>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
>>>>> > > > >> > >>
>>>>> > > > >> > >> I am struck by the number of individual calls
you're
>>>>> making to
>>>>> > > > >> > METviewer:
>>>>> > > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2
plot
>>>>> types =
>>>>> > 656
>>>>> > > > >> calls
>>>>> > > > >> > to
>>>>> > > > >> > >> METviewer
>>>>> > > > >> > >>
>>>>> > > > >> > >> When making plots via the METviewer GUI, it's true
that
>>>>> 1 XML
>>>>> > = 1
>>>>> > > > >> plot.
>>>>> > > > >> > >> However, when making plots via the batch engine,
the
>>>>> plot XML
>>>>> > can
>>>>> > > > be
>>>>> > > > >> set
>>>>> > > > >> > >> up
>>>>> > > > >> > >> to create many, many plots. While I have no direct
proof
>>>>> of
>>>>> > this,
>>>>> > > > it
>>>>> > > > >> > might
>>>>> > > > >> > >> be the case that executing 600+ batch jobs for
METviewer
>>>>> is
>>>>> > > leading
>>>>> > > > >> to
>>>>> > > > >> > >> trouble on the system. One thing to try would be
setting
>>>>> up the
>>>>> > > XML
>>>>> > > > >> to
>>>>> > > > >> > >> make
>>>>> > > > >> > >> many, many more plots in each call. Then check to
see if
>>>>> > running
>>>>> > > > >> > METviewer
>>>>> > > > >> > >> in this way is more stable.
>>>>> > > > >> > >>
>>>>> > > > >> > >> I don't think I fully appreciate all of the logic
that
>>>>> lives in
>>>>> > > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
>>>>> > > > >> test_Time_Series_AGG.sh.
>>>>> > > > >> > >> However, it *might* be possible to construct a
single
>>>>> XML file
>>>>> > > > which
>>>>> > > > >> > >> produces all 656 plots.
>>>>> > > > >> > >>
>>>>> > > > >> > >> Let me know what you think and how you'd like to
proceed.
>>>>> > > > >> > >>
>>>>> > > > >> > >> Thanks,
>>>>> > > > >> > >> John
>>>>> > > > >> > >>
>>>>> > > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach -
NOAA
>>>>> Affiliate
>>>>> > > via
>>>>> > > > >> RT <
>>>>> > > > >> > >> met_help at ucar.edu> wrote:
>>>>> > > > >> > >>
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > <URL:
>>>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>>>> > > >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > Hi John,
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > No problem.
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > I'm including my responses below:
>>>>> > > > >> > >> > Sorry for the delay. I took a look on wcoss
today. You
>>>>> > describe
>>>>> > > > >> that
>>>>> > > > >> > >> your
>>>>> > > > >> > >> > plotting script has stopped working but you're
not
>>>>> sure why.
>>>>> > > > >> > >> > It seems more intermittent. Take a look at the
example
>>>>> > below.
>>>>> > > > You
>>>>> > > > >> > can
>>>>> > > > >> > >> see
>>>>> > > > >> > >> > that for some reason there are gaps in the plots
being
>>>>> > > generated.
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105 Oct
4
>>>>> 14:52
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 41781 Oct 5 15:15
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 56691 Oct 19 15:02
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 55548 Oct 20 16:08
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 55598 Oct 21 16:08
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 55599 Oct 22 16:09
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 55352 Oct 24 16:34
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 26175 Oct 31 08:20
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 26178 Nov 1 08:20
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 26160 Nov 3 08:22
>>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 25354 Nov 3 08:22
>>>>> > > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
>>>>> > > > >> > >> > You can tell when a plot is successful by
looking at
>>>>> the file
>>>>> > > > >> size. I
>>>>> > > > >> > >> > highlighted an example of that in red. However,
the
>>>>> files
>>>>> > > > showing
>>>>> > > > >> > about
>>>>> > > > >> > >> > 25000 are the ones where no line plots are being
>>>>> produced.
>>>>> > You
>>>>> > > > can
>>>>> > > > >> > also
>>>>> > > > >> > >> > see that stat files are generated everyday, so
as long
>>>>> as I'm
>>>>> > > > >> loading
>>>>> > > > >> > >> the
>>>>> > > > >> > >> > data, which I am, I should be able to use the
xml
>>>>> files to
>>>>> > > > generate
>>>>> > > > >> > the
>>>>> > > > >> > >> > plots. I'm attaching a directory list from
wcoss that
>>>>> shows
>>>>> > > the
>>>>> > > > >> dates
>>>>> > > > >> > >> > where data is produced.
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > *20190801 20190817 20200624 20200710
20200726
>>>>> 20200812
>>>>> > > > >> 20200828
>>>>> > > > >> > >> > 20201003 2020101920190802 20190818 20200625
>>>>> 20200711
>>>>> > > > 20200727
>>>>> > > > >> > >> > 20200813 20200829 20201004 2020102020190803
>>>>> 20190819
>>>>> > > > 20200626
>>>>> > > > >> > >> > 20200712 20200728 20200814 20200830
20201005
>>>>> > > > 2020102120190804
>>>>> > > > >> > >> > 20190820 20200627 20200713 20200729
20200815
>>>>> 20200831
>>>>> > > > >> 20201006
>>>>> > > > >> > >> > 2020102220190805 20190821 20200628 20200714
>>>>> 20200730
>>>>> > > > 20200816
>>>>> > > > >> > >> > 20200921 20201007 2020102320190806 20190822
>>>>> 20200629
>>>>> > > > 20200715
>>>>> > > > >> > >> > 20200731 20200817 20200922 20201008
>>>>> 2020102420190807
>>>>> > > > 20190823
>>>>> > > > >> > >> > 20200630 20200716 20200801 20200818
20200923
>>>>> 20201009
>>>>> > > > >> > >> > 2020102520190808 20190824 20200701 20200717
>>>>> 20200802
>>>>> > > > 20200819
>>>>> > > > >> > >> > 20200924 20201010 2020102620190809 20190825
>>>>> 20200702
>>>>> > > > 20200718
>>>>> > > > >> > >> > 20200803 20200820 20200925 20201011
>>>>> 2020102720190810
>>>>> > > > 20190826
>>>>> > > > >> > >> > 20200703 20200719 20200804 20200821
20200926
>>>>> 20201012
>>>>> > > > >> > >> > 2020102820190811 20190827 20200704 20200720
>>>>> 20200805
>>>>> > > > 20200822
>>>>> > > > >> > >> > 20200927 20201013 2020102920190812 20190828
>>>>> 20200705
>>>>> > > > 20200721
>>>>> > > > >> > >> > 20200806 20200823 20200928 20201014
>>>>> 2020103020190813
>>>>> > > > 20190829
>>>>> > > > >> > >> > 20200706 20200722 20200807 20200824
20200929
>>>>> 20201015
>>>>> > > > >> > >> > 2020103120190814 20190830 20200707 20200723
>>>>> 20200808
>>>>> > > > 20200825
>>>>> > > > >> > >> > 20200930 20201016 2020110120190815 20200622
>>>>> 20200708
>>>>> > > > 20200724
>>>>> > > > >> > >> > 20200810 20200826 20201001 20201017
>>>>> 2020110220190816
>>>>> > > > 20200623
>>>>> > > > >> > >> > 20200709 20200725 20200811 20200827
20201002
>>>>> 20201018*
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > Furthermore, what's interesting is that I have
mixed
>>>>> success
>>>>> > > with
>>>>> > > > >> > other
>>>>> > > > >> > >> > cases. Using the same field - PMTF - for the
same
>>>>> cycle -
>>>>> > the
>>>>> > > > 06z
>>>>> > > > >> > >> cycle -
>>>>> > > > >> > >> > you can see that some of the dates where data
was
>>>>> plotted for
>>>>> > > > EAST
>>>>> > > > >> did
>>>>> > > > >> > >> not
>>>>> > > > >> > >> > necessarily plot for FULL. This surprises me
because
>>>>> EAST
>>>>> > is a
>>>>> > > > >> subset
>>>>> > > > >> > >> of
>>>>> > > > >> > >> > FULL. See below:
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647 Oct
4
>>>>> 14:54
>>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 49039 Oct 5 15:17
>>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 56791 Oct 20 16:09
>>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 56834 Oct 21 16:09
>>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 56921 Oct 22 16:11
>>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 56518 Oct 24 16:36
>>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 61187 Oct 31 08:21
>>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 26475 Nov 2 08:22
>>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-
r--r-- 1
>>>>> > > > >> Edward.Strobach
>>>>> > > > >> > >> > emcmodel 26455 Nov 3 08:23
>>>>> > > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > What I find equally concerning as that sometimes
plots
>>>>> are
>>>>> > > > >> incorrectly
>>>>> > > > >> > >> > stored in the wrong directory list as
highlighted in
>>>>> green
>>>>> > > above.
>>>>> > > > >> I've
>>>>> > > > >> > >> > consulted the plot xml file for PMTF, for
instance,
>>>>> and find
>>>>> > > > >> nothing
>>>>> > > > >> > >> > problematic. The only setting that might pose
an
>>>>> issue when
>>>>> > it
>>>>> > > > >> comes
>>>>> > > > >> > to
>>>>> > > > >> > >> > generating line plots is when I set event_equal
to
>>>>> true.
>>>>> > > > However,
>>>>> > > > >> all
>>>>> > > > >> > >> the
>>>>> > > > >> > >> > stat files are populated for all experiments. In
the
>>>>> case of
>>>>> > > > ozone
>>>>> > > > >> you
>>>>> > > > >> > >> see
>>>>> > > > >> > >> > something different. I find that ozone tends to
result
>>>>> in
>>>>> > > > populated
>>>>> > > > >> > >> plots
>>>>> > > > >> > >> > but with intermittent reporting different than
PMTF -
>>>>> see
>>>>> > > below:
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal
>>>>> > 6144-rw-r--r-- 1
>>>>> > > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 54318 Oct 6 10:51
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 54305 Oct 7 10:49
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 62179 Oct 13 13:25
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 63811 Oct 19 12:18
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 60112 Oct 20 12:21
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 60165 Oct 21 12:33
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 60160 Oct 22 12:21
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 60056 Oct 23 12:40
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 58517 Oct 24 13:31
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 58062 Oct 31 14:17
>>>>> > > > >> > >> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-
r--r-- 1
>>>>> > > > >> > Edward.Strobach
>>>>> > > > >> > >> > emcmodel 58119 Nov 1 14:19
>>>>> > > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > Similar issues have been found for other fields,
>>>>> cycles, and
>>>>> > > > >> regions
>>>>> > > > >> > >> > including verification being done for
meteorology.
>>>>> > > > >> > >> > I can't seem to reason this intermittency and
>>>>> inconsistency
>>>>> > > > between
>>>>> > > > >> > >> > reporting since the xml files are correct and
I'm
>>>>> using the
>>>>> > > same
>>>>> > > > >> > >> procedure
>>>>> > > > >> > >> > that I've always used. I do realize that I need
to
>>>>> start
>>>>> > > > creating
>>>>> > > > >> a
>>>>> > > > >> > >> better
>>>>> > > > >> > >> > way to manage all these files that are being
produced,
>>>>> both
>>>>> > in
>>>>> > > > >> terms
>>>>> > > > >> > of
>>>>> > > > >> > >> > data/plots and xml files.
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > I ran across a rather large directory:
>>>>> > > > >> > >> > ls
>>>>> > > > >>
>>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>>>>> > > > >> > |
>>>>> > > > >> > >> wc
>>>>> > > > >> > >> > -w
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > That has 71,398 xml files in there. But I see
>>>>> timestamps in
>>>>> > > those
>>>>> > > > >> > >> filenames
>>>>> > > > >> > >> > from 20200921 to 20200925.
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > Is it the case that this stopped working on
20200925?
>>>>> Or are
>>>>> > > > those
>>>>> > > > >> > dated
>>>>> > > > >> > >> > filenames remnants from older work?
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > So it hasn't been working for over a month. Is
that
>>>>> right?
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > There are a lot of files and I need to begin
managing
>>>>> that
>>>>> > > > >> better. I
>>>>> > > > >> > >> think
>>>>> > > > >> > >> > during that time there was a machine switch so
it
>>>>> stopped on
>>>>> > > that
>>>>> > > > >> day.
>>>>> > > > >> > >> > I've been able to produce plots even now, but
it's not
>>>>> > > consistent
>>>>> > > > >> in
>>>>> > > > >> > >> > reporting while some regions generate plots and
others
>>>>> do
>>>>> > not.
>>>>> > > > >> I've
>>>>> > > > >> > >> > reduced the level processing in hopes that I
don't hit
>>>>> a walk
>>>>> > > > clock
>>>>> > > > >> > >> issue
>>>>> > > > >> > >> > since I'm submitting batch jobs. However, I've
run
>>>>> this step
>>>>> > > > >> outside
>>>>> > > > >> > of
>>>>> > > > >> > >> > batch and it always seems to work. This issue
seems to
>>>>> happen
>>>>> > > > with
>>>>> > > > >> > batch
>>>>> > > > >> > >> > and it is not at all clear why. Let me know
what else
>>>>> you
>>>>> > need
>>>>> > > > >> from
>>>>> > > > >> > me
>>>>> > > > >> > >> to
>>>>> > > > >> > >> > better coordinate what I would need to do next.
>>>>> Thanks.
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley
Gotway via
>>>>> RT <
>>>>> > > > >> > >> > met_help at ucar.edu>
>>>>> > > > >> > >> > wrote:
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > > Hi Ed,
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > > Sorry for the delay. I took a look on wcoss
today.
>>>>> You
>>>>> > > describe
>>>>> > > > >> that
>>>>> > > > >> > >> your
>>>>> > > > >> > >> > > plotting script has stopped working but you're
not
>>>>> sure
>>>>> > why.
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > > I ran across a rather large directory:
>>>>> > > > >> > >> > > ls
>>>>> > > > >> >
>>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>>>>> > > > >> > >> |
>>>>> > > > >> > >> > wc
>>>>> > > > >> > >> > > -w
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > > That has 71,398 xml files in there. But I see
>>>>> timestamps in
>>>>> > > > those
>>>>> > > > >> > >> > filenames
>>>>> > > > >> > >> > > from 20200921 to 20200925.
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > > Is it the case that this stopped working on
>>>>> 20200925? Or
>>>>> > are
>>>>> > > > >> those
>>>>> > > > >> > >> dated
>>>>> > > > >> > >> > > filenames remnants from older work?
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > > So it hasn't been working for over a month. Is
that
>>>>> right?
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > > John
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System
itself
>>>>> via RT
>>>>> > <
>>>>> > > > >> > >> > > met_help at ucar.edu> wrote:
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > > >
>>>>> > > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283 was
acted
>>>>> upon.
>>>>> > > > >> > >> > > > Transaction: Given to johnhg (John Halley
Gotway)
>>>>> by
>>>>> > > > RT_System
>>>>> > > > >> > >> > > > Queue: met_help
>>>>> > > > >> > >> > > > Subject: plots no longer being
generated
>>>>> > > > >> > >> > > > Owner: johnhg
>>>>> > > > >> > >> > > > Requestors: edward.strobach at noaa.gov
>>>>> > > > >> > >> > > > Status: new
>>>>> > > > >> > >> > > > Ticket <URL:
>>>>> > > > >> > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > > >
>>>>> > > > >> > >> > > >
>>>>> > > > >> > >> > > > This transaction appears to have no content
>>>>> > > > >> > >> > > >
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> > >
>>>>> > > > >> > >> >
>>>>> > > > >> > >> > --
>>>>> > > > >> > >> > Edward Strobach
>>>>> > > > >> > >> > EMC/NCEP/NWS/
>>>>> > > > >> > >> > IMSG Contractor
>>>>> > > > >> > >> > Cubicle#: 2029
>>>>> > > > >> > >> > 301-683-3717
>>>>> > > > >> > >> >
>>>>> > > > >> > >> >
>>>>> > > > >> > >>
>>>>> > > > >> > >>
>>>>> > > > >> > >
>>>>> > > > >> > > --
>>>>> > > > >> > > Edward Strobach
>>>>> > > > >> > > EMC/NCEP/NWS/
>>>>> > > > >> > > IMSG Contractor
>>>>> > > > >> > > Cubicle#: 2029
>>>>> > > > >> > > 301-683-3717
>>>>> > > > >> > >
>>>>> > > > >> >
>>>>> > > > >> >
>>>>> > > > >> > --
>>>>> > > > >> > Edward Strobach
>>>>> > > > >> > EMC/NCEP/NWS/
>>>>> > > > >> > IMSG Contractor
>>>>> > > > >> > Cubicle#: 2029
>>>>> > > > >> > 301-683-3717
>>>>> > > > >> >
>>>>> > > > >> >
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >
>>>>> > > > > --
>>>>> > > > > Edward Strobach
>>>>> > > > > EMC/NCEP/NWS/
>>>>> > > > > IMSG Contractor
>>>>> > > > > Cubicle#: 2029
>>>>> > > > > 301-683-3717
>>>>> > > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > --
>>>>> > > > Edward Strobach
>>>>> > > > EMC/NCEP/NWS/
>>>>> > > > IMSG Contractor
>>>>> > > > Cubicle#: 2029
>>>>> > > > 301-683-3717
>>>>> > > >
>>>>> > > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>> > --
>>>>> > Edward Strobach
>>>>> > EMC/NCEP/NWS/
>>>>> > IMSG Contractor
>>>>> > Cubicle#: 2029
>>>>> > 301-683-3717
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>
>>>> --
>>>> Edward Strobach
>>>> EMC/NCEP/NWS/
>>>> IMSG Contractor
>>>> Cubicle#: 2029
>>>> 301-683-3717
>>>>
>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: John Halley Gotway
Time: Mon Nov 09 10:47:02 2020
Ed,
Yes, there's lots of moving parts here.
(1) Setting up the METviewer XML to run 1 or more plots.
(2) Running that XML remotely to generate plots.
(3) Transferring the resulting image files back.
I just wanted to make sure you know about the options for (1). And I'm
really glad to hear that you were able to see how to generate multiple
plots from a single METviewer XML!
I'd say the rest of the details about managing the plotting/image
transfer
logic is really up to you and your colleagues. If I were setting this
up, I
would consider having each batch job write to a single output
directory,
and then recursively transfer back that directory. That seems easier
than
dealing with lots of individual files. But it's really to you guys.
There is another detail about the METviewer plot XML that we haven't
talked
about yet, and that's plot "inheritance". In a single XML, you can
define
multiple <plot> tags and have one <plot> inherit settings from a
previous
one. I took this to the extreme for one project, setting up a pretty
complex XML that made like 6 different plot types using multiple
levels of
inheritance. In general, I'd advise against that. While it's cool that
it
works, it makes for complex logic that's difficult to maintain.
Instead,
I'd just look at any "for" loops you have in your plotting script and
move
as many as are easy and feasible into the XML directly... without
going
crazy! That would be like init hours, masking regions, and perhaps
statistic types. That should greatly reduce the number of individual
calls
to mv_batch.sh and hopefully provide for more stable results.
But if you ever want to learn about plot inheritance, I'm happy to
help you
with that too.
John
On Mon, Nov 9, 2020 at 8:02 AM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>
> FYI, when running test_v2.sh again after running it the first time
only one
> plot was copied over:
> PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> 100%
57KB
> 19.9MB/s 00:00
>
> On Mon, Nov 9, 2020 at 10:00 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > I did run some additional tests outside the main test*sh script.
> > Specifically I ran the following line several times to see what
plots
> would
> > be copied over to WCOSS:
> > bash
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> > edward.strobach
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls/Test.xml
> >
> > The first time around 3 plots were copied over:
> > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
> > 100% 51KB 18.6MB/s 00:00
> > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> > 100% 58KB 12.0MB/s 00:00
> > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> > 100% 57KB 2.8MB/s 00:00
> >
> > The second time 2 plots were copied over:
> > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> > 100% 58KB 2.6MB/s 00:00
> > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> > 100% 57KB 6.2MB/s 00:00
> >
> > The third time 3 plots were copied over:
> > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
> > 100%
51KB
> > 2.4MB/s 00:00
> > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> > 100%
58KB
> > 6.1MB/s 00:00
> > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> > 100%
57KB
> > 11.1MB/s 00:00
> >
> > The order of init_hour and vx_mask is as follows:
> > <06>
> > <12>
> > and
> > <Mixed_Forest>
> > <FULL>
> > <EAST>
> > <WEST>
> >
> > It does appear that the latest plots are copied over in the
process (i.e.
> > if two, then the last two; if three, then the last three).
> > I don't really have a feel for why two files are copied over for
some
> > instances and 3 for other instances. Lastly, I ran test_v2.sh
again, and
> > this time it copied over 4 plots. See below:
> >
> > PMTF_EAST_112020_2020-11-01_2020-11-06_t06z.png
> > 100%
53KB
> > 12.7MB/s 00:00
> > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
> > 100%
51KB
> > 9.7MB/s 00:00
> > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> > 100%
58KB
> > 11.3MB/s 00:00
> > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> > 100%
57KB
> > 11.4MB/s 00:00
> >
> > This is very bizarre to me.
> >
> >
> >
> > On Mon, Nov 9, 2020 at 9:26 AM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> >> Actually, I don't think I can do that. FBAR and OBAR are split
and
> >> adding RMSE as a third line might confuse the process.
> >>
> >> I was able to run a test with mixed success. I was successful at
> >> generating 8 plots (4 regions for 2 initial hours). The regions
were
> >> Mixed_Forest, FULL, EAST, and WEST while the initial hours were
06 and
> 12.
> >> The test ran well with indications that all plots were generated
and
> stored
> >> here: Created plot
> >>
> /data/mv_data/edward.strobach/plots/PMTF_Mixed_Forest_112020_2020-
11-01_2020-11-06_t06z.png
> >>
> >> However, I was not successful at saving the plots on Dell. Only
the last
> >> plot was saved here:
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011.
> >>
> >> It seems that in my bash script the line that produces the plots
that
> are
> >> then stored in the above directory must be done in a loop
otherwise it
> will
> >> only move the last plot over as suggested by the fact that only
the plot
> >> featuring WEST at 12z was moved over - the last plot of the
series of
> plots
> >> generated. I suppose I could create logic that extracts the
> region/initial
> >> hour information and construct a two part loop to move over all
plots?
> >>
> >> On Mon, Nov 9, 2020 at 8:19 AM Edward Strobach - NOAA Affiliate <
> >> edward.strobach at noaa.gov> wrote:
> >>
> >>> Just a follow-up on one other item..
> >>>
> >>> Can also list multiple stat types in the same way that I specify
the
> >>> init_hour and vx_mask? In other words, can I do the following:
> >>>
> >>> <dep1>
> >>> <fcst_var name="PMTF">
> >>> <stat>FBAR_OBAR</stat>
> >>> <stat>RMSE</stat>
> >>> </fcst_var>
> >>> </dep1>
> >>>
> >>> I know that the "set" block is not being used here.
> >>>
> >>> On Mon, Nov 9, 2020 at 7:41 AM Edward Strobach - NOAA Affiliate
<
> >>> edward.strobach at noaa.gov> wrote:
> >>>
> >>>> Thanks John. Let me try that and I'll let you know how it
goes.
> >>>>
> >>>> On Fri, Nov 6, 2020 at 10:53 AM John Halley Gotway via RT <
> >>>> met_help at ucar.edu> wrote:
> >>>>
> >>>>> Ed,
> >>>>>
> >>>>> You're right about the METviewer GUI grouping multiple
selections
> >>>>> together.
> >>>>> That looks like this in the XML using the <set> tag:
> >>>>> <field equalize="false" name="vx_mask">
> >>>>> <set name="vx_mask_0">
> >>>>> <val>FULL</val>
> >>>>> </set>
> >>>>> </field>
> >>>>>
> >>>>> But in the updates I sent, I remove the <set> tag:
> >>>>> <field equalize="false" name="vx_mask">
> >>>>> <val>FULL</val>
> >>>>> <val>CONUS</val>
> >>>>> <val>EAST</val>
> >>>>> <val>WEST</val>
> >>>>> </field>
> >>>>> So instead of processing this as 1 group of 4 values, it'll
process
> >>>>> the 4
> >>>>> values separately.
> >>>>>
> >>>>> The current vx_mask value is referenced using {vx_mask}
syntax. And I
> >>>>> included examples of using that in the output file names and
titles
> in
> >>>>> my
> >>>>> previous email:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
> >>>>>
> >>>>>
> >>>>>
>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
> >>>>>
> >>>>>
>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
> >>>>>
> >>>>> <title>Average PMTF (October 2020; {init_hour}z cycle;
{vx_mask}
> >>>>> Domain)</title>
> >>>>>
> >>>>> John
> >>>>>
> >>>>> On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA
Affiliate via
> RT
> >>>>> <
> >>>>> met_help at ucar.edu> wrote:
> >>>>>
> >>>>> >
> >>>>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >>>>> >
> >>>>> > Hi John,
> >>>>> >
> >>>>> > This definitely helps. I never thought about it this way
because I
> >>>>> > assumed, based on my experience with the interactive MET
GUI, that
> I
> >>>>> could
> >>>>> > only generate a single plot regardless of how I specified my
> >>>>> settings in
> >>>>> > the XML. For example, if I listed EAST and WEST under
vx_mask,
> then
> >>>>> my
> >>>>> > thinking would be that both those regions would be grouped
and a
> >>>>> single
> >>>>> > plot created. I guess I was wrong on that. The concern I
still
> >>>>> have,
> >>>>> > however, is that the plot title and plot name - the png file
- is
> >>>>> also set
> >>>>> > in the XML file. Even after adding the list you gave, I
can't see
> >>>>> how
> >>>>> > unique file names will be created. Is that true?
> >>>>> >
> >>>>> > On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT <
> >>>>> > met_help at ucar.edu>
> >>>>> > wrote:
> >>>>> >
> >>>>> > > Ed,
> >>>>> > >
> >>>>> > > Thanks for sending examples. My goal was to use
METviewer's XML
> >>>>> upload
> >>>>> > > functionality but I'm not able to upload a pdf version of
the xml
> >>>>> file.
> >>>>> > >
> >>>>> > > I did spend some time trying to replicate this exact plot
with
> >>>>> METviewer
> >>>>> > > at:
> >>>>> > > https://metviewer.nws.noaa.gov/metviewer1.jsp
> >>>>> > >
> >>>>> > > But I was never quite successful. So I made just a similar
> version
> >>>>> > instead.
> >>>>> > > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
> >>>>> > >
> >>>>> > > This plots FBAR and OBAR for 6 different models for the
month of
> >>>>> October
> >>>>> > > for all 06Z initializations over the FULL model domain.
> >>>>> > >
> >>>>> > > I wanted to show you how to modify this for batch mode...
and
> then
> >>>>> adapt
> >>>>> > it
> >>>>> > > to make multiple plots. But I'm not able to make batch
plots on
> >>>>> wcoss
> >>>>> > using
> >>>>> > > the AWS instance. I don't think I have the right
permissions to
> do
> >>>>> so.
> >>>>> > >
> >>>>> > > So let me just say this. Generally, you just update the
list of
> >>>>> items in
> >>>>> > > plot_fix section.
> >>>>> > >
> >>>>> > > Original from METviewer with 1 init hour and 1 mask:
> >>>>> > >
> >>>>> > > <field equalize="false" name="vx_mask">
> >>>>> > > <set name="vx_mask_0">
> >>>>> > > <val>FULL</val>
> >>>>> > > </set>
> >>>>> > > </field>
> >>>>> > > <field equalize="false" name="init_hour">
> >>>>> > > <set name="init_hour_1">
> >>>>> > > <val>06</val>
> >>>>> > > </set>
> >>>>> > > </field>
> >>>>> > >
> >>>>> > > Change to 2 init hours and 4 masks:
> >>>>> > >
> >>>>> > > <field equalize="false" name="vx_mask">
> >>>>> > > <val>FULL</val>
> >>>>> > > <val>CONUS</val>
> >>>>> > > <val>EAST</val>
> >>>>> > > <val>WEST</val>
> >>>>> > > </field>
> >>>>> > > <field equalize="false" name="init_hour">
> >>>>> > > <val>06</val>
> >>>>> > > <val>12</val>
> >>>>> > > </field>
> >>>>> > >
> >>>>> > > And then reference those values in the naming of the
titles and
> >>>>> output
> >>>>> > > files:
> >>>>> > >
> >>>>> > >
> >>>>> > >
> >>>>> > >
> >>>>> >
> >>>>>
>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
> >>>>> > >
> >>>>> > >
> >>>>> > >
> >>>>> >
> >>>>>
>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
> >>>>> > >
> >>>>> > >
> >>>>>
>
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
> >>>>> > >
> >>>>> > > <title>Average PMTF (October 2020; {init_hour}z cycle;
{vx_mask}
> >>>>> > > Domain)</title>
> >>>>> > >
> >>>>> > > So that's the general idea. And when you submit this to
mv_batch,
> >>>>> it'll
> >>>>> > > make 8 plots instead of 1.
> >>>>> > > And you can see how you'd easily expand this to many
plots.
> >>>>> > >
> >>>>> > > But I worry that this may not be sufficient information to
get
> you
> >>>>> going.
> >>>>> > >
> >>>>> > > John
> >>>>> > >
> >>>>> > > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA
Affiliate
> >>>>> via RT <
> >>>>> > > met_help at ucar.edu> wrote:
> >>>>> > >
> >>>>> > > >
> >>>>> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> >
> >>>>> > > >
> >>>>> > > > Hi John,
> >>>>> > > >
> >>>>> > > > I think I was able to cover your request okay. Below
are 4
> >>>>> attachments
> >>>>> > > > that include the test script that was used to generate
the
> >>>>> plots, a
> >>>>> > > sample
> >>>>> > > > XML file, and the two plots that were generated using
the test
> >>>>> run
> >>>>> > > script.
> >>>>> > > > These plots were not created previously using the batch
job but
> >>>>> uses
> >>>>> > the
> >>>>> > > > same exact logic as the original run script. Thanks
again for
> >>>>> working
> >>>>> > > with
> >>>>> > > > me on this. Please let me know if you need anything
else or
> >>>>> would like
> >>>>> > > me
> >>>>> > > > to test something.
> >>>>> > > >
> >>>>> > > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA
> Affiliate <
> >>>>> > > > edward.strobach at noaa.gov> wrote:
> >>>>> > > >
> >>>>> > > > > Ed,
> >>>>> > > > >
> >>>>> > > > > Image files showing up in an unexpected output
directory
> >>>>> likely point
> >>>>> > > to
> >>>>> > > > a
> >>>>> > > > > problem in the logic of the processing scripts. It's
pretty
> >>>>> unlikely
> >>>>> > > that
> >>>>> > > > > this behavior is caused by the METviewer code itself.
But
> I've
> >>>>> > > definitely
> >>>>> > > > > been surprised before!
> >>>>> > > > >
> >>>>> > > > > I tested the logic outside of a batch job and it works
> fine. I
> >>>>> > suspect
> >>>>> > > > > that the issuance of another batch job while the other
one is
> >>>>> getting
> >>>>> > > > > processed is causing some confusion in the processing.
These
> >>>>> issues
> >>>>> > > > don't
> >>>>> > > > > happen everyday and appear more sporadic than
anything.
> >>>>> > > > >
> >>>>> > > > > So what we really need here is an example of taking a
plot
> >>>>> from the
> >>>>> > > > > METviewer GUI and adapting it to make it generate
multiple
> >>>>> plots. I
> >>>>> > did
> >>>>> > > > go
> >>>>> > > > > through examples of doing this in previous tutorials,
but we
> >>>>> didn't
> >>>>> > > start
> >>>>> > > > > recording them until last summer. I checked the July
2019 NRL
> >>>>> > tutorial
> >>>>> > > > > videos and don't see it there.
> >>>>> > > > >
> >>>>> > > > > Is there something you need from me to help on this?
Just
> let
> >>>>> me
> >>>>> > know
> >>>>> > > > >
> >>>>> > > > > We've begun recording training videos for common
topics like
> >>>>> this.
> >>>>> > But
> >>>>> > > we
> >>>>> > > > > only have one for METviewer so far:
> >>>>> > > > >
> >>>>> > >
> >>>>>
> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> >>>>> > > > >
> >>>>> > > > > But this definitely seems like a good candidate for a
new
> >>>>> training
> >>>>> > > video.
> >>>>> > > > >
> >>>>> > > > > For now though, how about this...
> >>>>> > > > > - You use METviewer to manually make a single plot,
> formatting
> >>>>> it how
> >>>>> > > > you'd
> >>>>> > > > > like.
> >>>>> > > > > - You send me your sample plot and corresponding XML
file.
> >>>>> > > > > - I'll tweak it to generate many plots via batch
instead of
> >>>>> just one.
> >>>>> > > > > - And then you can continue tweaking to add more
> permutations.
> >>>>> > > > >
> >>>>> > > > > Okay, I'll work on that now. Thanks.
> >>>>> > > > >
> >>>>> > > > > John
> >>>>> > > > >
> >>>>> > > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway via
RT <
> >>>>> > > > > met_help at ucar.edu> wrote:
> >>>>> > > > >
> >>>>> > > > >> Ed,
> >>>>> > > > >>
> >>>>> > > > >> Image files showing up in an unexpected output
directory
> >>>>> likely
> >>>>> > point
> >>>>> > > > to a
> >>>>> > > > >> problem in the logic of the processing scripts. It's
pretty
> >>>>> unlikely
> >>>>> > > > that
> >>>>> > > > >> this behavior is caused by the METviewer code itself.
But
> I've
> >>>>> > > > definitely
> >>>>> > > > >> been surprised before!
> >>>>> > > > >>
> >>>>> > > > >> So what we really need here is an example of taking a
plot
> >>>>> from the
> >>>>> > > > >> METviewer GUI and adapting it to make it generate
multiple
> >>>>> plots. I
> >>>>> > > did
> >>>>> > > > go
> >>>>> > > > >> through examples of doing this in previous tutorials,
but we
> >>>>> didn't
> >>>>> > > > start
> >>>>> > > > >> recording them until last summer. I checked the July
2019
> NRL
> >>>>> > tutorial
> >>>>> > > > >> videos and don't see it there.
> >>>>> > > > >>
> >>>>> > > > >> We've begun recording training videos for common
topics like
> >>>>> this.
> >>>>> > But
> >>>>> > > > we
> >>>>> > > > >> only have one for METviewer so far:
> >>>>> > > > >>
> >>>>> > > >
> >>>>> >
> >>>>>
> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> >>>>> > > > >>
> >>>>> > > > >> But this definitely seems like a good candidate for a
new
> >>>>> training
> >>>>> > > > video.
> >>>>> > > > >>
> >>>>> > > > >> For now though, how about this...
> >>>>> > > > >> - You use METviewer to manually make a single plot,
> >>>>> formatting it
> >>>>> > how
> >>>>> > > > >> you'd
> >>>>> > > > >> like.
> >>>>> > > > >> - You send me your sample plot and corresponding XML
file.
> >>>>> > > > >> - I'll tweak it to generate many plots via batch
instead of
> >>>>> just
> >>>>> > one.
> >>>>> > > > >> - And then you can continue tweaking to add more
> permutations.
> >>>>> > > > >>
> >>>>> > > > >> John
> >>>>> > > > >>
> >>>>> > > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach - NOAA
> >>>>> Affiliate via
> >>>>> > > RT <
> >>>>> > > > >> met_help at ucar.edu> wrote:
> >>>>> > > > >>
> >>>>> > > > >> >
> >>>>> > > > >> > <URL:
> >>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >>>>> > > > >> >
> >>>>> > > > >> > Oh, and one more thing..
> >>>>> > > > >> >
> >>>>> > > > >> > The green text that didn't show up was meant to
highlight
> >>>>> the fact
> >>>>> > > > that
> >>>>> > > > >> > sometimes plots that shouldn't be stored in certain
> >>>>> directories
> >>>>> > end
> >>>>> > > up
> >>>>> > > > >> > there. The pasted example below shows one instance
where
> >>>>> OZMAX*
> >>>>> > was
> >>>>> > > > >> stored
> >>>>> > > > >> > where TMP* should have been stored.
> >>>>> > > > >> >
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct 5
10:52
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct 6
10:52
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct 7
10:00
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct 13
13:49
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct 14
11:31
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct 19
11:37
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct 20
11:35
> >>>>> > > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct 21
11:45
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct 22
11:42
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct 23
11:50
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct 24
12:57
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct 31
08:47
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov 2
08:50
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
> >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov 3
08:52
> >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
> >>>>> > > > >> >
> >>>>> > > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach -
NOAA
> >>>>> Affiliate <
> >>>>> > > > >> > edward.strobach at noaa.gov> wrote:
> >>>>> > > > >> >
> >>>>> > > > >> > > Hi John,
> >>>>> > > > >> > >
> >>>>> > > > >> > > I know there are a lot of calls and speculate
that this
> >>>>> could be
> >>>>> > > > part
> >>>>> > > > >> of
> >>>>> > > > >> > > the problem. I don't think that I'm following
best
> >>>>> practice by
> >>>>> > > > >> > approaching
> >>>>> > > > >> > > it in this way. Generating plots usually works
well if
> I
> >>>>> > process
> >>>>> > > a
> >>>>> > > > >> > subset
> >>>>> > > > >> > > of the computations outside of batch scripting.
The
> >>>>> python
> >>>>> > script
> >>>>> > > > was
> >>>>> > > > >> > > given to me by Logan, but I have since made
significant
> >>>>> changes
> >>>>> > to
> >>>>> > > > it
> >>>>> > > > >> in
> >>>>> > > > >> > > order to dynamically fill in the xml file based
on
> >>>>> parameters
> >>>>> > set.
> >>>>> > > > >> The
> >>>>> > > > >> > > test*.sh script was also given to me. I created
the
> >>>>> > > > Time_Series*bsub
> >>>>> > > > >> as
> >>>>> > > > >> > a
> >>>>> > > > >> > > way to process a series of plots based on cycle
time,
> >>>>> region,
> >>>>> > > etc. I
> >>>>> > > > >> > think
> >>>>> > > > >> > > the first two scripts are solid and do well for
what I
> >>>>> need to
> >>>>> > get
> >>>>> > > > >> done.
> >>>>> > > > >> > > It's the Time_Series*bsub script that bothers me
a bit.
> >>>>> > > > >> > >
> >>>>> > > > >> > > I can't see how a single XML file can process all
these
> >>>>> plots
> >>>>> > in a
> >>>>> > > > >> single
> >>>>> > > > >> > > setting based on what I've learned thus far. How
would
> >>>>> one go
> >>>>> > > about
> >>>>> > > > >> > that?
> >>>>> > > > >> > > I would be open going in that direction if you
can
> >>>>> provide some
> >>>>> > > > >> guidance.
> >>>>> > > > >> > >
> >>>>> > > > >> > > Unfortunately, I haven't gotten a lot of feedback
or
> >>>>> dialogue
> >>>>> > > > exchange
> >>>>> > > > >> > > about what I've done so far at EMC with this new
set-up.
> >>>>> This
> >>>>> > has
> >>>>> > > > >> also
> >>>>> > > > >> > > been part of the problem. I can't seem to get
them to
> >>>>> focus so
> >>>>> > > that
> >>>>> > > > >> we
> >>>>> > > > >> > can
> >>>>> > > > >> > > prioritize. I know that I'm doing the best I
can. If I
> >>>>> can
> >>>>> > learn
> >>>>> > > > >> how to
> >>>>> > > > >> > > go about this more efficiently it would be of
great
> >>>>> help. I can
> >>>>> > > > then
> >>>>> > > > >> > > expand this to other verification as venture to
other
> >>>>> projects
> >>>>> > in
> >>>>> > > > the
> >>>>> > > > >> > > future. I know you're busy with many things, so
I'm
> >>>>> willing to
> >>>>> > go
> >>>>> > > > >> along
> >>>>> > > > >> > > with your schedule as you find an opening.
> >>>>> > > > >> > >
> >>>>> > > > >> > > Thanks
> >>>>> > > > >> > >
> >>>>> > > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley
Gotway via
> RT
> >>>>> <
> >>>>> > > > >> > > met_help at ucar.edu> wrote:
> >>>>> > > > >> > >
> >>>>> > > > >> > >> Ed,
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> It's really difficult for me to see an obvious
problem
> >>>>> that's
> >>>>> > > > causing
> >>>>> > > > >> > the
> >>>>> > > > >> > >> behavior you describe.
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> Unfortunately, the red and green highlighting
you
> >>>>> describe
> >>>>> > > doesn't
> >>>>> > > > >> show
> >>>>> > > > >> > up
> >>>>> > > > >> > >> in the Gmail web client. There must be some mail
system
> >>>>> > > > >> incompatibility
> >>>>> > > > >> > >> there.
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> Looking at your bsub script:
> >>>>> > > > >> > >>
> >>>>> > > > >> > >>
> >>>>> > > > >> >
> >>>>> > > > >>
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> I am struck by the number of individual calls
you're
> >>>>> making to
> >>>>> > > > >> > METviewer:
> >>>>> > > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x 2
plot
> >>>>> types =
> >>>>> > 656
> >>>>> > > > >> calls
> >>>>> > > > >> > to
> >>>>> > > > >> > >> METviewer
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> When making plots via the METviewer GUI, it's
true that
> >>>>> 1 XML
> >>>>> > = 1
> >>>>> > > > >> plot.
> >>>>> > > > >> > >> However, when making plots via the batch engine,
the
> >>>>> plot XML
> >>>>> > can
> >>>>> > > > be
> >>>>> > > > >> set
> >>>>> > > > >> > >> up
> >>>>> > > > >> > >> to create many, many plots. While I have no
direct
> proof
> >>>>> of
> >>>>> > this,
> >>>>> > > > it
> >>>>> > > > >> > might
> >>>>> > > > >> > >> be the case that executing 600+ batch jobs for
> METviewer
> >>>>> is
> >>>>> > > leading
> >>>>> > > > >> to
> >>>>> > > > >> > >> trouble on the system. One thing to try would be
> setting
> >>>>> up the
> >>>>> > > XML
> >>>>> > > > >> to
> >>>>> > > > >> > >> make
> >>>>> > > > >> > >> many, many more plots in each call. Then check
to see
> if
> >>>>> > running
> >>>>> > > > >> > METviewer
> >>>>> > > > >> > >> in this way is more stable.
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> I don't think I fully appreciate all of the
logic that
> >>>>> lives in
> >>>>> > > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py, and
> >>>>> > > > >> test_Time_Series_AGG.sh.
> >>>>> > > > >> > >> However, it *might* be possible to construct a
single
> >>>>> XML file
> >>>>> > > > which
> >>>>> > > > >> > >> produces all 656 plots.
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> Let me know what you think and how you'd like to
> proceed.
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> Thanks,
> >>>>> > > > >> > >> John
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach -
NOAA
> >>>>> Affiliate
> >>>>> > > via
> >>>>> > > > >> RT <
> >>>>> > > > >> > >> met_help at ucar.edu> wrote:
> >>>>> > > > >> > >>
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > <URL:
> >>>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> >>>>> > > >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > Hi John,
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > No problem.
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > I'm including my responses below:
> >>>>> > > > >> > >> > Sorry for the delay. I took a look on wcoss
today.
> You
> >>>>> > describe
> >>>>> > > > >> that
> >>>>> > > > >> > >> your
> >>>>> > > > >> > >> > plotting script has stopped working but you're
not
> >>>>> sure why.
> >>>>> > > > >> > >> > It seems more intermittent. Take a look at
the
> example
> >>>>> > below.
> >>>>> > > > You
> >>>>> > > > >> > can
> >>>>> > > > >> > >> see
> >>>>> > > > >> > >> > that for some reason there are gaps in the
plots
> being
> >>>>> > > generated.
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105
Oct 4
> >>>>> 14:52
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 41781 Oct 5 15:15
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 56691 Oct 19 15:02
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 55548 Oct 20 16:08
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 55598 Oct 21 16:08
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 55599 Oct 22 16:09
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 55352 Oct 24 16:34
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 26175 Oct 31 08:20
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 26178 Nov 1 08:20
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 26160 Nov 3 08:22
> >>>>> > > > >> > >> > PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 25354 Nov 3 08:22
> >>>>> > > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
> >>>>> > > > >> > >> > You can tell when a plot is successful by
looking at
> >>>>> the file
> >>>>> > > > >> size. I
> >>>>> > > > >> > >> > highlighted an example of that in red.
However, the
> >>>>> files
> >>>>> > > > showing
> >>>>> > > > >> > about
> >>>>> > > > >> > >> > 25000 are the ones where no line plots are
being
> >>>>> produced.
> >>>>> > You
> >>>>> > > > can
> >>>>> > > > >> > also
> >>>>> > > > >> > >> > see that stat files are generated everyday, so
as
> long
> >>>>> as I'm
> >>>>> > > > >> loading
> >>>>> > > > >> > >> the
> >>>>> > > > >> > >> > data, which I am, I should be able to use the
xml
> >>>>> files to
> >>>>> > > > generate
> >>>>> > > > >> > the
> >>>>> > > > >> > >> > plots. I'm attaching a directory list from
wcoss
> that
> >>>>> shows
> >>>>> > > the
> >>>>> > > > >> dates
> >>>>> > > > >> > >> > where data is produced.
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > *20190801 20190817 20200624 20200710
20200726
> >>>>> 20200812
> >>>>> > > > >> 20200828
> >>>>> > > > >> > >> > 20201003 2020101920190802 20190818
20200625
> >>>>> 20200711
> >>>>> > > > 20200727
> >>>>> > > > >> > >> > 20200813 20200829 20201004
2020102020190803
> >>>>> 20190819
> >>>>> > > > 20200626
> >>>>> > > > >> > >> > 20200712 20200728 20200814 20200830
20201005
> >>>>> > > > 2020102120190804
> >>>>> > > > >> > >> > 20190820 20200627 20200713 20200729
20200815
> >>>>> 20200831
> >>>>> > > > >> 20201006
> >>>>> > > > >> > >> > 2020102220190805 20190821 20200628
20200714
> >>>>> 20200730
> >>>>> > > > 20200816
> >>>>> > > > >> > >> > 20200921 20201007 2020102320190806
20190822
> >>>>> 20200629
> >>>>> > > > 20200715
> >>>>> > > > >> > >> > 20200731 20200817 20200922 20201008
> >>>>> 2020102420190807
> >>>>> > > > 20190823
> >>>>> > > > >> > >> > 20200630 20200716 20200801 20200818
20200923
> >>>>> 20201009
> >>>>> > > > >> > >> > 2020102520190808 20190824 20200701
20200717
> >>>>> 20200802
> >>>>> > > > 20200819
> >>>>> > > > >> > >> > 20200924 20201010 2020102620190809
20190825
> >>>>> 20200702
> >>>>> > > > 20200718
> >>>>> > > > >> > >> > 20200803 20200820 20200925 20201011
> >>>>> 2020102720190810
> >>>>> > > > 20190826
> >>>>> > > > >> > >> > 20200703 20200719 20200804 20200821
20200926
> >>>>> 20201012
> >>>>> > > > >> > >> > 2020102820190811 20190827 20200704
20200720
> >>>>> 20200805
> >>>>> > > > 20200822
> >>>>> > > > >> > >> > 20200927 20201013 2020102920190812
20190828
> >>>>> 20200705
> >>>>> > > > 20200721
> >>>>> > > > >> > >> > 20200806 20200823 20200928 20201014
> >>>>> 2020103020190813
> >>>>> > > > 20190829
> >>>>> > > > >> > >> > 20200706 20200722 20200807 20200824
20200929
> >>>>> 20201015
> >>>>> > > > >> > >> > 2020103120190814 20190830 20200707
20200723
> >>>>> 20200808
> >>>>> > > > 20200825
> >>>>> > > > >> > >> > 20200930 20201016 2020110120190815
20200622
> >>>>> 20200708
> >>>>> > > > 20200724
> >>>>> > > > >> > >> > 20200810 20200826 20201001 20201017
> >>>>> 2020110220190816
> >>>>> > > > 20200623
> >>>>> > > > >> > >> > 20200709 20200725 20200811 20200827
20201002
> >>>>> 20201018*
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > Furthermore, what's interesting is that I have
mixed
> >>>>> success
> >>>>> > > with
> >>>>> > > > >> > other
> >>>>> > > > >> > >> > cases. Using the same field - PMTF - for the
same
> >>>>> cycle -
> >>>>> > the
> >>>>> > > > 06z
> >>>>> > > > >> > >> cycle -
> >>>>> > > > >> > >> > you can see that some of the dates where data
was
> >>>>> plotted for
> >>>>> > > > EAST
> >>>>> > > > >> did
> >>>>> > > > >> > >> not
> >>>>> > > > >> > >> > necessarily plot for FULL. This surprises me
because
> >>>>> EAST
> >>>>> > is a
> >>>>> > > > >> subset
> >>>>> > > > >> > >> of
> >>>>> > > > >> > >> > FULL. See below:
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647
Oct 4
> >>>>> 14:54
> >>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 49039 Oct 5 15:17
> >>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 56791 Oct 20 16:09
> >>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 56834 Oct 21 16:09
> >>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 56921 Oct 22 16:11
> >>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 56518 Oct 24 16:36
> >>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 61187 Oct 31 08:21
> >>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 26475 Nov 2 08:22
> >>>>> > > > >> > >> > PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-
r--r--
> 1
> >>>>> > > > >> Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 26455 Nov 3 08:23
> >>>>> > > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > What I find equally concerning as that
sometimes
> plots
> >>>>> are
> >>>>> > > > >> incorrectly
> >>>>> > > > >> > >> > stored in the wrong directory list as
highlighted in
> >>>>> green
> >>>>> > > above.
> >>>>> > > > >> I've
> >>>>> > > > >> > >> > consulted the plot xml file for PMTF, for
instance,
> >>>>> and find
> >>>>> > > > >> nothing
> >>>>> > > > >> > >> > problematic. The only setting that might pose
an
> >>>>> issue when
> >>>>> > it
> >>>>> > > > >> comes
> >>>>> > > > >> > to
> >>>>> > > > >> > >> > generating line plots is when I set
event_equal to
> >>>>> true.
> >>>>> > > > However,
> >>>>> > > > >> all
> >>>>> > > > >> > >> the
> >>>>> > > > >> > >> > stat files are populated for all experiments.
In the
> >>>>> case of
> >>>>> > > > ozone
> >>>>> > > > >> you
> >>>>> > > > >> > >> see
> >>>>> > > > >> > >> > something different. I find that ozone tends
to
> result
> >>>>> in
> >>>>> > > > populated
> >>>>> > > > >> > >> plots
> >>>>> > > > >> > >> > but with intermittent reporting different than
PMTF -
> >>>>> see
> >>>>> > > below:
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls -ltrtotal
> >>>>> > 6144-rw-r--r-- 1
> >>>>> > > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 54318 Oct 6 10:51
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 54305 Oct 7 10:49
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 62179 Oct 13 13:25
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 63811 Oct 19 12:18
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 60112 Oct 20 12:21
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 60165 Oct 21 12:33
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 60160 Oct 22 12:21
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 60056 Oct 23 12:40
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 58517 Oct 24 13:31
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 58062 Oct 31 14:17
> >>>>> > > > >> > >> >
> OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> >>>>> > > > >> > Edward.Strobach
> >>>>> > > > >> > >> > emcmodel 58119 Nov 1 14:19
> >>>>> > > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > Similar issues have been found for other
fields,
> >>>>> cycles, and
> >>>>> > > > >> regions
> >>>>> > > > >> > >> > including verification being done for
meteorology.
> >>>>> > > > >> > >> > I can't seem to reason this intermittency and
> >>>>> inconsistency
> >>>>> > > > between
> >>>>> > > > >> > >> > reporting since the xml files are correct and
I'm
> >>>>> using the
> >>>>> > > same
> >>>>> > > > >> > >> procedure
> >>>>> > > > >> > >> > that I've always used. I do realize that I
need to
> >>>>> start
> >>>>> > > > creating
> >>>>> > > > >> a
> >>>>> > > > >> > >> better
> >>>>> > > > >> > >> > way to manage all these files that are being
> produced,
> >>>>> both
> >>>>> > in
> >>>>> > > > >> terms
> >>>>> > > > >> > of
> >>>>> > > > >> > >> > data/plots and xml files.
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > I ran across a rather large directory:
> >>>>> > > > >> > >> > ls
> >>>>> > > > >>
> >>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> >>>>> > > > >> > |
> >>>>> > > > >> > >> wc
> >>>>> > > > >> > >> > -w
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > That has 71,398 xml files in there. But I see
> >>>>> timestamps in
> >>>>> > > those
> >>>>> > > > >> > >> filenames
> >>>>> > > > >> > >> > from 20200921 to 20200925.
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > Is it the case that this stopped working on
20200925?
> >>>>> Or are
> >>>>> > > > those
> >>>>> > > > >> > dated
> >>>>> > > > >> > >> > filenames remnants from older work?
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > So it hasn't been working for over a month. Is
that
> >>>>> right?
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > There are a lot of files and I need to begin
managing
> >>>>> that
> >>>>> > > > >> better. I
> >>>>> > > > >> > >> think
> >>>>> > > > >> > >> > during that time there was a machine switch so
it
> >>>>> stopped on
> >>>>> > > that
> >>>>> > > > >> day.
> >>>>> > > > >> > >> > I've been able to produce plots even now, but
it's
> not
> >>>>> > > consistent
> >>>>> > > > >> in
> >>>>> > > > >> > >> > reporting while some regions generate plots
and
> others
> >>>>> do
> >>>>> > not.
> >>>>> > > > >> I've
> >>>>> > > > >> > >> > reduced the level processing in hopes that I
don't
> hit
> >>>>> a walk
> >>>>> > > > clock
> >>>>> > > > >> > >> issue
> >>>>> > > > >> > >> > since I'm submitting batch jobs. However,
I've run
> >>>>> this step
> >>>>> > > > >> outside
> >>>>> > > > >> > of
> >>>>> > > > >> > >> > batch and it always seems to work. This issue
seems
> to
> >>>>> happen
> >>>>> > > > with
> >>>>> > > > >> > batch
> >>>>> > > > >> > >> > and it is not at all clear why. Let me know
what
> else
> >>>>> you
> >>>>> > need
> >>>>> > > > >> from
> >>>>> > > > >> > me
> >>>>> > > > >> > >> to
> >>>>> > > > >> > >> > better coordinate what I would need to do
next.
> >>>>> Thanks.
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley
Gotway via
> >>>>> RT <
> >>>>> > > > >> > >> > met_help at ucar.edu>
> >>>>> > > > >> > >> > wrote:
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > > Hi Ed,
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > > Sorry for the delay. I took a look on wcoss
today.
> >>>>> You
> >>>>> > > describe
> >>>>> > > > >> that
> >>>>> > > > >> > >> your
> >>>>> > > > >> > >> > > plotting script has stopped working but
you're not
> >>>>> sure
> >>>>> > why.
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > > I ran across a rather large directory:
> >>>>> > > > >> > >> > > ls
> >>>>> > > > >> >
> >>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> >>>>> > > > >> > >> |
> >>>>> > > > >> > >> > wc
> >>>>> > > > >> > >> > > -w
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > > That has 71,398 xml files in there. But I
see
> >>>>> timestamps in
> >>>>> > > > those
> >>>>> > > > >> > >> > filenames
> >>>>> > > > >> > >> > > from 20200921 to 20200925.
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > > Is it the case that this stopped working on
> >>>>> 20200925? Or
> >>>>> > are
> >>>>> > > > >> those
> >>>>> > > > >> > >> dated
> >>>>> > > > >> > >> > > filenames remnants from older work?
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > > So it hasn't been working for over a month.
Is that
> >>>>> right?
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > > John
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT System
itself
> >>>>> via RT
> >>>>> > <
> >>>>> > > > >> > >> > > met_help at ucar.edu> wrote:
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > > >
> >>>>> > > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283
was acted
> >>>>> upon.
> >>>>> > > > >> > >> > > > Transaction: Given to johnhg (John Halley
Gotway)
> >>>>> by
> >>>>> > > > RT_System
> >>>>> > > > >> > >> > > > Queue: met_help
> >>>>> > > > >> > >> > > > Subject: plots no longer being
generated
> >>>>> > > > >> > >> > > > Owner: johnhg
> >>>>> > > > >> > >> > > > Requestors: edward.strobach at noaa.gov
> >>>>> > > > >> > >> > > > Status: new
> >>>>> > > > >> > >> > > > Ticket <URL:
> >>>>> > > > >> > >>
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > > >
> >>>>> > > > >> > >> > > >
> >>>>> > > > >> > >> > > > This transaction appears to have no
content
> >>>>> > > > >> > >> > > >
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> > >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> > --
> >>>>> > > > >> > >> > Edward Strobach
> >>>>> > > > >> > >> > EMC/NCEP/NWS/
> >>>>> > > > >> > >> > IMSG Contractor
> >>>>> > > > >> > >> > Cubicle#: 2029
> >>>>> > > > >> > >> > 301-683-3717
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >> >
> >>>>> > > > >> > >>
> >>>>> > > > >> > >>
> >>>>> > > > >> > >
> >>>>> > > > >> > > --
> >>>>> > > > >> > > Edward Strobach
> >>>>> > > > >> > > EMC/NCEP/NWS/
> >>>>> > > > >> > > IMSG Contractor
> >>>>> > > > >> > > Cubicle#: 2029
> >>>>> > > > >> > > 301-683-3717
> >>>>> > > > >> > >
> >>>>> > > > >> >
> >>>>> > > > >> >
> >>>>> > > > >> > --
> >>>>> > > > >> > Edward Strobach
> >>>>> > > > >> > EMC/NCEP/NWS/
> >>>>> > > > >> > IMSG Contractor
> >>>>> > > > >> > Cubicle#: 2029
> >>>>> > > > >> > 301-683-3717
> >>>>> > > > >> >
> >>>>> > > > >> >
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >
> >>>>> > > > > --
> >>>>> > > > > Edward Strobach
> >>>>> > > > > EMC/NCEP/NWS/
> >>>>> > > > > IMSG Contractor
> >>>>> > > > > Cubicle#: 2029
> >>>>> > > > > 301-683-3717
> >>>>> > > > >
> >>>>> > > >
> >>>>> > > >
> >>>>> > > > --
> >>>>> > > > Edward Strobach
> >>>>> > > > EMC/NCEP/NWS/
> >>>>> > > > IMSG Contractor
> >>>>> > > > Cubicle#: 2029
> >>>>> > > > 301-683-3717
> >>>>> > > >
> >>>>> > > >
> >>>>> > >
> >>>>> > >
> >>>>> >
> >>>>> > --
> >>>>> > Edward Strobach
> >>>>> > EMC/NCEP/NWS/
> >>>>> > IMSG Contractor
> >>>>> > Cubicle#: 2029
> >>>>> > 301-683-3717
> >>>>> >
> >>>>> >
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Edward Strobach
> >>>> EMC/NCEP/NWS/
> >>>> IMSG Contractor
> >>>> Cubicle#: 2029
> >>>> 301-683-3717
> >>>>
> >>>
> >>>
> >>> --
> >>> Edward Strobach
> >>> EMC/NCEP/NWS/
> >>> IMSG Contractor
> >>> Cubicle#: 2029
> >>> 301-683-3717
> >>>
> >>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Mon Nov 09 12:09:50 2020
I am pleased about being able to do it this way. It seems like a
better
approach. However, It's unclear why, when running the same command to
process the same XML file, that files are not always being copied
over.
>From my earlier email you can see that sometimes 2 files are copied
over,
sometimes 4, and sometimes only 1. I was hoping all 8 would be copied
over. This is kind of weird and I'm not sure how one would go about
this.
On Mon, Nov 9, 2020 at 12:47 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Ed,
>
> Yes, there's lots of moving parts here.
> (1) Setting up the METviewer XML to run 1 or more plots.
> (2) Running that XML remotely to generate plots.
> (3) Transferring the resulting image files back.
>
> I just wanted to make sure you know about the options for (1). And
I'm
> really glad to hear that you were able to see how to generate
multiple
> plots from a single METviewer XML!
> I'd say the rest of the details about managing the plotting/image
transfer
> logic is really up to you and your colleagues. If I were setting
this up, I
> would consider having each batch job write to a single output
directory,
> and then recursively transfer back that directory. That seems easier
than
> dealing with lots of individual files. But it's really to you guys.
>
> There is another detail about the METviewer plot XML that we haven't
talked
> about yet, and that's plot "inheritance". In a single XML, you can
define
> multiple <plot> tags and have one <plot> inherit settings from a
previous
> one. I took this to the extreme for one project, setting up a pretty
> complex XML that made like 6 different plot types using multiple
levels of
> inheritance. In general, I'd advise against that. While it's cool
that it
> works, it makes for complex logic that's difficult to maintain.
Instead,
> I'd just look at any "for" loops you have in your plotting script
and move
> as many as are easy and feasible into the XML directly... without
going
> crazy! That would be like init hours, masking regions, and perhaps
> statistic types. That should greatly reduce the number of individual
calls
> to mv_batch.sh and hopefully provide for more stable results.
>
> But if you ever want to learn about plot inheritance, I'm happy to
help you
> with that too.
>
> John
>
> On Mon, Nov 9, 2020 at 8:02 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> >
> > FYI, when running test_v2.sh again after running it the first time
only
> one
> > plot was copied over:
> > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> > 100%
57KB
> > 19.9MB/s 00:00
> >
> > On Mon, Nov 9, 2020 at 10:00 AM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > I did run some additional tests outside the main test*sh script.
> > > Specifically I ran the following line several times to see what
plots
> > would
> > > be copied over to WCOSS:
> > > bash
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> > > edward.strobach
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls/Test.xml
> > >
> > > The first time around 3 plots were copied over:
> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
> > > 100% 51KB 18.6MB/s 00:00
> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> > > 100% 58KB 12.0MB/s 00:00
> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> > > 100% 57KB 2.8MB/s 00:00
> > >
> > > The second time 2 plots were copied over:
> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> > > 100% 58KB 2.6MB/s 00:00
> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> > > 100% 57KB 6.2MB/s 00:00
> > >
> > > The third time 3 plots were copied over:
> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
> > > 100%
51KB
> > > 2.4MB/s 00:00
> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> > > 100%
58KB
> > > 6.1MB/s 00:00
> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> > > 100%
57KB
> > > 11.1MB/s 00:00
> > >
> > > The order of init_hour and vx_mask is as follows:
> > > <06>
> > > <12>
> > > and
> > > <Mixed_Forest>
> > > <FULL>
> > > <EAST>
> > > <WEST>
> > >
> > > It does appear that the latest plots are copied over in the
process
> (i.e.
> > > if two, then the last two; if three, then the last three).
> > > I don't really have a feel for why two files are copied over for
some
> > > instances and 3 for other instances. Lastly, I ran test_v2.sh
again,
> and
> > > this time it copied over 4 plots. See below:
> > >
> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t06z.png
> > > 100%
53KB
> > > 12.7MB/s 00:00
> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
> > > 100%
51KB
> > > 9.7MB/s 00:00
> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
> > > 100%
58KB
> > > 11.3MB/s 00:00
> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
> > > 100%
57KB
> > > 11.4MB/s 00:00
> > >
> > > This is very bizarre to me.
> > >
> > >
> > >
> > > On Mon, Nov 9, 2020 at 9:26 AM Edward Strobach - NOAA Affiliate
<
> > > edward.strobach at noaa.gov> wrote:
> > >
> > >> Actually, I don't think I can do that. FBAR and OBAR are split
and
> > >> adding RMSE as a third line might confuse the process.
> > >>
> > >> I was able to run a test with mixed success. I was successful
at
> > >> generating 8 plots (4 regions for 2 initial hours). The
regions were
> > >> Mixed_Forest, FULL, EAST, and WEST while the initial hours were
06 and
> > 12.
> > >> The test ran well with indications that all plots were
generated and
> > stored
> > >> here: Created plot
> > >>
> >
> /data/mv_data/edward.strobach/plots/PMTF_Mixed_Forest_112020_2020-
11-01_2020-11-06_t06z.png
> > >>
> > >> However, I was not successful at saving the plots on Dell. Only
the
> last
> > >> plot was saved here:
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011.
> > >>
> > >> It seems that in my bash script the line that produces the
plots that
> > are
> > >> then stored in the above directory must be done in a loop
otherwise it
> > will
> > >> only move the last plot over as suggested by the fact that only
the
> plot
> > >> featuring WEST at 12z was moved over - the last plot of the
series of
> > plots
> > >> generated. I suppose I could create logic that extracts the
> > region/initial
> > >> hour information and construct a two part loop to move over all
plots?
> > >>
> > >> On Mon, Nov 9, 2020 at 8:19 AM Edward Strobach - NOAA Affiliate
<
> > >> edward.strobach at noaa.gov> wrote:
> > >>
> > >>> Just a follow-up on one other item..
> > >>>
> > >>> Can also list multiple stat types in the same way that I
specify the
> > >>> init_hour and vx_mask? In other words, can I do the
following:
> > >>>
> > >>> <dep1>
> > >>> <fcst_var name="PMTF">
> > >>> <stat>FBAR_OBAR</stat>
> > >>> <stat>RMSE</stat>
> > >>> </fcst_var>
> > >>> </dep1>
> > >>>
> > >>> I know that the "set" block is not being used here.
> > >>>
> > >>> On Mon, Nov 9, 2020 at 7:41 AM Edward Strobach - NOAA
Affiliate <
> > >>> edward.strobach at noaa.gov> wrote:
> > >>>
> > >>>> Thanks John. Let me try that and I'll let you know how it
goes.
> > >>>>
> > >>>> On Fri, Nov 6, 2020 at 10:53 AM John Halley Gotway via RT <
> > >>>> met_help at ucar.edu> wrote:
> > >>>>
> > >>>>> Ed,
> > >>>>>
> > >>>>> You're right about the METviewer GUI grouping multiple
selections
> > >>>>> together.
> > >>>>> That looks like this in the XML using the <set> tag:
> > >>>>> <field equalize="false" name="vx_mask">
> > >>>>> <set name="vx_mask_0">
> > >>>>> <val>FULL</val>
> > >>>>> </set>
> > >>>>> </field>
> > >>>>>
> > >>>>> But in the updates I sent, I remove the <set> tag:
> > >>>>> <field equalize="false" name="vx_mask">
> > >>>>> <val>FULL</val>
> > >>>>> <val>CONUS</val>
> > >>>>> <val>EAST</val>
> > >>>>> <val>WEST</val>
> > >>>>> </field>
> > >>>>> So instead of processing this as 1 group of 4 values, it'll
process
> > >>>>> the 4
> > >>>>> values separately.
> > >>>>>
> > >>>>> The current vx_mask value is referenced using {vx_mask}
syntax.
> And I
> > >>>>> included examples of using that in the output file names and
titles
> > in
> > >>>>> my
> > >>>>> previous email:
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> >
>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
> > >>>>>
> > >>>>>
> > >>>>>
> >
>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
> > >>>>>
> > >>>>>
> >
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
> > >>>>>
> > >>>>> <title>Average PMTF (October 2020; {init_hour}z cycle;
{vx_mask}
> > >>>>> Domain)</title>
> > >>>>>
> > >>>>> John
> > >>>>>
> > >>>>> On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA
Affiliate via
> > RT
> > >>>>> <
> > >>>>> met_help at ucar.edu> wrote:
> > >>>>>
> > >>>>> >
> > >>>>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> > >>>>> >
> > >>>>> > Hi John,
> > >>>>> >
> > >>>>> > This definitely helps. I never thought about it this way
> because I
> > >>>>> > assumed, based on my experience with the interactive MET
GUI,
> that
> > I
> > >>>>> could
> > >>>>> > only generate a single plot regardless of how I specified
my
> > >>>>> settings in
> > >>>>> > the XML. For example, if I listed EAST and WEST under
vx_mask,
> > then
> > >>>>> my
> > >>>>> > thinking would be that both those regions would be grouped
and a
> > >>>>> single
> > >>>>> > plot created. I guess I was wrong on that. The concern I
still
> > >>>>> have,
> > >>>>> > however, is that the plot title and plot name - the png
file - is
> > >>>>> also set
> > >>>>> > in the XML file. Even after adding the list you gave, I
can't
> see
> > >>>>> how
> > >>>>> > unique file names will be created. Is that true?
> > >>>>> >
> > >>>>> > On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT <
> > >>>>> > met_help at ucar.edu>
> > >>>>> > wrote:
> > >>>>> >
> > >>>>> > > Ed,
> > >>>>> > >
> > >>>>> > > Thanks for sending examples. My goal was to use
METviewer's XML
> > >>>>> upload
> > >>>>> > > functionality but I'm not able to upload a pdf version
of the
> xml
> > >>>>> file.
> > >>>>> > >
> > >>>>> > > I did spend some time trying to replicate this exact
plot with
> > >>>>> METviewer
> > >>>>> > > at:
> > >>>>> > > https://metviewer.nws.noaa.gov/metviewer1.jsp
> > >>>>> > >
> > >>>>> > > But I was never quite successful. So I made just a
similar
> > version
> > >>>>> > instead.
> > >>>>> > > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
> > >>>>> > >
> > >>>>> > > This plots FBAR and OBAR for 6 different models for the
month
> of
> > >>>>> October
> > >>>>> > > for all 06Z initializations over the FULL model domain.
> > >>>>> > >
> > >>>>> > > I wanted to show you how to modify this for batch
mode... and
> > then
> > >>>>> adapt
> > >>>>> > it
> > >>>>> > > to make multiple plots. But I'm not able to make batch
plots on
> > >>>>> wcoss
> > >>>>> > using
> > >>>>> > > the AWS instance. I don't think I have the right
permissions to
> > do
> > >>>>> so.
> > >>>>> > >
> > >>>>> > > So let me just say this. Generally, you just update the
list of
> > >>>>> items in
> > >>>>> > > plot_fix section.
> > >>>>> > >
> > >>>>> > > Original from METviewer with 1 init hour and 1 mask:
> > >>>>> > >
> > >>>>> > > <field equalize="false" name="vx_mask">
> > >>>>> > > <set name="vx_mask_0">
> > >>>>> > > <val>FULL</val>
> > >>>>> > > </set>
> > >>>>> > > </field>
> > >>>>> > > <field equalize="false" name="init_hour">
> > >>>>> > > <set name="init_hour_1">
> > >>>>> > > <val>06</val>
> > >>>>> > > </set>
> > >>>>> > > </field>
> > >>>>> > >
> > >>>>> > > Change to 2 init hours and 4 masks:
> > >>>>> > >
> > >>>>> > > <field equalize="false" name="vx_mask">
> > >>>>> > > <val>FULL</val>
> > >>>>> > > <val>CONUS</val>
> > >>>>> > > <val>EAST</val>
> > >>>>> > > <val>WEST</val>
> > >>>>> > > </field>
> > >>>>> > > <field equalize="false" name="init_hour">
> > >>>>> > > <val>06</val>
> > >>>>> > > <val>12</val>
> > >>>>> > > </field>
> > >>>>> > >
> > >>>>> > > And then reference those values in the naming of the
titles and
> > >>>>> output
> > >>>>> > > files:
> > >>>>> > >
> > >>>>> > >
> > >>>>> > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
> > >>>>> > >
> > >>>>> > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
> > >>>>> > >
> > >>>>> > >
> > >>>>>
> >
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
> > >>>>> > >
> > >>>>> > > <title>Average PMTF (October 2020; {init_hour}z cycle;
> {vx_mask}
> > >>>>> > > Domain)</title>
> > >>>>> > >
> > >>>>> > > So that's the general idea. And when you submit this to
> mv_batch,
> > >>>>> it'll
> > >>>>> > > make 8 plots instead of 1.
> > >>>>> > > And you can see how you'd easily expand this to many
plots.
> > >>>>> > >
> > >>>>> > > But I worry that this may not be sufficient information
to get
> > you
> > >>>>> going.
> > >>>>> > >
> > >>>>> > > John
> > >>>>> > >
> > >>>>> > > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA
> Affiliate
> > >>>>> via RT <
> > >>>>> > > met_help at ucar.edu> wrote:
> > >>>>> > >
> > >>>>> > > >
> > >>>>> > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > >
> > >>>>> > > >
> > >>>>> > > > Hi John,
> > >>>>> > > >
> > >>>>> > > > I think I was able to cover your request okay. Below
are 4
> > >>>>> attachments
> > >>>>> > > > that include the test script that was used to generate
the
> > >>>>> plots, a
> > >>>>> > > sample
> > >>>>> > > > XML file, and the two plots that were generated using
the
> test
> > >>>>> run
> > >>>>> > > script.
> > >>>>> > > > These plots were not created previously using the
batch job
> but
> > >>>>> uses
> > >>>>> > the
> > >>>>> > > > same exact logic as the original run script. Thanks
again
> for
> > >>>>> working
> > >>>>> > > with
> > >>>>> > > > me on this. Please let me know if you need anything
else or
> > >>>>> would like
> > >>>>> > > me
> > >>>>> > > > to test something.
> > >>>>> > > >
> > >>>>> > > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA
> > Affiliate <
> > >>>>> > > > edward.strobach at noaa.gov> wrote:
> > >>>>> > > >
> > >>>>> > > > > Ed,
> > >>>>> > > > >
> > >>>>> > > > > Image files showing up in an unexpected output
directory
> > >>>>> likely point
> > >>>>> > > to
> > >>>>> > > > a
> > >>>>> > > > > problem in the logic of the processing scripts. It's
pretty
> > >>>>> unlikely
> > >>>>> > > that
> > >>>>> > > > > this behavior is caused by the METviewer code
itself. But
> > I've
> > >>>>> > > definitely
> > >>>>> > > > > been surprised before!
> > >>>>> > > > >
> > >>>>> > > > > I tested the logic outside of a batch job and it
works
> > fine. I
> > >>>>> > suspect
> > >>>>> > > > > that the issuance of another batch job while the
other one
> is
> > >>>>> getting
> > >>>>> > > > > processed is causing some confusion in the
processing.
> These
> > >>>>> issues
> > >>>>> > > > don't
> > >>>>> > > > > happen everyday and appear more sporadic than
anything.
> > >>>>> > > > >
> > >>>>> > > > > So what we really need here is an example of taking
a plot
> > >>>>> from the
> > >>>>> > > > > METviewer GUI and adapting it to make it generate
multiple
> > >>>>> plots. I
> > >>>>> > did
> > >>>>> > > > go
> > >>>>> > > > > through examples of doing this in previous
tutorials, but
> we
> > >>>>> didn't
> > >>>>> > > start
> > >>>>> > > > > recording them until last summer. I checked the July
2019
> NRL
> > >>>>> > tutorial
> > >>>>> > > > > videos and don't see it there.
> > >>>>> > > > >
> > >>>>> > > > > Is there something you need from me to help on this?
Just
> > let
> > >>>>> me
> > >>>>> > know
> > >>>>> > > > >
> > >>>>> > > > > We've begun recording training videos for common
topics
> like
> > >>>>> this.
> > >>>>> > But
> > >>>>> > > we
> > >>>>> > > > > only have one for METviewer so far:
> > >>>>> > > > >
> > >>>>> > >
> > >>>>>
> > https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> > >>>>> > > > >
> > >>>>> > > > > But this definitely seems like a good candidate for
a new
> > >>>>> training
> > >>>>> > > video.
> > >>>>> > > > >
> > >>>>> > > > > For now though, how about this...
> > >>>>> > > > > - You use METviewer to manually make a single plot,
> > formatting
> > >>>>> it how
> > >>>>> > > > you'd
> > >>>>> > > > > like.
> > >>>>> > > > > - You send me your sample plot and corresponding XML
file.
> > >>>>> > > > > - I'll tweak it to generate many plots via batch
instead of
> > >>>>> just one.
> > >>>>> > > > > - And then you can continue tweaking to add more
> > permutations.
> > >>>>> > > > >
> > >>>>> > > > > Okay, I'll work on that now. Thanks.
> > >>>>> > > > >
> > >>>>> > > > > John
> > >>>>> > > > >
> > >>>>> > > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway
via RT <
> > >>>>> > > > > met_help at ucar.edu> wrote:
> > >>>>> > > > >
> > >>>>> > > > >> Ed,
> > >>>>> > > > >>
> > >>>>> > > > >> Image files showing up in an unexpected output
directory
> > >>>>> likely
> > >>>>> > point
> > >>>>> > > > to a
> > >>>>> > > > >> problem in the logic of the processing scripts.
It's
> pretty
> > >>>>> unlikely
> > >>>>> > > > that
> > >>>>> > > > >> this behavior is caused by the METviewer code
itself. But
> > I've
> > >>>>> > > > definitely
> > >>>>> > > > >> been surprised before!
> > >>>>> > > > >>
> > >>>>> > > > >> So what we really need here is an example of taking
a plot
> > >>>>> from the
> > >>>>> > > > >> METviewer GUI and adapting it to make it generate
multiple
> > >>>>> plots. I
> > >>>>> > > did
> > >>>>> > > > go
> > >>>>> > > > >> through examples of doing this in previous
tutorials, but
> we
> > >>>>> didn't
> > >>>>> > > > start
> > >>>>> > > > >> recording them until last summer. I checked the
July 2019
> > NRL
> > >>>>> > tutorial
> > >>>>> > > > >> videos and don't see it there.
> > >>>>> > > > >>
> > >>>>> > > > >> We've begun recording training videos for common
topics
> like
> > >>>>> this.
> > >>>>> > But
> > >>>>> > > > we
> > >>>>> > > > >> only have one for METviewer so far:
> > >>>>> > > > >>
> > >>>>> > > >
> > >>>>> >
> > >>>>>
> > https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
> > >>>>> > > > >>
> > >>>>> > > > >> But this definitely seems like a good candidate for
a new
> > >>>>> training
> > >>>>> > > > video.
> > >>>>> > > > >>
> > >>>>> > > > >> For now though, how about this...
> > >>>>> > > > >> - You use METviewer to manually make a single plot,
> > >>>>> formatting it
> > >>>>> > how
> > >>>>> > > > >> you'd
> > >>>>> > > > >> like.
> > >>>>> > > > >> - You send me your sample plot and corresponding
XML file.
> > >>>>> > > > >> - I'll tweak it to generate many plots via batch
instead
> of
> > >>>>> just
> > >>>>> > one.
> > >>>>> > > > >> - And then you can continue tweaking to add more
> > permutations.
> > >>>>> > > > >>
> > >>>>> > > > >> John
> > >>>>> > > > >>
> > >>>>> > > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach -
NOAA
> > >>>>> Affiliate via
> > >>>>> > > RT <
> > >>>>> > > > >> met_help at ucar.edu> wrote:
> > >>>>> > > > >>
> > >>>>> > > > >> >
> > >>>>> > > > >> > <URL:
> > >>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
> > >>>>> > > > >> >
> > >>>>> > > > >> > Oh, and one more thing..
> > >>>>> > > > >> >
> > >>>>> > > > >> > The green text that didn't show up was meant to
> highlight
> > >>>>> the fact
> > >>>>> > > > that
> > >>>>> > > > >> > sometimes plots that shouldn't be stored in
certain
> > >>>>> directories
> > >>>>> > end
> > >>>>> > > up
> > >>>>> > > > >> > there. The pasted example below shows one
instance
> where
> > >>>>> OZMAX*
> > >>>>> > was
> > >>>>> > > > >> stored
> > >>>>> > > > >> > where TMP* should have been stored.
> > >>>>> > > > >> >
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct
5 10:52
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct
6 10:52
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct
7 10:00
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct
13 13:49
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct
14 11:31
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct
19 11:37
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct
20 11:35
> > >>>>> > > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct
21 11:45
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct
22 11:42
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct
23 11:50
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct
24 12:57
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct
31 08:47
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov
2 08:50
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov
3 08:52
> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
> > >>>>> > > > >> >
> > >>>>> > > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach -
NOAA
> > >>>>> Affiliate <
> > >>>>> > > > >> > edward.strobach at noaa.gov> wrote:
> > >>>>> > > > >> >
> > >>>>> > > > >> > > Hi John,
> > >>>>> > > > >> > >
> > >>>>> > > > >> > > I know there are a lot of calls and speculate
that
> this
> > >>>>> could be
> > >>>>> > > > part
> > >>>>> > > > >> of
> > >>>>> > > > >> > > the problem. I don't think that I'm following
best
> > >>>>> practice by
> > >>>>> > > > >> > approaching
> > >>>>> > > > >> > > it in this way. Generating plots usually works
well
> if
> > I
> > >>>>> > process
> > >>>>> > > a
> > >>>>> > > > >> > subset
> > >>>>> > > > >> > > of the computations outside of batch scripting.
The
> > >>>>> python
> > >>>>> > script
> > >>>>> > > > was
> > >>>>> > > > >> > > given to me by Logan, but I have since made
> significant
> > >>>>> changes
> > >>>>> > to
> > >>>>> > > > it
> > >>>>> > > > >> in
> > >>>>> > > > >> > > order to dynamically fill in the xml file based
on
> > >>>>> parameters
> > >>>>> > set.
> > >>>>> > > > >> The
> > >>>>> > > > >> > > test*.sh script was also given to me. I
created the
> > >>>>> > > > Time_Series*bsub
> > >>>>> > > > >> as
> > >>>>> > > > >> > a
> > >>>>> > > > >> > > way to process a series of plots based on cycle
time,
> > >>>>> region,
> > >>>>> > > etc. I
> > >>>>> > > > >> > think
> > >>>>> > > > >> > > the first two scripts are solid and do well for
what I
> > >>>>> need to
> > >>>>> > get
> > >>>>> > > > >> done.
> > >>>>> > > > >> > > It's the Time_Series*bsub script that bothers
me a
> bit.
> > >>>>> > > > >> > >
> > >>>>> > > > >> > > I can't see how a single XML file can process
all
> these
> > >>>>> plots
> > >>>>> > in a
> > >>>>> > > > >> single
> > >>>>> > > > >> > > setting based on what I've learned thus far.
How
> would
> > >>>>> one go
> > >>>>> > > about
> > >>>>> > > > >> > that?
> > >>>>> > > > >> > > I would be open going in that direction if you
can
> > >>>>> provide some
> > >>>>> > > > >> guidance.
> > >>>>> > > > >> > >
> > >>>>> > > > >> > > Unfortunately, I haven't gotten a lot of
feedback or
> > >>>>> dialogue
> > >>>>> > > > exchange
> > >>>>> > > > >> > > about what I've done so far at EMC with this
new
> set-up.
> > >>>>> This
> > >>>>> > has
> > >>>>> > > > >> also
> > >>>>> > > > >> > > been part of the problem. I can't seem to get
them to
> > >>>>> focus so
> > >>>>> > > that
> > >>>>> > > > >> we
> > >>>>> > > > >> > can
> > >>>>> > > > >> > > prioritize. I know that I'm doing the best I
can.
> If I
> > >>>>> can
> > >>>>> > learn
> > >>>>> > > > >> how to
> > >>>>> > > > >> > > go about this more efficiently it would be of
great
> > >>>>> help. I can
> > >>>>> > > > then
> > >>>>> > > > >> > > expand this to other verification as venture to
other
> > >>>>> projects
> > >>>>> > in
> > >>>>> > > > the
> > >>>>> > > > >> > > future. I know you're busy with many things, so
I'm
> > >>>>> willing to
> > >>>>> > go
> > >>>>> > > > >> along
> > >>>>> > > > >> > > with your schedule as you find an opening.
> > >>>>> > > > >> > >
> > >>>>> > > > >> > > Thanks
> > >>>>> > > > >> > >
> > >>>>> > > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley
Gotway via
> > RT
> > >>>>> <
> > >>>>> > > > >> > > met_help at ucar.edu> wrote:
> > >>>>> > > > >> > >
> > >>>>> > > > >> > >> Ed,
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> It's really difficult for me to see an obvious
> problem
> > >>>>> that's
> > >>>>> > > > causing
> > >>>>> > > > >> > the
> > >>>>> > > > >> > >> behavior you describe.
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> Unfortunately, the red and green highlighting
you
> > >>>>> describe
> > >>>>> > > doesn't
> > >>>>> > > > >> show
> > >>>>> > > > >> > up
> > >>>>> > > > >> > >> in the Gmail web client. There must be some
mail
> system
> > >>>>> > > > >> incompatibility
> > >>>>> > > > >> > >> there.
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> Looking at your bsub script:
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >>
> > >>>>> > > > >> >
> > >>>>> > > > >>
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> I am struck by the number of individual calls
you're
> > >>>>> making to
> > >>>>> > > > >> > METviewer:
> > >>>>> > > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks x
2 plot
> > >>>>> types =
> > >>>>> > 656
> > >>>>> > > > >> calls
> > >>>>> > > > >> > to
> > >>>>> > > > >> > >> METviewer
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> When making plots via the METviewer GUI, it's
true
> that
> > >>>>> 1 XML
> > >>>>> > = 1
> > >>>>> > > > >> plot.
> > >>>>> > > > >> > >> However, when making plots via the batch
engine, the
> > >>>>> plot XML
> > >>>>> > can
> > >>>>> > > > be
> > >>>>> > > > >> set
> > >>>>> > > > >> > >> up
> > >>>>> > > > >> > >> to create many, many plots. While I have no
direct
> > proof
> > >>>>> of
> > >>>>> > this,
> > >>>>> > > > it
> > >>>>> > > > >> > might
> > >>>>> > > > >> > >> be the case that executing 600+ batch jobs for
> > METviewer
> > >>>>> is
> > >>>>> > > leading
> > >>>>> > > > >> to
> > >>>>> > > > >> > >> trouble on the system. One thing to try would
be
> > setting
> > >>>>> up the
> > >>>>> > > XML
> > >>>>> > > > >> to
> > >>>>> > > > >> > >> make
> > >>>>> > > > >> > >> many, many more plots in each call. Then check
to see
> > if
> > >>>>> > running
> > >>>>> > > > >> > METviewer
> > >>>>> > > > >> > >> in this way is more stable.
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> I don't think I fully appreciate all of the
logic
> that
> > >>>>> lives in
> > >>>>> > > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py,
and
> > >>>>> > > > >> test_Time_Series_AGG.sh.
> > >>>>> > > > >> > >> However, it *might* be possible to construct a
single
> > >>>>> XML file
> > >>>>> > > > which
> > >>>>> > > > >> > >> produces all 656 plots.
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> Let me know what you think and how you'd like
to
> > proceed.
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> Thanks,
> > >>>>> > > > >> > >> John
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward Strobach
- NOAA
> > >>>>> Affiliate
> > >>>>> > > via
> > >>>>> > > > >> RT <
> > >>>>> > > > >> > >> met_help at ucar.edu> wrote:
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > <URL:
> > >>>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > >>>>> > > >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > Hi John,
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > No problem.
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > I'm including my responses below:
> > >>>>> > > > >> > >> > Sorry for the delay. I took a look on wcoss
today.
> > You
> > >>>>> > describe
> > >>>>> > > > >> that
> > >>>>> > > > >> > >> your
> > >>>>> > > > >> > >> > plotting script has stopped working but
you're not
> > >>>>> sure why.
> > >>>>> > > > >> > >> > It seems more intermittent. Take a look at
the
> > example
> > >>>>> > below.
> > >>>>> > > > You
> > >>>>> > > > >> > can
> > >>>>> > > > >> > >> see
> > >>>>> > > > >> > >> > that for some reason there are gaps in the
plots
> > being
> > >>>>> > > generated.
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 34105
Oct 4
> > >>>>> 14:52
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 41781 Oct 5 15:15
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 56691 Oct 19 15:02
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 55548 Oct 20 16:08
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 55598 Oct 21 16:08
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 55599 Oct 22 16:09
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 55352 Oct 24 16:34
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 26175 Oct 31 08:20
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 26178 Nov 1 08:20
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 26160 Nov 3 08:22
> > >>>>> > > > >> > >> >
> PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 25354 Nov 3 08:22
> > >>>>> > > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
> > >>>>> > > > >> > >> > You can tell when a plot is successful by
looking
> at
> > >>>>> the file
> > >>>>> > > > >> size. I
> > >>>>> > > > >> > >> > highlighted an example of that in red.
However,
> the
> > >>>>> files
> > >>>>> > > > showing
> > >>>>> > > > >> > about
> > >>>>> > > > >> > >> > 25000 are the ones where no line plots are
being
> > >>>>> produced.
> > >>>>> > You
> > >>>>> > > > can
> > >>>>> > > > >> > also
> > >>>>> > > > >> > >> > see that stat files are generated everyday,
so as
> > long
> > >>>>> as I'm
> > >>>>> > > > >> loading
> > >>>>> > > > >> > >> the
> > >>>>> > > > >> > >> > data, which I am, I should be able to use
the xml
> > >>>>> files to
> > >>>>> > > > generate
> > >>>>> > > > >> > the
> > >>>>> > > > >> > >> > plots. I'm attaching a directory list from
wcoss
> > that
> > >>>>> shows
> > >>>>> > > the
> > >>>>> > > > >> dates
> > >>>>> > > > >> > >> > where data is produced.
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > *20190801 20190817 20200624 20200710
20200726
> > >>>>> 20200812
> > >>>>> > > > >> 20200828
> > >>>>> > > > >> > >> > 20201003 2020101920190802 20190818
20200625
> > >>>>> 20200711
> > >>>>> > > > 20200727
> > >>>>> > > > >> > >> > 20200813 20200829 20201004
2020102020190803
> > >>>>> 20190819
> > >>>>> > > > 20200626
> > >>>>> > > > >> > >> > 20200712 20200728 20200814 20200830
20201005
> > >>>>> > > > 2020102120190804
> > >>>>> > > > >> > >> > 20190820 20200627 20200713 20200729
20200815
> > >>>>> 20200831
> > >>>>> > > > >> 20201006
> > >>>>> > > > >> > >> > 2020102220190805 20190821 20200628
20200714
> > >>>>> 20200730
> > >>>>> > > > 20200816
> > >>>>> > > > >> > >> > 20200921 20201007 2020102320190806
20190822
> > >>>>> 20200629
> > >>>>> > > > 20200715
> > >>>>> > > > >> > >> > 20200731 20200817 20200922 20201008
> > >>>>> 2020102420190807
> > >>>>> > > > 20190823
> > >>>>> > > > >> > >> > 20200630 20200716 20200801 20200818
20200923
> > >>>>> 20201009
> > >>>>> > > > >> > >> > 2020102520190808 20190824 20200701
20200717
> > >>>>> 20200802
> > >>>>> > > > 20200819
> > >>>>> > > > >> > >> > 20200924 20201010 2020102620190809
20190825
> > >>>>> 20200702
> > >>>>> > > > 20200718
> > >>>>> > > > >> > >> > 20200803 20200820 20200925 20201011
> > >>>>> 2020102720190810
> > >>>>> > > > 20190826
> > >>>>> > > > >> > >> > 20200703 20200719 20200804 20200821
20200926
> > >>>>> 20201012
> > >>>>> > > > >> > >> > 2020102820190811 20190827 20200704
20200720
> > >>>>> 20200805
> > >>>>> > > > 20200822
> > >>>>> > > > >> > >> > 20200927 20201013 2020102920190812
20190828
> > >>>>> 20200705
> > >>>>> > > > 20200721
> > >>>>> > > > >> > >> > 20200806 20200823 20200928 20201014
> > >>>>> 2020103020190813
> > >>>>> > > > 20190829
> > >>>>> > > > >> > >> > 20200706 20200722 20200807 20200824
20200929
> > >>>>> 20201015
> > >>>>> > > > >> > >> > 2020103120190814 20190830 20200707
20200723
> > >>>>> 20200808
> > >>>>> > > > 20200825
> > >>>>> > > > >> > >> > 20200930 20201016 2020110120190815
20200622
> > >>>>> 20200708
> > >>>>> > > > 20200724
> > >>>>> > > > >> > >> > 20200810 20200826 20201001 20201017
> > >>>>> 2020110220190816
> > >>>>> > > > 20200623
> > >>>>> > > > >> > >> > 20200709 20200725 20200811 20200827
20201002
> > >>>>> 20201018*
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > Furthermore, what's interesting is that I
have
> mixed
> > >>>>> success
> > >>>>> > > with
> > >>>>> > > > >> > other
> > >>>>> > > > >> > >> > cases. Using the same field - PMTF - for
the same
> > >>>>> cycle -
> > >>>>> > the
> > >>>>> > > > 06z
> > >>>>> > > > >> > >> cycle -
> > >>>>> > > > >> > >> > you can see that some of the dates where
data was
> > >>>>> plotted for
> > >>>>> > > > EAST
> > >>>>> > > > >> did
> > >>>>> > > > >> > >> not
> > >>>>> > > > >> > >> > necessarily plot for FULL. This surprises
me
> because
> > >>>>> EAST
> > >>>>> > is a
> > >>>>> > > > >> subset
> > >>>>> > > > >> > >> of
> > >>>>> > > > >> > >> > FULL. See below:
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel 36647
Oct 4
> > >>>>> 14:54
> > >>>>> > > > >> > >> >
> PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 49039 Oct 5 15:17
> > >>>>> > > > >> > >> >
> PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 56791 Oct 20 16:09
> > >>>>> > > > >> > >> >
> PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 56834 Oct 21 16:09
> > >>>>> > > > >> > >> >
> PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 56921 Oct 22 16:11
> > >>>>> > > > >> > >> >
> PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 56518 Oct 24 16:36
> > >>>>> > > > >> > >> >
> PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 61187 Oct 31 08:21
> > >>>>> > > > >> > >> >
> PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 26475 Nov 2 08:22
> > >>>>> > > > >> > >> >
> PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r--
> > 1
> > >>>>> > > > >> Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 26455 Nov 3 08:23
> > >>>>> > > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > What I find equally concerning as that
sometimes
> > plots
> > >>>>> are
> > >>>>> > > > >> incorrectly
> > >>>>> > > > >> > >> > stored in the wrong directory list as
highlighted
> in
> > >>>>> green
> > >>>>> > > above.
> > >>>>> > > > >> I've
> > >>>>> > > > >> > >> > consulted the plot xml file for PMTF, for
instance,
> > >>>>> and find
> > >>>>> > > > >> nothing
> > >>>>> > > > >> > >> > problematic. The only setting that might
pose an
> > >>>>> issue when
> > >>>>> > it
> > >>>>> > > > >> comes
> > >>>>> > > > >> > to
> > >>>>> > > > >> > >> > generating line plots is when I set
event_equal to
> > >>>>> true.
> > >>>>> > > > However,
> > >>>>> > > > >> all
> > >>>>> > > > >> > >> the
> > >>>>> > > > >> > >> > stat files are populated for all
experiments. In
> the
> > >>>>> case of
> > >>>>> > > > ozone
> > >>>>> > > > >> you
> > >>>>> > > > >> > >> see
> > >>>>> > > > >> > >> > something different. I find that ozone tends
to
> > result
> > >>>>> in
> > >>>>> > > > populated
> > >>>>> > > > >> > >> plots
> > >>>>> > > > >> > >> > but with intermittent reporting different
than
> PMTF -
> > >>>>> see
> > >>>>> > > below:
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls
-ltrtotal
> > >>>>> > 6144-rw-r--r-- 1
> > >>>>> > > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 54318 Oct 6 10:51
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 54305 Oct 7 10:49
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 62179 Oct 13 13:25
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 63811 Oct 19 12:18
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 60112 Oct 20 12:21
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 60165 Oct 21 12:33
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 60160 Oct 22 12:21
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 60056 Oct 23 12:40
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 58517 Oct 24 13:31
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 58062 Oct 31 14:17
> > >>>>> > > > >> > >> >
> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
> > >>>>> > > > >> > Edward.Strobach
> > >>>>> > > > >> > >> > emcmodel 58119 Nov 1 14:19
> > >>>>> > > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > Similar issues have been found for other
fields,
> > >>>>> cycles, and
> > >>>>> > > > >> regions
> > >>>>> > > > >> > >> > including verification being done for
meteorology.
> > >>>>> > > > >> > >> > I can't seem to reason this intermittency
and
> > >>>>> inconsistency
> > >>>>> > > > between
> > >>>>> > > > >> > >> > reporting since the xml files are correct
and I'm
> > >>>>> using the
> > >>>>> > > same
> > >>>>> > > > >> > >> procedure
> > >>>>> > > > >> > >> > that I've always used. I do realize that I
need to
> > >>>>> start
> > >>>>> > > > creating
> > >>>>> > > > >> a
> > >>>>> > > > >> > >> better
> > >>>>> > > > >> > >> > way to manage all these files that are being
> > produced,
> > >>>>> both
> > >>>>> > in
> > >>>>> > > > >> terms
> > >>>>> > > > >> > of
> > >>>>> > > > >> > >> > data/plots and xml files.
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > I ran across a rather large directory:
> > >>>>> > > > >> > >> > ls
> > >>>>> > > > >>
> > >>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > >>>>> > > > >> > |
> > >>>>> > > > >> > >> wc
> > >>>>> > > > >> > >> > -w
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > That has 71,398 xml files in there. But I
see
> > >>>>> timestamps in
> > >>>>> > > those
> > >>>>> > > > >> > >> filenames
> > >>>>> > > > >> > >> > from 20200921 to 20200925.
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > Is it the case that this stopped working on
> 20200925?
> > >>>>> Or are
> > >>>>> > > > those
> > >>>>> > > > >> > dated
> > >>>>> > > > >> > >> > filenames remnants from older work?
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > So it hasn't been working for over a month.
Is that
> > >>>>> right?
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > There are a lot of files and I need to begin
> managing
> > >>>>> that
> > >>>>> > > > >> better. I
> > >>>>> > > > >> > >> think
> > >>>>> > > > >> > >> > during that time there was a machine switch
so it
> > >>>>> stopped on
> > >>>>> > > that
> > >>>>> > > > >> day.
> > >>>>> > > > >> > >> > I've been able to produce plots even now,
but it's
> > not
> > >>>>> > > consistent
> > >>>>> > > > >> in
> > >>>>> > > > >> > >> > reporting while some regions generate plots
and
> > others
> > >>>>> do
> > >>>>> > not.
> > >>>>> > > > >> I've
> > >>>>> > > > >> > >> > reduced the level processing in hopes that I
don't
> > hit
> > >>>>> a walk
> > >>>>> > > > clock
> > >>>>> > > > >> > >> issue
> > >>>>> > > > >> > >> > since I'm submitting batch jobs. However,
I've run
> > >>>>> this step
> > >>>>> > > > >> outside
> > >>>>> > > > >> > of
> > >>>>> > > > >> > >> > batch and it always seems to work. This
issue seems
> > to
> > >>>>> happen
> > >>>>> > > > with
> > >>>>> > > > >> > batch
> > >>>>> > > > >> > >> > and it is not at all clear why. Let me know
what
> > else
> > >>>>> you
> > >>>>> > need
> > >>>>> > > > >> from
> > >>>>> > > > >> > me
> > >>>>> > > > >> > >> to
> > >>>>> > > > >> > >> > better coordinate what I would need to do
next.
> > >>>>> Thanks.
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley
Gotway
> via
> > >>>>> RT <
> > >>>>> > > > >> > >> > met_help at ucar.edu>
> > >>>>> > > > >> > >> > wrote:
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > > Hi Ed,
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > > Sorry for the delay. I took a look on
wcoss
> today.
> > >>>>> You
> > >>>>> > > describe
> > >>>>> > > > >> that
> > >>>>> > > > >> > >> your
> > >>>>> > > > >> > >> > > plotting script has stopped working but
you're
> not
> > >>>>> sure
> > >>>>> > why.
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > > I ran across a rather large directory:
> > >>>>> > > > >> > >> > > ls
> > >>>>> > > > >> >
> > >>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
> > >>>>> > > > >> > >> |
> > >>>>> > > > >> > >> > wc
> > >>>>> > > > >> > >> > > -w
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > > That has 71,398 xml files in there. But I
see
> > >>>>> timestamps in
> > >>>>> > > > those
> > >>>>> > > > >> > >> > filenames
> > >>>>> > > > >> > >> > > from 20200921 to 20200925.
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > > Is it the case that this stopped working
on
> > >>>>> 20200925? Or
> > >>>>> > are
> > >>>>> > > > >> those
> > >>>>> > > > >> > >> dated
> > >>>>> > > > >> > >> > > filenames remnants from older work?
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > > So it hasn't been working for over a
month. Is
> that
> > >>>>> right?
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > > John
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT
System
> itself
> > >>>>> via RT
> > >>>>> > <
> > >>>>> > > > >> > >> > > met_help at ucar.edu> wrote:
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > > >
> > >>>>> > > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283
was
> acted
> > >>>>> upon.
> > >>>>> > > > >> > >> > > > Transaction: Given to johnhg (John
Halley
> Gotway)
> > >>>>> by
> > >>>>> > > > RT_System
> > >>>>> > > > >> > >> > > > Queue: met_help
> > >>>>> > > > >> > >> > > > Subject: plots no longer being
generated
> > >>>>> > > > >> > >> > > > Owner: johnhg
> > >>>>> > > > >> > >> > > > Requestors: edward.strobach at noaa.gov
> > >>>>> > > > >> > >> > > > Status: new
> > >>>>> > > > >> > >> > > > Ticket <URL:
> > >>>>> > > > >> > >>
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > > >
> > >>>>> > > > >> > >> > > >
> > >>>>> > > > >> > >> > > > This transaction appears to have no
content
> > >>>>> > > > >> > >> > > >
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> > >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> > --
> > >>>>> > > > >> > >> > Edward Strobach
> > >>>>> > > > >> > >> > EMC/NCEP/NWS/
> > >>>>> > > > >> > >> > IMSG Contractor
> > >>>>> > > > >> > >> > Cubicle#: 2029
> > >>>>> > > > >> > >> > 301-683-3717
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >> >
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >>
> > >>>>> > > > >> > >
> > >>>>> > > > >> > > --
> > >>>>> > > > >> > > Edward Strobach
> > >>>>> > > > >> > > EMC/NCEP/NWS/
> > >>>>> > > > >> > > IMSG Contractor
> > >>>>> > > > >> > > Cubicle#: 2029
> > >>>>> > > > >> > > 301-683-3717
> > >>>>> > > > >> > >
> > >>>>> > > > >> >
> > >>>>> > > > >> >
> > >>>>> > > > >> > --
> > >>>>> > > > >> > Edward Strobach
> > >>>>> > > > >> > EMC/NCEP/NWS/
> > >>>>> > > > >> > IMSG Contractor
> > >>>>> > > > >> > Cubicle#: 2029
> > >>>>> > > > >> > 301-683-3717
> > >>>>> > > > >> >
> > >>>>> > > > >> >
> > >>>>> > > > >>
> > >>>>> > > > >>
> > >>>>> > > > >
> > >>>>> > > > > --
> > >>>>> > > > > Edward Strobach
> > >>>>> > > > > EMC/NCEP/NWS/
> > >>>>> > > > > IMSG Contractor
> > >>>>> > > > > Cubicle#: 2029
> > >>>>> > > > > 301-683-3717
> > >>>>> > > > >
> > >>>>> > > >
> > >>>>> > > >
> > >>>>> > > > --
> > >>>>> > > > Edward Strobach
> > >>>>> > > > EMC/NCEP/NWS/
> > >>>>> > > > IMSG Contractor
> > >>>>> > > > Cubicle#: 2029
> > >>>>> > > > 301-683-3717
> > >>>>> > > >
> > >>>>> > > >
> > >>>>> > >
> > >>>>> > >
> > >>>>> >
> > >>>>> > --
> > >>>>> > Edward Strobach
> > >>>>> > EMC/NCEP/NWS/
> > >>>>> > IMSG Contractor
> > >>>>> > Cubicle#: 2029
> > >>>>> > 301-683-3717
> > >>>>> >
> > >>>>> >
> > >>>>>
> > >>>>>
> > >>>>
> > >>>> --
> > >>>> Edward Strobach
> > >>>> EMC/NCEP/NWS/
> > >>>> IMSG Contractor
> > >>>> Cubicle#: 2029
> > >>>> 301-683-3717
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Edward Strobach
> > >>> EMC/NCEP/NWS/
> > >>> IMSG Contractor
> > >>> Cubicle#: 2029
> > >>> 301-683-3717
> > >>>
> > >>
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Wed Nov 11 10:19:28 2020
Hi John,
Just a brief update..
I changed my set-up to reflect the changes you recommended. A sample
XML
file is created - refer to the *"XML_adjusted*" file attached below -
when
running Plot_XML_Builder_v2.py. The file that is submitted via batch
-
i.e. "*XML_Build_Plot_Create_Step"* - is also attached and creates 8
different XML files corresponding to the stat variable (FBAR_OBAR and
RMSE), the independent variable (fcst_lead v. fcst_valid_beg), and
forecast
variable (PMTF and OZCON). I find that the batch job is successful if
I do
not list more than one name under vx_mask with an underscore. In
other
words, if you look at the vx_mask list in the *"XML_adjusted*" file,
you
will see Bare_Tundra followed by Crop_Grassland. I tried removing
Crop_Grassland and ran it again but received an error related to
Crop_Woodland (the second vx_mask variable within the list that has an
underscore). The pasted output below shows the error message for the
second attempt. The first attempt would be the same but with
Crop_Grassland instead.
CALLING: scp
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls/Plot_Time_Series_OZCON_fcst_valid_beg_FBAR_OBAR_Aggregate_202011.xml
edward.strobach at 205.156.8.85:~
---- MVBatch ----
input file:
Plot_Time_Series_OZCON_fcst_valid_beg_FBAR_OBAR_Aggregate_202011.xml
** ERROR: String Crop_Woodland includes SQL unsafe word AND or
BETWEEN
---- MVBatch Done ----
Good new though! All the plots did save - at least for PMTF; haven't
checked OZCON yet - when submitting the batch job for only
geographical
regions, not dominant land use masks. This is a step in the right
direction. I'm not sure what to make out for the error above but
wonder if
you know of a workaround.
Some recommendations:
I don't think I can currently list a bunch of stat variables or
fcst_vars
under <dep1>. The example below shows that FBAR and OBAR are part of
a
list that is used for one plot. I wouldn't be able to list RMSE after
that, at least I don't think I could. If we were to use FBAR_OBAR
instead
of having them separate, then one might be able to list several <stat>
variables in sequence to produce plots for different stat types.
Similarly, one might be able to list several fcst_vars instead of just
one. I would propose the following changes if you think it would be
helpful and worth pursuing.
ORIGINAL:
<dep>
<dep1>
<fcst_var name="OZCON">
<stat>FBAR</stat>
<stat>OBAR</stat>
</fcst_var>
</dep1>
<dep2/>
</dep>
NEW:
<dep>
<dep1>
<fcst_var name="OZCON">
<stat>FBAR_OBAR</stat>
<stat>RMSE</stat>
</fcst_var>
<fcst_var name="PMTF">
<stat>FBAR_OBAR</stat>
<stat>RMSE</stat>
</fcst_var>
</dep1>
<dep2/>
</dep>
If the above formulation was incorporated then I would only produce
two XML
files corresponding to two independent variables: one fcst_lead and
one
for fcst_valid_beg. This may resolve some of the same batch issues
that I
was having earlier on since there were so many simultaneous plots
being
processed and stored using different calls.
On Mon, Nov 9, 2020 at 2:09 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> I am pleased about being able to do it this way. It seems like a
better
> approach. However, It's unclear why, when running the same command
to
> process the same XML file, that files are not always being copied
over.
> From my earlier email you can see that sometimes 2 files are copied
over,
> sometimes 4, and sometimes only 1. I was hoping all 8 would be
copied
> over. This is kind of weird and I'm not sure how one would go about
this.
>
> On Mon, Nov 9, 2020 at 12:47 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Ed,
>>
>> Yes, there's lots of moving parts here.
>> (1) Setting up the METviewer XML to run 1 or more plots.
>> (2) Running that XML remotely to generate plots.
>> (3) Transferring the resulting image files back.
>>
>> I just wanted to make sure you know about the options for (1). And
I'm
>> really glad to hear that you were able to see how to generate
multiple
>> plots from a single METviewer XML!
>> I'd say the rest of the details about managing the plotting/image
transfer
>> logic is really up to you and your colleagues. If I were setting
this up,
>> I
>> would consider having each batch job write to a single output
directory,
>> and then recursively transfer back that directory. That seems
easier than
>> dealing with lots of individual files. But it's really to you guys.
>>
>> There is another detail about the METviewer plot XML that we
haven't
>> talked
>> about yet, and that's plot "inheritance". In a single XML, you can
define
>> multiple <plot> tags and have one <plot> inherit settings from a
previous
>> one. I took this to the extreme for one project, setting up a
pretty
>> complex XML that made like 6 different plot types using multiple
levels of
>> inheritance. In general, I'd advise against that. While it's cool
that it
>> works, it makes for complex logic that's difficult to maintain.
Instead,
>> I'd just look at any "for" loops you have in your plotting script
and move
>> as many as are easy and feasible into the XML directly... without
going
>> crazy! That would be like init hours, masking regions, and perhaps
>> statistic types. That should greatly reduce the number of
individual calls
>> to mv_batch.sh and hopefully provide for more stable results.
>>
>> But if you ever want to learn about plot inheritance, I'm happy to
help
>> you
>> with that too.
>>
>> John
>>
>> On Mon, Nov 9, 2020 at 8:02 AM Edward Strobach - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>> >
>> > FYI, when running test_v2.sh again after running it the first
time only
>> one
>> > plot was copied over:
>> > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>> > 100%
57KB
>> > 19.9MB/s 00:00
>> >
>> > On Mon, Nov 9, 2020 at 10:00 AM Edward Strobach - NOAA Affiliate
<
>> > edward.strobach at noaa.gov> wrote:
>> >
>> > > I did run some additional tests outside the main test*sh
script.
>> > > Specifically I ran the following line several times to see what
plots
>> > would
>> > > be copied over to WCOSS:
>> > > bash
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>> > > edward.strobach
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls/Test.xml
>> > >
>> > > The first time around 3 plots were copied over:
>> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
>> > > 100% 51KB 18.6MB/s 00:00
>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
>> > > 100% 58KB 12.0MB/s 00:00
>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>> > > 100% 57KB 2.8MB/s 00:00
>> > >
>> > > The second time 2 plots were copied over:
>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
>> > > 100% 58KB 2.6MB/s 00:00
>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>> > > 100% 57KB 6.2MB/s 00:00
>> > >
>> > > The third time 3 plots were copied over:
>> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
>> > >
100%
>> 51KB
>> > > 2.4MB/s 00:00
>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
>> > >
100%
>> 58KB
>> > > 6.1MB/s 00:00
>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>> > >
100%
>> 57KB
>> > > 11.1MB/s 00:00
>> > >
>> > > The order of init_hour and vx_mask is as follows:
>> > > <06>
>> > > <12>
>> > > and
>> > > <Mixed_Forest>
>> > > <FULL>
>> > > <EAST>
>> > > <WEST>
>> > >
>> > > It does appear that the latest plots are copied over in the
process
>> (i.e.
>> > > if two, then the last two; if three, then the last three).
>> > > I don't really have a feel for why two files are copied over
for some
>> > > instances and 3 for other instances. Lastly, I ran test_v2.sh
again,
>> and
>> > > this time it copied over 4 plots. See below:
>> > >
>> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t06z.png
>> > >
100%
>> 53KB
>> > > 12.7MB/s 00:00
>> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
>> > >
100%
>> 51KB
>> > > 9.7MB/s 00:00
>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
>> > >
100%
>> 58KB
>> > > 11.3MB/s 00:00
>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>> > >
100%
>> 57KB
>> > > 11.4MB/s 00:00
>> > >
>> > > This is very bizarre to me.
>> > >
>> > >
>> > >
>> > > On Mon, Nov 9, 2020 at 9:26 AM Edward Strobach - NOAA Affiliate
<
>> > > edward.strobach at noaa.gov> wrote:
>> > >
>> > >> Actually, I don't think I can do that. FBAR and OBAR are
split and
>> > >> adding RMSE as a third line might confuse the process.
>> > >>
>> > >> I was able to run a test with mixed success. I was successful
at
>> > >> generating 8 plots (4 regions for 2 initial hours). The
regions were
>> > >> Mixed_Forest, FULL, EAST, and WEST while the initial hours
were 06
>> and
>> > 12.
>> > >> The test ran well with indications that all plots were
generated and
>> > stored
>> > >> here: Created plot
>> > >>
>> >
>> /data/mv_data/edward.strobach/plots/PMTF_Mixed_Forest_112020_2020-
11-01_2020-11-06_t06z.png
>> > >>
>> > >> However, I was not successful at saving the plots on Dell.
Only the
>> last
>> > >> plot was saved here:
>> > >>
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011.
>> > >>
>> > >> It seems that in my bash script the line that produces the
plots that
>> > are
>> > >> then stored in the above directory must be done in a loop
otherwise
>> it
>> > will
>> > >> only move the last plot over as suggested by the fact that
only the
>> plot
>> > >> featuring WEST at 12z was moved over - the last plot of the
series of
>> > plots
>> > >> generated. I suppose I could create logic that extracts the
>> > region/initial
>> > >> hour information and construct a two part loop to move over
all
>> plots?
>> > >>
>> > >> On Mon, Nov 9, 2020 at 8:19 AM Edward Strobach - NOAA
Affiliate <
>> > >> edward.strobach at noaa.gov> wrote:
>> > >>
>> > >>> Just a follow-up on one other item..
>> > >>>
>> > >>> Can also list multiple stat types in the same way that I
specify the
>> > >>> init_hour and vx_mask? In other words, can I do the
following:
>> > >>>
>> > >>> <dep1>
>> > >>> <fcst_var name="PMTF">
>> > >>> <stat>FBAR_OBAR</stat>
>> > >>> <stat>RMSE</stat>
>> > >>> </fcst_var>
>> > >>> </dep1>
>> > >>>
>> > >>> I know that the "set" block is not being used here.
>> > >>>
>> > >>> On Mon, Nov 9, 2020 at 7:41 AM Edward Strobach - NOAA
Affiliate <
>> > >>> edward.strobach at noaa.gov> wrote:
>> > >>>
>> > >>>> Thanks John. Let me try that and I'll let you know how it
goes.
>> > >>>>
>> > >>>> On Fri, Nov 6, 2020 at 10:53 AM John Halley Gotway via RT <
>> > >>>> met_help at ucar.edu> wrote:
>> > >>>>
>> > >>>>> Ed,
>> > >>>>>
>> > >>>>> You're right about the METviewer GUI grouping multiple
selections
>> > >>>>> together.
>> > >>>>> That looks like this in the XML using the <set> tag:
>> > >>>>> <field equalize="false" name="vx_mask">
>> > >>>>> <set name="vx_mask_0">
>> > >>>>> <val>FULL</val>
>> > >>>>> </set>
>> > >>>>> </field>
>> > >>>>>
>> > >>>>> But in the updates I sent, I remove the <set> tag:
>> > >>>>> <field equalize="false" name="vx_mask">
>> > >>>>> <val>FULL</val>
>> > >>>>> <val>CONUS</val>
>> > >>>>> <val>EAST</val>
>> > >>>>> <val>WEST</val>
>> > >>>>> </field>
>> > >>>>> So instead of processing this as 1 group of 4 values, it'll
>> process
>> > >>>>> the 4
>> > >>>>> values separately.
>> > >>>>>
>> > >>>>> The current vx_mask value is referenced using {vx_mask}
syntax.
>> And I
>> > >>>>> included examples of using that in the output file names
and
>> titles
>> > in
>> > >>>>> my
>> > >>>>> previous email:
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> >
>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> >
>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>> > >>>>>
>> > >>>>>
>> >
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>> > >>>>>
>> > >>>>> <title>Average PMTF (October 2020; {init_hour}z cycle;
{vx_mask}
>> > >>>>> Domain)</title>
>> > >>>>>
>> > >>>>> John
>> > >>>>>
>> > >>>>> On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA
Affiliate
>> via
>> > RT
>> > >>>>> <
>> > >>>>> met_help at ucar.edu> wrote:
>> > >>>>>
>> > >>>>> >
>> > >>>>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>> > >>>>> >
>> > >>>>> > Hi John,
>> > >>>>> >
>> > >>>>> > This definitely helps. I never thought about it this way
>> because I
>> > >>>>> > assumed, based on my experience with the interactive MET
GUI,
>> that
>> > I
>> > >>>>> could
>> > >>>>> > only generate a single plot regardless of how I specified
my
>> > >>>>> settings in
>> > >>>>> > the XML. For example, if I listed EAST and WEST under
vx_mask,
>> > then
>> > >>>>> my
>> > >>>>> > thinking would be that both those regions would be
grouped and a
>> > >>>>> single
>> > >>>>> > plot created. I guess I was wrong on that. The concern
I still
>> > >>>>> have,
>> > >>>>> > however, is that the plot title and plot name - the png
file -
>> is
>> > >>>>> also set
>> > >>>>> > in the XML file. Even after adding the list you gave, I
can't
>> see
>> > >>>>> how
>> > >>>>> > unique file names will be created. Is that true?
>> > >>>>> >
>> > >>>>> > On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT
<
>> > >>>>> > met_help at ucar.edu>
>> > >>>>> > wrote:
>> > >>>>> >
>> > >>>>> > > Ed,
>> > >>>>> > >
>> > >>>>> > > Thanks for sending examples. My goal was to use
METviewer's
>> XML
>> > >>>>> upload
>> > >>>>> > > functionality but I'm not able to upload a pdf version
of the
>> xml
>> > >>>>> file.
>> > >>>>> > >
>> > >>>>> > > I did spend some time trying to replicate this exact
plot with
>> > >>>>> METviewer
>> > >>>>> > > at:
>> > >>>>> > > https://metviewer.nws.noaa.gov/metviewer1.jsp
>> > >>>>> > >
>> > >>>>> > > But I was never quite successful. So I made just a
similar
>> > version
>> > >>>>> > instead.
>> > >>>>> > > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
>> > >>>>> > >
>> > >>>>> > > This plots FBAR and OBAR for 6 different models for the
month
>> of
>> > >>>>> October
>> > >>>>> > > for all 06Z initializations over the FULL model domain.
>> > >>>>> > >
>> > >>>>> > > I wanted to show you how to modify this for batch
mode... and
>> > then
>> > >>>>> adapt
>> > >>>>> > it
>> > >>>>> > > to make multiple plots. But I'm not able to make batch
plots
>> on
>> > >>>>> wcoss
>> > >>>>> > using
>> > >>>>> > > the AWS instance. I don't think I have the right
permissions
>> to
>> > do
>> > >>>>> so.
>> > >>>>> > >
>> > >>>>> > > So let me just say this. Generally, you just update the
list
>> of
>> > >>>>> items in
>> > >>>>> > > plot_fix section.
>> > >>>>> > >
>> > >>>>> > > Original from METviewer with 1 init hour and 1 mask:
>> > >>>>> > >
>> > >>>>> > > <field equalize="false" name="vx_mask">
>> > >>>>> > > <set name="vx_mask_0">
>> > >>>>> > > <val>FULL</val>
>> > >>>>> > > </set>
>> > >>>>> > > </field>
>> > >>>>> > > <field equalize="false" name="init_hour">
>> > >>>>> > > <set name="init_hour_1">
>> > >>>>> > > <val>06</val>
>> > >>>>> > > </set>
>> > >>>>> > > </field>
>> > >>>>> > >
>> > >>>>> > > Change to 2 init hours and 4 masks:
>> > >>>>> > >
>> > >>>>> > > <field equalize="false" name="vx_mask">
>> > >>>>> > > <val>FULL</val>
>> > >>>>> > > <val>CONUS</val>
>> > >>>>> > > <val>EAST</val>
>> > >>>>> > > <val>WEST</val>
>> > >>>>> > > </field>
>> > >>>>> > > <field equalize="false" name="init_hour">
>> > >>>>> > > <val>06</val>
>> > >>>>> > > <val>12</val>
>> > >>>>> > > </field>
>> > >>>>> > >
>> > >>>>> > > And then reference those values in the naming of the
titles
>> and
>> > >>>>> output
>> > >>>>> > > files:
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> >
>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> >
>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>>
>> >
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>> > >>>>> > >
>> > >>>>> > > <title>Average PMTF (October 2020; {init_hour}z cycle;
>> {vx_mask}
>> > >>>>> > > Domain)</title>
>> > >>>>> > >
>> > >>>>> > > So that's the general idea. And when you submit this to
>> mv_batch,
>> > >>>>> it'll
>> > >>>>> > > make 8 plots instead of 1.
>> > >>>>> > > And you can see how you'd easily expand this to many
plots.
>> > >>>>> > >
>> > >>>>> > > But I worry that this may not be sufficient information
to get
>> > you
>> > >>>>> going.
>> > >>>>> > >
>> > >>>>> > > John
>> > >>>>> > >
>> > >>>>> > > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA
>> Affiliate
>> > >>>>> via RT <
>> > >>>>> > > met_help at ucar.edu> wrote:
>> > >>>>> > >
>> > >>>>> > > >
>> > >>>>> > > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>> > >
>> > >>>>> > > >
>> > >>>>> > > > Hi John,
>> > >>>>> > > >
>> > >>>>> > > > I think I was able to cover your request okay. Below
are 4
>> > >>>>> attachments
>> > >>>>> > > > that include the test script that was used to
generate the
>> > >>>>> plots, a
>> > >>>>> > > sample
>> > >>>>> > > > XML file, and the two plots that were generated using
the
>> test
>> > >>>>> run
>> > >>>>> > > script.
>> > >>>>> > > > These plots were not created previously using the
batch job
>> but
>> > >>>>> uses
>> > >>>>> > the
>> > >>>>> > > > same exact logic as the original run script. Thanks
again
>> for
>> > >>>>> working
>> > >>>>> > > with
>> > >>>>> > > > me on this. Please let me know if you need anything
else or
>> > >>>>> would like
>> > >>>>> > > me
>> > >>>>> > > > to test something.
>> > >>>>> > > >
>> > >>>>> > > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach - NOAA
>> > Affiliate <
>> > >>>>> > > > edward.strobach at noaa.gov> wrote:
>> > >>>>> > > >
>> > >>>>> > > > > Ed,
>> > >>>>> > > > >
>> > >>>>> > > > > Image files showing up in an unexpected output
directory
>> > >>>>> likely point
>> > >>>>> > > to
>> > >>>>> > > > a
>> > >>>>> > > > > problem in the logic of the processing scripts.
It's
>> pretty
>> > >>>>> unlikely
>> > >>>>> > > that
>> > >>>>> > > > > this behavior is caused by the METviewer code
itself. But
>> > I've
>> > >>>>> > > definitely
>> > >>>>> > > > > been surprised before!
>> > >>>>> > > > >
>> > >>>>> > > > > I tested the logic outside of a batch job and it
works
>> > fine. I
>> > >>>>> > suspect
>> > >>>>> > > > > that the issuance of another batch job while the
other
>> one is
>> > >>>>> getting
>> > >>>>> > > > > processed is causing some confusion in the
processing.
>> These
>> > >>>>> issues
>> > >>>>> > > > don't
>> > >>>>> > > > > happen everyday and appear more sporadic than
anything.
>> > >>>>> > > > >
>> > >>>>> > > > > So what we really need here is an example of taking
a plot
>> > >>>>> from the
>> > >>>>> > > > > METviewer GUI and adapting it to make it generate
multiple
>> > >>>>> plots. I
>> > >>>>> > did
>> > >>>>> > > > go
>> > >>>>> > > > > through examples of doing this in previous
tutorials, but
>> we
>> > >>>>> didn't
>> > >>>>> > > start
>> > >>>>> > > > > recording them until last summer. I checked the
July 2019
>> NRL
>> > >>>>> > tutorial
>> > >>>>> > > > > videos and don't see it there.
>> > >>>>> > > > >
>> > >>>>> > > > > Is there something you need from me to help on
this? Just
>> > let
>> > >>>>> me
>> > >>>>> > know
>> > >>>>> > > > >
>> > >>>>> > > > > We've begun recording training videos for common
topics
>> like
>> > >>>>> this.
>> > >>>>> > But
>> > >>>>> > > we
>> > >>>>> > > > > only have one for METviewer so far:
>> > >>>>> > > > >
>> > >>>>> > >
>> > >>>>>
>> >
>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>> > >>>>> > > > >
>> > >>>>> > > > > But this definitely seems like a good candidate for
a new
>> > >>>>> training
>> > >>>>> > > video.
>> > >>>>> > > > >
>> > >>>>> > > > > For now though, how about this...
>> > >>>>> > > > > - You use METviewer to manually make a single plot,
>> > formatting
>> > >>>>> it how
>> > >>>>> > > > you'd
>> > >>>>> > > > > like.
>> > >>>>> > > > > - You send me your sample plot and corresponding
XML file.
>> > >>>>> > > > > - I'll tweak it to generate many plots via batch
instead
>> of
>> > >>>>> just one.
>> > >>>>> > > > > - And then you can continue tweaking to add more
>> > permutations.
>> > >>>>> > > > >
>> > >>>>> > > > > Okay, I'll work on that now. Thanks.
>> > >>>>> > > > >
>> > >>>>> > > > > John
>> > >>>>> > > > >
>> > >>>>> > > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway
via RT
>> <
>> > >>>>> > > > > met_help at ucar.edu> wrote:
>> > >>>>> > > > >
>> > >>>>> > > > >> Ed,
>> > >>>>> > > > >>
>> > >>>>> > > > >> Image files showing up in an unexpected output
directory
>> > >>>>> likely
>> > >>>>> > point
>> > >>>>> > > > to a
>> > >>>>> > > > >> problem in the logic of the processing scripts.
It's
>> pretty
>> > >>>>> unlikely
>> > >>>>> > > > that
>> > >>>>> > > > >> this behavior is caused by the METviewer code
itself. But
>> > I've
>> > >>>>> > > > definitely
>> > >>>>> > > > >> been surprised before!
>> > >>>>> > > > >>
>> > >>>>> > > > >> So what we really need here is an example of
taking a
>> plot
>> > >>>>> from the
>> > >>>>> > > > >> METviewer GUI and adapting it to make it generate
>> multiple
>> > >>>>> plots. I
>> > >>>>> > > did
>> > >>>>> > > > go
>> > >>>>> > > > >> through examples of doing this in previous
tutorials,
>> but we
>> > >>>>> didn't
>> > >>>>> > > > start
>> > >>>>> > > > >> recording them until last summer. I checked the
July 2019
>> > NRL
>> > >>>>> > tutorial
>> > >>>>> > > > >> videos and don't see it there.
>> > >>>>> > > > >>
>> > >>>>> > > > >> We've begun recording training videos for common
topics
>> like
>> > >>>>> this.
>> > >>>>> > But
>> > >>>>> > > > we
>> > >>>>> > > > >> only have one for METviewer so far:
>> > >>>>> > > > >>
>> > >>>>> > > >
>> > >>>>> >
>> > >>>>>
>> >
>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>> > >>>>> > > > >>
>> > >>>>> > > > >> But this definitely seems like a good candidate
for a new
>> > >>>>> training
>> > >>>>> > > > video.
>> > >>>>> > > > >>
>> > >>>>> > > > >> For now though, how about this...
>> > >>>>> > > > >> - You use METviewer to manually make a single
plot,
>> > >>>>> formatting it
>> > >>>>> > how
>> > >>>>> > > > >> you'd
>> > >>>>> > > > >> like.
>> > >>>>> > > > >> - You send me your sample plot and corresponding
XML
>> file.
>> > >>>>> > > > >> - I'll tweak it to generate many plots via batch
instead
>> of
>> > >>>>> just
>> > >>>>> > one.
>> > >>>>> > > > >> - And then you can continue tweaking to add more
>> > permutations.
>> > >>>>> > > > >>
>> > >>>>> > > > >> John
>> > >>>>> > > > >>
>> > >>>>> > > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach -
NOAA
>> > >>>>> Affiliate via
>> > >>>>> > > RT <
>> > >>>>> > > > >> met_help at ucar.edu> wrote:
>> > >>>>> > > > >>
>> > >>>>> > > > >> >
>> > >>>>> > > > >> > <URL:
>> > >>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>> > >>>>> > > > >> >
>> > >>>>> > > > >> > Oh, and one more thing..
>> > >>>>> > > > >> >
>> > >>>>> > > > >> > The green text that didn't show up was meant to
>> highlight
>> > >>>>> the fact
>> > >>>>> > > > that
>> > >>>>> > > > >> > sometimes plots that shouldn't be stored in
certain
>> > >>>>> directories
>> > >>>>> > end
>> > >>>>> > > up
>> > >>>>> > > > >> > there. The pasted example below shows one
instance
>> where
>> > >>>>> OZMAX*
>> > >>>>> > was
>> > >>>>> > > > >> stored
>> > >>>>> > > > >> > where TMP* should have been stored.
>> > >>>>> > > > >> >
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct
5
>> 10:52
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct
6
>> 10:52
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct
7
>> 10:00
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct
13
>> 13:49
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct
14
>> 11:31
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct
19
>> 11:37
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct
20
>> 11:35
>> > >>>>> > > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct
21
>> 11:45
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct
22
>> 11:42
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct
23
>> 11:50
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct
24
>> 12:57
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct
31
>> 08:47
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov
2
>> 08:50
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov
3
>> 08:52
>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
>> > >>>>> > > > >> >
>> > >>>>> > > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach
- NOAA
>> > >>>>> Affiliate <
>> > >>>>> > > > >> > edward.strobach at noaa.gov> wrote:
>> > >>>>> > > > >> >
>> > >>>>> > > > >> > > Hi John,
>> > >>>>> > > > >> > >
>> > >>>>> > > > >> > > I know there are a lot of calls and speculate
that
>> this
>> > >>>>> could be
>> > >>>>> > > > part
>> > >>>>> > > > >> of
>> > >>>>> > > > >> > > the problem. I don't think that I'm following
best
>> > >>>>> practice by
>> > >>>>> > > > >> > approaching
>> > >>>>> > > > >> > > it in this way. Generating plots usually
works well
>> if
>> > I
>> > >>>>> > process
>> > >>>>> > > a
>> > >>>>> > > > >> > subset
>> > >>>>> > > > >> > > of the computations outside of batch
scripting. The
>> > >>>>> python
>> > >>>>> > script
>> > >>>>> > > > was
>> > >>>>> > > > >> > > given to me by Logan, but I have since made
>> significant
>> > >>>>> changes
>> > >>>>> > to
>> > >>>>> > > > it
>> > >>>>> > > > >> in
>> > >>>>> > > > >> > > order to dynamically fill in the xml file
based on
>> > >>>>> parameters
>> > >>>>> > set.
>> > >>>>> > > > >> The
>> > >>>>> > > > >> > > test*.sh script was also given to me. I
created the
>> > >>>>> > > > Time_Series*bsub
>> > >>>>> > > > >> as
>> > >>>>> > > > >> > a
>> > >>>>> > > > >> > > way to process a series of plots based on
cycle time,
>> > >>>>> region,
>> > >>>>> > > etc. I
>> > >>>>> > > > >> > think
>> > >>>>> > > > >> > > the first two scripts are solid and do well
for what
>> I
>> > >>>>> need to
>> > >>>>> > get
>> > >>>>> > > > >> done.
>> > >>>>> > > > >> > > It's the Time_Series*bsub script that bothers
me a
>> bit.
>> > >>>>> > > > >> > >
>> > >>>>> > > > >> > > I can't see how a single XML file can process
all
>> these
>> > >>>>> plots
>> > >>>>> > in a
>> > >>>>> > > > >> single
>> > >>>>> > > > >> > > setting based on what I've learned thus far.
How
>> would
>> > >>>>> one go
>> > >>>>> > > about
>> > >>>>> > > > >> > that?
>> > >>>>> > > > >> > > I would be open going in that direction if you
can
>> > >>>>> provide some
>> > >>>>> > > > >> guidance.
>> > >>>>> > > > >> > >
>> > >>>>> > > > >> > > Unfortunately, I haven't gotten a lot of
feedback or
>> > >>>>> dialogue
>> > >>>>> > > > exchange
>> > >>>>> > > > >> > > about what I've done so far at EMC with this
new
>> set-up.
>> > >>>>> This
>> > >>>>> > has
>> > >>>>> > > > >> also
>> > >>>>> > > > >> > > been part of the problem. I can't seem to get
them
>> to
>> > >>>>> focus so
>> > >>>>> > > that
>> > >>>>> > > > >> we
>> > >>>>> > > > >> > can
>> > >>>>> > > > >> > > prioritize. I know that I'm doing the best I
can.
>> If I
>> > >>>>> can
>> > >>>>> > learn
>> > >>>>> > > > >> how to
>> > >>>>> > > > >> > > go about this more efficiently it would be of
great
>> > >>>>> help. I can
>> > >>>>> > > > then
>> > >>>>> > > > >> > > expand this to other verification as venture
to other
>> > >>>>> projects
>> > >>>>> > in
>> > >>>>> > > > the
>> > >>>>> > > > >> > > future. I know you're busy with many things,
so I'm
>> > >>>>> willing to
>> > >>>>> > go
>> > >>>>> > > > >> along
>> > >>>>> > > > >> > > with your schedule as you find an opening.
>> > >>>>> > > > >> > >
>> > >>>>> > > > >> > > Thanks
>> > >>>>> > > > >> > >
>> > >>>>> > > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley
Gotway
>> via
>> > RT
>> > >>>>> <
>> > >>>>> > > > >> > > met_help at ucar.edu> wrote:
>> > >>>>> > > > >> > >
>> > >>>>> > > > >> > >> Ed,
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> It's really difficult for me to see an
obvious
>> problem
>> > >>>>> that's
>> > >>>>> > > > causing
>> > >>>>> > > > >> > the
>> > >>>>> > > > >> > >> behavior you describe.
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> Unfortunately, the red and green highlighting
you
>> > >>>>> describe
>> > >>>>> > > doesn't
>> > >>>>> > > > >> show
>> > >>>>> > > > >> > up
>> > >>>>> > > > >> > >> in the Gmail web client. There must be some
mail
>> system
>> > >>>>> > > > >> incompatibility
>> > >>>>> > > > >> > >> there.
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> Looking at your bsub script:
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> >
>> > >>>>> > > > >>
>> > >>>>> > > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> I am struck by the number of individual calls
you're
>> > >>>>> making to
>> > >>>>> > > > >> > METviewer:
>> > >>>>> > > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks
x 2
>> plot
>> > >>>>> types =
>> > >>>>> > 656
>> > >>>>> > > > >> calls
>> > >>>>> > > > >> > to
>> > >>>>> > > > >> > >> METviewer
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> When making plots via the METviewer GUI, it's
true
>> that
>> > >>>>> 1 XML
>> > >>>>> > = 1
>> > >>>>> > > > >> plot.
>> > >>>>> > > > >> > >> However, when making plots via the batch
engine, the
>> > >>>>> plot XML
>> > >>>>> > can
>> > >>>>> > > > be
>> > >>>>> > > > >> set
>> > >>>>> > > > >> > >> up
>> > >>>>> > > > >> > >> to create many, many plots. While I have no
direct
>> > proof
>> > >>>>> of
>> > >>>>> > this,
>> > >>>>> > > > it
>> > >>>>> > > > >> > might
>> > >>>>> > > > >> > >> be the case that executing 600+ batch jobs
for
>> > METviewer
>> > >>>>> is
>> > >>>>> > > leading
>> > >>>>> > > > >> to
>> > >>>>> > > > >> > >> trouble on the system. One thing to try would
be
>> > setting
>> > >>>>> up the
>> > >>>>> > > XML
>> > >>>>> > > > >> to
>> > >>>>> > > > >> > >> make
>> > >>>>> > > > >> > >> many, many more plots in each call. Then
check to
>> see
>> > if
>> > >>>>> > running
>> > >>>>> > > > >> > METviewer
>> > >>>>> > > > >> > >> in this way is more stable.
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> I don't think I fully appreciate all of the
logic
>> that
>> > >>>>> lives in
>> > >>>>> > > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py,
and
>> > >>>>> > > > >> test_Time_Series_AGG.sh.
>> > >>>>> > > > >> > >> However, it *might* be possible to construct
a
>> single
>> > >>>>> XML file
>> > >>>>> > > > which
>> > >>>>> > > > >> > >> produces all 656 plots.
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> Let me know what you think and how you'd like
to
>> > proceed.
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> Thanks,
>> > >>>>> > > > >> > >> John
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward
Strobach -
>> NOAA
>> > >>>>> Affiliate
>> > >>>>> > > via
>> > >>>>> > > > >> RT <
>> > >>>>> > > > >> > >> met_help at ucar.edu> wrote:
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > <URL:
>> > >>>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>> > >>>>> > > >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > Hi John,
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > No problem.
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > I'm including my responses below:
>> > >>>>> > > > >> > >> > Sorry for the delay. I took a look on wcoss
today.
>> > You
>> > >>>>> > describe
>> > >>>>> > > > >> that
>> > >>>>> > > > >> > >> your
>> > >>>>> > > > >> > >> > plotting script has stopped working but
you're not
>> > >>>>> sure why.
>> > >>>>> > > > >> > >> > It seems more intermittent. Take a look at
the
>> > example
>> > >>>>> > below.
>> > >>>>> > > > You
>> > >>>>> > > > >> > can
>> > >>>>> > > > >> > >> see
>> > >>>>> > > > >> > >> > that for some reason there are gaps in the
plots
>> > being
>> > >>>>> > > generated.
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel
34105 Oct
>> 4
>> > >>>>> 14:52
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 41781 Oct 5 15:15
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 56691 Oct 19 15:02
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 55548 Oct 20 16:08
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 55598 Oct 21 16:08
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 55599 Oct 22 16:09
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 55352 Oct 24 16:34
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 26175 Oct 31 08:20
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 26178 Nov 1 08:20
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 26160 Nov 3 08:22
>> > >>>>> > > > >> > >> >
>> PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 25354 Nov 3 08:22
>> > >>>>> > > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
>> > >>>>> > > > >> > >> > You can tell when a plot is successful by
looking
>> at
>> > >>>>> the file
>> > >>>>> > > > >> size. I
>> > >>>>> > > > >> > >> > highlighted an example of that in red.
However,
>> the
>> > >>>>> files
>> > >>>>> > > > showing
>> > >>>>> > > > >> > about
>> > >>>>> > > > >> > >> > 25000 are the ones where no line plots are
being
>> > >>>>> produced.
>> > >>>>> > You
>> > >>>>> > > > can
>> > >>>>> > > > >> > also
>> > >>>>> > > > >> > >> > see that stat files are generated everyday,
so as
>> > long
>> > >>>>> as I'm
>> > >>>>> > > > >> loading
>> > >>>>> > > > >> > >> the
>> > >>>>> > > > >> > >> > data, which I am, I should be able to use
the xml
>> > >>>>> files to
>> > >>>>> > > > generate
>> > >>>>> > > > >> > the
>> > >>>>> > > > >> > >> > plots. I'm attaching a directory list from
wcoss
>> > that
>> > >>>>> shows
>> > >>>>> > > the
>> > >>>>> > > > >> dates
>> > >>>>> > > > >> > >> > where data is produced.
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > *20190801 20190817 20200624 20200710
20200726
>> > >>>>> 20200812
>> > >>>>> > > > >> 20200828
>> > >>>>> > > > >> > >> > 20201003 2020101920190802 20190818
20200625
>> > >>>>> 20200711
>> > >>>>> > > > 20200727
>> > >>>>> > > > >> > >> > 20200813 20200829 20201004
2020102020190803
>> > >>>>> 20190819
>> > >>>>> > > > 20200626
>> > >>>>> > > > >> > >> > 20200712 20200728 20200814 20200830
20201005
>> > >>>>> > > > 2020102120190804
>> > >>>>> > > > >> > >> > 20190820 20200627 20200713 20200729
20200815
>> > >>>>> 20200831
>> > >>>>> > > > >> 20201006
>> > >>>>> > > > >> > >> > 2020102220190805 20190821 20200628
20200714
>> > >>>>> 20200730
>> > >>>>> > > > 20200816
>> > >>>>> > > > >> > >> > 20200921 20201007 2020102320190806
20190822
>> > >>>>> 20200629
>> > >>>>> > > > 20200715
>> > >>>>> > > > >> > >> > 20200731 20200817 20200922 20201008
>> > >>>>> 2020102420190807
>> > >>>>> > > > 20190823
>> > >>>>> > > > >> > >> > 20200630 20200716 20200801 20200818
20200923
>> > >>>>> 20201009
>> > >>>>> > > > >> > >> > 2020102520190808 20190824 20200701
20200717
>> > >>>>> 20200802
>> > >>>>> > > > 20200819
>> > >>>>> > > > >> > >> > 20200924 20201010 2020102620190809
20190825
>> > >>>>> 20200702
>> > >>>>> > > > 20200718
>> > >>>>> > > > >> > >> > 20200803 20200820 20200925 20201011
>> > >>>>> 2020102720190810
>> > >>>>> > > > 20190826
>> > >>>>> > > > >> > >> > 20200703 20200719 20200804 20200821
20200926
>> > >>>>> 20201012
>> > >>>>> > > > >> > >> > 2020102820190811 20190827 20200704
20200720
>> > >>>>> 20200805
>> > >>>>> > > > 20200822
>> > >>>>> > > > >> > >> > 20200927 20201013 2020102920190812
20190828
>> > >>>>> 20200705
>> > >>>>> > > > 20200721
>> > >>>>> > > > >> > >> > 20200806 20200823 20200928 20201014
>> > >>>>> 2020103020190813
>> > >>>>> > > > 20190829
>> > >>>>> > > > >> > >> > 20200706 20200722 20200807 20200824
20200929
>> > >>>>> 20201015
>> > >>>>> > > > >> > >> > 2020103120190814 20190830 20200707
20200723
>> > >>>>> 20200808
>> > >>>>> > > > 20200825
>> > >>>>> > > > >> > >> > 20200930 20201016 2020110120190815
20200622
>> > >>>>> 20200708
>> > >>>>> > > > 20200724
>> > >>>>> > > > >> > >> > 20200810 20200826 20201001 20201017
>> > >>>>> 2020110220190816
>> > >>>>> > > > 20200623
>> > >>>>> > > > >> > >> > 20200709 20200725 20200811 20200827
20201002
>> > >>>>> 20201018*
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > Furthermore, what's interesting is that I
have
>> mixed
>> > >>>>> success
>> > >>>>> > > with
>> > >>>>> > > > >> > other
>> > >>>>> > > > >> > >> > cases. Using the same field - PMTF - for
the same
>> > >>>>> cycle -
>> > >>>>> > the
>> > >>>>> > > > 06z
>> > >>>>> > > > >> > >> cycle -
>> > >>>>> > > > >> > >> > you can see that some of the dates where
data was
>> > >>>>> plotted for
>> > >>>>> > > > EAST
>> > >>>>> > > > >> did
>> > >>>>> > > > >> > >> not
>> > >>>>> > > > >> > >> > necessarily plot for FULL. This surprises
me
>> because
>> > >>>>> EAST
>> > >>>>> > is a
>> > >>>>> > > > >> subset
>> > >>>>> > > > >> > >> of
>> > >>>>> > > > >> > >> > FULL. See below:
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel
36647 Oct
>> 4
>> > >>>>> 14:54
>> > >>>>> > > > >> > >> >
>> PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 49039 Oct 5 15:17
>> > >>>>> > > > >> > >> >
>> PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 56791 Oct 20 16:09
>> > >>>>> > > > >> > >> >
>> PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 56834 Oct 21 16:09
>> > >>>>> > > > >> > >> >
>> PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 56921 Oct 22 16:11
>> > >>>>> > > > >> > >> >
>> PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 56518 Oct 24 16:36
>> > >>>>> > > > >> > >> >
>> PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 61187 Oct 31 08:21
>> > >>>>> > > > >> > >> >
>> PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 26475 Nov 2 08:22
>> > >>>>> > > > >> > >> >
>> PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r--
>> > 1
>> > >>>>> > > > >> Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 26455 Nov 3 08:23
>> > >>>>> > > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > What I find equally concerning as that
sometimes
>> > plots
>> > >>>>> are
>> > >>>>> > > > >> incorrectly
>> > >>>>> > > > >> > >> > stored in the wrong directory list as
highlighted
>> in
>> > >>>>> green
>> > >>>>> > > above.
>> > >>>>> > > > >> I've
>> > >>>>> > > > >> > >> > consulted the plot xml file for PMTF, for
>> instance,
>> > >>>>> and find
>> > >>>>> > > > >> nothing
>> > >>>>> > > > >> > >> > problematic. The only setting that might
pose an
>> > >>>>> issue when
>> > >>>>> > it
>> > >>>>> > > > >> comes
>> > >>>>> > > > >> > to
>> > >>>>> > > > >> > >> > generating line plots is when I set
event_equal to
>> > >>>>> true.
>> > >>>>> > > > However,
>> > >>>>> > > > >> all
>> > >>>>> > > > >> > >> the
>> > >>>>> > > > >> > >> > stat files are populated for all
experiments. In
>> the
>> > >>>>> case of
>> > >>>>> > > > ozone
>> > >>>>> > > > >> you
>> > >>>>> > > > >> > >> see
>> > >>>>> > > > >> > >> > something different. I find that ozone
tends to
>> > result
>> > >>>>> in
>> > >>>>> > > > populated
>> > >>>>> > > > >> > >> plots
>> > >>>>> > > > >> > >> > but with intermittent reporting different
than
>> PMTF -
>> > >>>>> see
>> > >>>>> > > below:
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls
-ltrtotal
>> > >>>>> > 6144-rw-r--r-- 1
>> > >>>>> > > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4 10:36
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 54318 Oct 6 10:51
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 54305 Oct 7 10:49
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 62179 Oct 13 13:25
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 63811 Oct 19 12:18
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 60112 Oct 20 12:21
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 60165 Oct 21 12:33
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 60160 Oct 22 12:21
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 60056 Oct 23 12:40
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 58517 Oct 24 13:31
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 58062 Oct 31 14:17
>> > >>>>> > > > >> > >> >
>> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
>> > >>>>> > > > >> > Edward.Strobach
>> > >>>>> > > > >> > >> > emcmodel 58119 Nov 1 14:19
>> > >>>>> > > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > Similar issues have been found for other
fields,
>> > >>>>> cycles, and
>> > >>>>> > > > >> regions
>> > >>>>> > > > >> > >> > including verification being done for
meteorology.
>> > >>>>> > > > >> > >> > I can't seem to reason this intermittency
and
>> > >>>>> inconsistency
>> > >>>>> > > > between
>> > >>>>> > > > >> > >> > reporting since the xml files are correct
and I'm
>> > >>>>> using the
>> > >>>>> > > same
>> > >>>>> > > > >> > >> procedure
>> > >>>>> > > > >> > >> > that I've always used. I do realize that I
need
>> to
>> > >>>>> start
>> > >>>>> > > > creating
>> > >>>>> > > > >> a
>> > >>>>> > > > >> > >> better
>> > >>>>> > > > >> > >> > way to manage all these files that are
being
>> > produced,
>> > >>>>> both
>> > >>>>> > in
>> > >>>>> > > > >> terms
>> > >>>>> > > > >> > of
>> > >>>>> > > > >> > >> > data/plots and xml files.
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > I ran across a rather large directory:
>> > >>>>> > > > >> > >> > ls
>> > >>>>> > > > >>
>> > >>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>> > >>>>> > > > >> > |
>> > >>>>> > > > >> > >> wc
>> > >>>>> > > > >> > >> > -w
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > That has 71,398 xml files in there. But I
see
>> > >>>>> timestamps in
>> > >>>>> > > those
>> > >>>>> > > > >> > >> filenames
>> > >>>>> > > > >> > >> > from 20200921 to 20200925.
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > Is it the case that this stopped working on
>> 20200925?
>> > >>>>> Or are
>> > >>>>> > > > those
>> > >>>>> > > > >> > dated
>> > >>>>> > > > >> > >> > filenames remnants from older work?
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > So it hasn't been working for over a month.
Is
>> that
>> > >>>>> right?
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > There are a lot of files and I need to
begin
>> managing
>> > >>>>> that
>> > >>>>> > > > >> better. I
>> > >>>>> > > > >> > >> think
>> > >>>>> > > > >> > >> > during that time there was a machine switch
so it
>> > >>>>> stopped on
>> > >>>>> > > that
>> > >>>>> > > > >> day.
>> > >>>>> > > > >> > >> > I've been able to produce plots even now,
but it's
>> > not
>> > >>>>> > > consistent
>> > >>>>> > > > >> in
>> > >>>>> > > > >> > >> > reporting while some regions generate plots
and
>> > others
>> > >>>>> do
>> > >>>>> > not.
>> > >>>>> > > > >> I've
>> > >>>>> > > > >> > >> > reduced the level processing in hopes that
I don't
>> > hit
>> > >>>>> a walk
>> > >>>>> > > > clock
>> > >>>>> > > > >> > >> issue
>> > >>>>> > > > >> > >> > since I'm submitting batch jobs. However,
I've
>> run
>> > >>>>> this step
>> > >>>>> > > > >> outside
>> > >>>>> > > > >> > of
>> > >>>>> > > > >> > >> > batch and it always seems to work. This
issue
>> seems
>> > to
>> > >>>>> happen
>> > >>>>> > > > with
>> > >>>>> > > > >> > batch
>> > >>>>> > > > >> > >> > and it is not at all clear why. Let me
know what
>> > else
>> > >>>>> you
>> > >>>>> > need
>> > >>>>> > > > >> from
>> > >>>>> > > > >> > me
>> > >>>>> > > > >> > >> to
>> > >>>>> > > > >> > >> > better coordinate what I would need to do
next.
>> > >>>>> Thanks.
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley
Gotway
>> via
>> > >>>>> RT <
>> > >>>>> > > > >> > >> > met_help at ucar.edu>
>> > >>>>> > > > >> > >> > wrote:
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > > Hi Ed,
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > > Sorry for the delay. I took a look on
wcoss
>> today.
>> > >>>>> You
>> > >>>>> > > describe
>> > >>>>> > > > >> that
>> > >>>>> > > > >> > >> your
>> > >>>>> > > > >> > >> > > plotting script has stopped working but
you're
>> not
>> > >>>>> sure
>> > >>>>> > why.
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > > I ran across a rather large directory:
>> > >>>>> > > > >> > >> > > ls
>> > >>>>> > > > >> >
>> > >>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>> > >>>>> > > > >> > >> |
>> > >>>>> > > > >> > >> > wc
>> > >>>>> > > > >> > >> > > -w
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > > That has 71,398 xml files in there. But I
see
>> > >>>>> timestamps in
>> > >>>>> > > > those
>> > >>>>> > > > >> > >> > filenames
>> > >>>>> > > > >> > >> > > from 20200921 to 20200925.
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > > Is it the case that this stopped working
on
>> > >>>>> 20200925? Or
>> > >>>>> > are
>> > >>>>> > > > >> those
>> > >>>>> > > > >> > >> dated
>> > >>>>> > > > >> > >> > > filenames remnants from older work?
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > > So it hasn't been working for over a
month. Is
>> that
>> > >>>>> right?
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > > John
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT
System
>> itself
>> > >>>>> via RT
>> > >>>>> > <
>> > >>>>> > > > >> > >> > > met_help at ucar.edu> wrote:
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > > >
>> > >>>>> > > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request 97283
was
>> acted
>> > >>>>> upon.
>> > >>>>> > > > >> > >> > > > Transaction: Given to johnhg (John
Halley
>> Gotway)
>> > >>>>> by
>> > >>>>> > > > RT_System
>> > >>>>> > > > >> > >> > > > Queue: met_help
>> > >>>>> > > > >> > >> > > > Subject: plots no longer being
generated
>> > >>>>> > > > >> > >> > > > Owner: johnhg
>> > >>>>> > > > >> > >> > > > Requestors: edward.strobach at noaa.gov
>> > >>>>> > > > >> > >> > > > Status: new
>> > >>>>> > > > >> > >> > > > Ticket <URL:
>> > >>>>> > > > >> > >>
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > > >
>> > >>>>> > > > >> > >> > > >
>> > >>>>> > > > >> > >> > > > This transaction appears to have no
content
>> > >>>>> > > > >> > >> > > >
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> > >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> > --
>> > >>>>> > > > >> > >> > Edward Strobach
>> > >>>>> > > > >> > >> > EMC/NCEP/NWS/
>> > >>>>> > > > >> > >> > IMSG Contractor
>> > >>>>> > > > >> > >> > Cubicle#: 2029
>> > >>>>> > > > >> > >> > 301-683-3717
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >> >
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >>
>> > >>>>> > > > >> > >
>> > >>>>> > > > >> > > --
>> > >>>>> > > > >> > > Edward Strobach
>> > >>>>> > > > >> > > EMC/NCEP/NWS/
>> > >>>>> > > > >> > > IMSG Contractor
>> > >>>>> > > > >> > > Cubicle#: 2029
>> > >>>>> > > > >> > > 301-683-3717
>> > >>>>> > > > >> > >
>> > >>>>> > > > >> >
>> > >>>>> > > > >> >
>> > >>>>> > > > >> > --
>> > >>>>> > > > >> > Edward Strobach
>> > >>>>> > > > >> > EMC/NCEP/NWS/
>> > >>>>> > > > >> > IMSG Contractor
>> > >>>>> > > > >> > Cubicle#: 2029
>> > >>>>> > > > >> > 301-683-3717
>> > >>>>> > > > >> >
>> > >>>>> > > > >> >
>> > >>>>> > > > >>
>> > >>>>> > > > >>
>> > >>>>> > > > >
>> > >>>>> > > > > --
>> > >>>>> > > > > Edward Strobach
>> > >>>>> > > > > EMC/NCEP/NWS/
>> > >>>>> > > > > IMSG Contractor
>> > >>>>> > > > > Cubicle#: 2029
>> > >>>>> > > > > 301-683-3717
>> > >>>>> > > > >
>> > >>>>> > > >
>> > >>>>> > > >
>> > >>>>> > > > --
>> > >>>>> > > > Edward Strobach
>> > >>>>> > > > EMC/NCEP/NWS/
>> > >>>>> > > > IMSG Contractor
>> > >>>>> > > > Cubicle#: 2029
>> > >>>>> > > > 301-683-3717
>> > >>>>> > > >
>> > >>>>> > > >
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>> > --
>> > >>>>> > Edward Strobach
>> > >>>>> > EMC/NCEP/NWS/
>> > >>>>> > IMSG Contractor
>> > >>>>> > Cubicle#: 2029
>> > >>>>> > 301-683-3717
>> > >>>>> >
>> > >>>>> >
>> > >>>>>
>> > >>>>>
>> > >>>>
>> > >>>> --
>> > >>>> Edward Strobach
>> > >>>> EMC/NCEP/NWS/
>> > >>>> IMSG Contractor
>> > >>>> Cubicle#: 2029
>> > >>>> 301-683-3717
>> > >>>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Edward Strobach
>> > >>> EMC/NCEP/NWS/
>> > >>> IMSG Contractor
>> > >>> Cubicle#: 2029
>> > >>> 301-683-3717
>> > >>>
>> > >>
>> > >>
>> > >> --
>> > >> Edward Strobach
>> > >> EMC/NCEP/NWS/
>> > >> IMSG Contractor
>> > >> Cubicle#: 2029
>> > >> 301-683-3717
>> > >>
>> > >
>> > >
>> > > --
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/
>> > > IMSG Contractor
>> > > Cubicle#: 2029
>> > > 301-683-3717
>> > >
>> >
>> >
>> > --
>> > Edward Strobach
>> > EMC/NCEP/NWS/
>> > IMSG Contractor
>> > Cubicle#: 2029
>> > 301-683-3717
>> >
>> >
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: plots no longer being generated
From: Edward Strobach - NOAA Affiliate
Time: Wed Nov 11 10:20:27 2020
Oh, and the attachments.
On Wed, Nov 11, 2020 at 12:18 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Hi John,
>
> Just a brief update..
>
> I changed my set-up to reflect the changes you recommended. A
sample XML
> file is created - refer to the *"XML_adjusted*" file attached below
-
> when running Plot_XML_Builder_v2.py. The file that is submitted via
batch
> - i.e. "*XML_Build_Plot_Create_Step"* - is also attached and creates
8
> different XML files corresponding to the stat variable (FBAR_OBAR
and
> RMSE), the independent variable (fcst_lead v. fcst_valid_beg), and
forecast
> variable (PMTF and OZCON). I find that the batch job is successful
if I do
> not list more than one name under vx_mask with an underscore. In
other
> words, if you look at the vx_mask list in the *"XML_adjusted*" file,
you
> will see Bare_Tundra followed by Crop_Grassland. I tried removing
> Crop_Grassland and ran it again but received an error related to
> Crop_Woodland (the second vx_mask variable within the list that has
an
> underscore). The pasted output below shows the error message for
the
> second attempt. The first attempt would be the same but with
> Crop_Grassland instead.
>
> CALLING: scp
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls/Plot_Time_Series_OZCON_fcst_valid_beg_FBAR_OBAR_Aggregate_202011.xml
> edward.strobach at 205.156.8.85:~
> ---- MVBatch ----
>
> input file:
> Plot_Time_Series_OZCON_fcst_valid_beg_FBAR_OBAR_Aggregate_202011.xml
>
> ** ERROR: String Crop_Woodland includes SQL unsafe word AND or
BETWEEN
> ---- MVBatch Done ----
>
> Good new though! All the plots did save - at least for PMTF;
haven't
> checked OZCON yet - when submitting the batch job for only
geographical
> regions, not dominant land use masks. This is a step in the right
> direction. I'm not sure what to make out for the error above but
wonder if
> you know of a workaround.
>
> Some recommendations:
> I don't think I can currently list a bunch of stat variables or
fcst_vars
> under <dep1>. The example below shows that FBAR and OBAR are part
of a
> list that is used for one plot. I wouldn't be able to list RMSE
after
> that, at least I don't think I could. If we were to use FBAR_OBAR
instead
> of having them separate, then one might be able to list several
<stat>
> variables in sequence to produce plots for different stat types.
> Similarly, one might be able to list several fcst_vars instead of
just
> one. I would propose the following changes if you think it would be
> helpful and worth pursuing.
>
> ORIGINAL:
>
> <dep>
>
> <dep1>
>
> <fcst_var name="OZCON">
>
> <stat>FBAR</stat>
>
> <stat>OBAR</stat>
>
> </fcst_var>
>
> </dep1>
>
> <dep2/>
>
> </dep>
>
> NEW:
>
> <dep>
>
> <dep1>
>
> <fcst_var name="OZCON">
>
> <stat>FBAR_OBAR</stat>
>
> <stat>RMSE</stat>
>
> </fcst_var>
>
> <fcst_var name="PMTF">
>
> <stat>FBAR_OBAR</stat>
>
> <stat>RMSE</stat>
>
> </fcst_var>
>
> </dep1>
>
> <dep2/>
>
> </dep>
>
> If the above formulation was incorporated then I would only produce
two
> XML files corresponding to two independent variables: one fcst_lead
and
> one for fcst_valid_beg. This may resolve some of the same batch
issues
> that I was having earlier on since there were so many simultaneous
plots
> being processed and stored using different calls.
>
>
>
>
> On Mon, Nov 9, 2020 at 2:09 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> I am pleased about being able to do it this way. It seems like a
better
>> approach. However, It's unclear why, when running the same command
to
>> process the same XML file, that files are not always being copied
over.
>> From my earlier email you can see that sometimes 2 files are copied
over,
>> sometimes 4, and sometimes only 1. I was hoping all 8 would be
copied
>> over. This is kind of weird and I'm not sure how one would go
about this.
>>
>> On Mon, Nov 9, 2020 at 12:47 PM John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Ed,
>>>
>>> Yes, there's lots of moving parts here.
>>> (1) Setting up the METviewer XML to run 1 or more plots.
>>> (2) Running that XML remotely to generate plots.
>>> (3) Transferring the resulting image files back.
>>>
>>> I just wanted to make sure you know about the options for (1). And
I'm
>>> really glad to hear that you were able to see how to generate
multiple
>>> plots from a single METviewer XML!
>>> I'd say the rest of the details about managing the plotting/image
>>> transfer
>>> logic is really up to you and your colleagues. If I were setting
this
>>> up, I
>>> would consider having each batch job write to a single output
directory,
>>> and then recursively transfer back that directory. That seems
easier than
>>> dealing with lots of individual files. But it's really to you
guys.
>>>
>>> There is another detail about the METviewer plot XML that we
haven't
>>> talked
>>> about yet, and that's plot "inheritance". In a single XML, you can
define
>>> multiple <plot> tags and have one <plot> inherit settings from a
previous
>>> one. I took this to the extreme for one project, setting up a
pretty
>>> complex XML that made like 6 different plot types using multiple
levels
>>> of
>>> inheritance. In general, I'd advise against that. While it's cool
that it
>>> works, it makes for complex logic that's difficult to maintain.
Instead,
>>> I'd just look at any "for" loops you have in your plotting script
and
>>> move
>>> as many as are easy and feasible into the XML directly... without
going
>>> crazy! That would be like init hours, masking regions, and perhaps
>>> statistic types. That should greatly reduce the number of
individual
>>> calls
>>> to mv_batch.sh and hopefully provide for more stable results.
>>>
>>> But if you ever want to learn about plot inheritance, I'm happy to
help
>>> you
>>> with that too.
>>>
>>> John
>>>
>>> On Mon, Nov 9, 2020 at 8:02 AM Edward Strobach - NOAA Affiliate
via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>> >
>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>>> >
>>> > FYI, when running test_v2.sh again after running it the first
time
>>> only one
>>> > plot was copied over:
>>> > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>>> > 100%
57KB
>>> > 19.9MB/s 00:00
>>> >
>>> > On Mon, Nov 9, 2020 at 10:00 AM Edward Strobach - NOAA Affiliate
<
>>> > edward.strobach at noaa.gov> wrote:
>>> >
>>> > > I did run some additional tests outside the main test*sh
script.
>>> > > Specifically I ran the following line several times to see
what plots
>>> > would
>>> > > be copied over to WCOSS:
>>> > > bash
>>> > >
>>> >
>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>> > > edward.strobach
>>> > >
>>> >
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011
>>> > >
>>> >
>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls/Test.xml
>>> > >
>>> > > The first time around 3 plots were copied over:
>>> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
>>> > > 100% 51KB 18.6MB/s 00:00
>>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
>>> > > 100% 58KB 12.0MB/s 00:00
>>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>>> > > 100% 57KB 2.8MB/s 00:00
>>> > >
>>> > > The second time 2 plots were copied over:
>>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
>>> > > 100% 58KB 2.6MB/s 00:00
>>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>>> > > 100% 57KB 6.2MB/s 00:00
>>> > >
>>> > > The third time 3 plots were copied over:
>>> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
>>> > >
100%
>>> 51KB
>>> > > 2.4MB/s 00:00
>>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
>>> > >
100%
>>> 58KB
>>> > > 6.1MB/s 00:00
>>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>>> > >
100%
>>> 57KB
>>> > > 11.1MB/s 00:00
>>> > >
>>> > > The order of init_hour and vx_mask is as follows:
>>> > > <06>
>>> > > <12>
>>> > > and
>>> > > <Mixed_Forest>
>>> > > <FULL>
>>> > > <EAST>
>>> > > <WEST>
>>> > >
>>> > > It does appear that the latest plots are copied over in the
process
>>> (i.e.
>>> > > if two, then the last two; if three, then the last three).
>>> > > I don't really have a feel for why two files are copied over
for some
>>> > > instances and 3 for other instances. Lastly, I ran test_v2.sh
>>> again, and
>>> > > this time it copied over 4 plots. See below:
>>> > >
>>> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t06z.png
>>> > >
100%
>>> 53KB
>>> > > 12.7MB/s 00:00
>>> > > PMTF_EAST_112020_2020-11-01_2020-11-06_t12z.png
>>> > >
100%
>>> 51KB
>>> > > 9.7MB/s 00:00
>>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t06z.png
>>> > >
100%
>>> 58KB
>>> > > 11.3MB/s 00:00
>>> > > PMTF_WEST_112020_2020-11-01_2020-11-06_t12z.png
>>> > >
100%
>>> 57KB
>>> > > 11.4MB/s 00:00
>>> > >
>>> > > This is very bizarre to me.
>>> > >
>>> > >
>>> > >
>>> > > On Mon, Nov 9, 2020 at 9:26 AM Edward Strobach - NOAA
Affiliate <
>>> > > edward.strobach at noaa.gov> wrote:
>>> > >
>>> > >> Actually, I don't think I can do that. FBAR and OBAR are
split and
>>> > >> adding RMSE as a third line might confuse the process.
>>> > >>
>>> > >> I was able to run a test with mixed success. I was
successful at
>>> > >> generating 8 plots (4 regions for 2 initial hours). The
regions
>>> were
>>> > >> Mixed_Forest, FULL, EAST, and WEST while the initial hours
were 06
>>> and
>>> > 12.
>>> > >> The test ran well with indications that all plots were
generated and
>>> > stored
>>> > >> here: Created plot
>>> > >>
>>> >
>>> /data/mv_data/edward.strobach/plots/PMTF_Mixed_Forest_112020_2020-
11-01_2020-11-06_t06z.png
>>> > >>
>>> > >> However, I was not successful at saving the plots on Dell.
Only the
>>> last
>>> > >> plot was saved here:
>>> > >>
>>> >
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/plots/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate/202011.
>>> > >>
>>> > >> It seems that in my bash script the line that produces the
plots
>>> that
>>> > are
>>> > >> then stored in the above directory must be done in a loop
otherwise
>>> it
>>> > will
>>> > >> only move the last plot over as suggested by the fact that
only the
>>> plot
>>> > >> featuring WEST at 12z was moved over - the last plot of the
series
>>> of
>>> > plots
>>> > >> generated. I suppose I could create logic that extracts the
>>> > region/initial
>>> > >> hour information and construct a two part loop to move over
all
>>> plots?
>>> > >>
>>> > >> On Mon, Nov 9, 2020 at 8:19 AM Edward Strobach - NOAA
Affiliate <
>>> > >> edward.strobach at noaa.gov> wrote:
>>> > >>
>>> > >>> Just a follow-up on one other item..
>>> > >>>
>>> > >>> Can also list multiple stat types in the same way that I
specify
>>> the
>>> > >>> init_hour and vx_mask? In other words, can I do the
following:
>>> > >>>
>>> > >>> <dep1>
>>> > >>> <fcst_var name="PMTF">
>>> > >>> <stat>FBAR_OBAR</stat>
>>> > >>> <stat>RMSE</stat>
>>> > >>> </fcst_var>
>>> > >>> </dep1>
>>> > >>>
>>> > >>> I know that the "set" block is not being used here.
>>> > >>>
>>> > >>> On Mon, Nov 9, 2020 at 7:41 AM Edward Strobach - NOAA
Affiliate <
>>> > >>> edward.strobach at noaa.gov> wrote:
>>> > >>>
>>> > >>>> Thanks John. Let me try that and I'll let you know how it
goes.
>>> > >>>>
>>> > >>>> On Fri, Nov 6, 2020 at 10:53 AM John Halley Gotway via RT <
>>> > >>>> met_help at ucar.edu> wrote:
>>> > >>>>
>>> > >>>>> Ed,
>>> > >>>>>
>>> > >>>>> You're right about the METviewer GUI grouping multiple
selections
>>> > >>>>> together.
>>> > >>>>> That looks like this in the XML using the <set> tag:
>>> > >>>>> <field equalize="false" name="vx_mask">
>>> > >>>>> <set name="vx_mask_0">
>>> > >>>>> <val>FULL</val>
>>> > >>>>> </set>
>>> > >>>>> </field>
>>> > >>>>>
>>> > >>>>> But in the updates I sent, I remove the <set> tag:
>>> > >>>>> <field equalize="false" name="vx_mask">
>>> > >>>>> <val>FULL</val>
>>> > >>>>> <val>CONUS</val>
>>> > >>>>> <val>EAST</val>
>>> > >>>>> <val>WEST</val>
>>> > >>>>> </field>
>>> > >>>>> So instead of processing this as 1 group of 4 values,
it'll
>>> process
>>> > >>>>> the 4
>>> > >>>>> values separately.
>>> > >>>>>
>>> > >>>>> The current vx_mask value is referenced using {vx_mask}
syntax.
>>> And I
>>> > >>>>> included examples of using that in the output file names
and
>>> titles
>>> > in
>>> > >>>>> my
>>> > >>>>> previous email:
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> >
>>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> >
>>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>>> > >>>>>
>>> > >>>>>
>>> >
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>>> > >>>>>
>>> > >>>>> <title>Average PMTF (October 2020; {init_hour}z cycle;
{vx_mask}
>>> > >>>>> Domain)</title>
>>> > >>>>>
>>> > >>>>> John
>>> > >>>>>
>>> > >>>>> On Fri, Nov 6, 2020 at 3:49 AM Edward Strobach - NOAA
Affiliate
>>> via
>>> > RT
>>> > >>>>> <
>>> > >>>>> met_help at ucar.edu> wrote:
>>> > >>>>>
>>> > >>>>> >
>>> > >>>>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>> >
>>> > >>>>> >
>>> > >>>>> > Hi John,
>>> > >>>>> >
>>> > >>>>> > This definitely helps. I never thought about it this
way
>>> because I
>>> > >>>>> > assumed, based on my experience with the interactive MET
GUI,
>>> that
>>> > I
>>> > >>>>> could
>>> > >>>>> > only generate a single plot regardless of how I
specified my
>>> > >>>>> settings in
>>> > >>>>> > the XML. For example, if I listed EAST and WEST under
vx_mask,
>>> > then
>>> > >>>>> my
>>> > >>>>> > thinking would be that both those regions would be
grouped and
>>> a
>>> > >>>>> single
>>> > >>>>> > plot created. I guess I was wrong on that. The concern
I
>>> still
>>> > >>>>> have,
>>> > >>>>> > however, is that the plot title and plot name - the png
file -
>>> is
>>> > >>>>> also set
>>> > >>>>> > in the XML file. Even after adding the list you gave, I
can't
>>> see
>>> > >>>>> how
>>> > >>>>> > unique file names will be created. Is that true?
>>> > >>>>> >
>>> > >>>>> > On Thu, Nov 5, 2020 at 2:02 PM John Halley Gotway via RT
<
>>> > >>>>> > met_help at ucar.edu>
>>> > >>>>> > wrote:
>>> > >>>>> >
>>> > >>>>> > > Ed,
>>> > >>>>> > >
>>> > >>>>> > > Thanks for sending examples. My goal was to use
METviewer's
>>> XML
>>> > >>>>> upload
>>> > >>>>> > > functionality but I'm not able to upload a pdf version
of
>>> the xml
>>> > >>>>> file.
>>> > >>>>> > >
>>> > >>>>> > > I did spend some time trying to replicate this exact
plot
>>> with
>>> > >>>>> METviewer
>>> > >>>>> > > at:
>>> > >>>>> > > https://metviewer.nws.noaa.gov/metviewer1.jsp
>>> > >>>>> > >
>>> > >>>>> > > But I was never quite successful. So I made just a
similar
>>> > version
>>> > >>>>> > instead.
>>> > >>>>> > > See attached: *plot_CMAQ_Lead_Series_JHG.xml/.png*
>>> > >>>>> > >
>>> > >>>>> > > This plots FBAR and OBAR for 6 different models for
the
>>> month of
>>> > >>>>> October
>>> > >>>>> > > for all 06Z initializations over the FULL model
domain.
>>> > >>>>> > >
>>> > >>>>> > > I wanted to show you how to modify this for batch
mode... and
>>> > then
>>> > >>>>> adapt
>>> > >>>>> > it
>>> > >>>>> > > to make multiple plots. But I'm not able to make batch
plots
>>> on
>>> > >>>>> wcoss
>>> > >>>>> > using
>>> > >>>>> > > the AWS instance. I don't think I have the right
permissions
>>> to
>>> > do
>>> > >>>>> so.
>>> > >>>>> > >
>>> > >>>>> > > So let me just say this. Generally, you just update
the list
>>> of
>>> > >>>>> items in
>>> > >>>>> > > plot_fix section.
>>> > >>>>> > >
>>> > >>>>> > > Original from METviewer with 1 init hour and 1 mask:
>>> > >>>>> > >
>>> > >>>>> > > <field equalize="false" name="vx_mask">
>>> > >>>>> > > <set name="vx_mask_0">
>>> > >>>>> > > <val>FULL</val>
>>> > >>>>> > > </set>
>>> > >>>>> > > </field>
>>> > >>>>> > > <field equalize="false" name="init_hour">
>>> > >>>>> > > <set name="init_hour_1">
>>> > >>>>> > > <val>06</val>
>>> > >>>>> > > </set>
>>> > >>>>> > > </field>
>>> > >>>>> > >
>>> > >>>>> > > Change to 2 init hours and 4 masks:
>>> > >>>>> > >
>>> > >>>>> > > <field equalize="false" name="vx_mask">
>>> > >>>>> > > <val>FULL</val>
>>> > >>>>> > > <val>CONUS</val>
>>> > >>>>> > > <val>EAST</val>
>>> > >>>>> > > <val>WEST</val>
>>> > >>>>> > > </field>
>>> > >>>>> > > <field equalize="false" name="init_hour">
>>> > >>>>> > > <val>06</val>
>>> > >>>>> > > <val>12</val>
>>> > >>>>> > > </field>
>>> > >>>>> > >
>>> > >>>>> > > And then reference those values in the naming of the
titles
>>> and
>>> > >>>>> output
>>> > >>>>> > > files:
>>> > >>>>> > >
>>> > >>>>> > >
>>> > >>>>> > >
>>> > >>>>> > >
>>> > >>>>> >
>>> > >>>>>
>>> >
>>>
<data_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.data</data_file>
>>> > >>>>> > >
>>> > >>>>> > >
>>> > >>>>> > >
>>> > >>>>> >
>>> > >>>>>
>>> >
>>>
<plot_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.png</plot_file>
>>> > >>>>> > >
>>> > >>>>> > >
>>> > >>>>>
>>> >
<r_file>plot_CMAQ_Lead_Series_init{init_hour}_mask{vx_mask}.R</r_file>
>>> > >>>>> > >
>>> > >>>>> > > <title>Average PMTF (October 2020; {init_hour}z cycle;
>>> {vx_mask}
>>> > >>>>> > > Domain)</title>
>>> > >>>>> > >
>>> > >>>>> > > So that's the general idea. And when you submit this
to
>>> mv_batch,
>>> > >>>>> it'll
>>> > >>>>> > > make 8 plots instead of 1.
>>> > >>>>> > > And you can see how you'd easily expand this to many
plots.
>>> > >>>>> > >
>>> > >>>>> > > But I worry that this may not be sufficient
information to
>>> get
>>> > you
>>> > >>>>> going.
>>> > >>>>> > >
>>> > >>>>> > > John
>>> > >>>>> > >
>>> > >>>>> > > On Wed, Nov 4, 2020 at 11:32 AM Edward Strobach - NOAA
>>> Affiliate
>>> > >>>>> via RT <
>>> > >>>>> > > met_help at ucar.edu> wrote:
>>> > >>>>> > >
>>> > >>>>> > > >
>>> > >>>>> > > > <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>> > >
>>> > >>>>> > > >
>>> > >>>>> > > > Hi John,
>>> > >>>>> > > >
>>> > >>>>> > > > I think I was able to cover your request okay.
Below are 4
>>> > >>>>> attachments
>>> > >>>>> > > > that include the test script that was used to
generate the
>>> > >>>>> plots, a
>>> > >>>>> > > sample
>>> > >>>>> > > > XML file, and the two plots that were generated
using the
>>> test
>>> > >>>>> run
>>> > >>>>> > > script.
>>> > >>>>> > > > These plots were not created previously using the
batch
>>> job but
>>> > >>>>> uses
>>> > >>>>> > the
>>> > >>>>> > > > same exact logic as the original run script. Thanks
again
>>> for
>>> > >>>>> working
>>> > >>>>> > > with
>>> > >>>>> > > > me on this. Please let me know if you need anything
else
>>> or
>>> > >>>>> would like
>>> > >>>>> > > me
>>> > >>>>> > > > to test something.
>>> > >>>>> > > >
>>> > >>>>> > > > On Wed, Nov 4, 2020 at 1:06 PM Edward Strobach -
NOAA
>>> > Affiliate <
>>> > >>>>> > > > edward.strobach at noaa.gov> wrote:
>>> > >>>>> > > >
>>> > >>>>> > > > > Ed,
>>> > >>>>> > > > >
>>> > >>>>> > > > > Image files showing up in an unexpected output
directory
>>> > >>>>> likely point
>>> > >>>>> > > to
>>> > >>>>> > > > a
>>> > >>>>> > > > > problem in the logic of the processing scripts.
It's
>>> pretty
>>> > >>>>> unlikely
>>> > >>>>> > > that
>>> > >>>>> > > > > this behavior is caused by the METviewer code
itself. But
>>> > I've
>>> > >>>>> > > definitely
>>> > >>>>> > > > > been surprised before!
>>> > >>>>> > > > >
>>> > >>>>> > > > > I tested the logic outside of a batch job and it
works
>>> > fine. I
>>> > >>>>> > suspect
>>> > >>>>> > > > > that the issuance of another batch job while the
other
>>> one is
>>> > >>>>> getting
>>> > >>>>> > > > > processed is causing some confusion in the
processing.
>>> These
>>> > >>>>> issues
>>> > >>>>> > > > don't
>>> > >>>>> > > > > happen everyday and appear more sporadic than
anything.
>>> > >>>>> > > > >
>>> > >>>>> > > > > So what we really need here is an example of
taking a
>>> plot
>>> > >>>>> from the
>>> > >>>>> > > > > METviewer GUI and adapting it to make it generate
>>> multiple
>>> > >>>>> plots. I
>>> > >>>>> > did
>>> > >>>>> > > > go
>>> > >>>>> > > > > through examples of doing this in previous
tutorials,
>>> but we
>>> > >>>>> didn't
>>> > >>>>> > > start
>>> > >>>>> > > > > recording them until last summer. I checked the
July
>>> 2019 NRL
>>> > >>>>> > tutorial
>>> > >>>>> > > > > videos and don't see it there.
>>> > >>>>> > > > >
>>> > >>>>> > > > > Is there something you need from me to help on
this?
>>> Just
>>> > let
>>> > >>>>> me
>>> > >>>>> > know
>>> > >>>>> > > > >
>>> > >>>>> > > > > We've begun recording training videos for common
topics
>>> like
>>> > >>>>> this.
>>> > >>>>> > But
>>> > >>>>> > > we
>>> > >>>>> > > > > only have one for METviewer so far:
>>> > >>>>> > > > >
>>> > >>>>> > >
>>> > >>>>>
>>> >
>>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>>> > >>>>> > > > >
>>> > >>>>> > > > > But this definitely seems like a good candidate
for a new
>>> > >>>>> training
>>> > >>>>> > > video.
>>> > >>>>> > > > >
>>> > >>>>> > > > > For now though, how about this...
>>> > >>>>> > > > > - You use METviewer to manually make a single
plot,
>>> > formatting
>>> > >>>>> it how
>>> > >>>>> > > > you'd
>>> > >>>>> > > > > like.
>>> > >>>>> > > > > - You send me your sample plot and corresponding
XML
>>> file.
>>> > >>>>> > > > > - I'll tweak it to generate many plots via batch
instead
>>> of
>>> > >>>>> just one.
>>> > >>>>> > > > > - And then you can continue tweaking to add more
>>> > permutations.
>>> > >>>>> > > > >
>>> > >>>>> > > > > Okay, I'll work on that now. Thanks.
>>> > >>>>> > > > >
>>> > >>>>> > > > > John
>>> > >>>>> > > > >
>>> > >>>>> > > > > On Wed, Nov 4, 2020 at 12:14 PM John Halley Gotway
via
>>> RT <
>>> > >>>>> > > > > met_help at ucar.edu> wrote:
>>> > >>>>> > > > >
>>> > >>>>> > > > >> Ed,
>>> > >>>>> > > > >>
>>> > >>>>> > > > >> Image files showing up in an unexpected output
directory
>>> > >>>>> likely
>>> > >>>>> > point
>>> > >>>>> > > > to a
>>> > >>>>> > > > >> problem in the logic of the processing scripts.
It's
>>> pretty
>>> > >>>>> unlikely
>>> > >>>>> > > > that
>>> > >>>>> > > > >> this behavior is caused by the METviewer code
itself.
>>> But
>>> > I've
>>> > >>>>> > > > definitely
>>> > >>>>> > > > >> been surprised before!
>>> > >>>>> > > > >>
>>> > >>>>> > > > >> So what we really need here is an example of
taking a
>>> plot
>>> > >>>>> from the
>>> > >>>>> > > > >> METviewer GUI and adapting it to make it generate
>>> multiple
>>> > >>>>> plots. I
>>> > >>>>> > > did
>>> > >>>>> > > > go
>>> > >>>>> > > > >> through examples of doing this in previous
tutorials,
>>> but we
>>> > >>>>> didn't
>>> > >>>>> > > > start
>>> > >>>>> > > > >> recording them until last summer. I checked the
July
>>> 2019
>>> > NRL
>>> > >>>>> > tutorial
>>> > >>>>> > > > >> videos and don't see it there.
>>> > >>>>> > > > >>
>>> > >>>>> > > > >> We've begun recording training videos for common
topics
>>> like
>>> > >>>>> this.
>>> > >>>>> > But
>>> > >>>>> > > > we
>>> > >>>>> > > > >> only have one for METviewer so far:
>>> > >>>>> > > > >>
>>> > >>>>> > > >
>>> > >>>>> >
>>> > >>>>>
>>> >
>>> https://dtcenter.github.io/METplus-
Training/modules/METviewer/index.html
>>> > >>>>> > > > >>
>>> > >>>>> > > > >> But this definitely seems like a good candidate
for a
>>> new
>>> > >>>>> training
>>> > >>>>> > > > video.
>>> > >>>>> > > > >>
>>> > >>>>> > > > >> For now though, how about this...
>>> > >>>>> > > > >> - You use METviewer to manually make a single
plot,
>>> > >>>>> formatting it
>>> > >>>>> > how
>>> > >>>>> > > > >> you'd
>>> > >>>>> > > > >> like.
>>> > >>>>> > > > >> - You send me your sample plot and corresponding
XML
>>> file.
>>> > >>>>> > > > >> - I'll tweak it to generate many plots via batch
>>> instead of
>>> > >>>>> just
>>> > >>>>> > one.
>>> > >>>>> > > > >> - And then you can continue tweaking to add more
>>> > permutations.
>>> > >>>>> > > > >>
>>> > >>>>> > > > >> John
>>> > >>>>> > > > >>
>>> > >>>>> > > > >> On Wed, Nov 4, 2020 at 9:22 AM Edward Strobach -
NOAA
>>> > >>>>> Affiliate via
>>> > >>>>> > > RT <
>>> > >>>>> > > > >> met_help at ucar.edu> wrote:
>>> > >>>>> > > > >>
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >> > <URL:
>>> > >>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283 >
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >> > Oh, and one more thing..
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >> > The green text that didn't show up was meant to
>>> highlight
>>> > >>>>> the fact
>>> > >>>>> > > > that
>>> > >>>>> > > > >> > sometimes plots that shouldn't be stored in
certain
>>> > >>>>> directories
>>> > >>>>> > end
>>> > >>>>> > > up
>>> > >>>>> > > > >> > there. The pasted example below shows one
instance
>>> where
>>> > >>>>> OZMAX*
>>> > >>>>> > was
>>> > >>>>> > > > >> stored
>>> > >>>>> > > > >> > where TMP* should have been stored.
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24808 Oct
5
>>> 10:52
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-02.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 23323 Oct
6
>>> 10:52
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-03.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 24863 Oct
7
>>> 10:00
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-04.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61081 Oct
13
>>> 13:49
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-10.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 61377 Oct
14
>>> 11:31
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-11.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 60158 Oct
19
>>> 11:37
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-12.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25785 Oct
20
>>> 11:35
>>> > >>>>> > > > >> > OZMAX1_EAST_102020_2020-10-01_2020-10-17.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56642 Oct
21
>>> 11:45
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-18.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57008 Oct
22
>>> 11:42
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-19.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 57147 Oct
23
>>> 11:50
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-20.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 56661 Oct
24
>>> 12:57
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-21.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 55976 Oct
31
>>> 08:47
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-28.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25272 Nov
2
>>> 08:50
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-30.png
>>> > >>>>> > > > >> > -rw-r--r-- 1 Edward.Strobach emcmodel 25271 Nov
3
>>> 08:52
>>> > >>>>> > > > >> > TMP_FULL_102020_2020-10-01_2020-10-31.png
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >> > On Wed, Nov 4, 2020 at 11:15 AM Edward Strobach
- NOAA
>>> > >>>>> Affiliate <
>>> > >>>>> > > > >> > edward.strobach at noaa.gov> wrote:
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >> > > Hi John,
>>> > >>>>> > > > >> > >
>>> > >>>>> > > > >> > > I know there are a lot of calls and speculate
that
>>> this
>>> > >>>>> could be
>>> > >>>>> > > > part
>>> > >>>>> > > > >> of
>>> > >>>>> > > > >> > > the problem. I don't think that I'm
following best
>>> > >>>>> practice by
>>> > >>>>> > > > >> > approaching
>>> > >>>>> > > > >> > > it in this way. Generating plots usually
works
>>> well if
>>> > I
>>> > >>>>> > process
>>> > >>>>> > > a
>>> > >>>>> > > > >> > subset
>>> > >>>>> > > > >> > > of the computations outside of batch
scripting. The
>>> > >>>>> python
>>> > >>>>> > script
>>> > >>>>> > > > was
>>> > >>>>> > > > >> > > given to me by Logan, but I have since made
>>> significant
>>> > >>>>> changes
>>> > >>>>> > to
>>> > >>>>> > > > it
>>> > >>>>> > > > >> in
>>> > >>>>> > > > >> > > order to dynamically fill in the xml file
based on
>>> > >>>>> parameters
>>> > >>>>> > set.
>>> > >>>>> > > > >> The
>>> > >>>>> > > > >> > > test*.sh script was also given to me. I
created the
>>> > >>>>> > > > Time_Series*bsub
>>> > >>>>> > > > >> as
>>> > >>>>> > > > >> > a
>>> > >>>>> > > > >> > > way to process a series of plots based on
cycle
>>> time,
>>> > >>>>> region,
>>> > >>>>> > > etc. I
>>> > >>>>> > > > >> > think
>>> > >>>>> > > > >> > > the first two scripts are solid and do well
for
>>> what I
>>> > >>>>> need to
>>> > >>>>> > get
>>> > >>>>> > > > >> done.
>>> > >>>>> > > > >> > > It's the Time_Series*bsub script that bothers
me a
>>> bit.
>>> > >>>>> > > > >> > >
>>> > >>>>> > > > >> > > I can't see how a single XML file can process
all
>>> these
>>> > >>>>> plots
>>> > >>>>> > in a
>>> > >>>>> > > > >> single
>>> > >>>>> > > > >> > > setting based on what I've learned thus far.
How
>>> would
>>> > >>>>> one go
>>> > >>>>> > > about
>>> > >>>>> > > > >> > that?
>>> > >>>>> > > > >> > > I would be open going in that direction if
you can
>>> > >>>>> provide some
>>> > >>>>> > > > >> guidance.
>>> > >>>>> > > > >> > >
>>> > >>>>> > > > >> > > Unfortunately, I haven't gotten a lot of
feedback or
>>> > >>>>> dialogue
>>> > >>>>> > > > exchange
>>> > >>>>> > > > >> > > about what I've done so far at EMC with this
new
>>> set-up.
>>> > >>>>> This
>>> > >>>>> > has
>>> > >>>>> > > > >> also
>>> > >>>>> > > > >> > > been part of the problem. I can't seem to
get them
>>> to
>>> > >>>>> focus so
>>> > >>>>> > > that
>>> > >>>>> > > > >> we
>>> > >>>>> > > > >> > can
>>> > >>>>> > > > >> > > prioritize. I know that I'm doing the best I
can.
>>> If I
>>> > >>>>> can
>>> > >>>>> > learn
>>> > >>>>> > > > >> how to
>>> > >>>>> > > > >> > > go about this more efficiently it would be of
great
>>> > >>>>> help. I can
>>> > >>>>> > > > then
>>> > >>>>> > > > >> > > expand this to other verification as venture
to
>>> other
>>> > >>>>> projects
>>> > >>>>> > in
>>> > >>>>> > > > the
>>> > >>>>> > > > >> > > future. I know you're busy with many things,
so I'm
>>> > >>>>> willing to
>>> > >>>>> > go
>>> > >>>>> > > > >> along
>>> > >>>>> > > > >> > > with your schedule as you find an opening.
>>> > >>>>> > > > >> > >
>>> > >>>>> > > > >> > > Thanks
>>> > >>>>> > > > >> > >
>>> > >>>>> > > > >> > > On Wed, Nov 4, 2020 at 10:56 AM John Halley
Gotway
>>> via
>>> > RT
>>> > >>>>> <
>>> > >>>>> > > > >> > > met_help at ucar.edu> wrote:
>>> > >>>>> > > > >> > >
>>> > >>>>> > > > >> > >> Ed,
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> It's really difficult for me to see an
obvious
>>> problem
>>> > >>>>> that's
>>> > >>>>> > > > causing
>>> > >>>>> > > > >> > the
>>> > >>>>> > > > >> > >> behavior you describe.
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> Unfortunately, the red and green
highlighting you
>>> > >>>>> describe
>>> > >>>>> > > doesn't
>>> > >>>>> > > > >> show
>>> > >>>>> > > > >> > up
>>> > >>>>> > > > >> > >> in the Gmail web client. There must be some
mail
>>> system
>>> > >>>>> > > > >> incompatibility
>>> > >>>>> > > > >> > >> there.
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> Looking at your bsub script:
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >>
>>> > >>>>> > > >
>>> > >>>>> > >
>>> > >>>>> >
>>> > >>>>>
>>> >
>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plotting_batch_files/Time_Series_file.bsub
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> I am struck by the number of individual
calls
>>> you're
>>> > >>>>> making to
>>> > >>>>> > > > >> > METviewer:
>>> > >>>>> > > > >> > >> 2 fcst vars x 2 stats x 2 hours x 41 masks
x 2
>>> plot
>>> > >>>>> types =
>>> > >>>>> > 656
>>> > >>>>> > > > >> calls
>>> > >>>>> > > > >> > to
>>> > >>>>> > > > >> > >> METviewer
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> When making plots via the METviewer GUI,
it's true
>>> that
>>> > >>>>> 1 XML
>>> > >>>>> > = 1
>>> > >>>>> > > > >> plot.
>>> > >>>>> > > > >> > >> However, when making plots via the batch
engine,
>>> the
>>> > >>>>> plot XML
>>> > >>>>> > can
>>> > >>>>> > > > be
>>> > >>>>> > > > >> set
>>> > >>>>> > > > >> > >> up
>>> > >>>>> > > > >> > >> to create many, many plots. While I have no
direct
>>> > proof
>>> > >>>>> of
>>> > >>>>> > this,
>>> > >>>>> > > > it
>>> > >>>>> > > > >> > might
>>> > >>>>> > > > >> > >> be the case that executing 600+ batch jobs
for
>>> > METviewer
>>> > >>>>> is
>>> > >>>>> > > leading
>>> > >>>>> > > > >> to
>>> > >>>>> > > > >> > >> trouble on the system. One thing to try
would be
>>> > setting
>>> > >>>>> up the
>>> > >>>>> > > XML
>>> > >>>>> > > > >> to
>>> > >>>>> > > > >> > >> make
>>> > >>>>> > > > >> > >> many, many more plots in each call. Then
check to
>>> see
>>> > if
>>> > >>>>> > running
>>> > >>>>> > > > >> > METviewer
>>> > >>>>> > > > >> > >> in this way is more stable.
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> I don't think I fully appreciate all of the
logic
>>> that
>>> > >>>>> lives in
>>> > >>>>> > > > >> > >> Time_Series_file.bsub, Plot_XML_builder.py,
and
>>> > >>>>> > > > >> test_Time_Series_AGG.sh.
>>> > >>>>> > > > >> > >> However, it *might* be possible to construct
a
>>> single
>>> > >>>>> XML file
>>> > >>>>> > > > which
>>> > >>>>> > > > >> > >> produces all 656 plots.
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> Let me know what you think and how you'd
like to
>>> > proceed.
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> Thanks,
>>> > >>>>> > > > >> > >> John
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> On Wed, Nov 4, 2020 at 6:22 AM Edward
Strobach -
>>> NOAA
>>> > >>>>> Affiliate
>>> > >>>>> > > via
>>> > >>>>> > > > >> RT <
>>> > >>>>> > > > >> > >> met_help at ucar.edu> wrote:
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > <URL:
>>> > >>>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>> > >>>>> > > >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > Hi John,
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > No problem.
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > I'm including my responses below:
>>> > >>>>> > > > >> > >> > Sorry for the delay. I took a look on
wcoss
>>> today.
>>> > You
>>> > >>>>> > describe
>>> > >>>>> > > > >> that
>>> > >>>>> > > > >> > >> your
>>> > >>>>> > > > >> > >> > plotting script has stopped working but
you're
>>> not
>>> > >>>>> sure why.
>>> > >>>>> > > > >> > >> > It seems more intermittent. Take a look
at the
>>> > example
>>> > >>>>> > below.
>>> > >>>>> > > > You
>>> > >>>>> > > > >> > can
>>> > >>>>> > > > >> > >> see
>>> > >>>>> > > > >> > >> > that for some reason there are gaps in the
plots
>>> > being
>>> > >>>>> > > generated.
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel
34105
>>> Oct 4
>>> > >>>>> 14:52
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 41781 Oct 5 15:15
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-02.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 56691 Oct 19 15:02
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 55548 Oct 20 16:08
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 55598 Oct 21 16:08
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 55599 Oct 22 16:09
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 55352 Oct 24 16:34
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 26175 Oct 31 08:20
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 26178 Nov 1 08:20
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-29.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 26160 Nov 3 08:22
>>> > >>>>> > > > >> > >> >
>>> PMTF_FULL_102020_2020-10-01_2020-10-31.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 25354 Nov 3 08:22
>>> > >>>>> > > > >> TMP_FULL_102020_2020-10-01_2020-10-31.png*
>>> > >>>>> > > > >> > >> > You can tell when a plot is successful by
>>> looking at
>>> > >>>>> the file
>>> > >>>>> > > > >> size. I
>>> > >>>>> > > > >> > >> > highlighted an example of that in red.
However,
>>> the
>>> > >>>>> files
>>> > >>>>> > > > showing
>>> > >>>>> > > > >> > about
>>> > >>>>> > > > >> > >> > 25000 are the ones where no line plots are
being
>>> > >>>>> produced.
>>> > >>>>> > You
>>> > >>>>> > > > can
>>> > >>>>> > > > >> > also
>>> > >>>>> > > > >> > >> > see that stat files are generated
everyday, so as
>>> > long
>>> > >>>>> as I'm
>>> > >>>>> > > > >> loading
>>> > >>>>> > > > >> > >> the
>>> > >>>>> > > > >> > >> > data, which I am, I should be able to use
the xml
>>> > >>>>> files to
>>> > >>>>> > > > generate
>>> > >>>>> > > > >> > the
>>> > >>>>> > > > >> > >> > plots. I'm attaching a directory list
from wcoss
>>> > that
>>> > >>>>> shows
>>> > >>>>> > > the
>>> > >>>>> > > > >> dates
>>> > >>>>> > > > >> > >> > where data is produced.
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > *20190801 20190817 20200624 20200710
20200726
>>> > >>>>> 20200812
>>> > >>>>> > > > >> 20200828
>>> > >>>>> > > > >> > >> > 20201003 2020101920190802 20190818
20200625
>>> > >>>>> 20200711
>>> > >>>>> > > > 20200727
>>> > >>>>> > > > >> > >> > 20200813 20200829 20201004
2020102020190803
>>> > >>>>> 20190819
>>> > >>>>> > > > 20200626
>>> > >>>>> > > > >> > >> > 20200712 20200728 20200814 20200830
20201005
>>> > >>>>> > > > 2020102120190804
>>> > >>>>> > > > >> > >> > 20190820 20200627 20200713 20200729
20200815
>>> > >>>>> 20200831
>>> > >>>>> > > > >> 20201006
>>> > >>>>> > > > >> > >> > 2020102220190805 20190821 20200628
20200714
>>> > >>>>> 20200730
>>> > >>>>> > > > 20200816
>>> > >>>>> > > > >> > >> > 20200921 20201007 2020102320190806
20190822
>>> > >>>>> 20200629
>>> > >>>>> > > > 20200715
>>> > >>>>> > > > >> > >> > 20200731 20200817 20200922 20201008
>>> > >>>>> 2020102420190807
>>> > >>>>> > > > 20190823
>>> > >>>>> > > > >> > >> > 20200630 20200716 20200801 20200818
20200923
>>> > >>>>> 20201009
>>> > >>>>> > > > >> > >> > 2020102520190808 20190824 20200701
20200717
>>> > >>>>> 20200802
>>> > >>>>> > > > 20200819
>>> > >>>>> > > > >> > >> > 20200924 20201010 2020102620190809
20190825
>>> > >>>>> 20200702
>>> > >>>>> > > > 20200718
>>> > >>>>> > > > >> > >> > 20200803 20200820 20200925 20201011
>>> > >>>>> 2020102720190810
>>> > >>>>> > > > 20190826
>>> > >>>>> > > > >> > >> > 20200703 20200719 20200804 20200821
20200926
>>> > >>>>> 20201012
>>> > >>>>> > > > >> > >> > 2020102820190811 20190827 20200704
20200720
>>> > >>>>> 20200805
>>> > >>>>> > > > 20200822
>>> > >>>>> > > > >> > >> > 20200927 20201013 2020102920190812
20190828
>>> > >>>>> 20200705
>>> > >>>>> > > > 20200721
>>> > >>>>> > > > >> > >> > 20200806 20200823 20200928 20201014
>>> > >>>>> 2020103020190813
>>> > >>>>> > > > 20190829
>>> > >>>>> > > > >> > >> > 20200706 20200722 20200807 20200824
20200929
>>> > >>>>> 20201015
>>> > >>>>> > > > >> > >> > 2020103120190814 20190830 20200707
20200723
>>> > >>>>> 20200808
>>> > >>>>> > > > 20200825
>>> > >>>>> > > > >> > >> > 20200930 20201016 2020110120190815
20200622
>>> > >>>>> 20200708
>>> > >>>>> > > > 20200724
>>> > >>>>> > > > >> > >> > 20200810 20200826 20201001 20201017
>>> > >>>>> 2020110220190816
>>> > >>>>> > > > 20200623
>>> > >>>>> > > > >> > >> > 20200709 20200725 20200811 20200827
20201002
>>> > >>>>> 20201018*
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > Furthermore, what's interesting is that I
have
>>> mixed
>>> > >>>>> success
>>> > >>>>> > > with
>>> > >>>>> > > > >> > other
>>> > >>>>> > > > >> > >> > cases. Using the same field - PMTF - for
the
>>> same
>>> > >>>>> cycle -
>>> > >>>>> > the
>>> > >>>>> > > > 06z
>>> > >>>>> > > > >> > >> cycle -
>>> > >>>>> > > > >> > >> > you can see that some of the dates where
data was
>>> > >>>>> plotted for
>>> > >>>>> > > > EAST
>>> > >>>>> > > > >> did
>>> > >>>>> > > > >> > >> not
>>> > >>>>> > > > >> > >> > necessarily plot for FULL. This surprises
me
>>> because
>>> > >>>>> EAST
>>> > >>>>> > is a
>>> > >>>>> > > > >> subset
>>> > >>>>> > > > >> > >> of
>>> > >>>>> > > > >> > >> > FULL. See below:
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > *-rw-r--r-- 1 Edward.Strobach emcmodel
36647
>>> Oct 4
>>> > >>>>> 14:54
>>> > >>>>> > > > >> > >> >
>>> PMTF_EAST_102020_2020-10-01_2020-10-01.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 49039 Oct 5 15:17
>>> > >>>>> > > > >> > >> >
>>> PMTF_EAST_102020_2020-10-01_2020-10-02.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 56791 Oct 20 16:09
>>> > >>>>> > > > >> > >> >
>>> PMTF_EAST_102020_2020-10-01_2020-10-17.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 56834 Oct 21 16:09
>>> > >>>>> > > > >> > >> >
>>> PMTF_EAST_102020_2020-10-01_2020-10-18.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 56921 Oct 22 16:11
>>> > >>>>> > > > >> > >> >
>>> PMTF_EAST_102020_2020-10-01_2020-10-19.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 56518 Oct 24 16:36
>>> > >>>>> > > > >> > >> >
>>> PMTF_EAST_102020_2020-10-01_2020-10-21.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 61187 Oct 31 08:21
>>> > >>>>> > > > >> > >> >
>>> PMTF_EAST_102020_2020-10-01_2020-10-28.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 26475 Nov 2 08:22
>>> > >>>>> > > > >> > >> >
>>> PMTF_EAST_102020_2020-10-01_2020-10-30.png-rw-r--r--
>>> > 1
>>> > >>>>> > > > >> Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 26455 Nov 3 08:23
>>> > >>>>> > > > >> > PMTF_EAST_102020_2020-10-01_2020-10-31.png*
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > What I find equally concerning as that
sometimes
>>> > plots
>>> > >>>>> are
>>> > >>>>> > > > >> incorrectly
>>> > >>>>> > > > >> > >> > stored in the wrong directory list as
>>> highlighted in
>>> > >>>>> green
>>> > >>>>> > > above.
>>> > >>>>> > > > >> I've
>>> > >>>>> > > > >> > >> > consulted the plot xml file for PMTF, for
>>> instance,
>>> > >>>>> and find
>>> > >>>>> > > > >> nothing
>>> > >>>>> > > > >> > >> > problematic. The only setting that might
pose an
>>> > >>>>> issue when
>>> > >>>>> > it
>>> > >>>>> > > > >> comes
>>> > >>>>> > > > >> > to
>>> > >>>>> > > > >> > >> > generating line plots is when I set
event_equal
>>> to
>>> > >>>>> true.
>>> > >>>>> > > > However,
>>> > >>>>> > > > >> all
>>> > >>>>> > > > >> > >> the
>>> > >>>>> > > > >> > >> > stat files are populated for all
experiments. In
>>> the
>>> > >>>>> case of
>>> > >>>>> > > > ozone
>>> > >>>>> > > > >> you
>>> > >>>>> > > > >> > >> see
>>> > >>>>> > > > >> > >> > something different. I find that ozone
tends to
>>> > result
>>> > >>>>> in
>>> > >>>>> > > > populated
>>> > >>>>> > > > >> > >> plots
>>> > >>>>> > > > >> > >> > but with intermittent reporting different
than
>>> PMTF -
>>> > >>>>> see
>>> > >>>>> > > below:
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > *[Edward.Strobach at m72a1 202010]$ ls
-ltrtotal
>>> > >>>>> > 6144-rw-r--r-- 1
>>> > >>>>> > > > >> > >> > Edward.Strobach emcmodel 37861 Oct 4
10:36
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-01.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 54318 Oct 6 10:51
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-03.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 54305 Oct 7 10:49
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-04.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 62179 Oct 13 13:25
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-10.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 63811 Oct 19 12:18
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-12.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 60112 Oct 20 12:21
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-17.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 60165 Oct 21 12:33
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-18.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 60160 Oct 22 12:21
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-19.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 60056 Oct 23 12:40
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-20.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 58517 Oct 24 13:31
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-21.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 58062 Oct 31 14:17
>>> > >>>>> > > > >> > >> >
>>> > OZCON_FULL_102020_2020-10-01_2020-10-28.png-rw-r--r-- 1
>>> > >>>>> > > > >> > Edward.Strobach
>>> > >>>>> > > > >> > >> > emcmodel 58119 Nov 1 14:19
>>> > >>>>> > > > >> > OZCON_FULL_102020_2020-10-01_2020-10-29.png*
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > Similar issues have been found for other
fields,
>>> > >>>>> cycles, and
>>> > >>>>> > > > >> regions
>>> > >>>>> > > > >> > >> > including verification being done for
>>> meteorology.
>>> > >>>>> > > > >> > >> > I can't seem to reason this intermittency
and
>>> > >>>>> inconsistency
>>> > >>>>> > > > between
>>> > >>>>> > > > >> > >> > reporting since the xml files are correct
and I'm
>>> > >>>>> using the
>>> > >>>>> > > same
>>> > >>>>> > > > >> > >> procedure
>>> > >>>>> > > > >> > >> > that I've always used. I do realize that
I need
>>> to
>>> > >>>>> start
>>> > >>>>> > > > creating
>>> > >>>>> > > > >> a
>>> > >>>>> > > > >> > >> better
>>> > >>>>> > > > >> > >> > way to manage all these files that are
being
>>> > produced,
>>> > >>>>> both
>>> > >>>>> > in
>>> > >>>>> > > > >> terms
>>> > >>>>> > > > >> > of
>>> > >>>>> > > > >> > >> > data/plots and xml files.
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > I ran across a rather large directory:
>>> > >>>>> > > > >> > >> > ls
>>> > >>>>> > > > >>
>>> > >>>>>
>>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>>> > >>>>> > > > >> > |
>>> > >>>>> > > > >> > >> wc
>>> > >>>>> > > > >> > >> > -w
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > That has 71,398 xml files in there. But I
see
>>> > >>>>> timestamps in
>>> > >>>>> > > those
>>> > >>>>> > > > >> > >> filenames
>>> > >>>>> > > > >> > >> > from 20200921 to 20200925.
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > Is it the case that this stopped working
on
>>> 20200925?
>>> > >>>>> Or are
>>> > >>>>> > > > those
>>> > >>>>> > > > >> > dated
>>> > >>>>> > > > >> > >> > filenames remnants from older work?
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > So it hasn't been working for over a
month. Is
>>> that
>>> > >>>>> right?
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > There are a lot of files and I need to
begin
>>> managing
>>> > >>>>> that
>>> > >>>>> > > > >> better. I
>>> > >>>>> > > > >> > >> think
>>> > >>>>> > > > >> > >> > during that time there was a machine
switch so it
>>> > >>>>> stopped on
>>> > >>>>> > > that
>>> > >>>>> > > > >> day.
>>> > >>>>> > > > >> > >> > I've been able to produce plots even now,
but
>>> it's
>>> > not
>>> > >>>>> > > consistent
>>> > >>>>> > > > >> in
>>> > >>>>> > > > >> > >> > reporting while some regions generate
plots and
>>> > others
>>> > >>>>> do
>>> > >>>>> > not.
>>> > >>>>> > > > >> I've
>>> > >>>>> > > > >> > >> > reduced the level processing in hopes that
I
>>> don't
>>> > hit
>>> > >>>>> a walk
>>> > >>>>> > > > clock
>>> > >>>>> > > > >> > >> issue
>>> > >>>>> > > > >> > >> > since I'm submitting batch jobs. However,
I've
>>> run
>>> > >>>>> this step
>>> > >>>>> > > > >> outside
>>> > >>>>> > > > >> > of
>>> > >>>>> > > > >> > >> > batch and it always seems to work. This
issue
>>> seems
>>> > to
>>> > >>>>> happen
>>> > >>>>> > > > with
>>> > >>>>> > > > >> > batch
>>> > >>>>> > > > >> > >> > and it is not at all clear why. Let me
know what
>>> > else
>>> > >>>>> you
>>> > >>>>> > need
>>> > >>>>> > > > >> from
>>> > >>>>> > > > >> > me
>>> > >>>>> > > > >> > >> to
>>> > >>>>> > > > >> > >> > better coordinate what I would need to do
next.
>>> > >>>>> Thanks.
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > On Tue, Nov 3, 2020 at 7:28 PM John Halley
>>> Gotway via
>>> > >>>>> RT <
>>> > >>>>> > > > >> > >> > met_help at ucar.edu>
>>> > >>>>> > > > >> > >> > wrote:
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > > Hi Ed,
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > > Sorry for the delay. I took a look on
wcoss
>>> today.
>>> > >>>>> You
>>> > >>>>> > > describe
>>> > >>>>> > > > >> that
>>> > >>>>> > > > >> > >> your
>>> > >>>>> > > > >> > >> > > plotting script has stopped working but
you're
>>> not
>>> > >>>>> sure
>>> > >>>>> > why.
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > > I ran across a rather large directory:
>>> > >>>>> > > > >> > >> > > ls
>>> > >>>>> > > > >> >
>>> > >>>>>
>>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/plot_xmls
>>> > >>>>> > > > >> > >> |
>>> > >>>>> > > > >> > >> > wc
>>> > >>>>> > > > >> > >> > > -w
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > > That has 71,398 xml files in there. But
I see
>>> > >>>>> timestamps in
>>> > >>>>> > > > those
>>> > >>>>> > > > >> > >> > filenames
>>> > >>>>> > > > >> > >> > > from 20200921 to 20200925.
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > > Is it the case that this stopped working
on
>>> > >>>>> 20200925? Or
>>> > >>>>> > are
>>> > >>>>> > > > >> those
>>> > >>>>> > > > >> > >> dated
>>> > >>>>> > > > >> > >> > > filenames remnants from older work?
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > > So it hasn't been working for over a
month. Is
>>> that
>>> > >>>>> right?
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > > John
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > > On Sun, Nov 1, 2020 at 8:50 PM The RT
System
>>> itself
>>> > >>>>> via RT
>>> > >>>>> > <
>>> > >>>>> > > > >> > >> > > met_help at ucar.edu> wrote:
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > > >
>>> > >>>>> > > > >> > >> > > > Sun Nov 01 20:49:52 2020: Request
97283 was
>>> acted
>>> > >>>>> upon.
>>> > >>>>> > > > >> > >> > > > Transaction: Given to johnhg (John
Halley
>>> Gotway)
>>> > >>>>> by
>>> > >>>>> > > > RT_System
>>> > >>>>> > > > >> > >> > > > Queue: met_help
>>> > >>>>> > > > >> > >> > > > Subject: plots no longer being
generated
>>> > >>>>> > > > >> > >> > > > Owner: johnhg
>>> > >>>>> > > > >> > >> > > > Requestors: edward.strobach at noaa.gov
>>> > >>>>> > > > >> > >> > > > Status: new
>>> > >>>>> > > > >> > >> > > > Ticket <URL:
>>> > >>>>> > > > >> > >>
>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97283
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > > >
>>> > >>>>> > > > >> > >> > > >
>>> > >>>>> > > > >> > >> > > > This transaction appears to have no
content
>>> > >>>>> > > > >> > >> > > >
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> > >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> > --
>>> > >>>>> > > > >> > >> > Edward Strobach
>>> > >>>>> > > > >> > >> > EMC/NCEP/NWS/
>>> > >>>>> > > > >> > >> > IMSG Contractor
>>> > >>>>> > > > >> > >> > Cubicle#: 2029
>>> > >>>>> > > > >> > >> > 301-683-3717
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >> >
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >>
>>> > >>>>> > > > >> > >
>>> > >>>>> > > > >> > > --
>>> > >>>>> > > > >> > > Edward Strobach
>>> > >>>>> > > > >> > > EMC/NCEP/NWS/
>>> > >>>>> > > > >> > > IMSG Contractor
>>> > >>>>> > > > >> > > Cubicle#: 2029
>>> > >>>>> > > > >> > > 301-683-3717
>>> > >>>>> > > > >> > >
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >> > --
>>> > >>>>> > > > >> > Edward Strobach
>>> > >>>>> > > > >> > EMC/NCEP/NWS/
>>> > >>>>> > > > >> > IMSG Contractor
>>> > >>>>> > > > >> > Cubicle#: 2029
>>> > >>>>> > > > >> > 301-683-3717
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >> >
>>> > >>>>> > > > >>
>>> > >>>>> > > > >>
>>> > >>>>> > > > >
>>> > >>>>> > > > > --
>>> > >>>>> > > > > Edward Strobach
>>> > >>>>> > > > > EMC/NCEP/NWS/
>>> > >>>>> > > > > IMSG Contractor
>>> > >>>>> > > > > Cubicle#: 2029
>>> > >>>>> > > > > 301-683-3717
>>> > >>>>> > > > >
>>> > >>>>> > > >
>>> > >>>>> > > >
>>> > >>>>> > > > --
>>> > >>>>> > > > Edward Strobach
>>> > >>>>> > > > EMC/NCEP/NWS/
>>> > >>>>> > > > IMSG Contractor
>>> > >>>>> > > > Cubicle#: 2029
>>> > >>>>> > > > 301-683-3717
>>> > >>>>> > > >
>>> > >>>>> > > >
>>> > >>>>> > >
>>> > >>>>> > >
>>> > >>>>> >
>>> > >>>>> > --
>>> > >>>>> > Edward Strobach
>>> > >>>>> > EMC/NCEP/NWS/
>>> > >>>>> > IMSG Contractor
>>> > >>>>> > Cubicle#: 2029
>>> > >>>>> > 301-683-3717
>>> > >>>>> >
>>> > >>>>> >
>>> > >>>>>
>>> > >>>>>
>>> > >>>>
>>> > >>>> --
>>> > >>>> Edward Strobach
>>> > >>>> EMC/NCEP/NWS/
>>> > >>>> IMSG Contractor
>>> > >>>> Cubicle#: 2029
>>> > >>>> 301-683-3717
>>> > >>>>
>>> > >>>
>>> > >>>
>>> > >>> --
>>> > >>> Edward Strobach
>>> > >>> EMC/NCEP/NWS/
>>> > >>> IMSG Contractor
>>> > >>> Cubicle#: 2029
>>> > >>> 301-683-3717
>>> > >>>
>>> > >>
>>> > >>
>>> > >> --
>>> > >> Edward Strobach
>>> > >> EMC/NCEP/NWS/
>>> > >> IMSG Contractor
>>> > >> Cubicle#: 2029
>>> > >> 301-683-3717
>>> > >>
>>> > >
>>> > >
>>> > > --
>>> > > Edward Strobach
>>> > > EMC/NCEP/NWS/
>>> > > IMSG Contractor
>>> > > Cubicle#: 2029
>>> > > 301-683-3717
>>> > >
>>> >
>>> >
>>> > --
>>> > Edward Strobach
>>> > EMC/NCEP/NWS/
>>> > IMSG Contractor
>>> > Cubicle#: 2029
>>> > 301-683-3717
>>> >
>>> >
>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
More information about the Met_help
mailing list