[Met_help] [rt.rap.ucar.edu #95596] History for setting up batch mode; communicating results with metviewer online; other remaining issues
Tatiana Burek via RT
met_help at ucar.edu
Tue Oct 13 12:13:19 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning,
Over the last two weeks I've made an effort to understand metviewer, test
different cases using the history tab, changing information to see what
works and what does not, and make connections between the GUI settings and
the XML file. Although my comfort level with metviewer has significantly
increased during this time, there are some remaining issues that I'm hoping
to clear up.
1. Problem changing XML file and running with different statistic and plot
type:
Ho-Chun Huang, a colleague at EMC, provided me the working directory to get
started which I copied over:
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
He has a test.sh script that I adjusted and used to run his example.
running of the script "bash test.sh" produced a CSI plot that was stored
here:
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
Running the script produced the expected result. The next natural step was
to copy hochum_xml.template and create a new xml file called
Strobach_xml.template in order to create a time series where the
independent time is fcst_valid_beg instead of obs_thresh. Accordingly, I
made changes to the fixed fields (removing fcst_valid_beg) and changed the
statistic from CSI to FMEAN (FBAR was also tried). The error that resulted
is as follows:
AND BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00', '2020-05-01
14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00', '2020-05-01
17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00', '2020-05-01
20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00', '2020-05-01
23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00', '2020-05-02
02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00', '2020-05-02
05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00', '2020-05-02
08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00', '2020-05-02
11:00:00', '2020-05-02 12:00:00')
AND BINARY h.fcst_var = 'AOTK'
AND ld.stat_header_id = h.stat_header_id
AND ld.fmean != -9999;
No data was returned from database
2020-06-09
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
Source) -1 printFormattedTable - ** ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException in printFormattedTable(ResultSet
res): result set contained no data
** ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException in
printFormattedTable(ResultSet res): result set contained no data
** ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException in
printFormattedTable(ResultSet res): result set contained no data
Apparently there was no data during this time. I tried changing the range
of valid times and changed the valid time to different days just in case
data was missing. The issue was the same. To confirm my logic, I
reproduced my attempted approach on metviewer, which I found to work.
2. Running in batch mode
I think I lack an understanding of how to set up batch mode because I don't
see where these definitions are being explicitly defined:
<plot_spec>
<connection>
<host>rds_host:3306</host>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
<user>rds_user</user>
<password>rds_pwd</password>
</connection>
<rscript>Rscript</rscript>
<folders>
<r_tmpl>rds_R_tmpl</r_tmpl>
<r_work>rds_R_work</r_work>
<plots>rds_plots</plots>
<data>rds_data</data>
<scripts>rds_scripts</scripts>
</folders>
<plot>
<template>series_plot.R_tmpl</template>
I see the <folders> section, but grepping all available files does not
reveal how rds_R_work is being defined, as well as other similarly defined
variables. I think if I know how these are being defined, then I'll be
able to pipe in my information and process the data using met and produce
results in batch mode. Am I right on that? Also, if I want to use obs for
verification, do I add another line in the XML file that designates file
location of obs? I assume that other steps have to be run before all
this. If data is in prepbufr form, for example, I would have to separately
run PB2NC. If I want grid2obs, then I would want to do that as well.
3. Communicating results to metviewer online
Lastly, I see a lot of people/projects listed here - see attachment
How do I get the results I generate to be uploaded to this location? It's
not clear how others are setting this up.
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Fri Jun 12 10:51:21 2020
Hi Edward,
#1. I don't have an access to NOAA servers and can't view your XML for
the batch mode. Please, attache it to this issue and I will take a
look.
#2. All parameters in <folders> section should NOT be changed and will
be replaced by other scripts during the batch execution. The data for
the batch mode is coming from the database and not from files. The
name of the database is in <database> tag.
#3. Users rum MET and produce MET output files. They use METviewer
load mode to load these files to the specific database. In the config
file for this process they also can specify the database group. For
example database mv_dev is in Test12 group.
After the data has been loaded to the database it is visible and
available in the GUI and also can be used by the batch mode.
You can always add more data to the database using the same load mode.
Tatiana
On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> Good morning,
>
> Over the last two weeks I've made an effort to understand metviewer,
> test
> different cases using the history tab, changing information to see
> what
> works and what does not, and make connections between the GUI
settings
> and
> the XML file. Although my comfort level with metviewer has
> significantly
> increased during this time, there are some remaining issues that I'm
> hoping
> to clear up.
>
> 1. Problem changing XML file and running with different statistic
and
> plot
> type:
>
> Ho-Chun Huang, a colleague at EMC, provided me the working directory
> to get
> started which I copied over:
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>
> He has a test.sh script that I adjusted and used to run his example.
> running of the script "bash test.sh" produced a CSI plot that was
> stored
> here:
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>
> Running the script produced the expected result. The next natural
> step was
> to copy hochum_xml.template and create a new xml file called
> Strobach_xml.template in order to create a time series where the
> independent time is fcst_valid_beg instead of obs_thresh.
Accordingly,
> I
> made changes to the fixed fields (removing fcst_valid_beg) and
changed
> the
> statistic from CSI to FMEAN (FBAR was also tried). The error that
> resulted
> is as follows:
>
> AND BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00', '2020-05-
01
> 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00', '2020-05-01
> 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00', '2020-05-01
> 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00', '2020-05-01
> 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00', '2020-05-02
> 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00', '2020-05-02
> 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00', '2020-05-02
> 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00', '2020-05-02
> 11:00:00', '2020-05-02 12:00:00')
> AND BINARY h.fcst_var = 'AOTK'
> AND ld.stat_header_id = h.stat_header_id
> AND ld.fmean != -9999;
> No data was returned from database
> 2020-06-09
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> Source) -1 printFormattedTable - ** ERROR: Caught class
> edu.ucar.metviewer.EmptyResultSetException in
> printFormattedTable(ResultSet
> res): result set contained no data
> ** ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException
in
> printFormattedTable(ResultSet res): result set contained no data
> ** ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException
in
> printFormattedTable(ResultSet res): result set contained no data
>
> Apparently there was no data during this time. I tried changing the
> range
> of valid times and changed the valid time to different days just in
> case
> data was missing. The issue was the same. To confirm my logic, I
> reproduced my attempted approach on metviewer, which I found to
work.
>
> 2. Running in batch mode
>
> I think I lack an understanding of how to set up batch mode because
I
> don't
> see where these definitions are being explicitly defined:
>
> <plot_spec>
> <connection>
> <host>rds_host:3306</host>
> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> <user>rds_user</user>
> <password>rds_pwd</password>
> </connection>
> <rscript>Rscript</rscript>
> <folders>
> <r_tmpl>rds_R_tmpl</r_tmpl>
> <r_work>rds_R_work</r_work>
> <plots>rds_plots</plots>
> <data>rds_data</data>
> <scripts>rds_scripts</scripts>
> </folders>
> <plot>
> <template>series_plot.R_tmpl</template>
>
> I see the <folders> section, but grepping all available files does
not
> reveal how rds_R_work is being defined, as well as other similarly
> defined
> variables. I think if I know how these are being defined, then I'll
> be
> able to pipe in my information and process the data using met and
> produce
> results in batch mode. Am I right on that? Also, if I want to use
> obs for
> verification, do I add another line in the XML file that designates
> file
> location of obs? I assume that other steps have to be run before
all
> this. If data is in prepbufr form, for example, I would have to
> separately
> run PB2NC. If I want grid2obs, then I would want to do that as
well.
>
> 3. Communicating results to metviewer online
>
> Lastly, I see a lot of people/projects listed here - see attachment
>
> How do I get the results I generate to be uploaded to this location?
> It's
> not clear how others are setting this up.
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Jun 12 11:08:43 2020
Hi,
I'm attaching the xml below, but am having hard time following how to
go
about the other steps since I have zero experience on how to do these
things.
#2. All parameters in <folders> section should NOT be changed and will
be
replaced by other scripts during the batch execution. The data for the
batch mode is coming from the database and not from files. The name of
the
database is in <database> tag.
I understand that nothing should be changed in the folders section,
but
there is no indication how and where these variables are being
defined.
Grepping these names of the files available does not reveal formal
definitions of these variables. If I understood exactly how these
were
being defined, then I would be able connect this better.
#3. Users rum MET and produce MET output files. They use METviewer
load
mode to load these files to the specific database. In the config file
for
this process they also can specify the database group. For example
database
mv_dev is in Test12 group.
After the data has been loaded to the database it is visible and
available
in the GUI and also can be used by the batch mode.
You can always add more data to the database using the same load mode.
Is there a guide for me to understand the process that you are
referring to?
On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
> #1. I don't have an access to NOAA servers and can't view your XML
for the
> batch mode. Please, attache it to this issue and I will take a look.
>
> #2. All parameters in <folders> section should NOT be changed and
will be
> replaced by other scripts during the batch execution. The data for
the
> batch mode is coming from the database and not from files. The name
of the
> database is in <database> tag.
>
> #3. Users rum MET and produce MET output files. They use METviewer
load
> mode to load these files to the specific database. In the config
file for
> this process they also can specify the database group. For example
database
> mv_dev is in Test12 group.
> After the data has been loaded to the database it is visible and
available
> in the GUI and also can be used by the batch mode.
> You can always add more data to the database using the same load
mode.
>
> Tatiana
>
> On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> > Good morning,
> >
> > Over the last two weeks I've made an effort to understand
metviewer,
> > test
> > different cases using the history tab, changing information to see
> > what
> > works and what does not, and make connections between the GUI
settings
> > and
> > the XML file. Although my comfort level with metviewer has
> > significantly
> > increased during this time, there are some remaining issues that
I'm
> > hoping
> > to clear up.
> >
> > 1. Problem changing XML file and running with different statistic
and
> > plot
> > type:
> >
> > Ho-Chun Huang, a colleague at EMC, provided me the working
directory
> > to get
> > started which I copied over:
> > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >
> > He has a test.sh script that I adjusted and used to run his
example.
> > running of the script "bash test.sh" produced a CSI plot that was
> > stored
> > here:
> >
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >
> > Running the script produced the expected result. The next natural
> > step was
> > to copy hochum_xml.template and create a new xml file called
> > Strobach_xml.template in order to create a time series where the
> > independent time is fcst_valid_beg instead of obs_thresh.
Accordingly,
> > I
> > made changes to the fixed fields (removing fcst_valid_beg) and
changed
> > the
> > statistic from CSI to FMEAN (FBAR was also tried). The error that
> > resulted
> > is as follows:
> >
> > AND BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00', '2020-
05-01
> > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00', '2020-05-
01
> > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00', '2020-05-
01
> > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00', '2020-05-
01
> > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00', '2020-05-
02
> > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00', '2020-05-
02
> > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00', '2020-05-
02
> > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00', '2020-05-
02
> > 11:00:00', '2020-05-02 12:00:00')
> > AND BINARY h.fcst_var = 'AOTK'
> > AND ld.stat_header_id = h.stat_header_id
> > AND ld.fmean != -9999;
> > No data was returned from database
> > 2020-06-09
> >
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > Source) -1 printFormattedTable - ** ERROR: Caught class
> > edu.ucar.metviewer.EmptyResultSetException in
> > printFormattedTable(ResultSet
> > res): result set contained no data
> > ** ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException
in
> > printFormattedTable(ResultSet res): result set contained no data
> > ** ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException
in
> > printFormattedTable(ResultSet res): result set contained no data
> >
> > Apparently there was no data during this time. I tried changing
the
> > range
> > of valid times and changed the valid time to different days just
in
> > case
> > data was missing. The issue was the same. To confirm my logic, I
> > reproduced my attempted approach on metviewer, which I found to
work.
> >
> > 2. Running in batch mode
> >
> > I think I lack an understanding of how to set up batch mode
because I
> > don't
> > see where these definitions are being explicitly defined:
> >
> > <plot_spec>
> > <connection>
> > <host>rds_host:3306</host>
> > <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > <user>rds_user</user>
> > <password>rds_pwd</password>
> > </connection>
> > <rscript>Rscript</rscript>
> > <folders>
> > <r_tmpl>rds_R_tmpl</r_tmpl>
> > <r_work>rds_R_work</r_work>
> > <plots>rds_plots</plots>
> > <data>rds_data</data>
> > <scripts>rds_scripts</scripts>
> > </folders>
> > <plot>
> > <template>series_plot.R_tmpl</template>
> >
> > I see the <folders> section, but grepping all available files does
not
> > reveal how rds_R_work is being defined, as well as other similarly
> > defined
> > variables. I think if I know how these are being defined, then
I'll
> > be
> > able to pipe in my information and process the data using met and
> > produce
> > results in batch mode. Am I right on that? Also, if I want to
use
> > obs for
> > verification, do I add another line in the XML file that
designates
> > file
> > location of obs? I assume that other steps have to be run before
all
> > this. If data is in prepbufr form, for example, I would have to
> > separately
> > run PB2NC. If I want grid2obs, then I would want to do that as
well.
> >
> > 3. Communicating results to metviewer online
> >
> > Lastly, I see a lot of people/projects listed here - see
attachment
> >
> > How do I get the results I generate to be uploaded to this
location?
> > It's
> > not clear how others are setting this up.
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Fri Jun 12 12:22:08 2020
I found a few problems in your XML.
The database you are using doesn't have data with vx_mask 'abcREGabc'.
That is why METviewer returns an empty dataset and can't create a
plot.
The other problem is how you define fcst_lead. This database has
values like 10000, 20000 and not 010000, 020000. Since you include all
fcst_leads this variable can be omitted.
To minimize problems with variables values, I would suggest to create
XML using GUI and then edit it to fit batch standards.
I created a plot using you configurations with GUI (without vx_mack
and plot and axis titles) #20200612_173626. You can load it back to
the page by using the "History" tab (left bar on GUI). Find the plot,
open XML and click on "Load to page". After that you can recreate a
plot and edit the configurations. You can also download a new XML and
edit it for batch.
I attached the example of load XML to this message.
Description of rds parameters:
rds_R_tmpl and rds_R_work - location of METviewer R scripts
rds_plots - temporary directory where the plots will be generated
rds_data - temporary directory where the data retrieved from the
database and other intermediate files will be stored
rds_scripts - temporary directory where R scripts for this run will be
stored
After the execution is done all the files from temporary directories
would be removed.
We don't have the offitial user guide for METviewer. There are some
instructions on how to crate XML config files:
https://metviewer.nws.noaa.gov/doc/index.html
Tatiana
On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
> Hi,
>
> I'm attaching the xml below, but am having hard time following how
to
> go
> about the other steps since I have zero experience on how to do
these
> things.
>
> #2. All parameters in <folders> section should NOT be changed and
will
> be
> replaced by other scripts during the batch execution. The data for
the
> batch mode is coming from the database and not from files. The name
of
> the
> database is in <database> tag.
>
> I understand that nothing should be changed in the folders section,
> but
> there is no indication how and where these variables are being
> defined.
> Grepping these names of the files available does not reveal formal
> definitions of these variables. If I understood exactly how these
> were
> being defined, then I would be able connect this better.
>
> #3. Users rum MET and produce MET output files. They use METviewer
> load
> mode to load these files to the specific database. In the config
file
> for
> this process they also can specify the database group. For example
> database
> mv_dev is in Test12 group.
> After the data has been loaded to the database it is visible and
> available
> in the GUI and also can be used by the batch mode.
> You can always add more data to the database using the same load
mode.
>
> Is there a guide for me to understand the process that you are
> referring to?
>
>
>
>
>
>
>
>
>
>
> On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> > #1. I don't have an access to NOAA servers and can't view your XML
> > for the
> > batch mode. Please, attache it to this issue and I will take a
look.
> >
> > #2. All parameters in <folders> section should NOT be changed and
> > will be
> > replaced by other scripts during the batch execution. The data for
> > the
> > batch mode is coming from the database and not from files. The
name
> > of the
> > database is in <database> tag.
> >
> > #3. Users rum MET and produce MET output files. They use METviewer
> > load
> > mode to load these files to the specific database. In the config
file
> > for
> > this process they also can specify the database group. For example
> > database
> > mv_dev is in Test12 group.
> > After the data has been loaded to the database it is visible and
> > available
> > in the GUI and also can be used by the batch mode.
> > You can always add more data to the database using the same load
> > mode.
> >
> > Tatiana
> >
> > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> > > Good morning,
> > >
> > > Over the last two weeks I've made an effort to understand
> > > metviewer,
> > > test
> > > different cases using the history tab, changing information to
see
> > > what
> > > works and what does not, and make connections between the GUI
> > > settings
> > > and
> > > the XML file. Although my comfort level with metviewer has
> > > significantly
> > > increased during this time, there are some remaining issues that
> > > I'm
> > > hoping
> > > to clear up.
> > >
> > > 1. Problem changing XML file and running with different
statistic
> > > and
> > > plot
> > > type:
> > >
> > > Ho-Chun Huang, a colleague at EMC, provided me the working
> > > directory
> > > to get
> > > started which I copied over:
> > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >
> > > He has a test.sh script that I adjusted and used to run his
> > > example.
> > > running of the script "bash test.sh" produced a CSI plot that
was
> > > stored
> > > here:
> > >
> >
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >
> > > Running the script produced the expected result. The next
natural
> > > step was
> > > to copy hochum_xml.template and create a new xml file called
> > > Strobach_xml.template in order to create a time series where the
> > > independent time is fcst_valid_beg instead of obs_thresh.
> > > Accordingly,
> > > I
> > > made changes to the fixed fields (removing fcst_valid_beg) and
> > > changed
> > > the
> > > statistic from CSI to FMEAN (FBAR was also tried). The error
that
> > > resulted
> > > is as follows:
> > >
> > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00', '2020-
05-
> > > 01
> > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00', '2020-
05-
> > > 01
> > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00', '2020-
05-
> > > 01
> > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00', '2020-
05-
> > > 01
> > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00', '2020-
05-
> > > 02
> > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00', '2020-
05-
> > > 02
> > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00', '2020-
05-
> > > 02
> > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00', '2020-
05-
> > > 02
> > > 11:00:00', '2020-05-02 12:00:00')
> > > AND BINARY h.fcst_var = 'AOTK'
> > > AND ld.stat_header_id = h.stat_header_id
> > > AND ld.fmean != -9999;
> > > No data was returned from database
> > > 2020-06-09
> > >
> >
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > > Source) -1 printFormattedTable - ** ERROR: Caught class
> > > edu.ucar.metviewer.EmptyResultSetException in
> > > printFormattedTable(ResultSet
> > > res): result set contained no data
> > > ** ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
> > > in
> > > printFormattedTable(ResultSet res): result set contained no data
> > > ** ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
> > > in
> > > printFormattedTable(ResultSet res): result set contained no data
> > >
> > > Apparently there was no data during this time. I tried changing
> > > the
> > > range
> > > of valid times and changed the valid time to different days just
in
> > > case
> > > data was missing. The issue was the same. To confirm my logic,
I
> > > reproduced my attempted approach on metviewer, which I found to
> > > work.
> > >
> > > 2. Running in batch mode
> > >
> > > I think I lack an understanding of how to set up batch mode
because
> > > I
> > > don't
> > > see where these definitions are being explicitly defined:
> > >
> > > <plot_spec>
> > > <connection>
> > > <host>rds_host:3306</host>
> > >
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > > <user>rds_user</user>
> > > <password>rds_pwd</password>
> > > </connection>
> > > <rscript>Rscript</rscript>
> > > <folders>
> > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > > <r_work>rds_R_work</r_work>
> > > <plots>rds_plots</plots>
> > > <data>rds_data</data>
> > > <scripts>rds_scripts</scripts>
> > > </folders>
> > > <plot>
> > > <template>series_plot.R_tmpl</template>
> > >
> > > I see the <folders> section, but grepping all available files
does
> > > not
> > > reveal how rds_R_work is being defined, as well as other
similarly
> > > defined
> > > variables. I think if I know how these are being defined, then
> > > I'll
> > > be
> > > able to pipe in my information and process the data using met
and
> > > produce
> > > results in batch mode. Am I right on that? Also, if I want to
use
> > > obs for
> > > verification, do I add another line in the XML file that
designates
> > > file
> > > location of obs? I assume that other steps have to be run
before
> > > all
> > > this. If data is in prepbufr form, for example, I would have to
> > > separately
> > > run PB2NC. If I want grid2obs, then I would want to do that as
> > > well.
> > >
> > > 3. Communicating results to metviewer online
> > >
> > > Lastly, I see a lot of people/projects listed here - see
attachment
> > >
> > > How do I get the results I generate to be uploaded to this
> > > location?
> > > It's
> > > not clear how others are setting this up.
> >
> >
> >
> >
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Sun Jun 14 09:42:44 2020
Thanks Tatiana. These resources are helpful.
One last question. Simply typing lines like this in can take forever:
<plot_fix>
<field equalize="false" name="fcst_init_beg">
<set name="fcst_init_beg_0">
<val>2019-05-01 12:00:00</val>
<val>2019-05-02 06:00:00</val>
<val>2019-05-02 12:00:00</val>
<val>2019-05-03 06:00:00</val>
<val>2019-05-03 12:00:00</val>
Is there a better way? I see date_range, date_range_list, or
date_list.
Can this be used instead of typing individual values? This could
prove
pretty daunting if one would have to type in 700 entries each time.
On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> I found a few problems in your XML.
> The database you are using doesn't have data with vx_mask
'abcREGabc'.
> That is why METviewer returns an empty dataset and can't create a
plot.
> The other problem is how you define fcst_lead. This database has
values
> like 10000, 20000 and not 010000, 020000. Since you include all
fcst_leads
> this variable can be omitted.
> To minimize problems with variables values, I would suggest to
create XML
> using GUI and then edit it to fit batch standards.
> I created a plot using you configurations with GUI (without vx_mack
and
> plot and axis titles) #20200612_173626. You can load it back to the
page by
> using the "History" tab (left bar on GUI). Find the plot, open XML
and
> click on "Load to page". After that you can recreate a plot and edit
the
> configurations. You can also download a new XML and edit it for
batch.
> I attached the example of load XML to this message.
>
> Description of rds parameters:
> rds_R_tmpl and rds_R_work - location of METviewer R scripts
> rds_plots - temporary directory where the plots will be generated
> rds_data - temporary directory where the data retrieved from the
database
> and other intermediate files will be stored
> rds_scripts - temporary directory where R scripts for this run will
be
> stored
> After the execution is done all the files from temporary directories
would
> be removed.
>
>
> We don't have the offitial user guide for METviewer. There are some
> instructions on how to crate XML config files:
> https://metviewer.nws.noaa.gov/doc/index.html
>
> Tatiana
>
> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
> > Hi,
> >
> > I'm attaching the xml below, but am having hard time following how
to
> > go
> > about the other steps since I have zero experience on how to do
these
> > things.
> >
> > #2. All parameters in <folders> section should NOT be changed and
will
> > be
> > replaced by other scripts during the batch execution. The data for
the
> > batch mode is coming from the database and not from files. The
name of
> > the
> > database is in <database> tag.
> >
> > I understand that nothing should be changed in the folders
section,
> > but
> > there is no indication how and where these variables are being
> > defined.
> > Grepping these names of the files available does not reveal formal
> > definitions of these variables. If I understood exactly how these
> > were
> > being defined, then I would be able connect this better.
> >
> > #3. Users rum MET and produce MET output files. They use METviewer
> > load
> > mode to load these files to the specific database. In the config
file
> > for
> > this process they also can specify the database group. For example
> > database
> > mv_dev is in Test12 group.
> > After the data has been loaded to the database it is visible and
> > available
> > in the GUI and also can be used by the batch mode.
> > You can always add more data to the database using the same load
mode.
> >
> > Is there a guide for me to understand the process that you are
> > referring to?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward,
> > > #1. I don't have an access to NOAA servers and can't view your
XML
> > > for the
> > > batch mode. Please, attache it to this issue and I will take a
look.
> > >
> > > #2. All parameters in <folders> section should NOT be changed
and
> > > will be
> > > replaced by other scripts during the batch execution. The data
for
> > > the
> > > batch mode is coming from the database and not from files. The
name
> > > of the
> > > database is in <database> tag.
> > >
> > > #3. Users rum MET and produce MET output files. They use
METviewer
> > > load
> > > mode to load these files to the specific database. In the config
file
> > > for
> > > this process they also can specify the database group. For
example
> > > database
> > > mv_dev is in Test12 group.
> > > After the data has been loaded to the database it is visible and
> > > available
> > > in the GUI and also can be used by the batch mode.
> > > You can always add more data to the database using the same load
> > > mode.
> > >
> > > Tatiana
> > >
> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> > > > Good morning,
> > > >
> > > > Over the last two weeks I've made an effort to understand
> > > > metviewer,
> > > > test
> > > > different cases using the history tab, changing information to
see
> > > > what
> > > > works and what does not, and make connections between the GUI
> > > > settings
> > > > and
> > > > the XML file. Although my comfort level with metviewer has
> > > > significantly
> > > > increased during this time, there are some remaining issues
that
> > > > I'm
> > > > hoping
> > > > to clear up.
> > > >
> > > > 1. Problem changing XML file and running with different
statistic
> > > > and
> > > > plot
> > > > type:
> > > >
> > > > Ho-Chun Huang, a colleague at EMC, provided me the working
> > > > directory
> > > > to get
> > > > started which I copied over:
> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > >
> > > > He has a test.sh script that I adjusted and used to run his
> > > > example.
> > > > running of the script "bash test.sh" produced a CSI plot that
was
> > > > stored
> > > > here:
> > > >
> > >
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > > >
> > > > Running the script produced the expected result. The next
natural
> > > > step was
> > > > to copy hochum_xml.template and create a new xml file called
> > > > Strobach_xml.template in order to create a time series where
the
> > > > independent time is fcst_valid_beg instead of obs_thresh.
> > > > Accordingly,
> > > > I
> > > > made changes to the fixed fields (removing fcst_valid_beg) and
> > > > changed
> > > > the
> > > > statistic from CSI to FMEAN (FBAR was also tried). The error
that
> > > > resulted
> > > > is as follows:
> > > >
> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00',
'2020-05-
> > > > 01
> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
'2020-05-
> > > > 01
> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
'2020-05-
> > > > 01
> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
'2020-05-
> > > > 01
> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
'2020-05-
> > > > 02
> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
'2020-05-
> > > > 02
> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
'2020-05-
> > > > 02
> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
'2020-05-
> > > > 02
> > > > 11:00:00', '2020-05-02 12:00:00')
> > > > AND BINARY h.fcst_var = 'AOTK'
> > > > AND ld.stat_header_id = h.stat_header_id
> > > > AND ld.fmean != -9999;
> > > > No data was returned from database
> > > > 2020-06-09
> > > >
> > >
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > > > Source) -1 printFormattedTable - ** ERROR: Caught class
> > > > edu.ucar.metviewer.EmptyResultSetException in
> > > > printFormattedTable(ResultSet
> > > > res): result set contained no data
> > > > ** ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
> > > > in
> > > > printFormattedTable(ResultSet res): result set contained no
data
> > > > ** ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
> > > > in
> > > > printFormattedTable(ResultSet res): result set contained no
data
> > > >
> > > > Apparently there was no data during this time. I tried
changing
> > > > the
> > > > range
> > > > of valid times and changed the valid time to different days
just in
> > > > case
> > > > data was missing. The issue was the same. To confirm my
logic, I
> > > > reproduced my attempted approach on metviewer, which I found
to
> > > > work.
> > > >
> > > > 2. Running in batch mode
> > > >
> > > > I think I lack an understanding of how to set up batch mode
because
> > > > I
> > > > don't
> > > > see where these definitions are being explicitly defined:
> > > >
> > > > <plot_spec>
> > > > <connection>
> > > > <host>rds_host:3306</host>
> > > >
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > > > <user>rds_user</user>
> > > > <password>rds_pwd</password>
> > > > </connection>
> > > > <rscript>Rscript</rscript>
> > > > <folders>
> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > > > <r_work>rds_R_work</r_work>
> > > > <plots>rds_plots</plots>
> > > > <data>rds_data</data>
> > > > <scripts>rds_scripts</scripts>
> > > > </folders>
> > > > <plot>
> > > > <template>series_plot.R_tmpl</template>
> > > >
> > > > I see the <folders> section, but grepping all available files
does
> > > > not
> > > > reveal how rds_R_work is being defined, as well as other
similarly
> > > > defined
> > > > variables. I think if I know how these are being defined,
then
> > > > I'll
> > > > be
> > > > able to pipe in my information and process the data using met
and
> > > > produce
> > > > results in batch mode. Am I right on that? Also, if I want
to use
> > > > obs for
> > > > verification, do I add another line in the XML file that
designates
> > > > file
> > > > location of obs? I assume that other steps have to be run
before
> > > > all
> > > > this. If data is in prepbufr form, for example, I would have
to
> > > > separately
> > > > run PB2NC. If I want grid2obs, then I would want to do that
as
> > > > well.
> > > >
> > > > 3. Communicating results to metviewer online
> > > >
> > > > Lastly, I see a lot of people/projects listed here - see
attachment
> > > >
> > > > How do I get the results I generate to be uploaded to this
> > > > location?
> > > > It's
> > > > not clear how others are setting this up.
> > >
> > >
> > >
> > >
>
>
>
> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user rds_pwd
aurora
> Rscript rds_R_tmpl rds_R_work rds_plots rds_data rds_scripts
> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-05-12
> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12 20:00:00
2020-05-12
> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13 10:00:00
2020-05-13
> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13 14:00:00
2020-05-13
> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13 18:00:00
2020-05-13
> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13 22:00:00
2020-05-13
> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14 12:00:00
true
> median plot_20200612_173626.data plot_20200612_173626.png
> plot_20200612_173626.R test x_label test y_label
> false list() list() false false false false false false true false
false
> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0) 1 2
1.4 -2
> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc 1
listX 1 1
> 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8 1
#333333 0.8
> 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1) c(1,1)
c(1,2)
> c("","") c() c() 0.04 c()
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Sun Jun 14 09:48:24 2020
and if there isn't a better way, I wonder the whereabouts of a
separate
script that pipes dates (and other relevant information) into the XML
file
used to generate the results.
On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Thanks Tatiana. These resources are helpful.
>
> One last question. Simply typing lines like this in can take
forever:
>
> <plot_fix>
> <field equalize="false" name="fcst_init_beg">
> <set name="fcst_init_beg_0">
> <val>2019-05-01 12:00:00</val>
> <val>2019-05-02 06:00:00</val>
> <val>2019-05-02 12:00:00</val>
> <val>2019-05-03 06:00:00</val>
> <val>2019-05-03 12:00:00</val>
>
>
> Is there a better way? I see date_range, date_range_list, or
date_list.
> Can this be used instead of typing individual values? This could
prove
> pretty daunting if one would have to type in 700 entries each time.
>
> On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> I found a few problems in your XML.
>> The database you are using doesn't have data with vx_mask
'abcREGabc'.
>> That is why METviewer returns an empty dataset and can't create a
plot.
>> The other problem is how you define fcst_lead. This database has
values
>> like 10000, 20000 and not 010000, 020000. Since you include all
fcst_leads
>> this variable can be omitted.
>> To minimize problems with variables values, I would suggest to
create XML
>> using GUI and then edit it to fit batch standards.
>> I created a plot using you configurations with GUI (without vx_mack
and
>> plot and axis titles) #20200612_173626. You can load it back to the
page by
>> using the "History" tab (left bar on GUI). Find the plot, open XML
and
>> click on "Load to page". After that you can recreate a plot and
edit the
>> configurations. You can also download a new XML and edit it for
batch.
>> I attached the example of load XML to this message.
>>
>> Description of rds parameters:
>> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>> rds_plots - temporary directory where the plots will be generated
>> rds_data - temporary directory where the data retrieved from the
database
>> and other intermediate files will be stored
>> rds_scripts - temporary directory where R scripts for this run will
be
>> stored
>> After the execution is done all the files from temporary
directories
>> would be removed.
>>
>>
>> We don't have the offitial user guide for METviewer. There are some
>> instructions on how to crate XML config files:
>> https://metviewer.nws.noaa.gov/doc/index.html
>>
>> Tatiana
>>
>> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
>> > Hi,
>> >
>> > I'm attaching the xml below, but am having hard time following
how to
>> > go
>> > about the other steps since I have zero experience on how to do
these
>> > things.
>> >
>> > #2. All parameters in <folders> section should NOT be changed and
will
>> > be
>> > replaced by other scripts during the batch execution. The data
for the
>> > batch mode is coming from the database and not from files. The
name of
>> > the
>> > database is in <database> tag.
>> >
>> > I understand that nothing should be changed in the folders
section,
>> > but
>> > there is no indication how and where these variables are being
>> > defined.
>> > Grepping these names of the files available does not reveal
formal
>> > definitions of these variables. If I understood exactly how
these
>> > were
>> > being defined, then I would be able connect this better.
>> >
>> > #3. Users rum MET and produce MET output files. They use
METviewer
>> > load
>> > mode to load these files to the specific database. In the config
file
>> > for
>> > this process they also can specify the database group. For
example
>> > database
>> > mv_dev is in Test12 group.
>> > After the data has been loaded to the database it is visible and
>> > available
>> > in the GUI and also can be used by the batch mode.
>> > You can always add more data to the database using the same load
mode.
>> >
>> > Is there a guide for me to understand the process that you are
>> > referring to?
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>> > <met_help at ucar.edu>
>> > wrote:
>> >
>> > > Hi Edward,
>> > > #1. I don't have an access to NOAA servers and can't view your
XML
>> > > for the
>> > > batch mode. Please, attache it to this issue and I will take a
look.
>> > >
>> > > #2. All parameters in <folders> section should NOT be changed
and
>> > > will be
>> > > replaced by other scripts during the batch execution. The data
for
>> > > the
>> > > batch mode is coming from the database and not from files. The
name
>> > > of the
>> > > database is in <database> tag.
>> > >
>> > > #3. Users rum MET and produce MET output files. They use
METviewer
>> > > load
>> > > mode to load these files to the specific database. In the
config file
>> > > for
>> > > this process they also can specify the database group. For
example
>> > > database
>> > > mv_dev is in Test12 group.
>> > > After the data has been loaded to the database it is visible
and
>> > > available
>> > > in the GUI and also can be used by the batch mode.
>> > > You can always add more data to the database using the same
load
>> > > mode.
>> > >
>> > > Tatiana
>> > >
>> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
>> > > > Good morning,
>> > > >
>> > > > Over the last two weeks I've made an effort to understand
>> > > > metviewer,
>> > > > test
>> > > > different cases using the history tab, changing information
to see
>> > > > what
>> > > > works and what does not, and make connections between the GUI
>> > > > settings
>> > > > and
>> > > > the XML file. Although my comfort level with metviewer has
>> > > > significantly
>> > > > increased during this time, there are some remaining issues
that
>> > > > I'm
>> > > > hoping
>> > > > to clear up.
>> > > >
>> > > > 1. Problem changing XML file and running with different
statistic
>> > > > and
>> > > > plot
>> > > > type:
>> > > >
>> > > > Ho-Chun Huang, a colleague at EMC, provided me the working
>> > > > directory
>> > > > to get
>> > > > started which I copied over:
>> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > > >
>> > > > He has a test.sh script that I adjusted and used to run his
>> > > > example.
>> > > > running of the script "bash test.sh" produced a CSI plot that
was
>> > > > stored
>> > > > here:
>> > > >
>> > >
>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>> > > >
>> > > > Running the script produced the expected result. The next
natural
>> > > > step was
>> > > > to copy hochum_xml.template and create a new xml file called
>> > > > Strobach_xml.template in order to create a time series where
the
>> > > > independent time is fcst_valid_beg instead of obs_thresh.
>> > > > Accordingly,
>> > > > I
>> > > > made changes to the fixed fields (removing fcst_valid_beg)
and
>> > > > changed
>> > > > the
>> > > > statistic from CSI to FMEAN (FBAR was also tried). The error
that
>> > > > resulted
>> > > > is as follows:
>> > > >
>> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00',
'2020-05-
>> > > > 01
>> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
'2020-05-
>> > > > 01
>> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
'2020-05-
>> > > > 01
>> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
'2020-05-
>> > > > 01
>> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
'2020-05-
>> > > > 02
>> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
'2020-05-
>> > > > 02
>> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
'2020-05-
>> > > > 02
>> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
'2020-05-
>> > > > 02
>> > > > 11:00:00', '2020-05-02 12:00:00')
>> > > > AND BINARY h.fcst_var = 'AOTK'
>> > > > AND ld.stat_header_id = h.stat_header_id
>> > > > AND ld.fmean != -9999;
>> > > > No data was returned from database
>> > > > 2020-06-09
>> > > >
>> > >
>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>> > > > Source) -1 printFormattedTable - ** ERROR: Caught class
>> > > > edu.ucar.metviewer.EmptyResultSetException in
>> > > > printFormattedTable(ResultSet
>> > > > res): result set contained no data
>> > > > ** ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
>> > > > in
>> > > > printFormattedTable(ResultSet res): result set contained no
data
>> > > > ** ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
>> > > > in
>> > > > printFormattedTable(ResultSet res): result set contained no
data
>> > > >
>> > > > Apparently there was no data during this time. I tried
changing
>> > > > the
>> > > > range
>> > > > of valid times and changed the valid time to different days
just in
>> > > > case
>> > > > data was missing. The issue was the same. To confirm my
logic, I
>> > > > reproduced my attempted approach on metviewer, which I found
to
>> > > > work.
>> > > >
>> > > > 2. Running in batch mode
>> > > >
>> > > > I think I lack an understanding of how to set up batch mode
because
>> > > > I
>> > > > don't
>> > > > see where these definitions are being explicitly defined:
>> > > >
>> > > > <plot_spec>
>> > > > <connection>
>> > > > <host>rds_host:3306</host>
>> > > >
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>> > > > <user>rds_user</user>
>> > > > <password>rds_pwd</password>
>> > > > </connection>
>> > > > <rscript>Rscript</rscript>
>> > > > <folders>
>> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>> > > > <r_work>rds_R_work</r_work>
>> > > > <plots>rds_plots</plots>
>> > > > <data>rds_data</data>
>> > > > <scripts>rds_scripts</scripts>
>> > > > </folders>
>> > > > <plot>
>> > > > <template>series_plot.R_tmpl</template>
>> > > >
>> > > > I see the <folders> section, but grepping all available files
does
>> > > > not
>> > > > reveal how rds_R_work is being defined, as well as other
similarly
>> > > > defined
>> > > > variables. I think if I know how these are being defined,
then
>> > > > I'll
>> > > > be
>> > > > able to pipe in my information and process the data using met
and
>> > > > produce
>> > > > results in batch mode. Am I right on that? Also, if I want
to use
>> > > > obs for
>> > > > verification, do I add another line in the XML file that
designates
>> > > > file
>> > > > location of obs? I assume that other steps have to be run
before
>> > > > all
>> > > > this. If data is in prepbufr form, for example, I would have
to
>> > > > separately
>> > > > run PB2NC. If I want grid2obs, then I would want to do that
as
>> > > > well.
>> > > >
>> > > > 3. Communicating results to metviewer online
>> > > >
>> > > > Lastly, I see a lot of people/projects listed here - see
attachment
>> > > >
>> > > > How do I get the results I generate to be uploaded to this
>> > > > location?
>> > > > It's
>> > > > not clear how others are setting this up.
>> > >
>> > >
>> > >
>> > >
>>
>>
>>
>> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user rds_pwd
>> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data rds_scripts
>> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-05-12
>> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00 2020-05-12
>> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00 2020-05-13
>> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00 2020-05-13
>> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00 2020-05-13
>> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00 2020-05-13
>> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00 true
>> median plot_20200612_173626.data plot_20200612_173626.png
>> plot_20200612_173626.R test x_label test y_label
>> false list() list() false false false false false false true false
false
>> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0) 1 2
1.4 -2
>> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc 1
listX 1
>> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8 1
#333333
>> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1) c(1,1)
>> c(1,2) c("","") c() c() 0.04 c()
>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Mon Jun 15 11:55:54 2020
Hi,
Here is some examples how to use <date_range> and <date_range_list> in
batch XML.
First, you need to defined your range. You can have more then one
<date_range>:
<date_range name="may">
<start>2019-05-01 00:00:00</start>
<end>2019-05-31 23:59:59</end>
</date_range>
<date_range name="march">
<start>2019-03-01 00:00:00</start>
<end>2019-03-31 23:59:59</end>
</date_range>
and then use one or all date_ranges:
<plot_fix>
<field name="fcst_init_beg">
<date_range name="march"/>
<date_range name="may"/>
</field>...
This configuration would include all dates for March and May of 2019.
If you need more flexible configuration you can use date_range_list.
For example, you want to select first 6 hours every day beetween 2020-
01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look like this:
<date_range_list name = 'all'>
<range_start>2020-01-01 00:00:00</range_start>
<range_end>2020-01-05 00:00:00</range_end>
<range_length>360</range_length>
<inc>1440</inc>
</date_range_list>
where the 1st and 2nd values are the start and end,
range_length - the length of a range in minutes(6 houres)
inc - lenght beetween ranges in minutes (24 houres)
This example would create following ranges:
BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
Also you can add more parameters to <plot_fix> to filter your data.
For example, if you want to select only dates with 00:00 you can
specify
<field name="valid_hour">
<val>00</val>
</field>
Tatiana
On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> and if there isn't a better way, I wonder the whereabouts of a
> separate
> script that pipes dates (and other relevant information) into the
XML
> file
> used to generate the results.
>
> On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Thanks Tatiana. These resources are helpful.
> >
> > One last question. Simply typing lines like this in can take
> > forever:
> >
> > <plot_fix>
> > <field equalize="false" name="fcst_init_beg">
> > <set name="fcst_init_beg_0">
> > <val>2019-05-01 12:00:00</val>
> > <val>2019-05-02 06:00:00</val>
> > <val>2019-05-02 12:00:00</val>
> > <val>2019-05-03 06:00:00</val>
> > <val>2019-05-03 12:00:00</val>
> >
> >
> > Is there a better way? I see date_range, date_range_list, or
> > date_list.
> > Can this be used instead of typing individual values? This could
> > prove
> > pretty daunting if one would have to type in 700 entries each
time.
> >
> > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> >> I found a few problems in your XML.
> >> The database you are using doesn't have data with vx_mask
> >> 'abcREGabc'.
> >> That is why METviewer returns an empty dataset and can't create a
> >> plot.
> >> The other problem is how you define fcst_lead. This database has
> >> values
> >> like 10000, 20000 and not 010000, 020000. Since you include all
> >> fcst_leads
> >> this variable can be omitted.
> >> To minimize problems with variables values, I would suggest to
> >> create XML
> >> using GUI and then edit it to fit batch standards.
> >> I created a plot using you configurations with GUI (without
vx_mack
> >> and
> >> plot and axis titles) #20200612_173626. You can load it back to
the
> >> page by
> >> using the "History" tab (left bar on GUI). Find the plot, open
XML
> >> and
> >> click on "Load to page". After that you can recreate a plot and
edit
> >> the
> >> configurations. You can also download a new XML and edit it for
> >> batch.
> >> I attached the example of load XML to this message.
> >>
> >> Description of rds parameters:
> >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
> >> rds_plots - temporary directory where the plots will be generated
> >> rds_data - temporary directory where the data retrieved from the
> >> database
> >> and other intermediate files will be stored
> >> rds_scripts - temporary directory where R scripts for this run
will
> >> be
> >> stored
> >> After the execution is done all the files from temporary
directories
> >> would be removed.
> >>
> >>
> >> We don't have the offitial user guide for METviewer. There are
some
> >> instructions on how to crate XML config files:
> >> https://metviewer.nws.noaa.gov/doc/index.html
> >>
> >> Tatiana
> >>
> >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
> >> > Hi,
> >> >
> >> > I'm attaching the xml below, but am having hard time following
how
> >> > to
> >> > go
> >> > about the other steps since I have zero experience on how to do
> >> > these
> >> > things.
> >> >
> >> > #2. All parameters in <folders> section should NOT be changed
and
> >> > will
> >> > be
> >> > replaced by other scripts during the batch execution. The data
for
> >> > the
> >> > batch mode is coming from the database and not from files. The
> >> > name of
> >> > the
> >> > database is in <database> tag.
> >> >
> >> > I understand that nothing should be changed in the folders
> >> > section,
> >> > but
> >> > there is no indication how and where these variables are being
> >> > defined.
> >> > Grepping these names of the files available does not reveal
formal
> >> > definitions of these variables. If I understood exactly how
these
> >> > were
> >> > being defined, then I would be able connect this better.
> >> >
> >> > #3. Users rum MET and produce MET output files. They use
METviewer
> >> > load
> >> > mode to load these files to the specific database. In the
config
> >> > file
> >> > for
> >> > this process they also can specify the database group. For
example
> >> > database
> >> > mv_dev is in Test12 group.
> >> > After the data has been loaded to the database it is visible
and
> >> > available
> >> > in the GUI and also can be used by the batch mode.
> >> > You can always add more data to the database using the same
load
> >> > mode.
> >> >
> >> > Is there a guide for me to understand the process that you are
> >> > referring to?
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> >> > <met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Hi Edward,
> >> > > #1. I don't have an access to NOAA servers and can't view
your
> >> > > XML
> >> > > for the
> >> > > batch mode. Please, attache it to this issue and I will take
a
> >> > > look.
> >> > >
> >> > > #2. All parameters in <folders> section should NOT be changed
> >> > > and
> >> > > will be
> >> > > replaced by other scripts during the batch execution. The
data
> >> > > for
> >> > > the
> >> > > batch mode is coming from the database and not from files.
The
> >> > > name
> >> > > of the
> >> > > database is in <database> tag.
> >> > >
> >> > > #3. Users rum MET and produce MET output files. They use
> >> > > METviewer
> >> > > load
> >> > > mode to load these files to the specific database. In the
config
> >> > > file
> >> > > for
> >> > > this process they also can specify the database group. For
> >> > > example
> >> > > database
> >> > > mv_dev is in Test12 group.
> >> > > After the data has been loaded to the database it is visible
and
> >> > > available
> >> > > in the GUI and also can be used by the batch mode.
> >> > > You can always add more data to the database using the same
load
> >> > > mode.
> >> > >
> >> > > Tatiana
> >> > >
> >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> >> > > > Good morning,
> >> > > >
> >> > > > Over the last two weeks I've made an effort to understand
> >> > > > metviewer,
> >> > > > test
> >> > > > different cases using the history tab, changing information
to
> >> > > > see
> >> > > > what
> >> > > > works and what does not, and make connections between the
GUI
> >> > > > settings
> >> > > > and
> >> > > > the XML file. Although my comfort level with metviewer has
> >> > > > significantly
> >> > > > increased during this time, there are some remaining issues
> >> > > > that
> >> > > > I'm
> >> > > > hoping
> >> > > > to clear up.
> >> > > >
> >> > > > 1. Problem changing XML file and running with different
> >> > > > statistic
> >> > > > and
> >> > > > plot
> >> > > > type:
> >> > > >
> >> > > > Ho-Chun Huang, a colleague at EMC, provided me the working
> >> > > > directory
> >> > > > to get
> >> > > > started which I copied over:
> >> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >> > > >
> >> > > > He has a test.sh script that I adjusted and used to run his
> >> > > > example.
> >> > > > running of the script "bash test.sh" produced a CSI plot
that
> >> > > > was
> >> > > > stored
> >> > > > here:
> >> > > >
> >> > >
> >>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >> > > >
> >> > > > Running the script produced the expected result. The next
> >> > > > natural
> >> > > > step was
> >> > > > to copy hochum_xml.template and create a new xml file
called
> >> > > > Strobach_xml.template in order to create a time series
where
> >> > > > the
> >> > > > independent time is fcst_valid_beg instead of obs_thresh.
> >> > > > Accordingly,
> >> > > > I
> >> > > > made changes to the fixed fields (removing fcst_valid_beg)
and
> >> > > > changed
> >> > > > the
> >> > > > statistic from CSI to FMEAN (FBAR was also tried). The
error
> >> > > > that
> >> > > > resulted
> >> > > > is as follows:
> >> > > >
> >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00',
> >> > > > '2020-05-
> >> > > > 01
> >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
> >> > > > '2020-05-
> >> > > > 01
> >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
> >> > > > '2020-05-
> >> > > > 01
> >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
> >> > > > '2020-05-
> >> > > > 01
> >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
> >> > > > '2020-05-
> >> > > > 02
> >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
> >> > > > '2020-05-
> >> > > > 02
> >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
> >> > > > '2020-05-
> >> > > > 02
> >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
> >> > > > '2020-05-
> >> > > > 02
> >> > > > 11:00:00', '2020-05-02 12:00:00')
> >> > > > AND BINARY h.fcst_var = 'AOTK'
> >> > > > AND ld.stat_header_id = h.stat_header_id
> >> > > > AND ld.fmean != -9999;
> >> > > > No data was returned from database
> >> > > > 2020-06-09
> >> > > >
> >> > >
> >>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >> > > > Source) -1 printFormattedTable - ** ERROR: Caught class
> >> > > > edu.ucar.metviewer.EmptyResultSetException in
> >> > > > printFormattedTable(ResultSet
> >> > > > res): result set contained no data
> >> > > > ** ERROR: Caught class
> >> > > > edu.ucar.metviewer.EmptyResultSetException
> >> > > > in
> >> > > > printFormattedTable(ResultSet res): result set contained no
> >> > > > data
> >> > > > ** ERROR: Caught class
> >> > > > edu.ucar.metviewer.EmptyResultSetException
> >> > > > in
> >> > > > printFormattedTable(ResultSet res): result set contained no
> >> > > > data
> >> > > >
> >> > > > Apparently there was no data during this time. I tried
> >> > > > changing
> >> > > > the
> >> > > > range
> >> > > > of valid times and changed the valid time to different days
> >> > > > just in
> >> > > > case
> >> > > > data was missing. The issue was the same. To confirm my
> >> > > > logic, I
> >> > > > reproduced my attempted approach on metviewer, which I
found
> >> > > > to
> >> > > > work.
> >> > > >
> >> > > > 2. Running in batch mode
> >> > > >
> >> > > > I think I lack an understanding of how to set up batch mode
> >> > > > because
> >> > > > I
> >> > > > don't
> >> > > > see where these definitions are being explicitly defined:
> >> > > >
> >> > > > <plot_spec>
> >> > > > <connection>
> >> > > > <host>rds_host:3306</host>
> >> > > >
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >> > > > <user>rds_user</user>
> >> > > > <password>rds_pwd</password>
> >> > > > </connection>
> >> > > > <rscript>Rscript</rscript>
> >> > > > <folders>
> >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >> > > > <r_work>rds_R_work</r_work>
> >> > > > <plots>rds_plots</plots>
> >> > > > <data>rds_data</data>
> >> > > > <scripts>rds_scripts</scripts>
> >> > > > </folders>
> >> > > > <plot>
> >> > > > <template>series_plot.R_tmpl</template>
> >> > > >
> >> > > > I see the <folders> section, but grepping all available
files
> >> > > > does
> >> > > > not
> >> > > > reveal how rds_R_work is being defined, as well as other
> >> > > > similarly
> >> > > > defined
> >> > > > variables. I think if I know how these are being defined,
> >> > > > then
> >> > > > I'll
> >> > > > be
> >> > > > able to pipe in my information and process the data using
met
> >> > > > and
> >> > > > produce
> >> > > > results in batch mode. Am I right on that? Also, if I
want
> >> > > > to use
> >> > > > obs for
> >> > > > verification, do I add another line in the XML file that
> >> > > > designates
> >> > > > file
> >> > > > location of obs? I assume that other steps have to be run
> >> > > > before
> >> > > > all
> >> > > > this. If data is in prepbufr form, for example, I would
have
> >> > > > to
> >> > > > separately
> >> > > > run PB2NC. If I want grid2obs, then I would want to do
that
> >> > > > as
> >> > > > well.
> >> > > >
> >> > > > 3. Communicating results to metviewer online
> >> > > >
> >> > > > Lastly, I see a lot of people/projects listed here - see
> >> > > > attachment
> >> > > >
> >> > > > How do I get the results I generate to be uploaded to this
> >> > > > location?
> >> > > > It's
> >> > > > not clear how others are setting this up.
> >> > >
> >> > >
> >> > >
> >> > >
> >>
> >>
> >>
> >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
> >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
> >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-05-
12
> >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00
> >> 2020-05-12
> >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00
> >> 2020-05-13
> >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00
> >> 2020-05-13
> >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00
> >> 2020-05-13
> >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00
> >> 2020-05-13
> >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00
> >> true
> >> median plot_20200612_173626.data plot_20200612_173626.png
> >> plot_20200612_173626.R test x_label test y_label
> >> false list() list() false false false false false false true
false
> >> false
> >> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0) 1
2
> >> 1.4 -2
> >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc 1
> >> listX 1
> >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8 1
> >> #333333
> >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
> >> c(1,2) c("","") c() c() 0.04 c()
> >
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Mon Jun 15 14:28:45 2020
Thank you. This is a very helpful example
On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Hi,
> Here is some examples how to use <date_range> and <date_range_list>
in
> batch XML.
> First, you need to defined your range. You can have more then one
> <date_range>:
> <date_range name="may">
> <start>2019-05-01 00:00:00</start>
> <end>2019-05-31 23:59:59</end>
> </date_range>
> <date_range name="march">
> <start>2019-03-01 00:00:00</start>
> <end>2019-03-31 23:59:59</end>
> </date_range>
>
> and then use one or all date_ranges:
> <plot_fix>
> <field name="fcst_init_beg">
> <date_range name="march"/>
> <date_range name="may"/>
> </field>...
> This configuration would include all dates for March and May of
2019.
>
> If you need more flexible configuration you can use date_range_list.
> For example, you want to select first 6 hours every day beetween
> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
> <date_range_list name = 'all'>
> <range_start>2020-01-01 00:00:00</range_start>
> <range_end>2020-01-05 00:00:00</range_end>
> <range_length>360</range_length>
> <inc>1440</inc>
> </date_range_list>
> where the 1st and 2nd values are the start and end,
> range_length - the length of a range in minutes(6 houres)
> inc - lenght beetween ranges in minutes (24 houres)
> This example would create following ranges:
> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>
> Also you can add more parameters to <plot_fix> to filter your data.
For
> example, if you want to select only dates with 00:00 you can specify
> <field name="valid_hour">
> <val>00</val>
> </field>
>
> Tatiana
>
>
>
> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> > and if there isn't a better way, I wonder the whereabouts of a
> > separate
> > script that pipes dates (and other relevant information) into the
XML
> > file
> > used to generate the results.
> >
> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA Affiliate
<
> > edward.strobach at noaa.gov> wrote:
> >
> > > Thanks Tatiana. These resources are helpful.
> > >
> > > One last question. Simply typing lines like this in can take
> > > forever:
> > >
> > > <plot_fix>
> > > <field equalize="false" name="fcst_init_beg">
> > > <set name="fcst_init_beg_0">
> > > <val>2019-05-01 12:00:00</val>
> > > <val>2019-05-02 06:00:00</val>
> > > <val>2019-05-02 12:00:00</val>
> > > <val>2019-05-03 06:00:00</val>
> > > <val>2019-05-03 12:00:00</val>
> > >
> > >
> > > Is there a better way? I see date_range, date_range_list, or
> > > date_list.
> > > Can this be used instead of typing individual values? This
could
> > > prove
> > > pretty daunting if one would have to type in 700 entries each
time.
> > >
> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > >> I found a few problems in your XML.
> > >> The database you are using doesn't have data with vx_mask
> > >> 'abcREGabc'.
> > >> That is why METviewer returns an empty dataset and can't create
a
> > >> plot.
> > >> The other problem is how you define fcst_lead. This database
has
> > >> values
> > >> like 10000, 20000 and not 010000, 020000. Since you include all
> > >> fcst_leads
> > >> this variable can be omitted.
> > >> To minimize problems with variables values, I would suggest to
> > >> create XML
> > >> using GUI and then edit it to fit batch standards.
> > >> I created a plot using you configurations with GUI (without
vx_mack
> > >> and
> > >> plot and axis titles) #20200612_173626. You can load it back to
the
> > >> page by
> > >> using the "History" tab (left bar on GUI). Find the plot, open
XML
> > >> and
> > >> click on "Load to page". After that you can recreate a plot and
edit
> > >> the
> > >> configurations. You can also download a new XML and edit it for
> > >> batch.
> > >> I attached the example of load XML to this message.
> > >>
> > >> Description of rds parameters:
> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
> > >> rds_plots - temporary directory where the plots will be
generated
> > >> rds_data - temporary directory where the data retrieved from
the
> > >> database
> > >> and other intermediate files will be stored
> > >> rds_scripts - temporary directory where R scripts for this run
will
> > >> be
> > >> stored
> > >> After the execution is done all the files from temporary
directories
> > >> would be removed.
> > >>
> > >>
> > >> We don't have the offitial user guide for METviewer. There are
some
> > >> instructions on how to crate XML config files:
> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > >>
> > >> Tatiana
> > >>
> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
> > >> > Hi,
> > >> >
> > >> > I'm attaching the xml below, but am having hard time
following how
> > >> > to
> > >> > go
> > >> > about the other steps since I have zero experience on how to
do
> > >> > these
> > >> > things.
> > >> >
> > >> > #2. All parameters in <folders> section should NOT be changed
and
> > >> > will
> > >> > be
> > >> > replaced by other scripts during the batch execution. The
data for
> > >> > the
> > >> > batch mode is coming from the database and not from files.
The
> > >> > name of
> > >> > the
> > >> > database is in <database> tag.
> > >> >
> > >> > I understand that nothing should be changed in the folders
> > >> > section,
> > >> > but
> > >> > there is no indication how and where these variables are
being
> > >> > defined.
> > >> > Grepping these names of the files available does not reveal
formal
> > >> > definitions of these variables. If I understood exactly how
these
> > >> > were
> > >> > being defined, then I would be able connect this better.
> > >> >
> > >> > #3. Users rum MET and produce MET output files. They use
METviewer
> > >> > load
> > >> > mode to load these files to the specific database. In the
config
> > >> > file
> > >> > for
> > >> > this process they also can specify the database group. For
example
> > >> > database
> > >> > mv_dev is in Test12 group.
> > >> > After the data has been loaded to the database it is visible
and
> > >> > available
> > >> > in the GUI and also can be used by the batch mode.
> > >> > You can always add more data to the database using the same
load
> > >> > mode.
> > >> >
> > >> > Is there a guide for me to understand the process that you
are
> > >> > referring to?
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> > >> > <met_help at ucar.edu>
> > >> > wrote:
> > >> >
> > >> > > Hi Edward,
> > >> > > #1. I don't have an access to NOAA servers and can't view
your
> > >> > > XML
> > >> > > for the
> > >> > > batch mode. Please, attache it to this issue and I will
take a
> > >> > > look.
> > >> > >
> > >> > > #2. All parameters in <folders> section should NOT be
changed
> > >> > > and
> > >> > > will be
> > >> > > replaced by other scripts during the batch execution. The
data
> > >> > > for
> > >> > > the
> > >> > > batch mode is coming from the database and not from files.
The
> > >> > > name
> > >> > > of the
> > >> > > database is in <database> tag.
> > >> > >
> > >> > > #3. Users rum MET and produce MET output files. They use
> > >> > > METviewer
> > >> > > load
> > >> > > mode to load these files to the specific database. In the
config
> > >> > > file
> > >> > > for
> > >> > > this process they also can specify the database group. For
> > >> > > example
> > >> > > database
> > >> > > mv_dev is in Test12 group.
> > >> > > After the data has been loaded to the database it is
visible and
> > >> > > available
> > >> > > in the GUI and also can be used by the batch mode.
> > >> > > You can always add more data to the database using the same
load
> > >> > > mode.
> > >> > >
> > >> > > Tatiana
> > >> > >
> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
> > >> > > > Good morning,
> > >> > > >
> > >> > > > Over the last two weeks I've made an effort to understand
> > >> > > > metviewer,
> > >> > > > test
> > >> > > > different cases using the history tab, changing
information to
> > >> > > > see
> > >> > > > what
> > >> > > > works and what does not, and make connections between the
GUI
> > >> > > > settings
> > >> > > > and
> > >> > > > the XML file. Although my comfort level with metviewer
has
> > >> > > > significantly
> > >> > > > increased during this time, there are some remaining
issues
> > >> > > > that
> > >> > > > I'm
> > >> > > > hoping
> > >> > > > to clear up.
> > >> > > >
> > >> > > > 1. Problem changing XML file and running with different
> > >> > > > statistic
> > >> > > > and
> > >> > > > plot
> > >> > > > type:
> > >> > > >
> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
> > >> > > > directory
> > >> > > > to get
> > >> > > > started which I copied over:
> > >> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >> > > >
> > >> > > > He has a test.sh script that I adjusted and used to run
his
> > >> > > > example.
> > >> > > > running of the script "bash test.sh" produced a CSI plot
that
> > >> > > > was
> > >> > > > stored
> > >> > > > here:
> > >> > > >
> > >> > >
> > >>
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >> > > >
> > >> > > > Running the script produced the expected result. The
next
> > >> > > > natural
> > >> > > > step was
> > >> > > > to copy hochum_xml.template and create a new xml file
called
> > >> > > > Strobach_xml.template in order to create a time series
where
> > >> > > > the
> > >> > > > independent time is fcst_valid_beg instead of obs_thresh.
> > >> > > > Accordingly,
> > >> > > > I
> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg) and
> > >> > > > changed
> > >> > > > the
> > >> > > > statistic from CSI to FMEAN (FBAR was also tried). The
error
> > >> > > > that
> > >> > > > resulted
> > >> > > > is as follows:
> > >> > > >
> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00',
> > >> > > > '2020-05-
> > >> > > > 01
> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
> > >> > > > '2020-05-
> > >> > > > 01
> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
> > >> > > > '2020-05-
> > >> > > > 01
> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
> > >> > > > '2020-05-
> > >> > > > 01
> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
> > >> > > > '2020-05-
> > >> > > > 02
> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
> > >> > > > '2020-05-
> > >> > > > 02
> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
> > >> > > > '2020-05-
> > >> > > > 02
> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
> > >> > > > '2020-05-
> > >> > > > 02
> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > >> > > > AND ld.fmean != -9999;
> > >> > > > No data was returned from database
> > >> > > > 2020-06-09
> > >> > > >
> > >> > >
> > >>
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > >> > > > Source) -1 printFormattedTable - ** ERROR: Caught
class
> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> > >> > > > printFormattedTable(ResultSet
> > >> > > > res): result set contained no data
> > >> > > > ** ERROR: Caught class
> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >> > > > in
> > >> > > > printFormattedTable(ResultSet res): result set contained
no
> > >> > > > data
> > >> > > > ** ERROR: Caught class
> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >> > > > in
> > >> > > > printFormattedTable(ResultSet res): result set contained
no
> > >> > > > data
> > >> > > >
> > >> > > > Apparently there was no data during this time. I tried
> > >> > > > changing
> > >> > > > the
> > >> > > > range
> > >> > > > of valid times and changed the valid time to different
days
> > >> > > > just in
> > >> > > > case
> > >> > > > data was missing. The issue was the same. To confirm my
> > >> > > > logic, I
> > >> > > > reproduced my attempted approach on metviewer, which I
found
> > >> > > > to
> > >> > > > work.
> > >> > > >
> > >> > > > 2. Running in batch mode
> > >> > > >
> > >> > > > I think I lack an understanding of how to set up batch
mode
> > >> > > > because
> > >> > > > I
> > >> > > > don't
> > >> > > > see where these definitions are being explicitly defined:
> > >> > > >
> > >> > > > <plot_spec>
> > >> > > > <connection>
> > >> > > > <host>rds_host:3306</host>
> > >> > > >
> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > >> > > > <user>rds_user</user>
> > >> > > > <password>rds_pwd</password>
> > >> > > > </connection>
> > >> > > > <rscript>Rscript</rscript>
> > >> > > > <folders>
> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > >> > > > <r_work>rds_R_work</r_work>
> > >> > > > <plots>rds_plots</plots>
> > >> > > > <data>rds_data</data>
> > >> > > > <scripts>rds_scripts</scripts>
> > >> > > > </folders>
> > >> > > > <plot>
> > >> > > > <template>series_plot.R_tmpl</template>
> > >> > > >
> > >> > > > I see the <folders> section, but grepping all available
files
> > >> > > > does
> > >> > > > not
> > >> > > > reveal how rds_R_work is being defined, as well as other
> > >> > > > similarly
> > >> > > > defined
> > >> > > > variables. I think if I know how these are being
defined,
> > >> > > > then
> > >> > > > I'll
> > >> > > > be
> > >> > > > able to pipe in my information and process the data using
met
> > >> > > > and
> > >> > > > produce
> > >> > > > results in batch mode. Am I right on that? Also, if I
want
> > >> > > > to use
> > >> > > > obs for
> > >> > > > verification, do I add another line in the XML file that
> > >> > > > designates
> > >> > > > file
> > >> > > > location of obs? I assume that other steps have to be
run
> > >> > > > before
> > >> > > > all
> > >> > > > this. If data is in prepbufr form, for example, I would
have
> > >> > > > to
> > >> > > > separately
> > >> > > > run PB2NC. If I want grid2obs, then I would want to do
that
> > >> > > > as
> > >> > > > well.
> > >> > > >
> > >> > > > 3. Communicating results to metviewer online
> > >> > > >
> > >> > > > Lastly, I see a lot of people/projects listed here - see
> > >> > > > attachment
> > >> > > >
> > >> > > > How do I get the results I generate to be uploaded to
this
> > >> > > > location?
> > >> > > > It's
> > >> > > > not clear how others are setting this up.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >>
> > >>
> > >>
> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-
05-12
> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00
> > >> 2020-05-12
> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00
> > >> 2020-05-13
> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00
> > >> 2020-05-13
> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00
> > >> 2020-05-13
> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00
> > >> 2020-05-13
> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00
> > >> true
> > >> median plot_20200612_173626.data plot_20200612_173626.png
> > >> plot_20200612_173626.R test x_label test y_label
> > >> false list() list() false false false false false false true
false
> > >> false
> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0)
1 2
> > >> 1.4 -2
> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc
1
> > >> listX 1
> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8 1
> > >> #333333
> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
> > >> c(1,2) c("","") c() c() 0.04 c()
> > >
> > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Mon Jun 15 15:47:17 2020
I do plan to use date_range; however, in an example provided by the
link I
see this:
<load_spec>
<connection>
<host>kemosabe:3306</host>
<database>metvdb_hwt</database>
<user>pgoldenb</user>
<password>pgoldenb</password>
</connection>
<date_list name="folder_dates">
<start>2010051914V</start>
<end><date_offset><day_offset>0</day_offset><hour>6
</hour></date_offset></end>
<inc>3600</inc>
<format>yyyyMMddHH'V'</format>
</date_list>
<met_version>V3.0</met_version>
<verbose>false</verbose>
<insert_size>1</insert_size>
<mode_header_db_check>true</mode_header_db_check>
<drop_indexes>false</drop_indexes>
<apply_indexes>true</apply_indexes>
<group>Group name</group>
<load_stat>true</load_stat>
<load_mode>true</load_mode>
<load_mtd>true</load_mtd>
<load_mpr>false</load_mpr>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
<load_val>
<field name="model">
<val>arw</val>
<val>nmm</val>
</field>
<field name="valid_time">
<date_list name="folder_dates"/>
</field>
<field name="vx_mask">
<val>FULL</val>
<val>SWC</val>
</field>
</load_val>
</load_spec>
Are similar settings allowed with <plot_spec>? I see <connection>,
which
seems carried over. I'm looking to specify my folder_tmpl in
<plot_spec>,
which according to Tara, should be possible. Since my directory
structure
looks like this:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
I would do something like this I would think:
.......
<date_list name="folder_dates">
<start>20200612/start>
<end><date_offset><day_offset>0</day_offset></end>
<inc>86400</inc>
<format>yyyyMMdd</format>
</date_list>
<folder_tmpl>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
</folder_tmpl>
<load_val>
<field name="model">
<val>prod</val>
</field>
<field name="valid_time">
<date_list name="folder_dates"/>
</field>
......
On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Thank you. This is a very helpful example
>
> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> Hi,
>> Here is some examples how to use <date_range> and <date_range_list>
in
>> batch XML.
>> First, you need to defined your range. You can have more then one
>> <date_range>:
>> <date_range name="may">
>> <start>2019-05-01 00:00:00</start>
>> <end>2019-05-31 23:59:59</end>
>> </date_range>
>> <date_range name="march">
>> <start>2019-03-01 00:00:00</start>
>> <end>2019-03-31 23:59:59</end>
>> </date_range>
>>
>> and then use one or all date_ranges:
>> <plot_fix>
>> <field name="fcst_init_beg">
>> <date_range name="march"/>
>> <date_range name="may"/>
>> </field>...
>> This configuration would include all dates for March and May of
2019.
>>
>> If you need more flexible configuration you can use
date_range_list.
>> For example, you want to select first 6 hours every day beetween
>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
>> <date_range_list name = 'all'>
>> <range_start>2020-01-01 00:00:00</range_start>
>> <range_end>2020-01-05 00:00:00</range_end>
>> <range_length>360</range_length>
>> <inc>1440</inc>
>> </date_range_list>
>> where the 1st and 2nd values are the start and end,
>> range_length - the length of a range in minutes(6 houres)
>> inc - lenght beetween ranges in minutes (24 houres)
>> This example would create following ranges:
>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>>
>> Also you can add more parameters to <plot_fix> to filter your data.
For
>> example, if you want to select only dates with 00:00 you can
specify
>> <field name="valid_hour">
>> <val>00</val>
>> </field>
>>
>> Tatiana
>>
>>
>>
>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
>> > and if there isn't a better way, I wonder the whereabouts of a
>> > separate
>> > script that pipes dates (and other relevant information) into the
XML
>> > file
>> > used to generate the results.
>> >
>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA Affiliate
<
>> > edward.strobach at noaa.gov> wrote:
>> >
>> > > Thanks Tatiana. These resources are helpful.
>> > >
>> > > One last question. Simply typing lines like this in can take
>> > > forever:
>> > >
>> > > <plot_fix>
>> > > <field equalize="false" name="fcst_init_beg">
>> > > <set name="fcst_init_beg_0">
>> > > <val>2019-05-01 12:00:00</val>
>> > > <val>2019-05-02 06:00:00</val>
>> > > <val>2019-05-02 12:00:00</val>
>> > > <val>2019-05-03 06:00:00</val>
>> > > <val>2019-05-03 12:00:00</val>
>> > >
>> > >
>> > > Is there a better way? I see date_range, date_range_list, or
>> > > date_list.
>> > > Can this be used instead of typing individual values? This
could
>> > > prove
>> > > pretty daunting if one would have to type in 700 entries each
time.
>> > >
>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>> > > <met_help at ucar.edu>
>> > > wrote:
>> > >
>> > >> I found a few problems in your XML.
>> > >> The database you are using doesn't have data with vx_mask
>> > >> 'abcREGabc'.
>> > >> That is why METviewer returns an empty dataset and can't
create a
>> > >> plot.
>> > >> The other problem is how you define fcst_lead. This database
has
>> > >> values
>> > >> like 10000, 20000 and not 010000, 020000. Since you include
all
>> > >> fcst_leads
>> > >> this variable can be omitted.
>> > >> To minimize problems with variables values, I would suggest to
>> > >> create XML
>> > >> using GUI and then edit it to fit batch standards.
>> > >> I created a plot using you configurations with GUI (without
vx_mack
>> > >> and
>> > >> plot and axis titles) #20200612_173626. You can load it back
to the
>> > >> page by
>> > >> using the "History" tab (left bar on GUI). Find the plot, open
XML
>> > >> and
>> > >> click on "Load to page". After that you can recreate a plot
and edit
>> > >> the
>> > >> configurations. You can also download a new XML and edit it
for
>> > >> batch.
>> > >> I attached the example of load XML to this message.
>> > >>
>> > >> Description of rds parameters:
>> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>> > >> rds_plots - temporary directory where the plots will be
generated
>> > >> rds_data - temporary directory where the data retrieved from
the
>> > >> database
>> > >> and other intermediate files will be stored
>> > >> rds_scripts - temporary directory where R scripts for this run
will
>> > >> be
>> > >> stored
>> > >> After the execution is done all the files from temporary
directories
>> > >> would be removed.
>> > >>
>> > >>
>> > >> We don't have the offitial user guide for METviewer. There are
some
>> > >> instructions on how to crate XML config files:
>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>> > >>
>> > >> Tatiana
>> > >>
>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
>> > >> > Hi,
>> > >> >
>> > >> > I'm attaching the xml below, but am having hard time
following how
>> > >> > to
>> > >> > go
>> > >> > about the other steps since I have zero experience on how to
do
>> > >> > these
>> > >> > things.
>> > >> >
>> > >> > #2. All parameters in <folders> section should NOT be
changed and
>> > >> > will
>> > >> > be
>> > >> > replaced by other scripts during the batch execution. The
data for
>> > >> > the
>> > >> > batch mode is coming from the database and not from files.
The
>> > >> > name of
>> > >> > the
>> > >> > database is in <database> tag.
>> > >> >
>> > >> > I understand that nothing should be changed in the folders
>> > >> > section,
>> > >> > but
>> > >> > there is no indication how and where these variables are
being
>> > >> > defined.
>> > >> > Grepping these names of the files available does not reveal
formal
>> > >> > definitions of these variables. If I understood exactly how
these
>> > >> > were
>> > >> > being defined, then I would be able connect this better.
>> > >> >
>> > >> > #3. Users rum MET and produce MET output files. They use
METviewer
>> > >> > load
>> > >> > mode to load these files to the specific database. In the
config
>> > >> > file
>> > >> > for
>> > >> > this process they also can specify the database group. For
example
>> > >> > database
>> > >> > mv_dev is in Test12 group.
>> > >> > After the data has been loaded to the database it is visible
and
>> > >> > available
>> > >> > in the GUI and also can be used by the batch mode.
>> > >> > You can always add more data to the database using the same
load
>> > >> > mode.
>> > >> >
>> > >> > Is there a guide for me to understand the process that you
are
>> > >> > referring to?
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>> > >> > <met_help at ucar.edu>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Edward,
>> > >> > > #1. I don't have an access to NOAA servers and can't view
your
>> > >> > > XML
>> > >> > > for the
>> > >> > > batch mode. Please, attache it to this issue and I will
take a
>> > >> > > look.
>> > >> > >
>> > >> > > #2. All parameters in <folders> section should NOT be
changed
>> > >> > > and
>> > >> > > will be
>> > >> > > replaced by other scripts during the batch execution. The
data
>> > >> > > for
>> > >> > > the
>> > >> > > batch mode is coming from the database and not from files.
The
>> > >> > > name
>> > >> > > of the
>> > >> > > database is in <database> tag.
>> > >> > >
>> > >> > > #3. Users rum MET and produce MET output files. They use
>> > >> > > METviewer
>> > >> > > load
>> > >> > > mode to load these files to the specific database. In the
config
>> > >> > > file
>> > >> > > for
>> > >> > > this process they also can specify the database group. For
>> > >> > > example
>> > >> > > database
>> > >> > > mv_dev is in Test12 group.
>> > >> > > After the data has been loaded to the database it is
visible and
>> > >> > > available
>> > >> > > in the GUI and also can be used by the batch mode.
>> > >> > > You can always add more data to the database using the
same load
>> > >> > > mode.
>> > >> > >
>> > >> > > Tatiana
>> > >> > >
>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
>> > >> > > > Good morning,
>> > >> > > >
>> > >> > > > Over the last two weeks I've made an effort to
understand
>> > >> > > > metviewer,
>> > >> > > > test
>> > >> > > > different cases using the history tab, changing
information to
>> > >> > > > see
>> > >> > > > what
>> > >> > > > works and what does not, and make connections between
the GUI
>> > >> > > > settings
>> > >> > > > and
>> > >> > > > the XML file. Although my comfort level with metviewer
has
>> > >> > > > significantly
>> > >> > > > increased during this time, there are some remaining
issues
>> > >> > > > that
>> > >> > > > I'm
>> > >> > > > hoping
>> > >> > > > to clear up.
>> > >> > > >
>> > >> > > > 1. Problem changing XML file and running with different
>> > >> > > > statistic
>> > >> > > > and
>> > >> > > > plot
>> > >> > > > type:
>> > >> > > >
>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
>> > >> > > > directory
>> > >> > > > to get
>> > >> > > > started which I copied over:
>> > >> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > >> > > >
>> > >> > > > He has a test.sh script that I adjusted and used to run
his
>> > >> > > > example.
>> > >> > > > running of the script "bash test.sh" produced a CSI plot
that
>> > >> > > > was
>> > >> > > > stored
>> > >> > > > here:
>> > >> > > >
>> > >> > >
>> > >>
>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>> > >> > > >
>> > >> > > > Running the script produced the expected result. The
next
>> > >> > > > natural
>> > >> > > > step was
>> > >> > > > to copy hochum_xml.template and create a new xml file
called
>> > >> > > > Strobach_xml.template in order to create a time series
where
>> > >> > > > the
>> > >> > > > independent time is fcst_valid_beg instead of
obs_thresh.
>> > >> > > > Accordingly,
>> > >> > > > I
>> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg) and
>> > >> > > > changed
>> > >> > > > the
>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried). The
error
>> > >> > > > that
>> > >> > > > resulted
>> > >> > > > is as follows:
>> > >> > > >
>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
13:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 01
>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 01
>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 01
>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 01
>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 02
>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 02
>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 02
>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 02
>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>> > >> > > > AND ld.fmean != -9999;
>> > >> > > > No data was returned from database
>> > >> > > > 2020-06-09
>> > >> > > >
>> > >> > >
>> > >>
>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>> > >> > > > Source) -1 printFormattedTable - ** ERROR: Caught
class
>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>> > >> > > > printFormattedTable(ResultSet
>> > >> > > > res): result set contained no data
>> > >> > > > ** ERROR: Caught class
>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >> > > > in
>> > >> > > > printFormattedTable(ResultSet res): result set contained
no
>> > >> > > > data
>> > >> > > > ** ERROR: Caught class
>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >> > > > in
>> > >> > > > printFormattedTable(ResultSet res): result set contained
no
>> > >> > > > data
>> > >> > > >
>> > >> > > > Apparently there was no data during this time. I tried
>> > >> > > > changing
>> > >> > > > the
>> > >> > > > range
>> > >> > > > of valid times and changed the valid time to different
days
>> > >> > > > just in
>> > >> > > > case
>> > >> > > > data was missing. The issue was the same. To confirm
my
>> > >> > > > logic, I
>> > >> > > > reproduced my attempted approach on metviewer, which I
found
>> > >> > > > to
>> > >> > > > work.
>> > >> > > >
>> > >> > > > 2. Running in batch mode
>> > >> > > >
>> > >> > > > I think I lack an understanding of how to set up batch
mode
>> > >> > > > because
>> > >> > > > I
>> > >> > > > don't
>> > >> > > > see where these definitions are being explicitly
defined:
>> > >> > > >
>> > >> > > > <plot_spec>
>> > >> > > > <connection>
>> > >> > > > <host>rds_host:3306</host>
>> > >> > > >
>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>> > >> > > > <user>rds_user</user>
>> > >> > > > <password>rds_pwd</password>
>> > >> > > > </connection>
>> > >> > > > <rscript>Rscript</rscript>
>> > >> > > > <folders>
>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>> > >> > > > <r_work>rds_R_work</r_work>
>> > >> > > > <plots>rds_plots</plots>
>> > >> > > > <data>rds_data</data>
>> > >> > > > <scripts>rds_scripts</scripts>
>> > >> > > > </folders>
>> > >> > > > <plot>
>> > >> > > > <template>series_plot.R_tmpl</template>
>> > >> > > >
>> > >> > > > I see the <folders> section, but grepping all available
files
>> > >> > > > does
>> > >> > > > not
>> > >> > > > reveal how rds_R_work is being defined, as well as other
>> > >> > > > similarly
>> > >> > > > defined
>> > >> > > > variables. I think if I know how these are being
defined,
>> > >> > > > then
>> > >> > > > I'll
>> > >> > > > be
>> > >> > > > able to pipe in my information and process the data
using met
>> > >> > > > and
>> > >> > > > produce
>> > >> > > > results in batch mode. Am I right on that? Also, if I
want
>> > >> > > > to use
>> > >> > > > obs for
>> > >> > > > verification, do I add another line in the XML file that
>> > >> > > > designates
>> > >> > > > file
>> > >> > > > location of obs? I assume that other steps have to be
run
>> > >> > > > before
>> > >> > > > all
>> > >> > > > this. If data is in prepbufr form, for example, I would
have
>> > >> > > > to
>> > >> > > > separately
>> > >> > > > run PB2NC. If I want grid2obs, then I would want to do
that
>> > >> > > > as
>> > >> > > > well.
>> > >> > > >
>> > >> > > > 3. Communicating results to metviewer online
>> > >> > > >
>> > >> > > > Lastly, I see a lot of people/projects listed here - see
>> > >> > > > attachment
>> > >> > > >
>> > >> > > > How do I get the results I generate to be uploaded to
this
>> > >> > > > location?
>> > >> > > > It's
>> > >> > > > not clear how others are setting this up.
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >>
>> > >>
>> > >>
>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-
05-12
>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00
>> > >> 2020-05-12
>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00
>> > >> 2020-05-13
>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00
>> > >> 2020-05-13
>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00
>> > >> 2020-05-13
>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00
>> > >> 2020-05-13
>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00
>> > >> true
>> > >> median plot_20200612_173626.data plot_20200612_173626.png
>> > >> plot_20200612_173626.R test x_label test y_label
>> > >> false list() list() false false false false false false true
false
>> > >> false
>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0)
1 2
>> > >> 1.4 -2
>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc
1
>> > >> listX 1
>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8
1
>> > >> #333333
>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
>> > >> c(1,2) c("","") c() c() 0.04 c()
>> > >
>> > >
>> > >
>> > > --
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/
>> > > IMSG Contractor
>> > > Cubicle#: 2029
>> > > 301-683-3717
>> > >
>>
>>
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 07:13:37 2020
It seems that those options must be treated separately outside of
plot_spec. Is it acceptable to create both a load_spec and plot_spec
in
the same xml file? Would the definitions in load_spec be able to
carry
over into plot_spec? It appears that data loading and plotting are
treated
separately.
On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> I do plan to use date_range; however, in an example provided by the
link I
> see this:
>
> <load_spec>
> <connection>
> <host>kemosabe:3306</host>
> <database>metvdb_hwt</database>
> <user>pgoldenb</user>
> <password>pgoldenb</password>
> </connection>
>
> <date_list name="folder_dates">
> <start>2010051914V</start>
> <end><date_offset><day_offset>0</day_offset><hour>6
> </hour></date_offset></end>
> <inc>3600</inc>
> <format>yyyyMMddHH'V'</format>
> </date_list>
>
> <met_version>V3.0</met_version>
>
> <verbose>false</verbose>
> <insert_size>1</insert_size>
> <mode_header_db_check>true</mode_header_db_check>
> <drop_indexes>false</drop_indexes>
> <apply_indexes>true</apply_indexes>
> <group>Group name</group>
> <load_stat>true</load_stat>
> <load_mode>true</load_mode>
> <load_mtd>true</load_mtd>
> <load_mpr>false</load_mpr>
>
> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> <load_val>
> <field name="model">
> <val>arw</val>
> <val>nmm</val>
> </field>
>
> <field name="valid_time">
> <date_list name="folder_dates"/>
> </field>
>
> <field name="vx_mask">
> <val>FULL</val>
> <val>SWC</val>
> </field>
> </load_val>
> </load_spec>
>
> Are similar settings allowed with <plot_spec>? I see <connection>,
which
> seems carried over. I'm looking to specify my folder_tmpl in
<plot_spec>,
> which according to Tara, should be possible. Since my directory
structure
> looks like this:
>
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>
> I would do something like this I would think:
>
> .......
>
> <date_list name="folder_dates">
> <start>20200612/start>
> <end><date_offset><day_offset>0</day_offset></end>
> <inc>86400</inc>
> <format>yyyyMMdd</format>
> </date_list>
>
> <folder_tmpl>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> </folder_tmpl>
> <load_val>
> <field name="model">
> <val>prod</val>
> </field>
>
> <field name="valid_time">
> <date_list name="folder_dates"/>
> </field>
>
> ......
>
> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> Thank you. This is a very helpful example
>>
>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
>> wrote:
>>
>>> Hi,
>>> Here is some examples how to use <date_range> and
<date_range_list> in
>>> batch XML.
>>> First, you need to defined your range. You can have more then one
>>> <date_range>:
>>> <date_range name="may">
>>> <start>2019-05-01 00:00:00</start>
>>> <end>2019-05-31 23:59:59</end>
>>> </date_range>
>>> <date_range name="march">
>>> <start>2019-03-01 00:00:00</start>
>>> <end>2019-03-31 23:59:59</end>
>>> </date_range>
>>>
>>> and then use one or all date_ranges:
>>> <plot_fix>
>>> <field name="fcst_init_beg">
>>> <date_range name="march"/>
>>> <date_range name="may"/>
>>> </field>...
>>> This configuration would include all dates for March and May of
2019.
>>>
>>> If you need more flexible configuration you can use
date_range_list.
>>> For example, you want to select first 6 hours every day beetween
>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
>>> <date_range_list name = 'all'>
>>> <range_start>2020-01-01 00:00:00</range_start>
>>> <range_end>2020-01-05 00:00:00</range_end>
>>> <range_length>360</range_length>
>>> <inc>1440</inc>
>>> </date_range_list>
>>> where the 1st and 2nd values are the start and end,
>>> range_length - the length of a range in minutes(6 houres)
>>> inc - lenght beetween ranges in minutes (24 houres)
>>> This example would create following ranges:
>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>>>
>>> Also you can add more parameters to <plot_fix> to filter your
data. For
>>> example, if you want to select only dates with 00:00 you can
specify
>>> <field name="valid_hour">
>>> <val>00</val>
>>> </field>
>>>
>>> Tatiana
>>>
>>>
>>>
>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
>>> > and if there isn't a better way, I wonder the whereabouts of a
>>> > separate
>>> > script that pipes dates (and other relevant information) into
the XML
>>> > file
>>> > used to generate the results.
>>> >
>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
Affiliate <
>>> > edward.strobach at noaa.gov> wrote:
>>> >
>>> > > Thanks Tatiana. These resources are helpful.
>>> > >
>>> > > One last question. Simply typing lines like this in can take
>>> > > forever:
>>> > >
>>> > > <plot_fix>
>>> > > <field equalize="false" name="fcst_init_beg">
>>> > > <set name="fcst_init_beg_0">
>>> > > <val>2019-05-01 12:00:00</val>
>>> > > <val>2019-05-02 06:00:00</val>
>>> > > <val>2019-05-02 12:00:00</val>
>>> > > <val>2019-05-03 06:00:00</val>
>>> > > <val>2019-05-03 12:00:00</val>
>>> > >
>>> > >
>>> > > Is there a better way? I see date_range, date_range_list, or
>>> > > date_list.
>>> > > Can this be used instead of typing individual values? This
could
>>> > > prove
>>> > > pretty daunting if one would have to type in 700 entries each
time.
>>> > >
>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>>> > > <met_help at ucar.edu>
>>> > > wrote:
>>> > >
>>> > >> I found a few problems in your XML.
>>> > >> The database you are using doesn't have data with vx_mask
>>> > >> 'abcREGabc'.
>>> > >> That is why METviewer returns an empty dataset and can't
create a
>>> > >> plot.
>>> > >> The other problem is how you define fcst_lead. This database
has
>>> > >> values
>>> > >> like 10000, 20000 and not 010000, 020000. Since you include
all
>>> > >> fcst_leads
>>> > >> this variable can be omitted.
>>> > >> To minimize problems with variables values, I would suggest
to
>>> > >> create XML
>>> > >> using GUI and then edit it to fit batch standards.
>>> > >> I created a plot using you configurations with GUI (without
vx_mack
>>> > >> and
>>> > >> plot and axis titles) #20200612_173626. You can load it back
to the
>>> > >> page by
>>> > >> using the "History" tab (left bar on GUI). Find the plot,
open XML
>>> > >> and
>>> > >> click on "Load to page". After that you can recreate a plot
and edit
>>> > >> the
>>> > >> configurations. You can also download a new XML and edit it
for
>>> > >> batch.
>>> > >> I attached the example of load XML to this message.
>>> > >>
>>> > >> Description of rds parameters:
>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>>> > >> rds_plots - temporary directory where the plots will be
generated
>>> > >> rds_data - temporary directory where the data retrieved from
the
>>> > >> database
>>> > >> and other intermediate files will be stored
>>> > >> rds_scripts - temporary directory where R scripts for this
run will
>>> > >> be
>>> > >> stored
>>> > >> After the execution is done all the files from temporary
directories
>>> > >> would be removed.
>>> > >>
>>> > >>
>>> > >> We don't have the offitial user guide for METviewer. There
are some
>>> > >> instructions on how to crate XML config files:
>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>> > >>
>>> > >> Tatiana
>>> > >>
>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
>>> > >> > Hi,
>>> > >> >
>>> > >> > I'm attaching the xml below, but am having hard time
following how
>>> > >> > to
>>> > >> > go
>>> > >> > about the other steps since I have zero experience on how
to do
>>> > >> > these
>>> > >> > things.
>>> > >> >
>>> > >> > #2. All parameters in <folders> section should NOT be
changed and
>>> > >> > will
>>> > >> > be
>>> > >> > replaced by other scripts during the batch execution. The
data for
>>> > >> > the
>>> > >> > batch mode is coming from the database and not from files.
The
>>> > >> > name of
>>> > >> > the
>>> > >> > database is in <database> tag.
>>> > >> >
>>> > >> > I understand that nothing should be changed in the folders
>>> > >> > section,
>>> > >> > but
>>> > >> > there is no indication how and where these variables are
being
>>> > >> > defined.
>>> > >> > Grepping these names of the files available does not reveal
formal
>>> > >> > definitions of these variables. If I understood exactly
how these
>>> > >> > were
>>> > >> > being defined, then I would be able connect this better.
>>> > >> >
>>> > >> > #3. Users rum MET and produce MET output files. They use
METviewer
>>> > >> > load
>>> > >> > mode to load these files to the specific database. In the
config
>>> > >> > file
>>> > >> > for
>>> > >> > this process they also can specify the database group. For
example
>>> > >> > database
>>> > >> > mv_dev is in Test12 group.
>>> > >> > After the data has been loaded to the database it is
visible and
>>> > >> > available
>>> > >> > in the GUI and also can be used by the batch mode.
>>> > >> > You can always add more data to the database using the same
load
>>> > >> > mode.
>>> > >> >
>>> > >> > Is there a guide for me to understand the process that you
are
>>> > >> > referring to?
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>>> > >> > <met_help at ucar.edu>
>>> > >> > wrote:
>>> > >> >
>>> > >> > > Hi Edward,
>>> > >> > > #1. I don't have an access to NOAA servers and can't view
your
>>> > >> > > XML
>>> > >> > > for the
>>> > >> > > batch mode. Please, attache it to this issue and I will
take a
>>> > >> > > look.
>>> > >> > >
>>> > >> > > #2. All parameters in <folders> section should NOT be
changed
>>> > >> > > and
>>> > >> > > will be
>>> > >> > > replaced by other scripts during the batch execution. The
data
>>> > >> > > for
>>> > >> > > the
>>> > >> > > batch mode is coming from the database and not from
files. The
>>> > >> > > name
>>> > >> > > of the
>>> > >> > > database is in <database> tag.
>>> > >> > >
>>> > >> > > #3. Users rum MET and produce MET output files. They use
>>> > >> > > METviewer
>>> > >> > > load
>>> > >> > > mode to load these files to the specific database. In the
config
>>> > >> > > file
>>> > >> > > for
>>> > >> > > this process they also can specify the database group.
For
>>> > >> > > example
>>> > >> > > database
>>> > >> > > mv_dev is in Test12 group.
>>> > >> > > After the data has been loaded to the database it is
visible and
>>> > >> > > available
>>> > >> > > in the GUI and also can be used by the batch mode.
>>> > >> > > You can always add more data to the database using the
same load
>>> > >> > > mode.
>>> > >> > >
>>> > >> > > Tatiana
>>> > >> > >
>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
>>> > >> > > > Good morning,
>>> > >> > > >
>>> > >> > > > Over the last two weeks I've made an effort to
understand
>>> > >> > > > metviewer,
>>> > >> > > > test
>>> > >> > > > different cases using the history tab, changing
information to
>>> > >> > > > see
>>> > >> > > > what
>>> > >> > > > works and what does not, and make connections between
the GUI
>>> > >> > > > settings
>>> > >> > > > and
>>> > >> > > > the XML file. Although my comfort level with metviewer
has
>>> > >> > > > significantly
>>> > >> > > > increased during this time, there are some remaining
issues
>>> > >> > > > that
>>> > >> > > > I'm
>>> > >> > > > hoping
>>> > >> > > > to clear up.
>>> > >> > > >
>>> > >> > > > 1. Problem changing XML file and running with
different
>>> > >> > > > statistic
>>> > >> > > > and
>>> > >> > > > plot
>>> > >> > > > type:
>>> > >> > > >
>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
>>> > >> > > > directory
>>> > >> > > > to get
>>> > >> > > > started which I copied over:
>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>> > >> > > >
>>> > >> > > > He has a test.sh script that I adjusted and used to run
his
>>> > >> > > > example.
>>> > >> > > > running of the script "bash test.sh" produced a CSI
plot that
>>> > >> > > > was
>>> > >> > > > stored
>>> > >> > > > here:
>>> > >> > > >
>>> > >> > >
>>> > >>
>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>> > >> > > >
>>> > >> > > > Running the script produced the expected result. The
next
>>> > >> > > > natural
>>> > >> > > > step was
>>> > >> > > > to copy hochum_xml.template and create a new xml file
called
>>> > >> > > > Strobach_xml.template in order to create a time series
where
>>> > >> > > > the
>>> > >> > > > independent time is fcst_valid_beg instead of
obs_thresh.
>>> > >> > > > Accordingly,
>>> > >> > > > I
>>> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg) and
>>> > >> > > > changed
>>> > >> > > > the
>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried). The
error
>>> > >> > > > that
>>> > >> > > > resulted
>>> > >> > > > is as follows:
>>> > >> > > >
>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
13:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 01
>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
16:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 01
>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
19:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 01
>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
22:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 01
>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
01:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 02
>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
04:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 02
>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
07:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 02
>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
10:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 02
>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>> > >> > > > AND ld.fmean != -9999;
>>> > >> > > > No data was returned from database
>>> > >> > > > 2020-06-09
>>> > >> > > >
>>> > >> > >
>>> > >>
>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>> > >> > > > Source) -1 printFormattedTable - ** ERROR: Caught
class
>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>>> > >> > > > printFormattedTable(ResultSet
>>> > >> > > > res): result set contained no data
>>> > >> > > > ** ERROR: Caught class
>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>> > >> > > > in
>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>> > >> > > > data
>>> > >> > > > ** ERROR: Caught class
>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>> > >> > > > in
>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>> > >> > > > data
>>> > >> > > >
>>> > >> > > > Apparently there was no data during this time. I tried
>>> > >> > > > changing
>>> > >> > > > the
>>> > >> > > > range
>>> > >> > > > of valid times and changed the valid time to different
days
>>> > >> > > > just in
>>> > >> > > > case
>>> > >> > > > data was missing. The issue was the same. To confirm
my
>>> > >> > > > logic, I
>>> > >> > > > reproduced my attempted approach on metviewer, which I
found
>>> > >> > > > to
>>> > >> > > > work.
>>> > >> > > >
>>> > >> > > > 2. Running in batch mode
>>> > >> > > >
>>> > >> > > > I think I lack an understanding of how to set up batch
mode
>>> > >> > > > because
>>> > >> > > > I
>>> > >> > > > don't
>>> > >> > > > see where these definitions are being explicitly
defined:
>>> > >> > > >
>>> > >> > > > <plot_spec>
>>> > >> > > > <connection>
>>> > >> > > > <host>rds_host:3306</host>
>>> > >> > > >
>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>> > >> > > > <user>rds_user</user>
>>> > >> > > > <password>rds_pwd</password>
>>> > >> > > > </connection>
>>> > >> > > > <rscript>Rscript</rscript>
>>> > >> > > > <folders>
>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>> > >> > > > <r_work>rds_R_work</r_work>
>>> > >> > > > <plots>rds_plots</plots>
>>> > >> > > > <data>rds_data</data>
>>> > >> > > > <scripts>rds_scripts</scripts>
>>> > >> > > > </folders>
>>> > >> > > > <plot>
>>> > >> > > > <template>series_plot.R_tmpl</template>
>>> > >> > > >
>>> > >> > > > I see the <folders> section, but grepping all available
files
>>> > >> > > > does
>>> > >> > > > not
>>> > >> > > > reveal how rds_R_work is being defined, as well as
other
>>> > >> > > > similarly
>>> > >> > > > defined
>>> > >> > > > variables. I think if I know how these are being
defined,
>>> > >> > > > then
>>> > >> > > > I'll
>>> > >> > > > be
>>> > >> > > > able to pipe in my information and process the data
using met
>>> > >> > > > and
>>> > >> > > > produce
>>> > >> > > > results in batch mode. Am I right on that? Also, if I
want
>>> > >> > > > to use
>>> > >> > > > obs for
>>> > >> > > > verification, do I add another line in the XML file
that
>>> > >> > > > designates
>>> > >> > > > file
>>> > >> > > > location of obs? I assume that other steps have to be
run
>>> > >> > > > before
>>> > >> > > > all
>>> > >> > > > this. If data is in prepbufr form, for example, I
would have
>>> > >> > > > to
>>> > >> > > > separately
>>> > >> > > > run PB2NC. If I want grid2obs, then I would want to do
that
>>> > >> > > > as
>>> > >> > > > well.
>>> > >> > > >
>>> > >> > > > 3. Communicating results to metviewer online
>>> > >> > > >
>>> > >> > > > Lastly, I see a lot of people/projects listed here -
see
>>> > >> > > > attachment
>>> > >> > > >
>>> > >> > > > How do I get the results I generate to be uploaded to
this
>>> > >> > > > location?
>>> > >> > > > It's
>>> > >> > > > not clear how others are setting this up.
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >>
>>> > >>
>>> > >>
>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
2020-05-12
>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00
>>> > >> 2020-05-12
>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00
>>> > >> 2020-05-13
>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00
>>> > >> 2020-05-13
>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00
>>> > >> 2020-05-13
>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00
>>> > >> 2020-05-13
>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00
>>> > >> true
>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
>>> > >> plot_20200612_173626.R test x_label test y_label
>>> > >> false list() list() false false false false false false true
false
>>> > >> false
>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
c(1,1,0) 1 2
>>> > >> 1.4 -2
>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
#cccccc 1
>>> > >> listX 1
>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8
1
>>> > >> #333333
>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Edward Strobach
>>> > > EMC/NCEP/NWS/
>>> > > IMSG Contractor
>>> > > Cubicle#: 2029
>>> > > 301-683-3717
>>> > >
>>>
>>>
>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 07:35:03 2020
Actually, it appears that I need to run the database loading module
first
if I want to change/add a data source. From there I can run the
plotting
batch module. Is that right? So if I run the database loading
module,
then I can specify my dates and other related info specific to the
dataset. If I create my database with a unique name, then it should
be
recognized on the metviewer gui, presumably.
On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> It seems that those options must be treated separately outside of
> plot_spec. Is it acceptable to create both a load_spec and
plot_spec in
> the same xml file? Would the definitions in load_spec be able to
carry
> over into plot_spec? It appears that data loading and plotting are
treated
> separately.
>
> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> I do plan to use date_range; however, in an example provided by the
link
>> I see this:
>>
>> <load_spec>
>> <connection>
>> <host>kemosabe:3306</host>
>> <database>metvdb_hwt</database>
>> <user>pgoldenb</user>
>> <password>pgoldenb</password>
>> </connection>
>>
>> <date_list name="folder_dates">
>> <start>2010051914V</start>
>> <end><date_offset><day_offset>0</day_offset><hour>6
>> </hour></date_offset></end>
>> <inc>3600</inc>
>> <format>yyyyMMddHH'V'</format>
>> </date_list>
>>
>> <met_version>V3.0</met_version>
>>
>> <verbose>false</verbose>
>> <insert_size>1</insert_size>
>> <mode_header_db_check>true</mode_header_db_check>
>> <drop_indexes>false</drop_indexes>
>> <apply_indexes>true</apply_indexes>
>> <group>Group name</group>
>> <load_stat>true</load_stat>
>> <load_mode>true</load_mode>
>> <load_mtd>true</load_mtd>
>> <load_mpr>false</load_mpr>
>>
>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>> <load_val>
>> <field name="model">
>> <val>arw</val>
>> <val>nmm</val>
>> </field>
>>
>> <field name="valid_time">
>> <date_list name="folder_dates"/>
>> </field>
>>
>> <field name="vx_mask">
>> <val>FULL</val>
>> <val>SWC</val>
>> </field>
>> </load_val>
>> </load_spec>
>>
>> Are similar settings allowed with <plot_spec>? I see <connection>,
which
>> seems carried over. I'm looking to specify my folder_tmpl in
<plot_spec>,
>> which according to Tara, should be possible. Since my directory
structure
>> looks like this:
>>
>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>
>> I would do something like this I would think:
>>
>> .......
>>
>> <date_list name="folder_dates">
>> <start>20200612/start>
>> <end><date_offset><day_offset>0</day_offset></end>
>> <inc>86400</inc>
>> <format>yyyyMMdd</format>
>> </date_list>
>>
>> <folder_tmpl>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>> </folder_tmpl>
>> <load_val>
>> <field name="model">
>> <val>prod</val>
>> </field>
>>
>> <field name="valid_time">
>> <date_list name="folder_dates"/>
>> </field>
>>
>> ......
>>
>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> Thank you. This is a very helpful example
>>>
>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
>>> wrote:
>>>
>>>> Hi,
>>>> Here is some examples how to use <date_range> and
<date_range_list> in
>>>> batch XML.
>>>> First, you need to defined your range. You can have more then one
>>>> <date_range>:
>>>> <date_range name="may">
>>>> <start>2019-05-01 00:00:00</start>
>>>> <end>2019-05-31 23:59:59</end>
>>>> </date_range>
>>>> <date_range name="march">
>>>> <start>2019-03-01 00:00:00</start>
>>>> <end>2019-03-31 23:59:59</end>
>>>> </date_range>
>>>>
>>>> and then use one or all date_ranges:
>>>> <plot_fix>
>>>> <field name="fcst_init_beg">
>>>> <date_range name="march"/>
>>>> <date_range name="may"/>
>>>> </field>...
>>>> This configuration would include all dates for March and May of
2019.
>>>>
>>>> If you need more flexible configuration you can use
date_range_list.
>>>> For example, you want to select first 6 hours every day beetween
>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
>>>> <date_range_list name = 'all'>
>>>> <range_start>2020-01-01 00:00:00</range_start>
>>>> <range_end>2020-01-05 00:00:00</range_end>
>>>> <range_length>360</range_length>
>>>> <inc>1440</inc>
>>>> </date_range_list>
>>>> where the 1st and 2nd values are the start and end,
>>>> range_length - the length of a range in minutes(6 houres)
>>>> inc - lenght beetween ranges in minutes (24 houres)
>>>> This example would create following ranges:
>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>>>>
>>>> Also you can add more parameters to <plot_fix> to filter your
data. For
>>>> example, if you want to select only dates with 00:00 you can
specify
>>>> <field name="valid_hour">
>>>> <val>00</val>
>>>> </field>
>>>>
>>>> Tatiana
>>>>
>>>>
>>>>
>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
>>>> > and if there isn't a better way, I wonder the whereabouts of a
>>>> > separate
>>>> > script that pipes dates (and other relevant information) into
the XML
>>>> > file
>>>> > used to generate the results.
>>>> >
>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
Affiliate <
>>>> > edward.strobach at noaa.gov> wrote:
>>>> >
>>>> > > Thanks Tatiana. These resources are helpful.
>>>> > >
>>>> > > One last question. Simply typing lines like this in can take
>>>> > > forever:
>>>> > >
>>>> > > <plot_fix>
>>>> > > <field equalize="false" name="fcst_init_beg">
>>>> > > <set name="fcst_init_beg_0">
>>>> > > <val>2019-05-01 12:00:00</val>
>>>> > > <val>2019-05-02 06:00:00</val>
>>>> > > <val>2019-05-02 12:00:00</val>
>>>> > > <val>2019-05-03 06:00:00</val>
>>>> > > <val>2019-05-03 12:00:00</val>
>>>> > >
>>>> > >
>>>> > > Is there a better way? I see date_range, date_range_list,
or
>>>> > > date_list.
>>>> > > Can this be used instead of typing individual values? This
could
>>>> > > prove
>>>> > > pretty daunting if one would have to type in 700 entries each
time.
>>>> > >
>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>>>> > > <met_help at ucar.edu>
>>>> > > wrote:
>>>> > >
>>>> > >> I found a few problems in your XML.
>>>> > >> The database you are using doesn't have data with vx_mask
>>>> > >> 'abcREGabc'.
>>>> > >> That is why METviewer returns an empty dataset and can't
create a
>>>> > >> plot.
>>>> > >> The other problem is how you define fcst_lead. This database
has
>>>> > >> values
>>>> > >> like 10000, 20000 and not 010000, 020000. Since you include
all
>>>> > >> fcst_leads
>>>> > >> this variable can be omitted.
>>>> > >> To minimize problems with variables values, I would suggest
to
>>>> > >> create XML
>>>> > >> using GUI and then edit it to fit batch standards.
>>>> > >> I created a plot using you configurations with GUI (without
vx_mack
>>>> > >> and
>>>> > >> plot and axis titles) #20200612_173626. You can load it back
to the
>>>> > >> page by
>>>> > >> using the "History" tab (left bar on GUI). Find the plot,
open XML
>>>> > >> and
>>>> > >> click on "Load to page". After that you can recreate a plot
and
>>>> edit
>>>> > >> the
>>>> > >> configurations. You can also download a new XML and edit it
for
>>>> > >> batch.
>>>> > >> I attached the example of load XML to this message.
>>>> > >>
>>>> > >> Description of rds parameters:
>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>>>> > >> rds_plots - temporary directory where the plots will be
generated
>>>> > >> rds_data - temporary directory where the data retrieved from
the
>>>> > >> database
>>>> > >> and other intermediate files will be stored
>>>> > >> rds_scripts - temporary directory where R scripts for this
run will
>>>> > >> be
>>>> > >> stored
>>>> > >> After the execution is done all the files from temporary
>>>> directories
>>>> > >> would be removed.
>>>> > >>
>>>> > >>
>>>> > >> We don't have the offitial user guide for METviewer. There
are some
>>>> > >> instructions on how to crate XML config files:
>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>>> > >>
>>>> > >> Tatiana
>>>> > >>
>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
>>>> > >> > Hi,
>>>> > >> >
>>>> > >> > I'm attaching the xml below, but am having hard time
following
>>>> how
>>>> > >> > to
>>>> > >> > go
>>>> > >> > about the other steps since I have zero experience on how
to do
>>>> > >> > these
>>>> > >> > things.
>>>> > >> >
>>>> > >> > #2. All parameters in <folders> section should NOT be
changed and
>>>> > >> > will
>>>> > >> > be
>>>> > >> > replaced by other scripts during the batch execution. The
data
>>>> for
>>>> > >> > the
>>>> > >> > batch mode is coming from the database and not from files.
The
>>>> > >> > name of
>>>> > >> > the
>>>> > >> > database is in <database> tag.
>>>> > >> >
>>>> > >> > I understand that nothing should be changed in the folders
>>>> > >> > section,
>>>> > >> > but
>>>> > >> > there is no indication how and where these variables are
being
>>>> > >> > defined.
>>>> > >> > Grepping these names of the files available does not
reveal
>>>> formal
>>>> > >> > definitions of these variables. If I understood exactly
how
>>>> these
>>>> > >> > were
>>>> > >> > being defined, then I would be able connect this better.
>>>> > >> >
>>>> > >> > #3. Users rum MET and produce MET output files. They use
>>>> METviewer
>>>> > >> > load
>>>> > >> > mode to load these files to the specific database. In the
config
>>>> > >> > file
>>>> > >> > for
>>>> > >> > this process they also can specify the database group. For
>>>> example
>>>> > >> > database
>>>> > >> > mv_dev is in Test12 group.
>>>> > >> > After the data has been loaded to the database it is
visible and
>>>> > >> > available
>>>> > >> > in the GUI and also can be used by the batch mode.
>>>> > >> > You can always add more data to the database using the
same load
>>>> > >> > mode.
>>>> > >> >
>>>> > >> > Is there a guide for me to understand the process that you
are
>>>> > >> > referring to?
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>>>> > >> > <met_help at ucar.edu>
>>>> > >> > wrote:
>>>> > >> >
>>>> > >> > > Hi Edward,
>>>> > >> > > #1. I don't have an access to NOAA servers and can't
view your
>>>> > >> > > XML
>>>> > >> > > for the
>>>> > >> > > batch mode. Please, attache it to this issue and I will
take a
>>>> > >> > > look.
>>>> > >> > >
>>>> > >> > > #2. All parameters in <folders> section should NOT be
changed
>>>> > >> > > and
>>>> > >> > > will be
>>>> > >> > > replaced by other scripts during the batch execution.
The data
>>>> > >> > > for
>>>> > >> > > the
>>>> > >> > > batch mode is coming from the database and not from
files. The
>>>> > >> > > name
>>>> > >> > > of the
>>>> > >> > > database is in <database> tag.
>>>> > >> > >
>>>> > >> > > #3. Users rum MET and produce MET output files. They use
>>>> > >> > > METviewer
>>>> > >> > > load
>>>> > >> > > mode to load these files to the specific database. In
the
>>>> config
>>>> > >> > > file
>>>> > >> > > for
>>>> > >> > > this process they also can specify the database group.
For
>>>> > >> > > example
>>>> > >> > > database
>>>> > >> > > mv_dev is in Test12 group.
>>>> > >> > > After the data has been loaded to the database it is
visible
>>>> and
>>>> > >> > > available
>>>> > >> > > in the GUI and also can be used by the batch mode.
>>>> > >> > > You can always add more data to the database using the
same
>>>> load
>>>> > >> > > mode.
>>>> > >> > >
>>>> > >> > > Tatiana
>>>> > >> > >
>>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
>>>> > >> > > > Good morning,
>>>> > >> > > >
>>>> > >> > > > Over the last two weeks I've made an effort to
understand
>>>> > >> > > > metviewer,
>>>> > >> > > > test
>>>> > >> > > > different cases using the history tab, changing
information
>>>> to
>>>> > >> > > > see
>>>> > >> > > > what
>>>> > >> > > > works and what does not, and make connections between
the GUI
>>>> > >> > > > settings
>>>> > >> > > > and
>>>> > >> > > > the XML file. Although my comfort level with
metviewer has
>>>> > >> > > > significantly
>>>> > >> > > > increased during this time, there are some remaining
issues
>>>> > >> > > > that
>>>> > >> > > > I'm
>>>> > >> > > > hoping
>>>> > >> > > > to clear up.
>>>> > >> > > >
>>>> > >> > > > 1. Problem changing XML file and running with
different
>>>> > >> > > > statistic
>>>> > >> > > > and
>>>> > >> > > > plot
>>>> > >> > > > type:
>>>> > >> > > >
>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
>>>> > >> > > > directory
>>>> > >> > > > to get
>>>> > >> > > > started which I copied over:
>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>> > >> > > >
>>>> > >> > > > He has a test.sh script that I adjusted and used to
run his
>>>> > >> > > > example.
>>>> > >> > > > running of the script "bash test.sh" produced a CSI
plot that
>>>> > >> > > > was
>>>> > >> > > > stored
>>>> > >> > > > here:
>>>> > >> > > >
>>>> > >> > >
>>>> > >>
>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>> > >> > > >
>>>> > >> > > > Running the script produced the expected result. The
next
>>>> > >> > > > natural
>>>> > >> > > > step was
>>>> > >> > > > to copy hochum_xml.template and create a new xml file
called
>>>> > >> > > > Strobach_xml.template in order to create a time series
where
>>>> > >> > > > the
>>>> > >> > > > independent time is fcst_valid_beg instead of
obs_thresh.
>>>> > >> > > > Accordingly,
>>>> > >> > > > I
>>>> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg)
>>>> and
>>>> > >> > > > changed
>>>> > >> > > > the
>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).
The error
>>>> > >> > > > that
>>>> > >> > > > resulted
>>>> > >> > > > is as follows:
>>>> > >> > > >
>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
13:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 01
>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
16:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 01
>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
19:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 01
>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
22:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 01
>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
01:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 02
>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
04:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 02
>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
07:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 02
>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
10:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 02
>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>>> > >> > > > AND ld.fmean != -9999;
>>>> > >> > > > No data was returned from database
>>>> > >> > > > 2020-06-09
>>>> > >> > > >
>>>> > >> > >
>>>> > >>
>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>> > >> > > > Source) -1 printFormattedTable - ** ERROR: Caught
class
>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>>>> > >> > > > printFormattedTable(ResultSet
>>>> > >> > > > res): result set contained no data
>>>> > >> > > > ** ERROR: Caught class
>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>>> > >> > > > in
>>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>>> > >> > > > data
>>>> > >> > > > ** ERROR: Caught class
>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>>> > >> > > > in
>>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>>> > >> > > > data
>>>> > >> > > >
>>>> > >> > > > Apparently there was no data during this time. I
tried
>>>> > >> > > > changing
>>>> > >> > > > the
>>>> > >> > > > range
>>>> > >> > > > of valid times and changed the valid time to different
days
>>>> > >> > > > just in
>>>> > >> > > > case
>>>> > >> > > > data was missing. The issue was the same. To confirm
my
>>>> > >> > > > logic, I
>>>> > >> > > > reproduced my attempted approach on metviewer, which I
found
>>>> > >> > > > to
>>>> > >> > > > work.
>>>> > >> > > >
>>>> > >> > > > 2. Running in batch mode
>>>> > >> > > >
>>>> > >> > > > I think I lack an understanding of how to set up batch
mode
>>>> > >> > > > because
>>>> > >> > > > I
>>>> > >> > > > don't
>>>> > >> > > > see where these definitions are being explicitly
defined:
>>>> > >> > > >
>>>> > >> > > > <plot_spec>
>>>> > >> > > > <connection>
>>>> > >> > > > <host>rds_host:3306</host>
>>>> > >> > > >
>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>> > >> > > > <user>rds_user</user>
>>>> > >> > > > <password>rds_pwd</password>
>>>> > >> > > > </connection>
>>>> > >> > > > <rscript>Rscript</rscript>
>>>> > >> > > > <folders>
>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>> > >> > > > <plots>rds_plots</plots>
>>>> > >> > > > <data>rds_data</data>
>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>> > >> > > > </folders>
>>>> > >> > > > <plot>
>>>> > >> > > > <template>series_plot.R_tmpl</template>
>>>> > >> > > >
>>>> > >> > > > I see the <folders> section, but grepping all
available files
>>>> > >> > > > does
>>>> > >> > > > not
>>>> > >> > > > reveal how rds_R_work is being defined, as well as
other
>>>> > >> > > > similarly
>>>> > >> > > > defined
>>>> > >> > > > variables. I think if I know how these are being
defined,
>>>> > >> > > > then
>>>> > >> > > > I'll
>>>> > >> > > > be
>>>> > >> > > > able to pipe in my information and process the data
using met
>>>> > >> > > > and
>>>> > >> > > > produce
>>>> > >> > > > results in batch mode. Am I right on that? Also, if
I want
>>>> > >> > > > to use
>>>> > >> > > > obs for
>>>> > >> > > > verification, do I add another line in the XML file
that
>>>> > >> > > > designates
>>>> > >> > > > file
>>>> > >> > > > location of obs? I assume that other steps have to be
run
>>>> > >> > > > before
>>>> > >> > > > all
>>>> > >> > > > this. If data is in prepbufr form, for example, I
would have
>>>> > >> > > > to
>>>> > >> > > > separately
>>>> > >> > > > run PB2NC. If I want grid2obs, then I would want to
do that
>>>> > >> > > > as
>>>> > >> > > > well.
>>>> > >> > > >
>>>> > >> > > > 3. Communicating results to metviewer online
>>>> > >> > > >
>>>> > >> > > > Lastly, I see a lot of people/projects listed here -
see
>>>> > >> > > > attachment
>>>> > >> > > >
>>>> > >> > > > How do I get the results I generate to be uploaded to
this
>>>> > >> > > > location?
>>>> > >> > > > It's
>>>> > >> > > > not clear how others are setting this up.
>>>> > >> > >
>>>> > >> > >
>>>> > >> > >
>>>> > >> > >
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
2020-05-12
>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
>>>> 20:00:00
>>>> > >> 2020-05-12
>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
>>>> 10:00:00
>>>> > >> 2020-05-13
>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
>>>> 14:00:00
>>>> > >> 2020-05-13
>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
>>>> 18:00:00
>>>> > >> 2020-05-13
>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
>>>> 22:00:00
>>>> > >> 2020-05-13
>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
>>>> 12:00:00
>>>> > >> true
>>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
>>>> > >> plot_20200612_173626.R test x_label test y_label
>>>> > >> false list() list() false false false false false false true
false
>>>> > >> false
>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
c(1,1,0) 1 2
>>>> > >> 1.4 -2
>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
#cccccc 1
>>>> > >> listX 1
>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3
0.8 1
>>>> > >> #333333
>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Edward Strobach
>>>> > > EMC/NCEP/NWS/
>>>> > > IMSG Contractor
>>>> > > Cubicle#: 2029
>>>> > > 301-683-3717
>>>> > >
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 08:26:32 2020
lastly, it does seem thatmysql -u[db_username] -p[db_password]
-e'create
database [db_name]
would need to be used to create a database. I assume that an sql
module is
available on wcoss; I know you don't have access to those machines
Tatiana. If I create my own database with my own name, then I should
be
able to proceed with project specific tasks under that tab. Something
like
mv_EdwardStrobach.
I can then create mv_cmaq_prod_v_para which I add data to using
load_spec.
Once that step is done, then I can run plot_spec. That's my takeaway
currently
On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Actually, it appears that I need to run the database loading module
first
> if I want to change/add a data source. From there I can run the
plotting
> batch module. Is that right? So if I run the database loading
module,
> then I can specify my dates and other related info specific to the
> dataset. If I create my database with a unique name, then it should
be
> recognized on the metviewer gui, presumably.
>
> On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> It seems that those options must be treated separately outside of
>> plot_spec. Is it acceptable to create both a load_spec and
plot_spec in
>> the same xml file? Would the definitions in load_spec be able to
carry
>> over into plot_spec? It appears that data loading and plotting are
treated
>> separately.
>>
>> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> I do plan to use date_range; however, in an example provided by
the link
>>> I see this:
>>>
>>> <load_spec>
>>> <connection>
>>> <host>kemosabe:3306</host>
>>> <database>metvdb_hwt</database>
>>> <user>pgoldenb</user>
>>> <password>pgoldenb</password>
>>> </connection>
>>>
>>> <date_list name="folder_dates">
>>> <start>2010051914V</start>
>>> <end><date_offset><day_offset>0</day_offset><hour>6
>>> </hour></date_offset></end>
>>> <inc>3600</inc>
>>> <format>yyyyMMddHH'V'</format>
>>> </date_list>
>>>
>>> <met_version>V3.0</met_version>
>>>
>>> <verbose>false</verbose>
>>> <insert_size>1</insert_size>
>>> <mode_header_db_check>true</mode_header_db_check>
>>> <drop_indexes>false</drop_indexes>
>>> <apply_indexes>true</apply_indexes>
>>> <group>Group name</group>
>>> <load_stat>true</load_stat>
>>> <load_mode>true</load_mode>
>>> <load_mtd>true</load_mtd>
>>> <load_mpr>false</load_mpr>
>>>
>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>> <load_val>
>>> <field name="model">
>>> <val>arw</val>
>>> <val>nmm</val>
>>> </field>
>>>
>>> <field name="valid_time">
>>> <date_list name="folder_dates"/>
>>> </field>
>>>
>>> <field name="vx_mask">
>>> <val>FULL</val>
>>> <val>SWC</val>
>>> </field>
>>> </load_val>
>>> </load_spec>
>>>
>>> Are similar settings allowed with <plot_spec>? I see
<connection>,
>>> which seems carried over. I'm looking to specify my folder_tmpl
in
>>> <plot_spec>, which according to Tara, should be possible. Since
my
>>> directory structure looks like this:
>>>
>>>
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>
>>> I would do something like this I would think:
>>>
>>> .......
>>>
>>> <date_list name="folder_dates">
>>> <start>20200612/start>
>>> <end><date_offset><day_offset>0</day_offset></end>
>>> <inc>86400</inc>
>>> <format>yyyyMMdd</format>
>>> </date_list>
>>>
>>> <folder_tmpl>
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>> </folder_tmpl>
>>> <load_val>
>>> <field name="model">
>>> <val>prod</val>
>>> </field>
>>>
>>> <field name="valid_time">
>>> <date_list name="folder_dates"/>
>>> </field>
>>>
>>> ......
>>>
>>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate <
>>> edward.strobach at noaa.gov> wrote:
>>>
>>>> Thank you. This is a very helpful example
>>>>
>>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> Here is some examples how to use <date_range> and
<date_range_list> in
>>>>> batch XML.
>>>>> First, you need to defined your range. You can have more then
one
>>>>> <date_range>:
>>>>> <date_range name="may">
>>>>> <start>2019-05-01 00:00:00</start>
>>>>> <end>2019-05-31 23:59:59</end>
>>>>> </date_range>
>>>>> <date_range name="march">
>>>>> <start>2019-03-01 00:00:00</start>
>>>>> <end>2019-03-31 23:59:59</end>
>>>>> </date_range>
>>>>>
>>>>> and then use one or all date_ranges:
>>>>> <plot_fix>
>>>>> <field name="fcst_init_beg">
>>>>> <date_range name="march"/>
>>>>> <date_range name="may"/>
>>>>> </field>...
>>>>> This configuration would include all dates for March and May of
2019.
>>>>>
>>>>> If you need more flexible configuration you can use
date_range_list.
>>>>> For example, you want to select first 6 hours every day beetween
>>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
>>>>> <date_range_list name = 'all'>
>>>>> <range_start>2020-01-01 00:00:00</range_start>
>>>>> <range_end>2020-01-05 00:00:00</range_end>
>>>>> <range_length>360</range_length>
>>>>> <inc>1440</inc>
>>>>> </date_range_list>
>>>>> where the 1st and 2nd values are the start and end,
>>>>> range_length - the length of a range in minutes(6 houres)
>>>>> inc - lenght beetween ranges in minutes (24 houres)
>>>>> This example would create following ranges:
>>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>>>>>
>>>>> Also you can add more parameters to <plot_fix> to filter your
data.
>>>>> For example, if you want to select only dates with 00:00 you can
specify
>>>>> <field name="valid_hour">
>>>>> <val>00</val>
>>>>> </field>
>>>>>
>>>>> Tatiana
>>>>>
>>>>>
>>>>>
>>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
>>>>> > and if there isn't a better way, I wonder the whereabouts of a
>>>>> > separate
>>>>> > script that pipes dates (and other relevant information) into
the XML
>>>>> > file
>>>>> > used to generate the results.
>>>>> >
>>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
Affiliate <
>>>>> > edward.strobach at noaa.gov> wrote:
>>>>> >
>>>>> > > Thanks Tatiana. These resources are helpful.
>>>>> > >
>>>>> > > One last question. Simply typing lines like this in can
take
>>>>> > > forever:
>>>>> > >
>>>>> > > <plot_fix>
>>>>> > > <field equalize="false" name="fcst_init_beg">
>>>>> > > <set name="fcst_init_beg_0">
>>>>> > > <val>2019-05-01 12:00:00</val>
>>>>> > > <val>2019-05-02 06:00:00</val>
>>>>> > > <val>2019-05-02 12:00:00</val>
>>>>> > > <val>2019-05-03 06:00:00</val>
>>>>> > > <val>2019-05-03 12:00:00</val>
>>>>> > >
>>>>> > >
>>>>> > > Is there a better way? I see date_range, date_range_list,
or
>>>>> > > date_list.
>>>>> > > Can this be used instead of typing individual values? This
could
>>>>> > > prove
>>>>> > > pretty daunting if one would have to type in 700 entries
each time.
>>>>> > >
>>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>>>>> > > <met_help at ucar.edu>
>>>>> > > wrote:
>>>>> > >
>>>>> > >> I found a few problems in your XML.
>>>>> > >> The database you are using doesn't have data with vx_mask
>>>>> > >> 'abcREGabc'.
>>>>> > >> That is why METviewer returns an empty dataset and can't
create a
>>>>> > >> plot.
>>>>> > >> The other problem is how you define fcst_lead. This
database has
>>>>> > >> values
>>>>> > >> like 10000, 20000 and not 010000, 020000. Since you include
all
>>>>> > >> fcst_leads
>>>>> > >> this variable can be omitted.
>>>>> > >> To minimize problems with variables values, I would suggest
to
>>>>> > >> create XML
>>>>> > >> using GUI and then edit it to fit batch standards.
>>>>> > >> I created a plot using you configurations with GUI (without
>>>>> vx_mack
>>>>> > >> and
>>>>> > >> plot and axis titles) #20200612_173626. You can load it
back to
>>>>> the
>>>>> > >> page by
>>>>> > >> using the "History" tab (left bar on GUI). Find the plot,
open XML
>>>>> > >> and
>>>>> > >> click on "Load to page". After that you can recreate a plot
and
>>>>> edit
>>>>> > >> the
>>>>> > >> configurations. You can also download a new XML and edit it
for
>>>>> > >> batch.
>>>>> > >> I attached the example of load XML to this message.
>>>>> > >>
>>>>> > >> Description of rds parameters:
>>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>>>>> > >> rds_plots - temporary directory where the plots will be
generated
>>>>> > >> rds_data - temporary directory where the data retrieved
from the
>>>>> > >> database
>>>>> > >> and other intermediate files will be stored
>>>>> > >> rds_scripts - temporary directory where R scripts for this
run
>>>>> will
>>>>> > >> be
>>>>> > >> stored
>>>>> > >> After the execution is done all the files from temporary
>>>>> directories
>>>>> > >> would be removed.
>>>>> > >>
>>>>> > >>
>>>>> > >> We don't have the offitial user guide for METviewer. There
are
>>>>> some
>>>>> > >> instructions on how to crate XML config files:
>>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>>>> > >>
>>>>> > >> Tatiana
>>>>> > >>
>>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
>>>>> > >> > Hi,
>>>>> > >> >
>>>>> > >> > I'm attaching the xml below, but am having hard time
following
>>>>> how
>>>>> > >> > to
>>>>> > >> > go
>>>>> > >> > about the other steps since I have zero experience on how
to do
>>>>> > >> > these
>>>>> > >> > things.
>>>>> > >> >
>>>>> > >> > #2. All parameters in <folders> section should NOT be
changed
>>>>> and
>>>>> > >> > will
>>>>> > >> > be
>>>>> > >> > replaced by other scripts during the batch execution. The
data
>>>>> for
>>>>> > >> > the
>>>>> > >> > batch mode is coming from the database and not from
files. The
>>>>> > >> > name of
>>>>> > >> > the
>>>>> > >> > database is in <database> tag.
>>>>> > >> >
>>>>> > >> > I understand that nothing should be changed in the
folders
>>>>> > >> > section,
>>>>> > >> > but
>>>>> > >> > there is no indication how and where these variables are
being
>>>>> > >> > defined.
>>>>> > >> > Grepping these names of the files available does not
reveal
>>>>> formal
>>>>> > >> > definitions of these variables. If I understood exactly
how
>>>>> these
>>>>> > >> > were
>>>>> > >> > being defined, then I would be able connect this better.
>>>>> > >> >
>>>>> > >> > #3. Users rum MET and produce MET output files. They use
>>>>> METviewer
>>>>> > >> > load
>>>>> > >> > mode to load these files to the specific database. In the
config
>>>>> > >> > file
>>>>> > >> > for
>>>>> > >> > this process they also can specify the database group.
For
>>>>> example
>>>>> > >> > database
>>>>> > >> > mv_dev is in Test12 group.
>>>>> > >> > After the data has been loaded to the database it is
visible and
>>>>> > >> > available
>>>>> > >> > in the GUI and also can be used by the batch mode.
>>>>> > >> > You can always add more data to the database using the
same load
>>>>> > >> > mode.
>>>>> > >> >
>>>>> > >> > Is there a guide for me to understand the process that
you are
>>>>> > >> > referring to?
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>>>>> > >> > <met_help at ucar.edu>
>>>>> > >> > wrote:
>>>>> > >> >
>>>>> > >> > > Hi Edward,
>>>>> > >> > > #1. I don't have an access to NOAA servers and can't
view your
>>>>> > >> > > XML
>>>>> > >> > > for the
>>>>> > >> > > batch mode. Please, attache it to this issue and I will
take a
>>>>> > >> > > look.
>>>>> > >> > >
>>>>> > >> > > #2. All parameters in <folders> section should NOT be
changed
>>>>> > >> > > and
>>>>> > >> > > will be
>>>>> > >> > > replaced by other scripts during the batch execution.
The data
>>>>> > >> > > for
>>>>> > >> > > the
>>>>> > >> > > batch mode is coming from the database and not from
files. The
>>>>> > >> > > name
>>>>> > >> > > of the
>>>>> > >> > > database is in <database> tag.
>>>>> > >> > >
>>>>> > >> > > #3. Users rum MET and produce MET output files. They
use
>>>>> > >> > > METviewer
>>>>> > >> > > load
>>>>> > >> > > mode to load these files to the specific database. In
the
>>>>> config
>>>>> > >> > > file
>>>>> > >> > > for
>>>>> > >> > > this process they also can specify the database group.
For
>>>>> > >> > > example
>>>>> > >> > > database
>>>>> > >> > > mv_dev is in Test12 group.
>>>>> > >> > > After the data has been loaded to the database it is
visible
>>>>> and
>>>>> > >> > > available
>>>>> > >> > > in the GUI and also can be used by the batch mode.
>>>>> > >> > > You can always add more data to the database using the
same
>>>>> load
>>>>> > >> > > mode.
>>>>> > >> > >
>>>>> > >> > > Tatiana
>>>>> > >> > >
>>>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
>>>>> > >> > > > Good morning,
>>>>> > >> > > >
>>>>> > >> > > > Over the last two weeks I've made an effort to
understand
>>>>> > >> > > > metviewer,
>>>>> > >> > > > test
>>>>> > >> > > > different cases using the history tab, changing
information
>>>>> to
>>>>> > >> > > > see
>>>>> > >> > > > what
>>>>> > >> > > > works and what does not, and make connections between
the
>>>>> GUI
>>>>> > >> > > > settings
>>>>> > >> > > > and
>>>>> > >> > > > the XML file. Although my comfort level with
metviewer has
>>>>> > >> > > > significantly
>>>>> > >> > > > increased during this time, there are some remaining
issues
>>>>> > >> > > > that
>>>>> > >> > > > I'm
>>>>> > >> > > > hoping
>>>>> > >> > > > to clear up.
>>>>> > >> > > >
>>>>> > >> > > > 1. Problem changing XML file and running with
different
>>>>> > >> > > > statistic
>>>>> > >> > > > and
>>>>> > >> > > > plot
>>>>> > >> > > > type:
>>>>> > >> > > >
>>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
>>>>> > >> > > > directory
>>>>> > >> > > > to get
>>>>> > >> > > > started which I copied over:
>>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>>> > >> > > >
>>>>> > >> > > > He has a test.sh script that I adjusted and used to
run his
>>>>> > >> > > > example.
>>>>> > >> > > > running of the script "bash test.sh" produced a CSI
plot
>>>>> that
>>>>> > >> > > > was
>>>>> > >> > > > stored
>>>>> > >> > > > here:
>>>>> > >> > > >
>>>>> > >> > >
>>>>> > >>
>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>>> > >> > > >
>>>>> > >> > > > Running the script produced the expected result. The
next
>>>>> > >> > > > natural
>>>>> > >> > > > step was
>>>>> > >> > > > to copy hochum_xml.template and create a new xml file
called
>>>>> > >> > > > Strobach_xml.template in order to create a time
series where
>>>>> > >> > > > the
>>>>> > >> > > > independent time is fcst_valid_beg instead of
obs_thresh.
>>>>> > >> > > > Accordingly,
>>>>> > >> > > > I
>>>>> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg)
>>>>> and
>>>>> > >> > > > changed
>>>>> > >> > > > the
>>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).
The
>>>>> error
>>>>> > >> > > > that
>>>>> > >> > > > resulted
>>>>> > >> > > > is as follows:
>>>>> > >> > > >
>>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
13:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 01
>>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
16:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 01
>>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
19:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 01
>>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
22:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 01
>>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
01:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 02
>>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
04:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 02
>>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
07:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 02
>>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
10:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 02
>>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>>>> > >> > > > AND ld.fmean != -9999;
>>>>> > >> > > > No data was returned from database
>>>>> > >> > > > 2020-06-09
>>>>> > >> > > >
>>>>> > >> > >
>>>>> > >>
>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>>> > >> > > > Source) -1 printFormattedTable - ** ERROR: Caught
class
>>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>>>>> > >> > > > printFormattedTable(ResultSet
>>>>> > >> > > > res): result set contained no data
>>>>> > >> > > > ** ERROR: Caught class
>>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>>>> > >> > > > in
>>>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>>>> > >> > > > data
>>>>> > >> > > > ** ERROR: Caught class
>>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>>>> > >> > > > in
>>>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>>>> > >> > > > data
>>>>> > >> > > >
>>>>> > >> > > > Apparently there was no data during this time. I
tried
>>>>> > >> > > > changing
>>>>> > >> > > > the
>>>>> > >> > > > range
>>>>> > >> > > > of valid times and changed the valid time to
different days
>>>>> > >> > > > just in
>>>>> > >> > > > case
>>>>> > >> > > > data was missing. The issue was the same. To
confirm my
>>>>> > >> > > > logic, I
>>>>> > >> > > > reproduced my attempted approach on metviewer, which
I found
>>>>> > >> > > > to
>>>>> > >> > > > work.
>>>>> > >> > > >
>>>>> > >> > > > 2. Running in batch mode
>>>>> > >> > > >
>>>>> > >> > > > I think I lack an understanding of how to set up
batch mode
>>>>> > >> > > > because
>>>>> > >> > > > I
>>>>> > >> > > > don't
>>>>> > >> > > > see where these definitions are being explicitly
defined:
>>>>> > >> > > >
>>>>> > >> > > > <plot_spec>
>>>>> > >> > > > <connection>
>>>>> > >> > > > <host>rds_host:3306</host>
>>>>> > >> > > >
>>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>>> > >> > > > <user>rds_user</user>
>>>>> > >> > > > <password>rds_pwd</password>
>>>>> > >> > > > </connection>
>>>>> > >> > > > <rscript>Rscript</rscript>
>>>>> > >> > > > <folders>
>>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>>> > >> > > > <plots>rds_plots</plots>
>>>>> > >> > > > <data>rds_data</data>
>>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>>> > >> > > > </folders>
>>>>> > >> > > > <plot>
>>>>> > >> > > > <template>series_plot.R_tmpl</template>
>>>>> > >> > > >
>>>>> > >> > > > I see the <folders> section, but grepping all
available
>>>>> files
>>>>> > >> > > > does
>>>>> > >> > > > not
>>>>> > >> > > > reveal how rds_R_work is being defined, as well as
other
>>>>> > >> > > > similarly
>>>>> > >> > > > defined
>>>>> > >> > > > variables. I think if I know how these are being
defined,
>>>>> > >> > > > then
>>>>> > >> > > > I'll
>>>>> > >> > > > be
>>>>> > >> > > > able to pipe in my information and process the data
using
>>>>> met
>>>>> > >> > > > and
>>>>> > >> > > > produce
>>>>> > >> > > > results in batch mode. Am I right on that? Also, if
I want
>>>>> > >> > > > to use
>>>>> > >> > > > obs for
>>>>> > >> > > > verification, do I add another line in the XML file
that
>>>>> > >> > > > designates
>>>>> > >> > > > file
>>>>> > >> > > > location of obs? I assume that other steps have to
be run
>>>>> > >> > > > before
>>>>> > >> > > > all
>>>>> > >> > > > this. If data is in prepbufr form, for example, I
would
>>>>> have
>>>>> > >> > > > to
>>>>> > >> > > > separately
>>>>> > >> > > > run PB2NC. If I want grid2obs, then I would want to
do that
>>>>> > >> > > > as
>>>>> > >> > > > well.
>>>>> > >> > > >
>>>>> > >> > > > 3. Communicating results to metviewer online
>>>>> > >> > > >
>>>>> > >> > > > Lastly, I see a lot of people/projects listed here -
see
>>>>> > >> > > > attachment
>>>>> > >> > > >
>>>>> > >> > > > How do I get the results I generate to be uploaded to
this
>>>>> > >> > > > location?
>>>>> > >> > > > It's
>>>>> > >> > > > not clear how others are setting this up.
>>>>> > >> > >
>>>>> > >> > >
>>>>> > >> > >
>>>>> > >> > >
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
>>>>> rds_pwd
>>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
>>>>> rds_scripts
>>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
>>>>> 2020-05-12
>>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
>>>>> 20:00:00
>>>>> > >> 2020-05-12
>>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
>>>>> 10:00:00
>>>>> > >> 2020-05-13
>>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
>>>>> 14:00:00
>>>>> > >> 2020-05-13
>>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
>>>>> 18:00:00
>>>>> > >> 2020-05-13
>>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
>>>>> 22:00:00
>>>>> > >> 2020-05-13
>>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
>>>>> 12:00:00
>>>>> > >> true
>>>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
>>>>> > >> plot_20200612_173626.R test x_label test y_label
>>>>> > >> false list() list() false false false false false false
true false
>>>>> > >> false
>>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
c(1,1,0) 1 2
>>>>> > >> 1.4 -2
>>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
#cccccc 1
>>>>> > >> listX 1
>>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3
0.8 1
>>>>> > >> #333333
>>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
>>>>> c(1,1)
>>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > Edward Strobach
>>>>> > > EMC/NCEP/NWS/
>>>>> > > IMSG Contractor
>>>>> > > Cubicle#: 2029
>>>>> > > 301-683-3717
>>>>> > >
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Edward Strobach
>>>> EMC/NCEP/NWS/
>>>> IMSG Contractor
>>>> Cubicle#: 2029
>>>> 301-683-3717
>>>>
>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Tue Jun 16 14:32:04 2020
You are correct. You need to run the data loading first and then the
plotting.
You do need to have two separate XML for each stage. XML for the
loading and plotting look similar but they describe different
information and can not be reused.
Loading XML describes the location and the directory structure of the
data. The XML doesn't describes the data itself. Plotting XML
describes the data needed for the particular plot and the potting
specifics.
If you create a new database an load data to it you should be able to
view that data and use it for potting.
Tatiana
On Tue Jun 16 07:35:03 2020, edward.strobach at noaa.gov wrote:
> Actually, it appears that I need to run the database loading module
> first
> if I want to change/add a data source. From there I can run the
> plotting
> batch module. Is that right? So if I run the database loading
> module,
> then I can specify my dates and other related info specific to the
> dataset. If I create my database with a unique name, then it should
> be
> recognized on the metviewer gui, presumably.
>
> On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > It seems that those options must be treated separately outside of
> > plot_spec. Is it acceptable to create both a load_spec and
plot_spec
> > in
> > the same xml file? Would the definitions in load_spec be able to
> > carry
> > over into plot_spec? It appears that data loading and plotting
are
> > treated
> > separately.
> >
> > On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> >> I do plan to use date_range; however, in an example provided by
the
> >> link
> >> I see this:
> >>
> >> <load_spec>
> >> <connection>
> >> <host>kemosabe:3306</host>
> >> <database>metvdb_hwt</database>
> >> <user>pgoldenb</user>
> >> <password>pgoldenb</password>
> >> </connection>
> >>
> >> <date_list name="folder_dates">
> >> <start>2010051914V</start>
> >> <end><date_offset><day_offset>0</day_offset><hour>6
> >> </hour></date_offset></end>
> >> <inc>3600</inc>
> >> <format>yyyyMMddHH'V'</format>
> >> </date_list>
> >>
> >> <met_version>V3.0</met_version>
> >>
> >> <verbose>false</verbose>
> >> <insert_size>1</insert_size>
> >> <mode_header_db_check>true</mode_header_db_check>
> >> <drop_indexes>false</drop_indexes>
> >> <apply_indexes>true</apply_indexes>
> >> <group>Group name</group>
> >> <load_stat>true</load_stat>
> >> <load_mode>true</load_mode>
> >> <load_mtd>true</load_mtd>
> >> <load_mpr>false</load_mpr>
> >>
> >>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> >> <load_val>
> >> <field name="model">
> >> <val>arw</val>
> >> <val>nmm</val>
> >> </field>
> >>
> >> <field name="valid_time">
> >> <date_list name="folder_dates"/>
> >> </field>
> >>
> >> <field name="vx_mask">
> >> <val>FULL</val>
> >> <val>SWC</val>
> >> </field>
> >> </load_val>
> >> </load_spec>
> >>
> >> Are similar settings allowed with <plot_spec>? I see
<connection>,
> >> which
> >> seems carried over. I'm looking to specify my folder_tmpl in
> >> <plot_spec>,
> >> which according to Tara, should be possible. Since my directory
> >> structure
> >> looks like this:
> >>
> >>
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> >>
> >> I would do something like this I would think:
> >>
> >> .......
> >>
> >> <date_list name="folder_dates">
> >> <start>20200612/start>
> >> <end><date_offset><day_offset>0</day_offset></end>
> >> <inc>86400</inc>
> >> <format>yyyyMMdd</format>
> >> </date_list>
> >>
> >> <folder_tmpl>
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> >> </folder_tmpl>
> >> <load_val>
> >> <field name="model">
> >> <val>prod</val>
> >> </field>
> >>
> >> <field name="valid_time">
> >> <date_list name="folder_dates"/>
> >> </field>
> >>
> >> ......
> >>
> >> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate
<
> >> edward.strobach at noaa.gov> wrote:
> >>
> >>> Thank you. This is a very helpful example
> >>>
> >>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> >>> <met_help at ucar.edu>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>> Here is some examples how to use <date_range> and
> >>>> <date_range_list> in
> >>>> batch XML.
> >>>> First, you need to defined your range. You can have more then
one
> >>>> <date_range>:
> >>>> <date_range name="may">
> >>>> <start>2019-05-01 00:00:00</start>
> >>>> <end>2019-05-31 23:59:59</end>
> >>>> </date_range>
> >>>> <date_range name="march">
> >>>> <start>2019-03-01 00:00:00</start>
> >>>> <end>2019-03-31 23:59:59</end>
> >>>> </date_range>
> >>>>
> >>>> and then use one or all date_ranges:
> >>>> <plot_fix>
> >>>> <field name="fcst_init_beg">
> >>>> <date_range name="march"/>
> >>>> <date_range name="may"/>
> >>>> </field>...
> >>>> This configuration would include all dates for March and May of
> >>>> 2019.
> >>>>
> >>>> If you need more flexible configuration you can use
> >>>> date_range_list.
> >>>> For example, you want to select first 6 hours every day
beetween
> >>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would
look
> >>>> like this:
> >>>> <date_range_list name = 'all'>
> >>>> <range_start>2020-01-01 00:00:00</range_start>
> >>>> <range_end>2020-01-05 00:00:00</range_end>
> >>>> <range_length>360</range_length>
> >>>> <inc>1440</inc>
> >>>> </date_range_list>
> >>>> where the 1st and 2nd values are the start and end,
> >>>> range_length - the length of a range in minutes(6 houres)
> >>>> inc - lenght beetween ranges in minutes (24 houres)
> >>>> This example would create following ranges:
> >>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> >>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> >>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> >>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> >>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> >>>>
> >>>> Also you can add more parameters to <plot_fix> to filter your
> >>>> data. For
> >>>> example, if you want to select only dates with 00:00 you can
> >>>> specify
> >>>> <field name="valid_hour">
> >>>> <val>00</val>
> >>>> </field>
> >>>>
> >>>> Tatiana
> >>>>
> >>>>
> >>>>
> >>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> >>>> > and if there isn't a better way, I wonder the whereabouts of
a
> >>>> > separate
> >>>> > script that pipes dates (and other relevant information) into
> >>>> > the XML
> >>>> > file
> >>>> > used to generate the results.
> >>>> >
> >>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> >>>> > Affiliate <
> >>>> > edward.strobach at noaa.gov> wrote:
> >>>> >
> >>>> > > Thanks Tatiana. These resources are helpful.
> >>>> > >
> >>>> > > One last question. Simply typing lines like this in can
take
> >>>> > > forever:
> >>>> > >
> >>>> > > <plot_fix>
> >>>> > > <field equalize="false" name="fcst_init_beg">
> >>>> > > <set name="fcst_init_beg_0">
> >>>> > > <val>2019-05-01 12:00:00</val>
> >>>> > > <val>2019-05-02 06:00:00</val>
> >>>> > > <val>2019-05-02 12:00:00</val>
> >>>> > > <val>2019-05-03 06:00:00</val>
> >>>> > > <val>2019-05-03 12:00:00</val>
> >>>> > >
> >>>> > >
> >>>> > > Is there a better way? I see date_range, date_range_list,
or
> >>>> > > date_list.
> >>>> > > Can this be used instead of typing individual values? This
> >>>> > > could
> >>>> > > prove
> >>>> > > pretty daunting if one would have to type in 700 entries
each
> >>>> > > time.
> >>>> > >
> >>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> >>>> > > <met_help at ucar.edu>
> >>>> > > wrote:
> >>>> > >
> >>>> > >> I found a few problems in your XML.
> >>>> > >> The database you are using doesn't have data with vx_mask
> >>>> > >> 'abcREGabc'.
> >>>> > >> That is why METviewer returns an empty dataset and can't
> >>>> > >> create a
> >>>> > >> plot.
> >>>> > >> The other problem is how you define fcst_lead. This
database
> >>>> > >> has
> >>>> > >> values
> >>>> > >> like 10000, 20000 and not 010000, 020000. Since you
include
> >>>> > >> all
> >>>> > >> fcst_leads
> >>>> > >> this variable can be omitted.
> >>>> > >> To minimize problems with variables values, I would
suggest
> >>>> > >> to
> >>>> > >> create XML
> >>>> > >> using GUI and then edit it to fit batch standards.
> >>>> > >> I created a plot using you configurations with GUI
(without
> >>>> > >> vx_mack
> >>>> > >> and
> >>>> > >> plot and axis titles) #20200612_173626. You can load it
back
> >>>> > >> to the
> >>>> > >> page by
> >>>> > >> using the "History" tab (left bar on GUI). Find the plot,
> >>>> > >> open XML
> >>>> > >> and
> >>>> > >> click on "Load to page". After that you can recreate a
plot
> >>>> > >> and
> >>>> edit
> >>>> > >> the
> >>>> > >> configurations. You can also download a new XML and edit
it
> >>>> > >> for
> >>>> > >> batch.
> >>>> > >> I attached the example of load XML to this message.
> >>>> > >>
> >>>> > >> Description of rds parameters:
> >>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
scripts
> >>>> > >> rds_plots - temporary directory where the plots will be
> >>>> > >> generated
> >>>> > >> rds_data - temporary directory where the data retrieved
from
> >>>> > >> the
> >>>> > >> database
> >>>> > >> and other intermediate files will be stored
> >>>> > >> rds_scripts - temporary directory where R scripts for this
> >>>> > >> run will
> >>>> > >> be
> >>>> > >> stored
> >>>> > >> After the execution is done all the files from temporary
> >>>> directories
> >>>> > >> would be removed.
> >>>> > >>
> >>>> > >>
> >>>> > >> We don't have the offitial user guide for METviewer. There
> >>>> > >> are some
> >>>> > >> instructions on how to crate XML config files:
> >>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> >>>> > >>
> >>>> > >> Tatiana
> >>>> > >>
> >>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
> >>>> > >> > Hi,
> >>>> > >> >
> >>>> > >> > I'm attaching the xml below, but am having hard time
> >>>> > >> > following
> >>>> how
> >>>> > >> > to
> >>>> > >> > go
> >>>> > >> > about the other steps since I have zero experience on
how
> >>>> > >> > to do
> >>>> > >> > these
> >>>> > >> > things.
> >>>> > >> >
> >>>> > >> > #2. All parameters in <folders> section should NOT be
> >>>> > >> > changed and
> >>>> > >> > will
> >>>> > >> > be
> >>>> > >> > replaced by other scripts during the batch execution.
The
> >>>> > >> > data
> >>>> for
> >>>> > >> > the
> >>>> > >> > batch mode is coming from the database and not from
files.
> >>>> > >> > The
> >>>> > >> > name of
> >>>> > >> > the
> >>>> > >> > database is in <database> tag.
> >>>> > >> >
> >>>> > >> > I understand that nothing should be changed in the
folders
> >>>> > >> > section,
> >>>> > >> > but
> >>>> > >> > there is no indication how and where these variables are
> >>>> > >> > being
> >>>> > >> > defined.
> >>>> > >> > Grepping these names of the files available does not
reveal
> >>>> formal
> >>>> > >> > definitions of these variables. If I understood exactly
> >>>> > >> > how
> >>>> these
> >>>> > >> > were
> >>>> > >> > being defined, then I would be able connect this better.
> >>>> > >> >
> >>>> > >> > #3. Users rum MET and produce MET output files. They use
> >>>> METviewer
> >>>> > >> > load
> >>>> > >> > mode to load these files to the specific database. In
the
> >>>> > >> > config
> >>>> > >> > file
> >>>> > >> > for
> >>>> > >> > this process they also can specify the database group.
For
> >>>> example
> >>>> > >> > database
> >>>> > >> > mv_dev is in Test12 group.
> >>>> > >> > After the data has been loaded to the database it is
> >>>> > >> > visible and
> >>>> > >> > available
> >>>> > >> > in the GUI and also can be used by the batch mode.
> >>>> > >> > You can always add more data to the database using the
same
> >>>> > >> > load
> >>>> > >> > mode.
> >>>> > >> >
> >>>> > >> > Is there a guide for me to understand the process that
you
> >>>> > >> > are
> >>>> > >> > referring to?
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> >>>> > >> > <met_help at ucar.edu>
> >>>> > >> > wrote:
> >>>> > >> >
> >>>> > >> > > Hi Edward,
> >>>> > >> > > #1. I don't have an access to NOAA servers and can't
view
> >>>> > >> > > your
> >>>> > >> > > XML
> >>>> > >> > > for the
> >>>> > >> > > batch mode. Please, attache it to this issue and I
will
> >>>> > >> > > take a
> >>>> > >> > > look.
> >>>> > >> > >
> >>>> > >> > > #2. All parameters in <folders> section should NOT be
> >>>> > >> > > changed
> >>>> > >> > > and
> >>>> > >> > > will be
> >>>> > >> > > replaced by other scripts during the batch execution.
The
> >>>> > >> > > data
> >>>> > >> > > for
> >>>> > >> > > the
> >>>> > >> > > batch mode is coming from the database and not from
> >>>> > >> > > files. The
> >>>> > >> > > name
> >>>> > >> > > of the
> >>>> > >> > > database is in <database> tag.
> >>>> > >> > >
> >>>> > >> > > #3. Users rum MET and produce MET output files. They
use
> >>>> > >> > > METviewer
> >>>> > >> > > load
> >>>> > >> > > mode to load these files to the specific database. In
the
> >>>> config
> >>>> > >> > > file
> >>>> > >> > > for
> >>>> > >> > > this process they also can specify the database group.
> >>>> > >> > > For
> >>>> > >> > > example
> >>>> > >> > > database
> >>>> > >> > > mv_dev is in Test12 group.
> >>>> > >> > > After the data has been loaded to the database it is
> >>>> > >> > > visible
> >>>> and
> >>>> > >> > > available
> >>>> > >> > > in the GUI and also can be used by the batch mode.
> >>>> > >> > > You can always add more data to the database using the
> >>>> > >> > > same
> >>>> load
> >>>> > >> > > mode.
> >>>> > >> > >
> >>>> > >> > > Tatiana
> >>>> > >> > >
> >>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
> >>>> > >> > > wrote:
> >>>> > >> > > > Good morning,
> >>>> > >> > > >
> >>>> > >> > > > Over the last two weeks I've made an effort to
> >>>> > >> > > > understand
> >>>> > >> > > > metviewer,
> >>>> > >> > > > test
> >>>> > >> > > > different cases using the history tab, changing
> >>>> > >> > > > information
> >>>> to
> >>>> > >> > > > see
> >>>> > >> > > > what
> >>>> > >> > > > works and what does not, and make connections
between
> >>>> > >> > > > the GUI
> >>>> > >> > > > settings
> >>>> > >> > > > and
> >>>> > >> > > > the XML file. Although my comfort level with
metviewer
> >>>> > >> > > > has
> >>>> > >> > > > significantly
> >>>> > >> > > > increased during this time, there are some remaining
> >>>> > >> > > > issues
> >>>> > >> > > > that
> >>>> > >> > > > I'm
> >>>> > >> > > > hoping
> >>>> > >> > > > to clear up.
> >>>> > >> > > >
> >>>> > >> > > > 1. Problem changing XML file and running with
> >>>> > >> > > > different
> >>>> > >> > > > statistic
> >>>> > >> > > > and
> >>>> > >> > > > plot
> >>>> > >> > > > type:
> >>>> > >> > > >
> >>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
> >>>> > >> > > > working
> >>>> > >> > > > directory
> >>>> > >> > > > to get
> >>>> > >> > > > started which I copied over:
> >>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >>>> > >> > > >
> >>>> > >> > > > He has a test.sh script that I adjusted and used to
run
> >>>> > >> > > > his
> >>>> > >> > > > example.
> >>>> > >> > > > running of the script "bash test.sh" produced a CSI
> >>>> > >> > > > plot that
> >>>> > >> > > > was
> >>>> > >> > > > stored
> >>>> > >> > > > here:
> >>>> > >> > > >
> >>>> > >> > >
> >>>> > >>
> >>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >>>> > >> > > >
> >>>> > >> > > > Running the script produced the expected result.
The
> >>>> > >> > > > next
> >>>> > >> > > > natural
> >>>> > >> > > > step was
> >>>> > >> > > > to copy hochum_xml.template and create a new xml
file
> >>>> > >> > > > called
> >>>> > >> > > > Strobach_xml.template in order to create a time
series
> >>>> > >> > > > where
> >>>> > >> > > > the
> >>>> > >> > > > independent time is fcst_valid_beg instead of
> >>>> > >> > > > obs_thresh.
> >>>> > >> > > > Accordingly,
> >>>> > >> > > > I
> >>>> > >> > > > made changes to the fixed fields (removing
> >>>> > >> > > > fcst_valid_beg)
> >>>> and
> >>>> > >> > > > changed
> >>>> > >> > > > the
> >>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).
The
> >>>> > >> > > > error
> >>>> > >> > > > that
> >>>> > >> > > > resulted
> >>>> > >> > > > is as follows:
> >>>> > >> > > >
> >>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
> >>>> > >> > > > 13:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 01
> >>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> >>>> > >> > > > 16:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 01
> >>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> >>>> > >> > > > 19:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 01
> >>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> >>>> > >> > > > 22:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 01
> >>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> >>>> > >> > > > 01:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 02
> >>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> >>>> > >> > > > 04:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 02
> >>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> >>>> > >> > > > 07:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 02
> >>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> >>>> > >> > > > 10:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 02
> >>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> >>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> >>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> >>>> > >> > > > AND ld.fmean != -9999;
> >>>> > >> > > > No data was returned from database
> >>>> > >> > > > 2020-06-09
> >>>> > >> > > >
> >>>> > >> > >
> >>>> > >>
> >>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >>>> > >> > > > Source) -1 printFormattedTable - ** ERROR: Caught
> >>>> > >> > > > class
> >>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> >>>> > >> > > > printFormattedTable(ResultSet
> >>>> > >> > > > res): result set contained no data
> >>>> > >> > > > ** ERROR: Caught class
> >>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >>>> > >> > > > in
> >>>> > >> > > > printFormattedTable(ResultSet res): result set
> >>>> > >> > > > contained no
> >>>> > >> > > > data
> >>>> > >> > > > ** ERROR: Caught class
> >>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >>>> > >> > > > in
> >>>> > >> > > > printFormattedTable(ResultSet res): result set
> >>>> > >> > > > contained no
> >>>> > >> > > > data
> >>>> > >> > > >
> >>>> > >> > > > Apparently there was no data during this time. I
tried
> >>>> > >> > > > changing
> >>>> > >> > > > the
> >>>> > >> > > > range
> >>>> > >> > > > of valid times and changed the valid time to
different
> >>>> > >> > > > days
> >>>> > >> > > > just in
> >>>> > >> > > > case
> >>>> > >> > > > data was missing. The issue was the same. To
confirm
> >>>> > >> > > > my
> >>>> > >> > > > logic, I
> >>>> > >> > > > reproduced my attempted approach on metviewer, which
I
> >>>> > >> > > > found
> >>>> > >> > > > to
> >>>> > >> > > > work.
> >>>> > >> > > >
> >>>> > >> > > > 2. Running in batch mode
> >>>> > >> > > >
> >>>> > >> > > > I think I lack an understanding of how to set up
batch
> >>>> > >> > > > mode
> >>>> > >> > > > because
> >>>> > >> > > > I
> >>>> > >> > > > don't
> >>>> > >> > > > see where these definitions are being explicitly
> >>>> > >> > > > defined:
> >>>> > >> > > >
> >>>> > >> > > > <plot_spec>
> >>>> > >> > > > <connection>
> >>>> > >> > > > <host>rds_host:3306</host>
> >>>> > >> > > >
> >>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >>>> > >> > > > <user>rds_user</user>
> >>>> > >> > > > <password>rds_pwd</password>
> >>>> > >> > > > </connection>
> >>>> > >> > > > <rscript>Rscript</rscript>
> >>>> > >> > > > <folders>
> >>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >>>> > >> > > > <r_work>rds_R_work</r_work>
> >>>> > >> > > > <plots>rds_plots</plots>
> >>>> > >> > > > <data>rds_data</data>
> >>>> > >> > > > <scripts>rds_scripts</scripts>
> >>>> > >> > > > </folders>
> >>>> > >> > > > <plot>
> >>>> > >> > > > <template>series_plot.R_tmpl</template>
> >>>> > >> > > >
> >>>> > >> > > > I see the <folders> section, but grepping all
available
> >>>> > >> > > > files
> >>>> > >> > > > does
> >>>> > >> > > > not
> >>>> > >> > > > reveal how rds_R_work is being defined, as well as
> >>>> > >> > > > other
> >>>> > >> > > > similarly
> >>>> > >> > > > defined
> >>>> > >> > > > variables. I think if I know how these are being
> >>>> > >> > > > defined,
> >>>> > >> > > > then
> >>>> > >> > > > I'll
> >>>> > >> > > > be
> >>>> > >> > > > able to pipe in my information and process the data
> >>>> > >> > > > using met
> >>>> > >> > > > and
> >>>> > >> > > > produce
> >>>> > >> > > > results in batch mode. Am I right on that? Also,
if I
> >>>> > >> > > > want
> >>>> > >> > > > to use
> >>>> > >> > > > obs for
> >>>> > >> > > > verification, do I add another line in the XML file
> >>>> > >> > > > that
> >>>> > >> > > > designates
> >>>> > >> > > > file
> >>>> > >> > > > location of obs? I assume that other steps have to
be
> >>>> > >> > > > run
> >>>> > >> > > > before
> >>>> > >> > > > all
> >>>> > >> > > > this. If data is in prepbufr form, for example, I
> >>>> > >> > > > would have
> >>>> > >> > > > to
> >>>> > >> > > > separately
> >>>> > >> > > > run PB2NC. If I want grid2obs, then I would want to
do
> >>>> > >> > > > that
> >>>> > >> > > > as
> >>>> > >> > > > well.
> >>>> > >> > > >
> >>>> > >> > > > 3. Communicating results to metviewer online
> >>>> > >> > > >
> >>>> > >> > > > Lastly, I see a lot of people/projects listed here -
> >>>> > >> > > > see
> >>>> > >> > > > attachment
> >>>> > >> > > >
> >>>> > >> > > > How do I get the results I generate to be uploaded
to
> >>>> > >> > > > this
> >>>> > >> > > > location?
> >>>> > >> > > > It's
> >>>> > >> > > > not clear how others are setting this up.
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
rds_user
> >>>> > >> rds_pwd
> >>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
> >>>> > >> rds_scripts
> >>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
> >>>> > >> 2020-05-12
> >>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-
12
> >>>> 20:00:00
> >>>> > >> 2020-05-12
> >>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-
13
> >>>> 10:00:00
> >>>> > >> 2020-05-13
> >>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-
13
> >>>> 14:00:00
> >>>> > >> 2020-05-13
> >>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-
13
> >>>> 18:00:00
> >>>> > >> 2020-05-13
> >>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-
13
> >>>> 22:00:00
> >>>> > >> 2020-05-13
> >>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-
14
> >>>> 12:00:00
> >>>> > >> true
> >>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
> >>>> > >> plot_20200612_173626.R test x_label test y_label
> >>>> > >> false list() list() false false false false false false
true
> >>>> > >> false
> >>>> > >> false
> >>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
> >>>> > >> c(1,1,0) 1 2
> >>>> > >> 1.4 -2
> >>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
> >>>> > >> #cccccc 1
> >>>> > >> listX 1
> >>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3
0.8
> >>>> > >> 1
> >>>> > >> #333333
> >>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> >>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
> >>>> > >> c(1,1)
> >>>> > >> c(1,2) c("","") c() c() 0.04 c()
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > --
> >>>> > > Edward Strobach
> >>>> > > EMC/NCEP/NWS/
> >>>> > > IMSG Contractor
> >>>> > > Cubicle#: 2029
> >>>> > > 301-683-3717
> >>>> > >
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> Edward Strobach
> >>> EMC/NCEP/NWS/
> >>> IMSG Contractor
> >>> Cubicle#: 2029
> >>> 301-683-3717
> >>>
> >>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Tue Jun 16 14:45:34 2020
We don't have database and METviewer running on wcoss.
Instead, users use METviewer-wcoss scripts that help to connect to and
use METviewer that is running on AWS. That is why XML connection and
folder sections look somewhat different than once from the Docs page.
To create a database on AWS you need to run mv_create_db_on_aws.sh
mv_create_db_on_aws.sh <user_name> <database_name>
For data loading :
mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
And for batch:
mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
These scripts copy XML and data files if needed to AWS, adjust data
path and users credentials, run METviewer and copy results back to
wcoss.
Tatiana
On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> lastly, it does seem thatmysql -u[db_username] -p[db_password]
> -e'create
> database [db_name]
>
> would need to be used to create a database. I assume that an sql
> module is
> available on wcoss; I know you don't have access to those machines
> Tatiana. If I create my own database with my own name, then I
should
> be
> able to proceed with project specific tasks under that tab.
Something
> like
> mv_EdwardStrobach.
>
> I can then create mv_cmaq_prod_v_para which I add data to using
> load_spec.
> Once that step is done, then I can run plot_spec. That's my
takeaway
> currently
>
> On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Actually, it appears that I need to run the database loading
module
> > first
> > if I want to change/add a data source. From there I can run the
> > plotting
> > batch module. Is that right? So if I run the database loading
> > module,
> > then I can specify my dates and other related info specific to the
> > dataset. If I create my database with a unique name, then it
should
> > be
> > recognized on the metviewer gui, presumably.
> >
> > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> >> It seems that those options must be treated separately outside of
> >> plot_spec. Is it acceptable to create both a load_spec and
> >> plot_spec in
> >> the same xml file? Would the definitions in load_spec be able to
> >> carry
> >> over into plot_spec? It appears that data loading and plotting
are
> >> treated
> >> separately.
> >>
> >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate
<
> >> edward.strobach at noaa.gov> wrote:
> >>
> >>> I do plan to use date_range; however, in an example provided by
the
> >>> link
> >>> I see this:
> >>>
> >>> <load_spec>
> >>> <connection>
> >>> <host>kemosabe:3306</host>
> >>> <database>metvdb_hwt</database>
> >>> <user>pgoldenb</user>
> >>> <password>pgoldenb</password>
> >>> </connection>
> >>>
> >>> <date_list name="folder_dates">
> >>> <start>2010051914V</start>
> >>> <end><date_offset><day_offset>0</day_offset><hour>6
> >>> </hour></date_offset></end>
> >>> <inc>3600</inc>
> >>> <format>yyyyMMddHH'V'</format>
> >>> </date_list>
> >>>
> >>> <met_version>V3.0</met_version>
> >>>
> >>> <verbose>false</verbose>
> >>> <insert_size>1</insert_size>
> >>> <mode_header_db_check>true</mode_header_db_check>
> >>> <drop_indexes>false</drop_indexes>
> >>> <apply_indexes>true</apply_indexes>
> >>> <group>Group name</group>
> >>> <load_stat>true</load_stat>
> >>> <load_mode>true</load_mode>
> >>> <load_mtd>true</load_mtd>
> >>> <load_mpr>false</load_mpr>
> >>>
> >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> >>> <load_val>
> >>> <field name="model">
> >>> <val>arw</val>
> >>> <val>nmm</val>
> >>> </field>
> >>>
> >>> <field name="valid_time">
> >>> <date_list name="folder_dates"/>
> >>> </field>
> >>>
> >>> <field name="vx_mask">
> >>> <val>FULL</val>
> >>> <val>SWC</val>
> >>> </field>
> >>> </load_val>
> >>> </load_spec>
> >>>
> >>> Are similar settings allowed with <plot_spec>? I see
<connection>,
> >>> which seems carried over. I'm looking to specify my folder_tmpl
in
> >>> <plot_spec>, which according to Tara, should be possible. Since
my
> >>> directory structure looks like this:
> >>>
> >>>
> >>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> >>>
> >>> I would do something like this I would think:
> >>>
> >>> .......
> >>>
> >>> <date_list name="folder_dates">
> >>> <start>20200612/start>
> >>> <end><date_offset><day_offset>0</day_offset></end>
> >>> <inc>86400</inc>
> >>> <format>yyyyMMdd</format>
> >>> </date_list>
> >>>
> >>> <folder_tmpl>
> >>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> >>> </folder_tmpl>
> >>> <load_val>
> >>> <field name="model">
> >>> <val>prod</val>
> >>> </field>
> >>>
> >>> <field name="valid_time">
> >>> <date_list name="folder_dates"/>
> >>> </field>
> >>>
> >>> ......
> >>>
> >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate
<
> >>> edward.strobach at noaa.gov> wrote:
> >>>
> >>>> Thank you. This is a very helpful example
> >>>>
> >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> >>>> <met_help at ucar.edu>
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>> Here is some examples how to use <date_range> and
> >>>>> <date_range_list> in
> >>>>> batch XML.
> >>>>> First, you need to defined your range. You can have more then
one
> >>>>> <date_range>:
> >>>>> <date_range name="may">
> >>>>> <start>2019-05-01 00:00:00</start>
> >>>>> <end>2019-05-31 23:59:59</end>
> >>>>> </date_range>
> >>>>> <date_range name="march">
> >>>>> <start>2019-03-01 00:00:00</start>
> >>>>> <end>2019-03-31 23:59:59</end>
> >>>>> </date_range>
> >>>>>
> >>>>> and then use one or all date_ranges:
> >>>>> <plot_fix>
> >>>>> <field name="fcst_init_beg">
> >>>>> <date_range name="march"/>
> >>>>> <date_range name="may"/>
> >>>>> </field>...
> >>>>> This configuration would include all dates for March and May
of
> >>>>> 2019.
> >>>>>
> >>>>> If you need more flexible configuration you can use
> >>>>> date_range_list.
> >>>>> For example, you want to select first 6 hours every day
beetween
> >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would
look
> >>>>> like this:
> >>>>> <date_range_list name = 'all'>
> >>>>> <range_start>2020-01-01 00:00:00</range_start>
> >>>>> <range_end>2020-01-05 00:00:00</range_end>
> >>>>> <range_length>360</range_length>
> >>>>> <inc>1440</inc>
> >>>>> </date_range_list>
> >>>>> where the 1st and 2nd values are the start and end,
> >>>>> range_length - the length of a range in minutes(6 houres)
> >>>>> inc - lenght beetween ranges in minutes (24 houres)
> >>>>> This example would create following ranges:
> >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> >>>>>
> >>>>> Also you can add more parameters to <plot_fix> to filter your
> >>>>> data.
> >>>>> For example, if you want to select only dates with 00:00 you
can
> >>>>> specify
> >>>>> <field name="valid_hour">
> >>>>> <val>00</val>
> >>>>> </field>
> >>>>>
> >>>>> Tatiana
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> >>>>> > and if there isn't a better way, I wonder the whereabouts of
a
> >>>>> > separate
> >>>>> > script that pipes dates (and other relevant information)
into
> >>>>> > the XML
> >>>>> > file
> >>>>> > used to generate the results.
> >>>>> >
> >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> >>>>> > Affiliate <
> >>>>> > edward.strobach at noaa.gov> wrote:
> >>>>> >
> >>>>> > > Thanks Tatiana. These resources are helpful.
> >>>>> > >
> >>>>> > > One last question. Simply typing lines like this in can
take
> >>>>> > > forever:
> >>>>> > >
> >>>>> > > <plot_fix>
> >>>>> > > <field equalize="false" name="fcst_init_beg">
> >>>>> > > <set name="fcst_init_beg_0">
> >>>>> > > <val>2019-05-01 12:00:00</val>
> >>>>> > > <val>2019-05-02 06:00:00</val>
> >>>>> > > <val>2019-05-02 12:00:00</val>
> >>>>> > > <val>2019-05-03 06:00:00</val>
> >>>>> > > <val>2019-05-03 12:00:00</val>
> >>>>> > >
> >>>>> > >
> >>>>> > > Is there a better way? I see date_range,
date_range_list,
> >>>>> > > or
> >>>>> > > date_list.
> >>>>> > > Can this be used instead of typing individual values?
This
> >>>>> > > could
> >>>>> > > prove
> >>>>> > > pretty daunting if one would have to type in 700 entries
each
> >>>>> > > time.
> >>>>> > >
> >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> >>>>> > > <met_help at ucar.edu>
> >>>>> > > wrote:
> >>>>> > >
> >>>>> > >> I found a few problems in your XML.
> >>>>> > >> The database you are using doesn't have data with vx_mask
> >>>>> > >> 'abcREGabc'.
> >>>>> > >> That is why METviewer returns an empty dataset and can't
> >>>>> > >> create a
> >>>>> > >> plot.
> >>>>> > >> The other problem is how you define fcst_lead. This
database
> >>>>> > >> has
> >>>>> > >> values
> >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
include
> >>>>> > >> all
> >>>>> > >> fcst_leads
> >>>>> > >> this variable can be omitted.
> >>>>> > >> To minimize problems with variables values, I would
suggest
> >>>>> > >> to
> >>>>> > >> create XML
> >>>>> > >> using GUI and then edit it to fit batch standards.
> >>>>> > >> I created a plot using you configurations with GUI
(without
> >>>>> vx_mack
> >>>>> > >> and
> >>>>> > >> plot and axis titles) #20200612_173626. You can load it
back
> >>>>> > >> to
> >>>>> the
> >>>>> > >> page by
> >>>>> > >> using the "History" tab (left bar on GUI). Find the plot,
> >>>>> > >> open XML
> >>>>> > >> and
> >>>>> > >> click on "Load to page". After that you can recreate a
plot
> >>>>> > >> and
> >>>>> edit
> >>>>> > >> the
> >>>>> > >> configurations. You can also download a new XML and edit
it
> >>>>> > >> for
> >>>>> > >> batch.
> >>>>> > >> I attached the example of load XML to this message.
> >>>>> > >>
> >>>>> > >> Description of rds parameters:
> >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
scripts
> >>>>> > >> rds_plots - temporary directory where the plots will be
> >>>>> > >> generated
> >>>>> > >> rds_data - temporary directory where the data retrieved
from
> >>>>> > >> the
> >>>>> > >> database
> >>>>> > >> and other intermediate files will be stored
> >>>>> > >> rds_scripts - temporary directory where R scripts for
this
> >>>>> > >> run
> >>>>> will
> >>>>> > >> be
> >>>>> > >> stored
> >>>>> > >> After the execution is done all the files from temporary
> >>>>> directories
> >>>>> > >> would be removed.
> >>>>> > >>
> >>>>> > >>
> >>>>> > >> We don't have the offitial user guide for METviewer.
There
> >>>>> > >> are
> >>>>> some
> >>>>> > >> instructions on how to crate XML config files:
> >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> >>>>> > >>
> >>>>> > >> Tatiana
> >>>>> > >>
> >>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
> >>>>> > >> > Hi,
> >>>>> > >> >
> >>>>> > >> > I'm attaching the xml below, but am having hard time
> >>>>> > >> > following
> >>>>> how
> >>>>> > >> > to
> >>>>> > >> > go
> >>>>> > >> > about the other steps since I have zero experience on
how
> >>>>> > >> > to do
> >>>>> > >> > these
> >>>>> > >> > things.
> >>>>> > >> >
> >>>>> > >> > #2. All parameters in <folders> section should NOT be
> >>>>> > >> > changed
> >>>>> and
> >>>>> > >> > will
> >>>>> > >> > be
> >>>>> > >> > replaced by other scripts during the batch execution.
The
> >>>>> > >> > data
> >>>>> for
> >>>>> > >> > the
> >>>>> > >> > batch mode is coming from the database and not from
files.
> >>>>> > >> > The
> >>>>> > >> > name of
> >>>>> > >> > the
> >>>>> > >> > database is in <database> tag.
> >>>>> > >> >
> >>>>> > >> > I understand that nothing should be changed in the
folders
> >>>>> > >> > section,
> >>>>> > >> > but
> >>>>> > >> > there is no indication how and where these variables
are
> >>>>> > >> > being
> >>>>> > >> > defined.
> >>>>> > >> > Grepping these names of the files available does not
> >>>>> > >> > reveal
> >>>>> formal
> >>>>> > >> > definitions of these variables. If I understood
exactly
> >>>>> > >> > how
> >>>>> these
> >>>>> > >> > were
> >>>>> > >> > being defined, then I would be able connect this
better.
> >>>>> > >> >
> >>>>> > >> > #3. Users rum MET and produce MET output files. They
use
> >>>>> METviewer
> >>>>> > >> > load
> >>>>> > >> > mode to load these files to the specific database. In
the
> >>>>> > >> > config
> >>>>> > >> > file
> >>>>> > >> > for
> >>>>> > >> > this process they also can specify the database group.
For
> >>>>> example
> >>>>> > >> > database
> >>>>> > >> > mv_dev is in Test12 group.
> >>>>> > >> > After the data has been loaded to the database it is
> >>>>> > >> > visible and
> >>>>> > >> > available
> >>>>> > >> > in the GUI and also can be used by the batch mode.
> >>>>> > >> > You can always add more data to the database using the
> >>>>> > >> > same load
> >>>>> > >> > mode.
> >>>>> > >> >
> >>>>> > >> > Is there a guide for me to understand the process that
you
> >>>>> > >> > are
> >>>>> > >> > referring to?
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> >>>>> > >> > <met_help at ucar.edu>
> >>>>> > >> > wrote:
> >>>>> > >> >
> >>>>> > >> > > Hi Edward,
> >>>>> > >> > > #1. I don't have an access to NOAA servers and can't
> >>>>> > >> > > view your
> >>>>> > >> > > XML
> >>>>> > >> > > for the
> >>>>> > >> > > batch mode. Please, attache it to this issue and I
will
> >>>>> > >> > > take a
> >>>>> > >> > > look.
> >>>>> > >> > >
> >>>>> > >> > > #2. All parameters in <folders> section should NOT be
> >>>>> > >> > > changed
> >>>>> > >> > > and
> >>>>> > >> > > will be
> >>>>> > >> > > replaced by other scripts during the batch execution.
> >>>>> > >> > > The data
> >>>>> > >> > > for
> >>>>> > >> > > the
> >>>>> > >> > > batch mode is coming from the database and not from
> >>>>> > >> > > files. The
> >>>>> > >> > > name
> >>>>> > >> > > of the
> >>>>> > >> > > database is in <database> tag.
> >>>>> > >> > >
> >>>>> > >> > > #3. Users rum MET and produce MET output files. They
use
> >>>>> > >> > > METviewer
> >>>>> > >> > > load
> >>>>> > >> > > mode to load these files to the specific database. In
> >>>>> > >> > > the
> >>>>> config
> >>>>> > >> > > file
> >>>>> > >> > > for
> >>>>> > >> > > this process they also can specify the database
group.
> >>>>> > >> > > For
> >>>>> > >> > > example
> >>>>> > >> > > database
> >>>>> > >> > > mv_dev is in Test12 group.
> >>>>> > >> > > After the data has been loaded to the database it is
> >>>>> > >> > > visible
> >>>>> and
> >>>>> > >> > > available
> >>>>> > >> > > in the GUI and also can be used by the batch mode.
> >>>>> > >> > > You can always add more data to the database using
the
> >>>>> > >> > > same
> >>>>> load
> >>>>> > >> > > mode.
> >>>>> > >> > >
> >>>>> > >> > > Tatiana
> >>>>> > >> > >
> >>>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
> >>>>> > >> > > wrote:
> >>>>> > >> > > > Good morning,
> >>>>> > >> > > >
> >>>>> > >> > > > Over the last two weeks I've made an effort to
> >>>>> > >> > > > understand
> >>>>> > >> > > > metviewer,
> >>>>> > >> > > > test
> >>>>> > >> > > > different cases using the history tab, changing
> >>>>> > >> > > > information
> >>>>> to
> >>>>> > >> > > > see
> >>>>> > >> > > > what
> >>>>> > >> > > > works and what does not, and make connections
between
> >>>>> > >> > > > the
> >>>>> GUI
> >>>>> > >> > > > settings
> >>>>> > >> > > > and
> >>>>> > >> > > > the XML file. Although my comfort level with
> >>>>> > >> > > > metviewer has
> >>>>> > >> > > > significantly
> >>>>> > >> > > > increased during this time, there are some
remaining
> >>>>> > >> > > > issues
> >>>>> > >> > > > that
> >>>>> > >> > > > I'm
> >>>>> > >> > > > hoping
> >>>>> > >> > > > to clear up.
> >>>>> > >> > > >
> >>>>> > >> > > > 1. Problem changing XML file and running with
> >>>>> > >> > > > different
> >>>>> > >> > > > statistic
> >>>>> > >> > > > and
> >>>>> > >> > > > plot
> >>>>> > >> > > > type:
> >>>>> > >> > > >
> >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
> >>>>> > >> > > > working
> >>>>> > >> > > > directory
> >>>>> > >> > > > to get
> >>>>> > >> > > > started which I copied over:
> >>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >>>>> > >> > > >
> >>>>> > >> > > > He has a test.sh script that I adjusted and used to
> >>>>> > >> > > > run his
> >>>>> > >> > > > example.
> >>>>> > >> > > > running of the script "bash test.sh" produced a CSI
> >>>>> > >> > > > plot
> >>>>> that
> >>>>> > >> > > > was
> >>>>> > >> > > > stored
> >>>>> > >> > > > here:
> >>>>> > >> > > >
> >>>>> > >> > >
> >>>>> > >>
> >>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >>>>> > >> > > >
> >>>>> > >> > > > Running the script produced the expected result.
The
> >>>>> > >> > > > next
> >>>>> > >> > > > natural
> >>>>> > >> > > > step was
> >>>>> > >> > > > to copy hochum_xml.template and create a new xml
file
> >>>>> > >> > > > called
> >>>>> > >> > > > Strobach_xml.template in order to create a time
series
> >>>>> > >> > > > where
> >>>>> > >> > > > the
> >>>>> > >> > > > independent time is fcst_valid_beg instead of
> >>>>> > >> > > > obs_thresh.
> >>>>> > >> > > > Accordingly,
> >>>>> > >> > > > I
> >>>>> > >> > > > made changes to the fixed fields (removing
> >>>>> > >> > > > fcst_valid_beg)
> >>>>> and
> >>>>> > >> > > > changed
> >>>>> > >> > > > the
> >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).
> >>>>> > >> > > > The
> >>>>> error
> >>>>> > >> > > > that
> >>>>> > >> > > > resulted
> >>>>> > >> > > > is as follows:
> >>>>> > >> > > >
> >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
> >>>>> > >> > > > 13:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 01
> >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> >>>>> > >> > > > 16:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 01
> >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> >>>>> > >> > > > 19:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 01
> >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> >>>>> > >> > > > 22:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 01
> >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> >>>>> > >> > > > 01:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 02
> >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> >>>>> > >> > > > 04:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 02
> >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> >>>>> > >> > > > 07:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 02
> >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> >>>>> > >> > > > 10:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 02
> >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> >>>>> > >> > > > AND ld.fmean != -9999;
> >>>>> > >> > > > No data was returned from database
> >>>>> > >> > > > 2020-06-09
> >>>>> > >> > > >
> >>>>> > >> > >
> >>>>> > >>
> >>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >>>>> > >> > > > Source) -1 printFormattedTable - ** ERROR:
Caught
> >>>>> > >> > > > class
> >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> >>>>> > >> > > > printFormattedTable(ResultSet
> >>>>> > >> > > > res): result set contained no data
> >>>>> > >> > > > ** ERROR: Caught class
> >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >>>>> > >> > > > in
> >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> >>>>> > >> > > > contained no
> >>>>> > >> > > > data
> >>>>> > >> > > > ** ERROR: Caught class
> >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >>>>> > >> > > > in
> >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> >>>>> > >> > > > contained no
> >>>>> > >> > > > data
> >>>>> > >> > > >
> >>>>> > >> > > > Apparently there was no data during this time. I
> >>>>> > >> > > > tried
> >>>>> > >> > > > changing
> >>>>> > >> > > > the
> >>>>> > >> > > > range
> >>>>> > >> > > > of valid times and changed the valid time to
different
> >>>>> > >> > > > days
> >>>>> > >> > > > just in
> >>>>> > >> > > > case
> >>>>> > >> > > > data was missing. The issue was the same. To
confirm
> >>>>> > >> > > > my
> >>>>> > >> > > > logic, I
> >>>>> > >> > > > reproduced my attempted approach on metviewer,
which I
> >>>>> > >> > > > found
> >>>>> > >> > > > to
> >>>>> > >> > > > work.
> >>>>> > >> > > >
> >>>>> > >> > > > 2. Running in batch mode
> >>>>> > >> > > >
> >>>>> > >> > > > I think I lack an understanding of how to set up
batch
> >>>>> > >> > > > mode
> >>>>> > >> > > > because
> >>>>> > >> > > > I
> >>>>> > >> > > > don't
> >>>>> > >> > > > see where these definitions are being explicitly
> >>>>> > >> > > > defined:
> >>>>> > >> > > >
> >>>>> > >> > > > <plot_spec>
> >>>>> > >> > > > <connection>
> >>>>> > >> > > > <host>rds_host:3306</host>
> >>>>> > >> > > >
> >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >>>>> > >> > > > <user>rds_user</user>
> >>>>> > >> > > > <password>rds_pwd</password>
> >>>>> > >> > > > </connection>
> >>>>> > >> > > > <rscript>Rscript</rscript>
> >>>>> > >> > > > <folders>
> >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >>>>> > >> > > > <r_work>rds_R_work</r_work>
> >>>>> > >> > > > <plots>rds_plots</plots>
> >>>>> > >> > > > <data>rds_data</data>
> >>>>> > >> > > > <scripts>rds_scripts</scripts>
> >>>>> > >> > > > </folders>
> >>>>> > >> > > > <plot>
> >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> >>>>> > >> > > >
> >>>>> > >> > > > I see the <folders> section, but grepping all
> >>>>> > >> > > > available
> >>>>> files
> >>>>> > >> > > > does
> >>>>> > >> > > > not
> >>>>> > >> > > > reveal how rds_R_work is being defined, as well as
> >>>>> > >> > > > other
> >>>>> > >> > > > similarly
> >>>>> > >> > > > defined
> >>>>> > >> > > > variables. I think if I know how these are being
> >>>>> > >> > > > defined,
> >>>>> > >> > > > then
> >>>>> > >> > > > I'll
> >>>>> > >> > > > be
> >>>>> > >> > > > able to pipe in my information and process the data
> >>>>> > >> > > > using
> >>>>> met
> >>>>> > >> > > > and
> >>>>> > >> > > > produce
> >>>>> > >> > > > results in batch mode. Am I right on that? Also,
if
> >>>>> > >> > > > I want
> >>>>> > >> > > > to use
> >>>>> > >> > > > obs for
> >>>>> > >> > > > verification, do I add another line in the XML file
> >>>>> > >> > > > that
> >>>>> > >> > > > designates
> >>>>> > >> > > > file
> >>>>> > >> > > > location of obs? I assume that other steps have to
be
> >>>>> > >> > > > run
> >>>>> > >> > > > before
> >>>>> > >> > > > all
> >>>>> > >> > > > this. If data is in prepbufr form, for example, I
> >>>>> > >> > > > would
> >>>>> have
> >>>>> > >> > > > to
> >>>>> > >> > > > separately
> >>>>> > >> > > > run PB2NC. If I want grid2obs, then I would want
to
> >>>>> > >> > > > do that
> >>>>> > >> > > > as
> >>>>> > >> > > > well.
> >>>>> > >> > > >
> >>>>> > >> > > > 3. Communicating results to metviewer online
> >>>>> > >> > > >
> >>>>> > >> > > > Lastly, I see a lot of people/projects listed here
-
> >>>>> > >> > > > see
> >>>>> > >> > > > attachment
> >>>>> > >> > > >
> >>>>> > >> > > > How do I get the results I generate to be uploaded
to
> >>>>> > >> > > > this
> >>>>> > >> > > > location?
> >>>>> > >> > > > It's
> >>>>> > >> > > > not clear how others are setting this up.
> >>>>> > >> > >
> >>>>> > >> > >
> >>>>> > >> > >
> >>>>> > >> > >
> >>>>> > >>
> >>>>> > >>
> >>>>> > >>
> >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
rds_user
> >>>>> rds_pwd
> >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
> >>>>> rds_scripts
> >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
> >>>>> 2020-05-12
> >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-
12
> >>>>> 20:00:00
> >>>>> > >> 2020-05-12
> >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-
13
> >>>>> 10:00:00
> >>>>> > >> 2020-05-13
> >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-
13
> >>>>> 14:00:00
> >>>>> > >> 2020-05-13
> >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-
13
> >>>>> 18:00:00
> >>>>> > >> 2020-05-13
> >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-
13
> >>>>> 22:00:00
> >>>>> > >> 2020-05-13
> >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-
14
> >>>>> 12:00:00
> >>>>> > >> true
> >>>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
> >>>>> > >> plot_20200612_173626.R test x_label test y_label
> >>>>> > >> false list() list() false false false false false false
true
> >>>>> > >> false
> >>>>> > >> false
> >>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
> >>>>> > >> c(1,1,0) 1 2
> >>>>> > >> 1.4 -2
> >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
> >>>>> > >> #cccccc 1
> >>>>> > >> listX 1
> >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3
> >>>>> > >> 0.8 1
> >>>>> > >> #333333
> >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
> >>>>> c(1,1)
> >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> >>>>> > >
> >>>>> > >
> >>>>> > >
> >>>>> > > --
> >>>>> > > Edward Strobach
> >>>>> > > EMC/NCEP/NWS/
> >>>>> > > IMSG Contractor
> >>>>> > > Cubicle#: 2029
> >>>>> > > 301-683-3717
> >>>>> > >
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Edward Strobach
> >>>> EMC/NCEP/NWS/
> >>>> IMSG Contractor
> >>>> Cubicle#: 2029
> >>>> 301-683-3717
> >>>>
> >>>
> >>>
> >>> --
> >>> Edward Strobach
> >>> EMC/NCEP/NWS/
> >>> IMSG Contractor
> >>> Cubicle#: 2029
> >>> 301-683-3717
> >>>
> >>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 15:40:50 2020
Thanks Tatiana. Yes, I've created a database which seems different
from
mv_group. How can one create their own group so that I can store and
access
the database. The database exists but cannot be viewed on the
metviewer
GUI. The list starts with EMC HREFv3 and ends with vhagerty. I named
my
database Ed_Strobach, but that was really intended for the group name.
On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> We don't have database and METviewer running on wcoss.
> Instead, users use METviewer-wcoss scripts that help to connect to
and use
> METviewer that is running on AWS. That is why XML connection and
folder
> sections look somewhat different than once from the Docs page.
> To create a database on AWS you need to run mv_create_db_on_aws.sh
> mv_create_db_on_aws.sh <user_name> <database_name>
> For data loading :
> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> And for batch:
> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>
> These scripts copy XML and data files if needed to AWS, adjust data
path
> and users credentials, run METviewer and copy results back to wcoss.
>
> Tatiana
> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> > lastly, it does seem thatmysql -u[db_username] -p[db_password]
> > -e'create
> > database [db_name]
> >
> > would need to be used to create a database. I assume that an sql
> > module is
> > available on wcoss; I know you don't have access to those machines
> > Tatiana. If I create my own database with my own name, then I
should
> > be
> > able to proceed with project specific tasks under that tab.
Something
> > like
> > mv_EdwardStrobach.
> >
> > I can then create mv_cmaq_prod_v_para which I add data to using
> > load_spec.
> > Once that step is done, then I can run plot_spec. That's my
takeaway
> > currently
> >
> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > Actually, it appears that I need to run the database loading
module
> > > first
> > > if I want to change/add a data source. From there I can run the
> > > plotting
> > > batch module. Is that right? So if I run the database loading
> > > module,
> > > then I can specify my dates and other related info specific to
the
> > > dataset. If I create my database with a unique name, then it
should
> > > be
> > > recognized on the metviewer gui, presumably.
> > >
> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate
<
> > > edward.strobach at noaa.gov> wrote:
> > >
> > >> It seems that those options must be treated separately outside
of
> > >> plot_spec. Is it acceptable to create both a load_spec and
> > >> plot_spec in
> > >> the same xml file? Would the definitions in load_spec be able
to
> > >> carry
> > >> over into plot_spec? It appears that data loading and plotting
are
> > >> treated
> > >> separately.
> > >>
> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
Affiliate <
> > >> edward.strobach at noaa.gov> wrote:
> > >>
> > >>> I do plan to use date_range; however, in an example provided
by the
> > >>> link
> > >>> I see this:
> > >>>
> > >>> <load_spec>
> > >>> <connection>
> > >>> <host>kemosabe:3306</host>
> > >>> <database>metvdb_hwt</database>
> > >>> <user>pgoldenb</user>
> > >>> <password>pgoldenb</password>
> > >>> </connection>
> > >>>
> > >>> <date_list name="folder_dates">
> > >>> <start>2010051914V</start>
> > >>> <end><date_offset><day_offset>0</day_offset><hour>6
> > >>> </hour></date_offset></end>
> > >>> <inc>3600</inc>
> > >>> <format>yyyyMMddHH'V'</format>
> > >>> </date_list>
> > >>>
> > >>> <met_version>V3.0</met_version>
> > >>>
> > >>> <verbose>false</verbose>
> > >>> <insert_size>1</insert_size>
> > >>> <mode_header_db_check>true</mode_header_db_check>
> > >>> <drop_indexes>false</drop_indexes>
> > >>> <apply_indexes>true</apply_indexes>
> > >>> <group>Group name</group>
> > >>> <load_stat>true</load_stat>
> > >>> <load_mode>true</load_mode>
> > >>> <load_mtd>true</load_mtd>
> > >>> <load_mpr>false</load_mpr>
> > >>>
> > >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > >>> <load_val>
> > >>> <field name="model">
> > >>> <val>arw</val>
> > >>> <val>nmm</val>
> > >>> </field>
> > >>>
> > >>> <field name="valid_time">
> > >>> <date_list name="folder_dates"/>
> > >>> </field>
> > >>>
> > >>> <field name="vx_mask">
> > >>> <val>FULL</val>
> > >>> <val>SWC</val>
> > >>> </field>
> > >>> </load_val>
> > >>> </load_spec>
> > >>>
> > >>> Are similar settings allowed with <plot_spec>? I see
<connection>,
> > >>> which seems carried over. I'm looking to specify my
folder_tmpl in
> > >>> <plot_spec>, which according to Tara, should be possible.
Since my
> > >>> directory structure looks like this:
> > >>>
> > >>>
> > >>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > >>>
> > >>> I would do something like this I would think:
> > >>>
> > >>> .......
> > >>>
> > >>> <date_list name="folder_dates">
> > >>> <start>20200612/start>
> > >>> <end><date_offset><day_offset>0</day_offset></end>
> > >>> <inc>86400</inc>
> > >>> <format>yyyyMMdd</format>
> > >>> </date_list>
> > >>>
> > >>> <folder_tmpl>
> > >>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > >>> </folder_tmpl>
> > >>> <load_val>
> > >>> <field name="model">
> > >>> <val>prod</val>
> > >>> </field>
> > >>>
> > >>> <field name="valid_time">
> > >>> <date_list name="folder_dates"/>
> > >>> </field>
> > >>>
> > >>> ......
> > >>>
> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
Affiliate <
> > >>> edward.strobach at noaa.gov> wrote:
> > >>>
> > >>>> Thank you. This is a very helpful example
> > >>>>
> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> > >>>> <met_help at ucar.edu>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>> Here is some examples how to use <date_range> and
> > >>>>> <date_range_list> in
> > >>>>> batch XML.
> > >>>>> First, you need to defined your range. You can have more
then one
> > >>>>> <date_range>:
> > >>>>> <date_range name="may">
> > >>>>> <start>2019-05-01 00:00:00</start>
> > >>>>> <end>2019-05-31 23:59:59</end>
> > >>>>> </date_range>
> > >>>>> <date_range name="march">
> > >>>>> <start>2019-03-01 00:00:00</start>
> > >>>>> <end>2019-03-31 23:59:59</end>
> > >>>>> </date_range>
> > >>>>>
> > >>>>> and then use one or all date_ranges:
> > >>>>> <plot_fix>
> > >>>>> <field name="fcst_init_beg">
> > >>>>> <date_range name="march"/>
> > >>>>> <date_range name="may"/>
> > >>>>> </field>...
> > >>>>> This configuration would include all dates for March and May
of
> > >>>>> 2019.
> > >>>>>
> > >>>>> If you need more flexible configuration you can use
> > >>>>> date_range_list.
> > >>>>> For example, you want to select first 6 hours every day
beetween
> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would
look
> > >>>>> like this:
> > >>>>> <date_range_list name = 'all'>
> > >>>>> <range_start>2020-01-01 00:00:00</range_start>
> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
> > >>>>> <range_length>360</range_length>
> > >>>>> <inc>1440</inc>
> > >>>>> </date_range_list>
> > >>>>> where the 1st and 2nd values are the start and end,
> > >>>>> range_length - the length of a range in minutes(6 houres)
> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> > >>>>> This example would create following ranges:
> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> > >>>>>
> > >>>>> Also you can add more parameters to <plot_fix> to filter
your
> > >>>>> data.
> > >>>>> For example, if you want to select only dates with 00:00 you
can
> > >>>>> specify
> > >>>>> <field name="valid_hour">
> > >>>>> <val>00</val>
> > >>>>> </field>
> > >>>>>
> > >>>>> Tatiana
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> > >>>>> > and if there isn't a better way, I wonder the whereabouts
of a
> > >>>>> > separate
> > >>>>> > script that pipes dates (and other relevant information)
into
> > >>>>> > the XML
> > >>>>> > file
> > >>>>> > used to generate the results.
> > >>>>> >
> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> > >>>>> > Affiliate <
> > >>>>> > edward.strobach at noaa.gov> wrote:
> > >>>>> >
> > >>>>> > > Thanks Tatiana. These resources are helpful.
> > >>>>> > >
> > >>>>> > > One last question. Simply typing lines like this in can
take
> > >>>>> > > forever:
> > >>>>> > >
> > >>>>> > > <plot_fix>
> > >>>>> > > <field equalize="false" name="fcst_init_beg">
> > >>>>> > > <set name="fcst_init_beg_0">
> > >>>>> > > <val>2019-05-01 12:00:00</val>
> > >>>>> > > <val>2019-05-02 06:00:00</val>
> > >>>>> > > <val>2019-05-02 12:00:00</val>
> > >>>>> > > <val>2019-05-03 06:00:00</val>
> > >>>>> > > <val>2019-05-03 12:00:00</val>
> > >>>>> > >
> > >>>>> > >
> > >>>>> > > Is there a better way? I see date_range,
date_range_list,
> > >>>>> > > or
> > >>>>> > > date_list.
> > >>>>> > > Can this be used instead of typing individual values?
This
> > >>>>> > > could
> > >>>>> > > prove
> > >>>>> > > pretty daunting if one would have to type in 700 entries
each
> > >>>>> > > time.
> > >>>>> > >
> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > >>>>> > > <met_help at ucar.edu>
> > >>>>> > > wrote:
> > >>>>> > >
> > >>>>> > >> I found a few problems in your XML.
> > >>>>> > >> The database you are using doesn't have data with
vx_mask
> > >>>>> > >> 'abcREGabc'.
> > >>>>> > >> That is why METviewer returns an empty dataset and
can't
> > >>>>> > >> create a
> > >>>>> > >> plot.
> > >>>>> > >> The other problem is how you define fcst_lead. This
database
> > >>>>> > >> has
> > >>>>> > >> values
> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
include
> > >>>>> > >> all
> > >>>>> > >> fcst_leads
> > >>>>> > >> this variable can be omitted.
> > >>>>> > >> To minimize problems with variables values, I would
suggest
> > >>>>> > >> to
> > >>>>> > >> create XML
> > >>>>> > >> using GUI and then edit it to fit batch standards.
> > >>>>> > >> I created a plot using you configurations with GUI
(without
> > >>>>> vx_mack
> > >>>>> > >> and
> > >>>>> > >> plot and axis titles) #20200612_173626. You can load it
back
> > >>>>> > >> to
> > >>>>> the
> > >>>>> > >> page by
> > >>>>> > >> using the "History" tab (left bar on GUI). Find the
plot,
> > >>>>> > >> open XML
> > >>>>> > >> and
> > >>>>> > >> click on "Load to page". After that you can recreate a
plot
> > >>>>> > >> and
> > >>>>> edit
> > >>>>> > >> the
> > >>>>> > >> configurations. You can also download a new XML and
edit it
> > >>>>> > >> for
> > >>>>> > >> batch.
> > >>>>> > >> I attached the example of load XML to this message.
> > >>>>> > >>
> > >>>>> > >> Description of rds parameters:
> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
scripts
> > >>>>> > >> rds_plots - temporary directory where the plots will be
> > >>>>> > >> generated
> > >>>>> > >> rds_data - temporary directory where the data retrieved
from
> > >>>>> > >> the
> > >>>>> > >> database
> > >>>>> > >> and other intermediate files will be stored
> > >>>>> > >> rds_scripts - temporary directory where R scripts for
this
> > >>>>> > >> run
> > >>>>> will
> > >>>>> > >> be
> > >>>>> > >> stored
> > >>>>> > >> After the execution is done all the files from
temporary
> > >>>>> directories
> > >>>>> > >> would be removed.
> > >>>>> > >>
> > >>>>> > >>
> > >>>>> > >> We don't have the offitial user guide for METviewer.
There
> > >>>>> > >> are
> > >>>>> some
> > >>>>> > >> instructions on how to crate XML config files:
> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > >>>>> > >>
> > >>>>> > >> Tatiana
> > >>>>> > >>
> > >>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
> > >>>>> > >> > Hi,
> > >>>>> > >> >
> > >>>>> > >> > I'm attaching the xml below, but am having hard time
> > >>>>> > >> > following
> > >>>>> how
> > >>>>> > >> > to
> > >>>>> > >> > go
> > >>>>> > >> > about the other steps since I have zero experience on
how
> > >>>>> > >> > to do
> > >>>>> > >> > these
> > >>>>> > >> > things.
> > >>>>> > >> >
> > >>>>> > >> > #2. All parameters in <folders> section should NOT be
> > >>>>> > >> > changed
> > >>>>> and
> > >>>>> > >> > will
> > >>>>> > >> > be
> > >>>>> > >> > replaced by other scripts during the batch execution.
The
> > >>>>> > >> > data
> > >>>>> for
> > >>>>> > >> > the
> > >>>>> > >> > batch mode is coming from the database and not from
files.
> > >>>>> > >> > The
> > >>>>> > >> > name of
> > >>>>> > >> > the
> > >>>>> > >> > database is in <database> tag.
> > >>>>> > >> >
> > >>>>> > >> > I understand that nothing should be changed in the
folders
> > >>>>> > >> > section,
> > >>>>> > >> > but
> > >>>>> > >> > there is no indication how and where these variables
are
> > >>>>> > >> > being
> > >>>>> > >> > defined.
> > >>>>> > >> > Grepping these names of the files available does not
> > >>>>> > >> > reveal
> > >>>>> formal
> > >>>>> > >> > definitions of these variables. If I understood
exactly
> > >>>>> > >> > how
> > >>>>> these
> > >>>>> > >> > were
> > >>>>> > >> > being defined, then I would be able connect this
better.
> > >>>>> > >> >
> > >>>>> > >> > #3. Users rum MET and produce MET output files. They
use
> > >>>>> METviewer
> > >>>>> > >> > load
> > >>>>> > >> > mode to load these files to the specific database. In
the
> > >>>>> > >> > config
> > >>>>> > >> > file
> > >>>>> > >> > for
> > >>>>> > >> > this process they also can specify the database
group. For
> > >>>>> example
> > >>>>> > >> > database
> > >>>>> > >> > mv_dev is in Test12 group.
> > >>>>> > >> > After the data has been loaded to the database it is
> > >>>>> > >> > visible and
> > >>>>> > >> > available
> > >>>>> > >> > in the GUI and also can be used by the batch mode.
> > >>>>> > >> > You can always add more data to the database using
the
> > >>>>> > >> > same load
> > >>>>> > >> > mode.
> > >>>>> > >> >
> > >>>>> > >> > Is there a guide for me to understand the process
that you
> > >>>>> > >> > are
> > >>>>> > >> > referring to?
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> > >>>>> > >> > <met_help at ucar.edu>
> > >>>>> > >> > wrote:
> > >>>>> > >> >
> > >>>>> > >> > > Hi Edward,
> > >>>>> > >> > > #1. I don't have an access to NOAA servers and
can't
> > >>>>> > >> > > view your
> > >>>>> > >> > > XML
> > >>>>> > >> > > for the
> > >>>>> > >> > > batch mode. Please, attache it to this issue and I
will
> > >>>>> > >> > > take a
> > >>>>> > >> > > look.
> > >>>>> > >> > >
> > >>>>> > >> > > #2. All parameters in <folders> section should NOT
be
> > >>>>> > >> > > changed
> > >>>>> > >> > > and
> > >>>>> > >> > > will be
> > >>>>> > >> > > replaced by other scripts during the batch
execution.
> > >>>>> > >> > > The data
> > >>>>> > >> > > for
> > >>>>> > >> > > the
> > >>>>> > >> > > batch mode is coming from the database and not from
> > >>>>> > >> > > files. The
> > >>>>> > >> > > name
> > >>>>> > >> > > of the
> > >>>>> > >> > > database is in <database> tag.
> > >>>>> > >> > >
> > >>>>> > >> > > #3. Users rum MET and produce MET output files.
They use
> > >>>>> > >> > > METviewer
> > >>>>> > >> > > load
> > >>>>> > >> > > mode to load these files to the specific database.
In
> > >>>>> > >> > > the
> > >>>>> config
> > >>>>> > >> > > file
> > >>>>> > >> > > for
> > >>>>> > >> > > this process they also can specify the database
group.
> > >>>>> > >> > > For
> > >>>>> > >> > > example
> > >>>>> > >> > > database
> > >>>>> > >> > > mv_dev is in Test12 group.
> > >>>>> > >> > > After the data has been loaded to the database it
is
> > >>>>> > >> > > visible
> > >>>>> and
> > >>>>> > >> > > available
> > >>>>> > >> > > in the GUI and also can be used by the batch mode.
> > >>>>> > >> > > You can always add more data to the database using
the
> > >>>>> > >> > > same
> > >>>>> load
> > >>>>> > >> > > mode.
> > >>>>> > >> > >
> > >>>>> > >> > > Tatiana
> > >>>>> > >> > >
> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
edward.strobach at noaa.gov
> > >>>>> > >> > > wrote:
> > >>>>> > >> > > > Good morning,
> > >>>>> > >> > > >
> > >>>>> > >> > > > Over the last two weeks I've made an effort to
> > >>>>> > >> > > > understand
> > >>>>> > >> > > > metviewer,
> > >>>>> > >> > > > test
> > >>>>> > >> > > > different cases using the history tab, changing
> > >>>>> > >> > > > information
> > >>>>> to
> > >>>>> > >> > > > see
> > >>>>> > >> > > > what
> > >>>>> > >> > > > works and what does not, and make connections
between
> > >>>>> > >> > > > the
> > >>>>> GUI
> > >>>>> > >> > > > settings
> > >>>>> > >> > > > and
> > >>>>> > >> > > > the XML file. Although my comfort level with
> > >>>>> > >> > > > metviewer has
> > >>>>> > >> > > > significantly
> > >>>>> > >> > > > increased during this time, there are some
remaining
> > >>>>> > >> > > > issues
> > >>>>> > >> > > > that
> > >>>>> > >> > > > I'm
> > >>>>> > >> > > > hoping
> > >>>>> > >> > > > to clear up.
> > >>>>> > >> > > >
> > >>>>> > >> > > > 1. Problem changing XML file and running with
> > >>>>> > >> > > > different
> > >>>>> > >> > > > statistic
> > >>>>> > >> > > > and
> > >>>>> > >> > > > plot
> > >>>>> > >> > > > type:
> > >>>>> > >> > > >
> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me
the
> > >>>>> > >> > > > working
> > >>>>> > >> > > > directory
> > >>>>> > >> > > > to get
> > >>>>> > >> > > > started which I copied over:
> > >>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >>>>> > >> > > >
> > >>>>> > >> > > > He has a test.sh script that I adjusted and used
to
> > >>>>> > >> > > > run his
> > >>>>> > >> > > > example.
> > >>>>> > >> > > > running of the script "bash test.sh" produced a
CSI
> > >>>>> > >> > > > plot
> > >>>>> that
> > >>>>> > >> > > > was
> > >>>>> > >> > > > stored
> > >>>>> > >> > > > here:
> > >>>>> > >> > > >
> > >>>>> > >> > >
> > >>>>> > >>
> > >>>>>
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >>>>> > >> > > >
> > >>>>> > >> > > > Running the script produced the expected result.
The
> > >>>>> > >> > > > next
> > >>>>> > >> > > > natural
> > >>>>> > >> > > > step was
> > >>>>> > >> > > > to copy hochum_xml.template and create a new xml
file
> > >>>>> > >> > > > called
> > >>>>> > >> > > > Strobach_xml.template in order to create a time
series
> > >>>>> > >> > > > where
> > >>>>> > >> > > > the
> > >>>>> > >> > > > independent time is fcst_valid_beg instead of
> > >>>>> > >> > > > obs_thresh.
> > >>>>> > >> > > > Accordingly,
> > >>>>> > >> > > > I
> > >>>>> > >> > > > made changes to the fixed fields (removing
> > >>>>> > >> > > > fcst_valid_beg)
> > >>>>> and
> > >>>>> > >> > > > changed
> > >>>>> > >> > > > the
> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
tried).
> > >>>>> > >> > > > The
> > >>>>> error
> > >>>>> > >> > > > that
> > >>>>> > >> > > > resulted
> > >>>>> > >> > > > is as follows:
> > >>>>> > >> > > >
> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
> > >>>>> > >> > > > 13:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 01
> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> > >>>>> > >> > > > 16:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 01
> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> > >>>>> > >> > > > 19:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 01
> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> > >>>>> > >> > > > 22:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 01
> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> > >>>>> > >> > > > 01:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 02
> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> > >>>>> > >> > > > 04:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 02
> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> > >>>>> > >> > > > 07:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 02
> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> > >>>>> > >> > > > 10:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 02
> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > >>>>> > >> > > > AND ld.fmean != -9999;
> > >>>>> > >> > > > No data was returned from database
> > >>>>> > >> > > > 2020-06-09
> > >>>>> > >> > > >
> > >>>>> > >> > >
> > >>>>> > >>
> > >>>>>
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > >>>>> > >> > > > Source) -1 printFormattedTable - ** ERROR:
Caught
> > >>>>> > >> > > > class
> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> > >>>>> > >> > > > printFormattedTable(ResultSet
> > >>>>> > >> > > > res): result set contained no data
> > >>>>> > >> > > > ** ERROR: Caught class
> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >>>>> > >> > > > in
> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> > >>>>> > >> > > > contained no
> > >>>>> > >> > > > data
> > >>>>> > >> > > > ** ERROR: Caught class
> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >>>>> > >> > > > in
> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> > >>>>> > >> > > > contained no
> > >>>>> > >> > > > data
> > >>>>> > >> > > >
> > >>>>> > >> > > > Apparently there was no data during this time. I
> > >>>>> > >> > > > tried
> > >>>>> > >> > > > changing
> > >>>>> > >> > > > the
> > >>>>> > >> > > > range
> > >>>>> > >> > > > of valid times and changed the valid time to
different
> > >>>>> > >> > > > days
> > >>>>> > >> > > > just in
> > >>>>> > >> > > > case
> > >>>>> > >> > > > data was missing. The issue was the same. To
confirm
> > >>>>> > >> > > > my
> > >>>>> > >> > > > logic, I
> > >>>>> > >> > > > reproduced my attempted approach on metviewer,
which I
> > >>>>> > >> > > > found
> > >>>>> > >> > > > to
> > >>>>> > >> > > > work.
> > >>>>> > >> > > >
> > >>>>> > >> > > > 2. Running in batch mode
> > >>>>> > >> > > >
> > >>>>> > >> > > > I think I lack an understanding of how to set up
batch
> > >>>>> > >> > > > mode
> > >>>>> > >> > > > because
> > >>>>> > >> > > > I
> > >>>>> > >> > > > don't
> > >>>>> > >> > > > see where these definitions are being explicitly
> > >>>>> > >> > > > defined:
> > >>>>> > >> > > >
> > >>>>> > >> > > > <plot_spec>
> > >>>>> > >> > > > <connection>
> > >>>>> > >> > > > <host>rds_host:3306</host>
> > >>>>> > >> > > >
> > >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > >>>>> > >> > > > <user>rds_user</user>
> > >>>>> > >> > > > <password>rds_pwd</password>
> > >>>>> > >> > > > </connection>
> > >>>>> > >> > > > <rscript>Rscript</rscript>
> > >>>>> > >> > > > <folders>
> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > >>>>> > >> > > > <plots>rds_plots</plots>
> > >>>>> > >> > > > <data>rds_data</data>
> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > >>>>> > >> > > > </folders>
> > >>>>> > >> > > > <plot>
> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> > >>>>> > >> > > >
> > >>>>> > >> > > > I see the <folders> section, but grepping all
> > >>>>> > >> > > > available
> > >>>>> files
> > >>>>> > >> > > > does
> > >>>>> > >> > > > not
> > >>>>> > >> > > > reveal how rds_R_work is being defined, as well
as
> > >>>>> > >> > > > other
> > >>>>> > >> > > > similarly
> > >>>>> > >> > > > defined
> > >>>>> > >> > > > variables. I think if I know how these are being
> > >>>>> > >> > > > defined,
> > >>>>> > >> > > > then
> > >>>>> > >> > > > I'll
> > >>>>> > >> > > > be
> > >>>>> > >> > > > able to pipe in my information and process the
data
> > >>>>> > >> > > > using
> > >>>>> met
> > >>>>> > >> > > > and
> > >>>>> > >> > > > produce
> > >>>>> > >> > > > results in batch mode. Am I right on that?
Also, if
> > >>>>> > >> > > > I want
> > >>>>> > >> > > > to use
> > >>>>> > >> > > > obs for
> > >>>>> > >> > > > verification, do I add another line in the XML
file
> > >>>>> > >> > > > that
> > >>>>> > >> > > > designates
> > >>>>> > >> > > > file
> > >>>>> > >> > > > location of obs? I assume that other steps have
to be
> > >>>>> > >> > > > run
> > >>>>> > >> > > > before
> > >>>>> > >> > > > all
> > >>>>> > >> > > > this. If data is in prepbufr form, for example,
I
> > >>>>> > >> > > > would
> > >>>>> have
> > >>>>> > >> > > > to
> > >>>>> > >> > > > separately
> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I would want
to
> > >>>>> > >> > > > do that
> > >>>>> > >> > > > as
> > >>>>> > >> > > > well.
> > >>>>> > >> > > >
> > >>>>> > >> > > > 3. Communicating results to metviewer online
> > >>>>> > >> > > >
> > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
here -
> > >>>>> > >> > > > see
> > >>>>> > >> > > > attachment
> > >>>>> > >> > > >
> > >>>>> > >> > > > How do I get the results I generate to be
uploaded to
> > >>>>> > >> > > > this
> > >>>>> > >> > > > location?
> > >>>>> > >> > > > It's
> > >>>>> > >> > > > not clear how others are setting this up.
> > >>>>> > >> > >
> > >>>>> > >> > >
> > >>>>> > >> > >
> > >>>>> > >> > >
> > >>>>> > >>
> > >>>>> > >>
> > >>>>> > >>
> > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
rds_user
> > >>>>> rds_pwd
> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
> > >>>>> rds_scripts
> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
CMAQAODPRODHIGH
> > >>>>> 2020-05-12
> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-
05-12
> > >>>>> 20:00:00
> > >>>>> > >> 2020-05-12
> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-
05-13
> > >>>>> 10:00:00
> > >>>>> > >> 2020-05-13
> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-
05-13
> > >>>>> 14:00:00
> > >>>>> > >> 2020-05-13
> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-
05-13
> > >>>>> 18:00:00
> > >>>>> > >> 2020-05-13
> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-
05-13
> > >>>>> 22:00:00
> > >>>>> > >> 2020-05-13
> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-
05-14
> > >>>>> 12:00:00
> > >>>>> > >> true
> > >>>>> > >> median plot_20200612_173626.data
plot_20200612_173626.png
> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> > >>>>> > >> false list() list() false false false false false false
true
> > >>>>> > >> false
> > >>>>> > >> false
> > >>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
> > >>>>> > >> c(1,1,0) 1 2
> > >>>>> > >> 1.4 -2
> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
> > >>>>> > >> #cccccc 1
> > >>>>> > >> listX 1
> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25)
3
> > >>>>> > >> 0.8 1
> > >>>>> > >> #333333
> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
c(TRUE,TRUE)
> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
> > >>>>> c(1,1)
> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > >>>>> > >
> > >>>>> > >
> > >>>>> > >
> > >>>>> > > --
> > >>>>> > > Edward Strobach
> > >>>>> > > EMC/NCEP/NWS/
> > >>>>> > > IMSG Contractor
> > >>>>> > > Cubicle#: 2029
> > >>>>> > > 301-683-3717
> > >>>>> > >
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>> --
> > >>>> Edward Strobach
> > >>>> EMC/NCEP/NWS/
> > >>>> IMSG Contractor
> > >>>> Cubicle#: 2029
> > >>>> 301-683-3717
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Edward Strobach
> > >>> EMC/NCEP/NWS/
> > >>> IMSG Contractor
> > >>> Cubicle#: 2029
> > >>> 301-683-3717
> > >>>
> > >>
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 18:41:10 2020
I was basically able to build my xml file but it says I have no
available
space. I'm not sure how that can be since I just started adding data
to my
account
scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space left
on
device
ERROR: Command returned with non-zero status (1): scp -r ./*
edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
4) Check METviewer AWS database list using mv_db_size_on_aws.sh
TABLE_SCHEMA USER SIZE_MB
mv_met_g2o_gefs binbin.zhou 140027.31
mv_met_system_hrefv2_v3 binbin.zhou 71986.11
On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Thanks Tatiana. Yes, I've created a database which seems different
from
> mv_group. How can one create their own group so that I can store and
access
> the database. The database exists but cannot be viewed on the
metviewer
> GUI. The list starts with EMC HREFv3 and ends with vhagerty. I
named my
> database Ed_Strobach, but that was really intended for the group
name.
>
> On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> We don't have database and METviewer running on wcoss.
>> Instead, users use METviewer-wcoss scripts that help to connect to
and
>> use METviewer that is running on AWS. That is why XML connection
and folder
>> sections look somewhat different than once from the Docs page.
>> To create a database on AWS you need to run mv_create_db_on_aws.sh
>> mv_create_db_on_aws.sh <user_name> <database_name>
>> For data loading :
>> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>> And for batch:
>> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>
>> These scripts copy XML and data files if needed to AWS, adjust data
path
>> and users credentials, run METviewer and copy results back to
wcoss.
>>
>> Tatiana
>> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
>> > lastly, it does seem thatmysql -u[db_username] -p[db_password]
>> > -e'create
>> > database [db_name]
>> >
>> > would need to be used to create a database. I assume that an sql
>> > module is
>> > available on wcoss; I know you don't have access to those
machines
>> > Tatiana. If I create my own database with my own name, then I
should
>> > be
>> > able to proceed with project specific tasks under that tab.
Something
>> > like
>> > mv_EdwardStrobach.
>> >
>> > I can then create mv_cmaq_prod_v_para which I add data to using
>> > load_spec.
>> > Once that step is done, then I can run plot_spec. That's my
takeaway
>> > currently
>> >
>> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate
<
>> > edward.strobach at noaa.gov> wrote:
>> >
>> > > Actually, it appears that I need to run the database loading
module
>> > > first
>> > > if I want to change/add a data source. From there I can run
the
>> > > plotting
>> > > batch module. Is that right? So if I run the database
loading
>> > > module,
>> > > then I can specify my dates and other related info specific to
the
>> > > dataset. If I create my database with a unique name, then it
should
>> > > be
>> > > recognized on the metviewer gui, presumably.
>> > >
>> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate <
>> > > edward.strobach at noaa.gov> wrote:
>> > >
>> > >> It seems that those options must be treated separately outside
of
>> > >> plot_spec. Is it acceptable to create both a load_spec and
>> > >> plot_spec in
>> > >> the same xml file? Would the definitions in load_spec be able
to
>> > >> carry
>> > >> over into plot_spec? It appears that data loading and
plotting are
>> > >> treated
>> > >> separately.
>> > >>
>> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
Affiliate <
>> > >> edward.strobach at noaa.gov> wrote:
>> > >>
>> > >>> I do plan to use date_range; however, in an example provided
by the
>> > >>> link
>> > >>> I see this:
>> > >>>
>> > >>> <load_spec>
>> > >>> <connection>
>> > >>> <host>kemosabe:3306</host>
>> > >>> <database>metvdb_hwt</database>
>> > >>> <user>pgoldenb</user>
>> > >>> <password>pgoldenb</password>
>> > >>> </connection>
>> > >>>
>> > >>> <date_list name="folder_dates">
>> > >>> <start>2010051914V</start>
>> > >>> <end><date_offset><day_offset>0</day_offset><hour>6
>> > >>> </hour></date_offset></end>
>> > >>> <inc>3600</inc>
>> > >>> <format>yyyyMMddHH'V'</format>
>> > >>> </date_list>
>> > >>>
>> > >>> <met_version>V3.0</met_version>
>> > >>>
>> > >>> <verbose>false</verbose>
>> > >>> <insert_size>1</insert_size>
>> > >>> <mode_header_db_check>true</mode_header_db_check>
>> > >>> <drop_indexes>false</drop_indexes>
>> > >>> <apply_indexes>true</apply_indexes>
>> > >>> <group>Group name</group>
>> > >>> <load_stat>true</load_stat>
>> > >>> <load_mode>true</load_mode>
>> > >>> <load_mtd>true</load_mtd>
>> > >>> <load_mpr>false</load_mpr>
>> > >>>
>> > >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>> > >>> <load_val>
>> > >>> <field name="model">
>> > >>> <val>arw</val>
>> > >>> <val>nmm</val>
>> > >>> </field>
>> > >>>
>> > >>> <field name="valid_time">
>> > >>> <date_list name="folder_dates"/>
>> > >>> </field>
>> > >>>
>> > >>> <field name="vx_mask">
>> > >>> <val>FULL</val>
>> > >>> <val>SWC</val>
>> > >>> </field>
>> > >>> </load_val>
>> > >>> </load_spec>
>> > >>>
>> > >>> Are similar settings allowed with <plot_spec>? I see
<connection>,
>> > >>> which seems carried over. I'm looking to specify my
folder_tmpl in
>> > >>> <plot_spec>, which according to Tara, should be possible.
Since my
>> > >>> directory structure looks like this:
>> > >>>
>> > >>>
>> > >>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>> > >>>
>> > >>> I would do something like this I would think:
>> > >>>
>> > >>> .......
>> > >>>
>> > >>> <date_list name="folder_dates">
>> > >>> <start>20200612/start>
>> > >>> <end><date_offset><day_offset>0</day_offset></end>
>> > >>> <inc>86400</inc>
>> > >>> <format>yyyyMMdd</format>
>> > >>> </date_list>
>> > >>>
>> > >>> <folder_tmpl>
>> > >>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>> > >>> </folder_tmpl>
>> > >>> <load_val>
>> > >>> <field name="model">
>> > >>> <val>prod</val>
>> > >>> </field>
>> > >>>
>> > >>> <field name="valid_time">
>> > >>> <date_list name="folder_dates"/>
>> > >>> </field>
>> > >>>
>> > >>> ......
>> > >>>
>> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
Affiliate <
>> > >>> edward.strobach at noaa.gov> wrote:
>> > >>>
>> > >>>> Thank you. This is a very helpful example
>> > >>>>
>> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
>> > >>>> <met_help at ucar.edu>
>> > >>>> wrote:
>> > >>>>
>> > >>>>> Hi,
>> > >>>>> Here is some examples how to use <date_range> and
>> > >>>>> <date_range_list> in
>> > >>>>> batch XML.
>> > >>>>> First, you need to defined your range. You can have more
then one
>> > >>>>> <date_range>:
>> > >>>>> <date_range name="may">
>> > >>>>> <start>2019-05-01 00:00:00</start>
>> > >>>>> <end>2019-05-31 23:59:59</end>
>> > >>>>> </date_range>
>> > >>>>> <date_range name="march">
>> > >>>>> <start>2019-03-01 00:00:00</start>
>> > >>>>> <end>2019-03-31 23:59:59</end>
>> > >>>>> </date_range>
>> > >>>>>
>> > >>>>> and then use one or all date_ranges:
>> > >>>>> <plot_fix>
>> > >>>>> <field name="fcst_init_beg">
>> > >>>>> <date_range name="march"/>
>> > >>>>> <date_range name="may"/>
>> > >>>>> </field>...
>> > >>>>> This configuration would include all dates for March and
May of
>> > >>>>> 2019.
>> > >>>>>
>> > >>>>> If you need more flexible configuration you can use
>> > >>>>> date_range_list.
>> > >>>>> For example, you want to select first 6 hours every day
beetween
>> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would
look
>> > >>>>> like this:
>> > >>>>> <date_range_list name = 'all'>
>> > >>>>> <range_start>2020-01-01 00:00:00</range_start>
>> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
>> > >>>>> <range_length>360</range_length>
>> > >>>>> <inc>1440</inc>
>> > >>>>> </date_range_list>
>> > >>>>> where the 1st and 2nd values are the start and end,
>> > >>>>> range_length - the length of a range in minutes(6 houres)
>> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
>> > >>>>> This example would create following ranges:
>> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>> > >>>>>
>> > >>>>> Also you can add more parameters to <plot_fix> to filter
your
>> > >>>>> data.
>> > >>>>> For example, if you want to select only dates with 00:00
you can
>> > >>>>> specify
>> > >>>>> <field name="valid_hour">
>> > >>>>> <val>00</val>
>> > >>>>> </field>
>> > >>>>>
>> > >>>>> Tatiana
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
>> > >>>>> > and if there isn't a better way, I wonder the whereabouts
of a
>> > >>>>> > separate
>> > >>>>> > script that pipes dates (and other relevant information)
into
>> > >>>>> > the XML
>> > >>>>> > file
>> > >>>>> > used to generate the results.
>> > >>>>> >
>> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
>> > >>>>> > Affiliate <
>> > >>>>> > edward.strobach at noaa.gov> wrote:
>> > >>>>> >
>> > >>>>> > > Thanks Tatiana. These resources are helpful.
>> > >>>>> > >
>> > >>>>> > > One last question. Simply typing lines like this in
can take
>> > >>>>> > > forever:
>> > >>>>> > >
>> > >>>>> > > <plot_fix>
>> > >>>>> > > <field equalize="false" name="fcst_init_beg">
>> > >>>>> > > <set name="fcst_init_beg_0">
>> > >>>>> > > <val>2019-05-01 12:00:00</val>
>> > >>>>> > > <val>2019-05-02 06:00:00</val>
>> > >>>>> > > <val>2019-05-02 12:00:00</val>
>> > >>>>> > > <val>2019-05-03 06:00:00</val>
>> > >>>>> > > <val>2019-05-03 12:00:00</val>
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > > Is there a better way? I see date_range,
date_range_list,
>> > >>>>> > > or
>> > >>>>> > > date_list.
>> > >>>>> > > Can this be used instead of typing individual values?
This
>> > >>>>> > > could
>> > >>>>> > > prove
>> > >>>>> > > pretty daunting if one would have to type in 700
entries each
>> > >>>>> > > time.
>> > >>>>> > >
>> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>> > >>>>> > > <met_help at ucar.edu>
>> > >>>>> > > wrote:
>> > >>>>> > >
>> > >>>>> > >> I found a few problems in your XML.
>> > >>>>> > >> The database you are using doesn't have data with
vx_mask
>> > >>>>> > >> 'abcREGabc'.
>> > >>>>> > >> That is why METviewer returns an empty dataset and
can't
>> > >>>>> > >> create a
>> > >>>>> > >> plot.
>> > >>>>> > >> The other problem is how you define fcst_lead. This
database
>> > >>>>> > >> has
>> > >>>>> > >> values
>> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
include
>> > >>>>> > >> all
>> > >>>>> > >> fcst_leads
>> > >>>>> > >> this variable can be omitted.
>> > >>>>> > >> To minimize problems with variables values, I would
suggest
>> > >>>>> > >> to
>> > >>>>> > >> create XML
>> > >>>>> > >> using GUI and then edit it to fit batch standards.
>> > >>>>> > >> I created a plot using you configurations with GUI
(without
>> > >>>>> vx_mack
>> > >>>>> > >> and
>> > >>>>> > >> plot and axis titles) #20200612_173626. You can load
it back
>> > >>>>> > >> to
>> > >>>>> the
>> > >>>>> > >> page by
>> > >>>>> > >> using the "History" tab (left bar on GUI). Find the
plot,
>> > >>>>> > >> open XML
>> > >>>>> > >> and
>> > >>>>> > >> click on "Load to page". After that you can recreate a
plot
>> > >>>>> > >> and
>> > >>>>> edit
>> > >>>>> > >> the
>> > >>>>> > >> configurations. You can also download a new XML and
edit it
>> > >>>>> > >> for
>> > >>>>> > >> batch.
>> > >>>>> > >> I attached the example of load XML to this message.
>> > >>>>> > >>
>> > >>>>> > >> Description of rds parameters:
>> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
scripts
>> > >>>>> > >> rds_plots - temporary directory where the plots will
be
>> > >>>>> > >> generated
>> > >>>>> > >> rds_data - temporary directory where the data
retrieved from
>> > >>>>> > >> the
>> > >>>>> > >> database
>> > >>>>> > >> and other intermediate files will be stored
>> > >>>>> > >> rds_scripts - temporary directory where R scripts for
this
>> > >>>>> > >> run
>> > >>>>> will
>> > >>>>> > >> be
>> > >>>>> > >> stored
>> > >>>>> > >> After the execution is done all the files from
temporary
>> > >>>>> directories
>> > >>>>> > >> would be removed.
>> > >>>>> > >>
>> > >>>>> > >>
>> > >>>>> > >> We don't have the offitial user guide for METviewer.
There
>> > >>>>> > >> are
>> > >>>>> some
>> > >>>>> > >> instructions on how to crate XML config files:
>> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>> > >>>>> > >>
>> > >>>>> > >> Tatiana
>> > >>>>> > >>
>> > >>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
>> > >>>>> > >> > Hi,
>> > >>>>> > >> >
>> > >>>>> > >> > I'm attaching the xml below, but am having hard time
>> > >>>>> > >> > following
>> > >>>>> how
>> > >>>>> > >> > to
>> > >>>>> > >> > go
>> > >>>>> > >> > about the other steps since I have zero experience
on how
>> > >>>>> > >> > to do
>> > >>>>> > >> > these
>> > >>>>> > >> > things.
>> > >>>>> > >> >
>> > >>>>> > >> > #2. All parameters in <folders> section should NOT
be
>> > >>>>> > >> > changed
>> > >>>>> and
>> > >>>>> > >> > will
>> > >>>>> > >> > be
>> > >>>>> > >> > replaced by other scripts during the batch
execution. The
>> > >>>>> > >> > data
>> > >>>>> for
>> > >>>>> > >> > the
>> > >>>>> > >> > batch mode is coming from the database and not from
files.
>> > >>>>> > >> > The
>> > >>>>> > >> > name of
>> > >>>>> > >> > the
>> > >>>>> > >> > database is in <database> tag.
>> > >>>>> > >> >
>> > >>>>> > >> > I understand that nothing should be changed in the
folders
>> > >>>>> > >> > section,
>> > >>>>> > >> > but
>> > >>>>> > >> > there is no indication how and where these variables
are
>> > >>>>> > >> > being
>> > >>>>> > >> > defined.
>> > >>>>> > >> > Grepping these names of the files available does not
>> > >>>>> > >> > reveal
>> > >>>>> formal
>> > >>>>> > >> > definitions of these variables. If I understood
exactly
>> > >>>>> > >> > how
>> > >>>>> these
>> > >>>>> > >> > were
>> > >>>>> > >> > being defined, then I would be able connect this
better.
>> > >>>>> > >> >
>> > >>>>> > >> > #3. Users rum MET and produce MET output files. They
use
>> > >>>>> METviewer
>> > >>>>> > >> > load
>> > >>>>> > >> > mode to load these files to the specific database.
In the
>> > >>>>> > >> > config
>> > >>>>> > >> > file
>> > >>>>> > >> > for
>> > >>>>> > >> > this process they also can specify the database
group. For
>> > >>>>> example
>> > >>>>> > >> > database
>> > >>>>> > >> > mv_dev is in Test12 group.
>> > >>>>> > >> > After the data has been loaded to the database it is
>> > >>>>> > >> > visible and
>> > >>>>> > >> > available
>> > >>>>> > >> > in the GUI and also can be used by the batch mode.
>> > >>>>> > >> > You can always add more data to the database using
the
>> > >>>>> > >> > same load
>> > >>>>> > >> > mode.
>> > >>>>> > >> >
>> > >>>>> > >> > Is there a guide for me to understand the process
that you
>> > >>>>> > >> > are
>> > >>>>> > >> > referring to?
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via
RT
>> > >>>>> > >> > <met_help at ucar.edu>
>> > >>>>> > >> > wrote:
>> > >>>>> > >> >
>> > >>>>> > >> > > Hi Edward,
>> > >>>>> > >> > > #1. I don't have an access to NOAA servers and
can't
>> > >>>>> > >> > > view your
>> > >>>>> > >> > > XML
>> > >>>>> > >> > > for the
>> > >>>>> > >> > > batch mode. Please, attache it to this issue and I
will
>> > >>>>> > >> > > take a
>> > >>>>> > >> > > look.
>> > >>>>> > >> > >
>> > >>>>> > >> > > #2. All parameters in <folders> section should NOT
be
>> > >>>>> > >> > > changed
>> > >>>>> > >> > > and
>> > >>>>> > >> > > will be
>> > >>>>> > >> > > replaced by other scripts during the batch
execution.
>> > >>>>> > >> > > The data
>> > >>>>> > >> > > for
>> > >>>>> > >> > > the
>> > >>>>> > >> > > batch mode is coming from the database and not
from
>> > >>>>> > >> > > files. The
>> > >>>>> > >> > > name
>> > >>>>> > >> > > of the
>> > >>>>> > >> > > database is in <database> tag.
>> > >>>>> > >> > >
>> > >>>>> > >> > > #3. Users rum MET and produce MET output files.
They use
>> > >>>>> > >> > > METviewer
>> > >>>>> > >> > > load
>> > >>>>> > >> > > mode to load these files to the specific database.
In
>> > >>>>> > >> > > the
>> > >>>>> config
>> > >>>>> > >> > > file
>> > >>>>> > >> > > for
>> > >>>>> > >> > > this process they also can specify the database
group.
>> > >>>>> > >> > > For
>> > >>>>> > >> > > example
>> > >>>>> > >> > > database
>> > >>>>> > >> > > mv_dev is in Test12 group.
>> > >>>>> > >> > > After the data has been loaded to the database it
is
>> > >>>>> > >> > > visible
>> > >>>>> and
>> > >>>>> > >> > > available
>> > >>>>> > >> > > in the GUI and also can be used by the batch mode.
>> > >>>>> > >> > > You can always add more data to the database using
the
>> > >>>>> > >> > > same
>> > >>>>> load
>> > >>>>> > >> > > mode.
>> > >>>>> > >> > >
>> > >>>>> > >> > > Tatiana
>> > >>>>> > >> > >
>> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
edward.strobach at noaa.gov
>> > >>>>> > >> > > wrote:
>> > >>>>> > >> > > > Good morning,
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Over the last two weeks I've made an effort to
>> > >>>>> > >> > > > understand
>> > >>>>> > >> > > > metviewer,
>> > >>>>> > >> > > > test
>> > >>>>> > >> > > > different cases using the history tab, changing
>> > >>>>> > >> > > > information
>> > >>>>> to
>> > >>>>> > >> > > > see
>> > >>>>> > >> > > > what
>> > >>>>> > >> > > > works and what does not, and make connections
between
>> > >>>>> > >> > > > the
>> > >>>>> GUI
>> > >>>>> > >> > > > settings
>> > >>>>> > >> > > > and
>> > >>>>> > >> > > > the XML file. Although my comfort level with
>> > >>>>> > >> > > > metviewer has
>> > >>>>> > >> > > > significantly
>> > >>>>> > >> > > > increased during this time, there are some
remaining
>> > >>>>> > >> > > > issues
>> > >>>>> > >> > > > that
>> > >>>>> > >> > > > I'm
>> > >>>>> > >> > > > hoping
>> > >>>>> > >> > > > to clear up.
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > 1. Problem changing XML file and running with
>> > >>>>> > >> > > > different
>> > >>>>> > >> > > > statistic
>> > >>>>> > >> > > > and
>> > >>>>> > >> > > > plot
>> > >>>>> > >> > > > type:
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me
the
>> > >>>>> > >> > > > working
>> > >>>>> > >> > > > directory
>> > >>>>> > >> > > > to get
>> > >>>>> > >> > > > started which I copied over:
>> > >>>>> > >> > > >
>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > He has a test.sh script that I adjusted and used
to
>> > >>>>> > >> > > > run his
>> > >>>>> > >> > > > example.
>> > >>>>> > >> > > > running of the script "bash test.sh" produced a
CSI
>> > >>>>> > >> > > > plot
>> > >>>>> that
>> > >>>>> > >> > > > was
>> > >>>>> > >> > > > stored
>> > >>>>> > >> > > > here:
>> > >>>>> > >> > > >
>> > >>>>> > >> > >
>> > >>>>> > >>
>> > >>>>>
>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Running the script produced the expected result.
The
>> > >>>>> > >> > > > next
>> > >>>>> > >> > > > natural
>> > >>>>> > >> > > > step was
>> > >>>>> > >> > > > to copy hochum_xml.template and create a new xml
file
>> > >>>>> > >> > > > called
>> > >>>>> > >> > > > Strobach_xml.template in order to create a time
series
>> > >>>>> > >> > > > where
>> > >>>>> > >> > > > the
>> > >>>>> > >> > > > independent time is fcst_valid_beg instead of
>> > >>>>> > >> > > > obs_thresh.
>> > >>>>> > >> > > > Accordingly,
>> > >>>>> > >> > > > I
>> > >>>>> > >> > > > made changes to the fixed fields (removing
>> > >>>>> > >> > > > fcst_valid_beg)
>> > >>>>> and
>> > >>>>> > >> > > > changed
>> > >>>>> > >> > > > the
>> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
tried).
>> > >>>>> > >> > > > The
>> > >>>>> error
>> > >>>>> > >> > > > that
>> > >>>>> > >> > > > resulted
>> > >>>>> > >> > > > is as follows:
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
>> > >>>>> > >> > > > 13:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 01
>> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
>> > >>>>> > >> > > > 16:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 01
>> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
>> > >>>>> > >> > > > 19:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 01
>> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
>> > >>>>> > >> > > > 22:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 01
>> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
>> > >>>>> > >> > > > 01:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 02
>> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
>> > >>>>> > >> > > > 04:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 02
>> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
>> > >>>>> > >> > > > 07:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 02
>> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
>> > >>>>> > >> > > > 10:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 02
>> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>> > >>>>> > >> > > > AND ld.fmean != -9999;
>> > >>>>> > >> > > > No data was returned from database
>> > >>>>> > >> > > > 2020-06-09
>> > >>>>> > >> > > >
>> > >>>>> > >> > >
>> > >>>>> > >>
>> > >>>>>
>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>> > >>>>> > >> > > > Source) -1 printFormattedTable - ** ERROR:
Caught
>> > >>>>> > >> > > > class
>> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>> > >>>>> > >> > > > printFormattedTable(ResultSet
>> > >>>>> > >> > > > res): result set contained no data
>> > >>>>> > >> > > > ** ERROR: Caught class
>> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >>>>> > >> > > > in
>> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
>> > >>>>> > >> > > > contained no
>> > >>>>> > >> > > > data
>> > >>>>> > >> > > > ** ERROR: Caught class
>> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >>>>> > >> > > > in
>> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
>> > >>>>> > >> > > > contained no
>> > >>>>> > >> > > > data
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Apparently there was no data during this time.
I
>> > >>>>> > >> > > > tried
>> > >>>>> > >> > > > changing
>> > >>>>> > >> > > > the
>> > >>>>> > >> > > > range
>> > >>>>> > >> > > > of valid times and changed the valid time to
different
>> > >>>>> > >> > > > days
>> > >>>>> > >> > > > just in
>> > >>>>> > >> > > > case
>> > >>>>> > >> > > > data was missing. The issue was the same. To
confirm
>> > >>>>> > >> > > > my
>> > >>>>> > >> > > > logic, I
>> > >>>>> > >> > > > reproduced my attempted approach on metviewer,
which I
>> > >>>>> > >> > > > found
>> > >>>>> > >> > > > to
>> > >>>>> > >> > > > work.
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > 2. Running in batch mode
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > I think I lack an understanding of how to set up
batch
>> > >>>>> > >> > > > mode
>> > >>>>> > >> > > > because
>> > >>>>> > >> > > > I
>> > >>>>> > >> > > > don't
>> > >>>>> > >> > > > see where these definitions are being explicitly
>> > >>>>> > >> > > > defined:
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > <plot_spec>
>> > >>>>> > >> > > > <connection>
>> > >>>>> > >> > > > <host>rds_host:3306</host>
>> > >>>>> > >> > > >
>> > >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>> > >>>>> > >> > > > <user>rds_user</user>
>> > >>>>> > >> > > > <password>rds_pwd</password>
>> > >>>>> > >> > > > </connection>
>> > >>>>> > >> > > > <rscript>Rscript</rscript>
>> > >>>>> > >> > > > <folders>
>> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>> > >>>>> > >> > > > <plots>rds_plots</plots>
>> > >>>>> > >> > > > <data>rds_data</data>
>> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>> > >>>>> > >> > > > </folders>
>> > >>>>> > >> > > > <plot>
>> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > I see the <folders> section, but grepping all
>> > >>>>> > >> > > > available
>> > >>>>> files
>> > >>>>> > >> > > > does
>> > >>>>> > >> > > > not
>> > >>>>> > >> > > > reveal how rds_R_work is being defined, as well
as
>> > >>>>> > >> > > > other
>> > >>>>> > >> > > > similarly
>> > >>>>> > >> > > > defined
>> > >>>>> > >> > > > variables. I think if I know how these are
being
>> > >>>>> > >> > > > defined,
>> > >>>>> > >> > > > then
>> > >>>>> > >> > > > I'll
>> > >>>>> > >> > > > be
>> > >>>>> > >> > > > able to pipe in my information and process the
data
>> > >>>>> > >> > > > using
>> > >>>>> met
>> > >>>>> > >> > > > and
>> > >>>>> > >> > > > produce
>> > >>>>> > >> > > > results in batch mode. Am I right on that?
Also, if
>> > >>>>> > >> > > > I want
>> > >>>>> > >> > > > to use
>> > >>>>> > >> > > > obs for
>> > >>>>> > >> > > > verification, do I add another line in the XML
file
>> > >>>>> > >> > > > that
>> > >>>>> > >> > > > designates
>> > >>>>> > >> > > > file
>> > >>>>> > >> > > > location of obs? I assume that other steps have
to be
>> > >>>>> > >> > > > run
>> > >>>>> > >> > > > before
>> > >>>>> > >> > > > all
>> > >>>>> > >> > > > this. If data is in prepbufr form, for example,
I
>> > >>>>> > >> > > > would
>> > >>>>> have
>> > >>>>> > >> > > > to
>> > >>>>> > >> > > > separately
>> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I would
want to
>> > >>>>> > >> > > > do that
>> > >>>>> > >> > > > as
>> > >>>>> > >> > > > well.
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > 3. Communicating results to metviewer online
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
here -
>> > >>>>> > >> > > > see
>> > >>>>> > >> > > > attachment
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > How do I get the results I generate to be
uploaded to
>> > >>>>> > >> > > > this
>> > >>>>> > >> > > > location?
>> > >>>>> > >> > > > It's
>> > >>>>> > >> > > > not clear how others are setting this up.
>> > >>>>> > >> > >
>> > >>>>> > >> > >
>> > >>>>> > >> > >
>> > >>>>> > >> > >
>> > >>>>> > >>
>> > >>>>> > >>
>> > >>>>> > >>
>> > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
rds_user
>> > >>>>> rds_pwd
>> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
>> > >>>>> rds_scripts
>> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
CMAQAODPRODHIGH
>> > >>>>> 2020-05-12
>> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-
05-12
>> > >>>>> 20:00:00
>> > >>>>> > >> 2020-05-12
>> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-
05-13
>> > >>>>> 10:00:00
>> > >>>>> > >> 2020-05-13
>> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-
05-13
>> > >>>>> 14:00:00
>> > >>>>> > >> 2020-05-13
>> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-
05-13
>> > >>>>> 18:00:00
>> > >>>>> > >> 2020-05-13
>> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-
05-13
>> > >>>>> 22:00:00
>> > >>>>> > >> 2020-05-13
>> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-
05-14
>> > >>>>> 12:00:00
>> > >>>>> > >> true
>> > >>>>> > >> median plot_20200612_173626.data
plot_20200612_173626.png
>> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
>> > >>>>> > >> false list() list() false false false false false
false true
>> > >>>>> > >> false
>> > >>>>> > >> false
>> > >>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
>> > >>>>> > >> c(1,1,0) 1 2
>> > >>>>> > >> 1.4 -2
>> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
>> > >>>>> > >> #cccccc 1
>> > >>>>> > >> listX 1
>> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25) 3
>> > >>>>> > >> 0.8 1
>> > >>>>> > >> #333333
>> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
c(TRUE,TRUE)
>> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
>> > >>>>> c(1,1)
>> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > > --
>> > >>>>> > > Edward Strobach
>> > >>>>> > > EMC/NCEP/NWS/
>> > >>>>> > > IMSG Contractor
>> > >>>>> > > Cubicle#: 2029
>> > >>>>> > > 301-683-3717
>> > >>>>> > >
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>
>> > >>>> --
>> > >>>> Edward Strobach
>> > >>>> EMC/NCEP/NWS/
>> > >>>> IMSG Contractor
>> > >>>> Cubicle#: 2029
>> > >>>> 301-683-3717
>> > >>>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Edward Strobach
>> > >>> EMC/NCEP/NWS/
>> > >>> IMSG Contractor
>> > >>> Cubicle#: 2029
>> > >>> 301-683-3717
>> > >>>
>> > >>
>> > >>
>> > >> --
>> > >> Edward Strobach
>> > >> EMC/NCEP/NWS/
>> > >> IMSG Contractor
>> > >> Cubicle#: 2029
>> > >> 301-683-3717
>> > >>
>> > >
>> > >
>> > > --
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/
>> > > IMSG Contractor
>> > > Cubicle#: 2029
>> > > 301-683-3717
>> > >
>>
>>
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Wed Jun 17 09:40:52 2020
Only databases that have a name that starts with 'mv_'are visible in
GUI. That is why 'Ed_Strobach' is not on the list.
If database doesn't have a group it will end up in "NO GROUP" group.
You specify the name for the group you want to see your database in
when you load data.
There is a tag <group> where you put the group name.
Tatiana
On Tue Jun 16 15:40:50 2020, edward.strobach at noaa.gov wrote:
> Thanks Tatiana. Yes, I've created a database which seems different
> from
> mv_group. How can one create their own group so that I can store and
> access
> the database. The database exists but cannot be viewed on the
> metviewer
> GUI. The list starts with EMC HREFv3 and ends with vhagerty. I
named
> my
> database Ed_Strobach, but that was really intended for the group
name.
>
> On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > We don't have database and METviewer running on wcoss.
> > Instead, users use METviewer-wcoss scripts that help to connect to
> > and use
> > METviewer that is running on AWS. That is why XML connection and
> > folder
> > sections look somewhat different than once from the Docs page.
> > To create a database on AWS you need to run mv_create_db_on_aws.sh
> > mv_create_db_on_aws.sh <user_name> <database_name>
> > For data loading :
> > mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> > And for batch:
> > mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> >
> > These scripts copy XML and data files if needed to AWS, adjust
data
> > path
> > and users credentials, run METviewer and copy results back to
wcoss.
> >
> > Tatiana
> > On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> > > lastly, it does seem thatmysql -u[db_username] -p[db_password]
> > > -e'create
> > > database [db_name]
> > >
> > > would need to be used to create a database. I assume that an
sql
> > > module is
> > > available on wcoss; I know you don't have access to those
machines
> > > Tatiana. If I create my own database with my own name, then I
> > > should
> > > be
> > > able to proceed with project specific tasks under that tab.
> > > Something
> > > like
> > > mv_EdwardStrobach.
> > >
> > > I can then create mv_cmaq_prod_v_para which I add data to using
> > > load_spec.
> > > Once that step is done, then I can run plot_spec. That's my
> > > takeaway
> > > currently
> > >
> > > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate
<
> > > edward.strobach at noaa.gov> wrote:
> > >
> > > > Actually, it appears that I need to run the database loading
> > > > module
> > > > first
> > > > if I want to change/add a data source. From there I can run
the
> > > > plotting
> > > > batch module. Is that right? So if I run the database
loading
> > > > module,
> > > > then I can specify my dates and other related info specific to
> > > > the
> > > > dataset. If I create my database with a unique name, then it
> > > > should
> > > > be
> > > > recognized on the metviewer gui, presumably.
> > > >
> > > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
> > > > <
> > > > edward.strobach at noaa.gov> wrote:
> > > >
> > > >> It seems that those options must be treated separately
outside
> > > >> of
> > > >> plot_spec. Is it acceptable to create both a load_spec and
> > > >> plot_spec in
> > > >> the same xml file? Would the definitions in load_spec be
able
> > > >> to
> > > >> carry
> > > >> over into plot_spec? It appears that data loading and
plotting
> > > >> are
> > > >> treated
> > > >> separately.
> > > >>
> > > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
Affiliate
> > > >> <
> > > >> edward.strobach at noaa.gov> wrote:
> > > >>
> > > >>> I do plan to use date_range; however, in an example provided
by
> > > >>> the
> > > >>> link
> > > >>> I see this:
> > > >>>
> > > >>> <load_spec>
> > > >>> <connection>
> > > >>> <host>kemosabe:3306</host>
> > > >>> <database>metvdb_hwt</database>
> > > >>> <user>pgoldenb</user>
> > > >>> <password>pgoldenb</password>
> > > >>> </connection>
> > > >>>
> > > >>> <date_list name="folder_dates">
> > > >>> <start>2010051914V</start>
> > > >>> <end><date_offset><day_offset>0</day_offset><hour>6
> > > >>> </hour></date_offset></end>
> > > >>> <inc>3600</inc>
> > > >>> <format>yyyyMMddHH'V'</format>
> > > >>> </date_list>
> > > >>>
> > > >>> <met_version>V3.0</met_version>
> > > >>>
> > > >>> <verbose>false</verbose>
> > > >>> <insert_size>1</insert_size>
> > > >>> <mode_header_db_check>true</mode_header_db_check>
> > > >>> <drop_indexes>false</drop_indexes>
> > > >>> <apply_indexes>true</apply_indexes>
> > > >>> <group>Group name</group>
> > > >>> <load_stat>true</load_stat>
> > > >>> <load_mode>true</load_mode>
> > > >>> <load_mtd>true</load_mtd>
> > > >>> <load_mpr>false</load_mpr>
> > > >>>
> > > >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > > >>> <load_val>
> > > >>> <field name="model">
> > > >>> <val>arw</val>
> > > >>> <val>nmm</val>
> > > >>> </field>
> > > >>>
> > > >>> <field name="valid_time">
> > > >>> <date_list name="folder_dates"/>
> > > >>> </field>
> > > >>>
> > > >>> <field name="vx_mask">
> > > >>> <val>FULL</val>
> > > >>> <val>SWC</val>
> > > >>> </field>
> > > >>> </load_val>
> > > >>> </load_spec>
> > > >>>
> > > >>> Are similar settings allowed with <plot_spec>? I see
> > > >>> <connection>,
> > > >>> which seems carried over. I'm looking to specify my
> > > >>> folder_tmpl in
> > > >>> <plot_spec>, which according to Tara, should be possible.
> > > >>> Since my
> > > >>> directory structure looks like this:
> > > >>>
> > > >>>
> > > >>>
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > > >>>
> > > >>> I would do something like this I would think:
> > > >>>
> > > >>> .......
> > > >>>
> > > >>> <date_list name="folder_dates">
> > > >>> <start>20200612/start>
> > > >>> <end><date_offset><day_offset>0</day_offset></end>
> > > >>> <inc>86400</inc>
> > > >>> <format>yyyyMMdd</format>
> > > >>> </date_list>
> > > >>>
> > > >>> <folder_tmpl>
> > > >>>
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > > >>> </folder_tmpl>
> > > >>> <load_val>
> > > >>> <field name="model">
> > > >>> <val>prod</val>
> > > >>> </field>
> > > >>>
> > > >>> <field name="valid_time">
> > > >>> <date_list name="folder_dates"/>
> > > >>> </field>
> > > >>>
> > > >>> ......
> > > >>>
> > > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
> > > >>> Affiliate <
> > > >>> edward.strobach at noaa.gov> wrote:
> > > >>>
> > > >>>> Thank you. This is a very helpful example
> > > >>>>
> > > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> > > >>>> <met_help at ucar.edu>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi,
> > > >>>>> Here is some examples how to use <date_range> and
> > > >>>>> <date_range_list> in
> > > >>>>> batch XML.
> > > >>>>> First, you need to defined your range. You can have more
then
> > > >>>>> one
> > > >>>>> <date_range>:
> > > >>>>> <date_range name="may">
> > > >>>>> <start>2019-05-01 00:00:00</start>
> > > >>>>> <end>2019-05-31 23:59:59</end>
> > > >>>>> </date_range>
> > > >>>>> <date_range name="march">
> > > >>>>> <start>2019-03-01 00:00:00</start>
> > > >>>>> <end>2019-03-31 23:59:59</end>
> > > >>>>> </date_range>
> > > >>>>>
> > > >>>>> and then use one or all date_ranges:
> > > >>>>> <plot_fix>
> > > >>>>> <field name="fcst_init_beg">
> > > >>>>> <date_range name="march"/>
> > > >>>>> <date_range name="may"/>
> > > >>>>> </field>...
> > > >>>>> This configuration would include all dates for March and
May
> > > >>>>> of
> > > >>>>> 2019.
> > > >>>>>
> > > >>>>> If you need more flexible configuration you can use
> > > >>>>> date_range_list.
> > > >>>>> For example, you want to select first 6 hours every day
> > > >>>>> beetween
> > > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
> > > >>>>> look
> > > >>>>> like this:
> > > >>>>> <date_range_list name = 'all'>
> > > >>>>> <range_start>2020-01-01 00:00:00</range_start>
> > > >>>>> <range_end>2020-01-05 00:00:00</range_end>
> > > >>>>> <range_length>360</range_length>
> > > >>>>> <inc>1440</inc>
> > > >>>>> </date_range_list>
> > > >>>>> where the 1st and 2nd values are the start and end,
> > > >>>>> range_length - the length of a range in minutes(6 houres)
> > > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> > > >>>>> This example would create following ranges:
> > > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> > > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> > > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> > > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> > > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> > > >>>>>
> > > >>>>> Also you can add more parameters to <plot_fix> to filter
your
> > > >>>>> data.
> > > >>>>> For example, if you want to select only dates with 00:00
you
> > > >>>>> can
> > > >>>>> specify
> > > >>>>> <field name="valid_hour">
> > > >>>>> <val>00</val>
> > > >>>>> </field>
> > > >>>>>
> > > >>>>> Tatiana
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
> > > >>>>> > and if there isn't a better way, I wonder the
whereabouts
> > > >>>>> > of a
> > > >>>>> > separate
> > > >>>>> > script that pipes dates (and other relevant information)
> > > >>>>> > into
> > > >>>>> > the XML
> > > >>>>> > file
> > > >>>>> > used to generate the results.
> > > >>>>> >
> > > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> > > >>>>> > Affiliate <
> > > >>>>> > edward.strobach at noaa.gov> wrote:
> > > >>>>> >
> > > >>>>> > > Thanks Tatiana. These resources are helpful.
> > > >>>>> > >
> > > >>>>> > > One last question. Simply typing lines like this in
can
> > > >>>>> > > take
> > > >>>>> > > forever:
> > > >>>>> > >
> > > >>>>> > > <plot_fix>
> > > >>>>> > > <field equalize="false" name="fcst_init_beg">
> > > >>>>> > > <set name="fcst_init_beg_0">
> > > >>>>> > > <val>2019-05-01 12:00:00</val>
> > > >>>>> > > <val>2019-05-02 06:00:00</val>
> > > >>>>> > > <val>2019-05-02 12:00:00</val>
> > > >>>>> > > <val>2019-05-03 06:00:00</val>
> > > >>>>> > > <val>2019-05-03 12:00:00</val>
> > > >>>>> > >
> > > >>>>> > >
> > > >>>>> > > Is there a better way? I see date_range,
> > > >>>>> > > date_range_list,
> > > >>>>> > > or
> > > >>>>> > > date_list.
> > > >>>>> > > Can this be used instead of typing individual values?
> > > >>>>> > > This
> > > >>>>> > > could
> > > >>>>> > > prove
> > > >>>>> > > pretty daunting if one would have to type in 700
entries
> > > >>>>> > > each
> > > >>>>> > > time.
> > > >>>>> > >
> > > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > > >>>>> > > <met_help at ucar.edu>
> > > >>>>> > > wrote:
> > > >>>>> > >
> > > >>>>> > >> I found a few problems in your XML.
> > > >>>>> > >> The database you are using doesn't have data with
> > > >>>>> > >> vx_mask
> > > >>>>> > >> 'abcREGabc'.
> > > >>>>> > >> That is why METviewer returns an empty dataset and
can't
> > > >>>>> > >> create a
> > > >>>>> > >> plot.
> > > >>>>> > >> The other problem is how you define fcst_lead. This
> > > >>>>> > >> database
> > > >>>>> > >> has
> > > >>>>> > >> values
> > > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
> > > >>>>> > >> include
> > > >>>>> > >> all
> > > >>>>> > >> fcst_leads
> > > >>>>> > >> this variable can be omitted.
> > > >>>>> > >> To minimize problems with variables values, I would
> > > >>>>> > >> suggest
> > > >>>>> > >> to
> > > >>>>> > >> create XML
> > > >>>>> > >> using GUI and then edit it to fit batch standards.
> > > >>>>> > >> I created a plot using you configurations with GUI
> > > >>>>> > >> (without
> > > >>>>> vx_mack
> > > >>>>> > >> and
> > > >>>>> > >> plot and axis titles) #20200612_173626. You can load
it
> > > >>>>> > >> back
> > > >>>>> > >> to
> > > >>>>> the
> > > >>>>> > >> page by
> > > >>>>> > >> using the "History" tab (left bar on GUI). Find the
> > > >>>>> > >> plot,
> > > >>>>> > >> open XML
> > > >>>>> > >> and
> > > >>>>> > >> click on "Load to page". After that you can recreate
a
> > > >>>>> > >> plot
> > > >>>>> > >> and
> > > >>>>> edit
> > > >>>>> > >> the
> > > >>>>> > >> configurations. You can also download a new XML and
edit
> > > >>>>> > >> it
> > > >>>>> > >> for
> > > >>>>> > >> batch.
> > > >>>>> > >> I attached the example of load XML to this message.
> > > >>>>> > >>
> > > >>>>> > >> Description of rds parameters:
> > > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
> > > >>>>> > >> scripts
> > > >>>>> > >> rds_plots - temporary directory where the plots will
be
> > > >>>>> > >> generated
> > > >>>>> > >> rds_data - temporary directory where the data
retrieved
> > > >>>>> > >> from
> > > >>>>> > >> the
> > > >>>>> > >> database
> > > >>>>> > >> and other intermediate files will be stored
> > > >>>>> > >> rds_scripts - temporary directory where R scripts for
> > > >>>>> > >> this
> > > >>>>> > >> run
> > > >>>>> will
> > > >>>>> > >> be
> > > >>>>> > >> stored
> > > >>>>> > >> After the execution is done all the files from
temporary
> > > >>>>> directories
> > > >>>>> > >> would be removed.
> > > >>>>> > >>
> > > >>>>> > >>
> > > >>>>> > >> We don't have the offitial user guide for METviewer.
> > > >>>>> > >> There
> > > >>>>> > >> are
> > > >>>>> some
> > > >>>>> > >> instructions on how to crate XML config files:
> > > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > > >>>>> > >>
> > > >>>>> > >> Tatiana
> > > >>>>> > >>
> > > >>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
> > > >>>>> > >> wrote:
> > > >>>>> > >> > Hi,
> > > >>>>> > >> >
> > > >>>>> > >> > I'm attaching the xml below, but am having hard
time
> > > >>>>> > >> > following
> > > >>>>> how
> > > >>>>> > >> > to
> > > >>>>> > >> > go
> > > >>>>> > >> > about the other steps since I have zero experience
on
> > > >>>>> > >> > how
> > > >>>>> > >> > to do
> > > >>>>> > >> > these
> > > >>>>> > >> > things.
> > > >>>>> > >> >
> > > >>>>> > >> > #2. All parameters in <folders> section should NOT
be
> > > >>>>> > >> > changed
> > > >>>>> and
> > > >>>>> > >> > will
> > > >>>>> > >> > be
> > > >>>>> > >> > replaced by other scripts during the batch
execution.
> > > >>>>> > >> > The
> > > >>>>> > >> > data
> > > >>>>> for
> > > >>>>> > >> > the
> > > >>>>> > >> > batch mode is coming from the database and not from
> > > >>>>> > >> > files.
> > > >>>>> > >> > The
> > > >>>>> > >> > name of
> > > >>>>> > >> > the
> > > >>>>> > >> > database is in <database> tag.
> > > >>>>> > >> >
> > > >>>>> > >> > I understand that nothing should be changed in the
> > > >>>>> > >> > folders
> > > >>>>> > >> > section,
> > > >>>>> > >> > but
> > > >>>>> > >> > there is no indication how and where these
variables
> > > >>>>> > >> > are
> > > >>>>> > >> > being
> > > >>>>> > >> > defined.
> > > >>>>> > >> > Grepping these names of the files available does
not
> > > >>>>> > >> > reveal
> > > >>>>> formal
> > > >>>>> > >> > definitions of these variables. If I understood
> > > >>>>> > >> > exactly
> > > >>>>> > >> > how
> > > >>>>> these
> > > >>>>> > >> > were
> > > >>>>> > >> > being defined, then I would be able connect this
> > > >>>>> > >> > better.
> > > >>>>> > >> >
> > > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
> > > >>>>> > >> > use
> > > >>>>> METviewer
> > > >>>>> > >> > load
> > > >>>>> > >> > mode to load these files to the specific database.
In
> > > >>>>> > >> > the
> > > >>>>> > >> > config
> > > >>>>> > >> > file
> > > >>>>> > >> > for
> > > >>>>> > >> > this process they also can specify the database
group.
> > > >>>>> > >> > For
> > > >>>>> example
> > > >>>>> > >> > database
> > > >>>>> > >> > mv_dev is in Test12 group.
> > > >>>>> > >> > After the data has been loaded to the database it
is
> > > >>>>> > >> > visible and
> > > >>>>> > >> > available
> > > >>>>> > >> > in the GUI and also can be used by the batch mode.
> > > >>>>> > >> > You can always add more data to the database using
the
> > > >>>>> > >> > same load
> > > >>>>> > >> > mode.
> > > >>>>> > >> >
> > > >>>>> > >> > Is there a guide for me to understand the process
that
> > > >>>>> > >> > you
> > > >>>>> > >> > are
> > > >>>>> > >> > referring to?
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via
RT
> > > >>>>> > >> > <met_help at ucar.edu>
> > > >>>>> > >> > wrote:
> > > >>>>> > >> >
> > > >>>>> > >> > > Hi Edward,
> > > >>>>> > >> > > #1. I don't have an access to NOAA servers and
can't
> > > >>>>> > >> > > view your
> > > >>>>> > >> > > XML
> > > >>>>> > >> > > for the
> > > >>>>> > >> > > batch mode. Please, attache it to this issue and
I
> > > >>>>> > >> > > will
> > > >>>>> > >> > > take a
> > > >>>>> > >> > > look.
> > > >>>>> > >> > >
> > > >>>>> > >> > > #2. All parameters in <folders> section should
NOT
> > > >>>>> > >> > > be
> > > >>>>> > >> > > changed
> > > >>>>> > >> > > and
> > > >>>>> > >> > > will be
> > > >>>>> > >> > > replaced by other scripts during the batch
> > > >>>>> > >> > > execution.
> > > >>>>> > >> > > The data
> > > >>>>> > >> > > for
> > > >>>>> > >> > > the
> > > >>>>> > >> > > batch mode is coming from the database and not
from
> > > >>>>> > >> > > files. The
> > > >>>>> > >> > > name
> > > >>>>> > >> > > of the
> > > >>>>> > >> > > database is in <database> tag.
> > > >>>>> > >> > >
> > > >>>>> > >> > > #3. Users rum MET and produce MET output files.
They
> > > >>>>> > >> > > use
> > > >>>>> > >> > > METviewer
> > > >>>>> > >> > > load
> > > >>>>> > >> > > mode to load these files to the specific
database.
> > > >>>>> > >> > > In
> > > >>>>> > >> > > the
> > > >>>>> config
> > > >>>>> > >> > > file
> > > >>>>> > >> > > for
> > > >>>>> > >> > > this process they also can specify the database
> > > >>>>> > >> > > group.
> > > >>>>> > >> > > For
> > > >>>>> > >> > > example
> > > >>>>> > >> > > database
> > > >>>>> > >> > > mv_dev is in Test12 group.
> > > >>>>> > >> > > After the data has been loaded to the database it
is
> > > >>>>> > >> > > visible
> > > >>>>> and
> > > >>>>> > >> > > available
> > > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
> > > >>>>> > >> > > You can always add more data to the database
using
> > > >>>>> > >> > > the
> > > >>>>> > >> > > same
> > > >>>>> load
> > > >>>>> > >> > > mode.
> > > >>>>> > >> > >
> > > >>>>> > >> > > Tatiana
> > > >>>>> > >> > >
> > > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> > > >>>>> > >> > > edward.strobach at noaa.gov
> > > >>>>> > >> > > wrote:
> > > >>>>> > >> > > > Good morning,
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Over the last two weeks I've made an effort to
> > > >>>>> > >> > > > understand
> > > >>>>> > >> > > > metviewer,
> > > >>>>> > >> > > > test
> > > >>>>> > >> > > > different cases using the history tab, changing
> > > >>>>> > >> > > > information
> > > >>>>> to
> > > >>>>> > >> > > > see
> > > >>>>> > >> > > > what
> > > >>>>> > >> > > > works and what does not, and make connections
> > > >>>>> > >> > > > between
> > > >>>>> > >> > > > the
> > > >>>>> GUI
> > > >>>>> > >> > > > settings
> > > >>>>> > >> > > > and
> > > >>>>> > >> > > > the XML file. Although my comfort level with
> > > >>>>> > >> > > > metviewer has
> > > >>>>> > >> > > > significantly
> > > >>>>> > >> > > > increased during this time, there are some
> > > >>>>> > >> > > > remaining
> > > >>>>> > >> > > > issues
> > > >>>>> > >> > > > that
> > > >>>>> > >> > > > I'm
> > > >>>>> > >> > > > hoping
> > > >>>>> > >> > > > to clear up.
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > 1. Problem changing XML file and running with
> > > >>>>> > >> > > > different
> > > >>>>> > >> > > > statistic
> > > >>>>> > >> > > > and
> > > >>>>> > >> > > > plot
> > > >>>>> > >> > > > type:
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me
the
> > > >>>>> > >> > > > working
> > > >>>>> > >> > > > directory
> > > >>>>> > >> > > > to get
> > > >>>>> > >> > > > started which I copied over:
> > > >>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > run his
> > > >>>>> > >> > > > example.
> > > >>>>> > >> > > > running of the script "bash test.sh" produced a
> > > >>>>> > >> > > > CSI
> > > >>>>> > >> > > > plot
> > > >>>>> that
> > > >>>>> > >> > > > was
> > > >>>>> > >> > > > stored
> > > >>>>> > >> > > > here:
> > > >>>>> > >> > > >
> > > >>>>> > >> > >
> > > >>>>> > >>
> > > >>>>>
> >
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Running the script produced the expected
result.
> > > >>>>> > >> > > > The
> > > >>>>> > >> > > > next
> > > >>>>> > >> > > > natural
> > > >>>>> > >> > > > step was
> > > >>>>> > >> > > > to copy hochum_xml.template and create a new
xml
> > > >>>>> > >> > > > file
> > > >>>>> > >> > > > called
> > > >>>>> > >> > > > Strobach_xml.template in order to create a time
> > > >>>>> > >> > > > series
> > > >>>>> > >> > > > where
> > > >>>>> > >> > > > the
> > > >>>>> > >> > > > independent time is fcst_valid_beg instead of
> > > >>>>> > >> > > > obs_thresh.
> > > >>>>> > >> > > > Accordingly,
> > > >>>>> > >> > > > I
> > > >>>>> > >> > > > made changes to the fixed fields (removing
> > > >>>>> > >> > > > fcst_valid_beg)
> > > >>>>> and
> > > >>>>> > >> > > > changed
> > > >>>>> > >> > > > the
> > > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
tried).
> > > >>>>> > >> > > > The
> > > >>>>> error
> > > >>>>> > >> > > > that
> > > >>>>> > >> > > > resulted
> > > >>>>> > >> > > > is as follows:
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
> > > >>>>> > >> > > > 13:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 01
> > > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> > > >>>>> > >> > > > 16:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 01
> > > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> > > >>>>> > >> > > > 19:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 01
> > > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> > > >>>>> > >> > > > 22:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 01
> > > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> > > >>>>> > >> > > > 01:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 02
> > > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> > > >>>>> > >> > > > 04:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 02
> > > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> > > >>>>> > >> > > > 07:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 02
> > > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> > > >>>>> > >> > > > 10:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 02
> > > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> > > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > > >>>>> > >> > > > AND ld.fmean != -9999;
> > > >>>>> > >> > > > No data was returned from database
> > > >>>>> > >> > > > 2020-06-09
> > > >>>>> > >> > > >
> > > >>>>> > >> > >
> > > >>>>> > >>
> > > >>>>>
> >
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > > >>>>> > >> > > > Source) -1 printFormattedTable - ** ERROR:
> > > >>>>> > >> > > > Caught
> > > >>>>> > >> > > > class
> > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> > > >>>>> > >> > > > printFormattedTable(ResultSet
> > > >>>>> > >> > > > res): result set contained no data
> > > >>>>> > >> > > > ** ERROR: Caught class
> > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > > >>>>> > >> > > > in
> > > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> > > >>>>> > >> > > > contained no
> > > >>>>> > >> > > > data
> > > >>>>> > >> > > > ** ERROR: Caught class
> > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > > >>>>> > >> > > > in
> > > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> > > >>>>> > >> > > > contained no
> > > >>>>> > >> > > > data
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Apparently there was no data during this time.
I
> > > >>>>> > >> > > > tried
> > > >>>>> > >> > > > changing
> > > >>>>> > >> > > > the
> > > >>>>> > >> > > > range
> > > >>>>> > >> > > > of valid times and changed the valid time to
> > > >>>>> > >> > > > different
> > > >>>>> > >> > > > days
> > > >>>>> > >> > > > just in
> > > >>>>> > >> > > > case
> > > >>>>> > >> > > > data was missing. The issue was the same. To
> > > >>>>> > >> > > > confirm
> > > >>>>> > >> > > > my
> > > >>>>> > >> > > > logic, I
> > > >>>>> > >> > > > reproduced my attempted approach on metviewer,
> > > >>>>> > >> > > > which I
> > > >>>>> > >> > > > found
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > work.
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > 2. Running in batch mode
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > I think I lack an understanding of how to set
up
> > > >>>>> > >> > > > batch
> > > >>>>> > >> > > > mode
> > > >>>>> > >> > > > because
> > > >>>>> > >> > > > I
> > > >>>>> > >> > > > don't
> > > >>>>> > >> > > > see where these definitions are being
explicitly
> > > >>>>> > >> > > > defined:
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > <plot_spec>
> > > >>>>> > >> > > > <connection>
> > > >>>>> > >> > > > <host>rds_host:3306</host>
> > > >>>>> > >> > > >
> > > >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > > >>>>> > >> > > > <user>rds_user</user>
> > > >>>>> > >> > > > <password>rds_pwd</password>
> > > >>>>> > >> > > > </connection>
> > > >>>>> > >> > > > <rscript>Rscript</rscript>
> > > >>>>> > >> > > > <folders>
> > > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > > >>>>> > >> > > > <plots>rds_plots</plots>
> > > >>>>> > >> > > > <data>rds_data</data>
> > > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > > >>>>> > >> > > > </folders>
> > > >>>>> > >> > > > <plot>
> > > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > I see the <folders> section, but grepping all
> > > >>>>> > >> > > > available
> > > >>>>> files
> > > >>>>> > >> > > > does
> > > >>>>> > >> > > > not
> > > >>>>> > >> > > > reveal how rds_R_work is being defined, as well
as
> > > >>>>> > >> > > > other
> > > >>>>> > >> > > > similarly
> > > >>>>> > >> > > > defined
> > > >>>>> > >> > > > variables. I think if I know how these are
being
> > > >>>>> > >> > > > defined,
> > > >>>>> > >> > > > then
> > > >>>>> > >> > > > I'll
> > > >>>>> > >> > > > be
> > > >>>>> > >> > > > able to pipe in my information and process the
> > > >>>>> > >> > > > data
> > > >>>>> > >> > > > using
> > > >>>>> met
> > > >>>>> > >> > > > and
> > > >>>>> > >> > > > produce
> > > >>>>> > >> > > > results in batch mode. Am I right on that?
Also,
> > > >>>>> > >> > > > if
> > > >>>>> > >> > > > I want
> > > >>>>> > >> > > > to use
> > > >>>>> > >> > > > obs for
> > > >>>>> > >> > > > verification, do I add another line in the XML
> > > >>>>> > >> > > > file
> > > >>>>> > >> > > > that
> > > >>>>> > >> > > > designates
> > > >>>>> > >> > > > file
> > > >>>>> > >> > > > location of obs? I assume that other steps
have
> > > >>>>> > >> > > > to be
> > > >>>>> > >> > > > run
> > > >>>>> > >> > > > before
> > > >>>>> > >> > > > all
> > > >>>>> > >> > > > this. If data is in prepbufr form, for
example, I
> > > >>>>> > >> > > > would
> > > >>>>> have
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > separately
> > > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I would
want
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > do that
> > > >>>>> > >> > > > as
> > > >>>>> > >> > > > well.
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > 3. Communicating results to metviewer online
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
here
> > > >>>>> > >> > > > -
> > > >>>>> > >> > > > see
> > > >>>>> > >> > > > attachment
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > How do I get the results I generate to be
uploaded
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > this
> > > >>>>> > >> > > > location?
> > > >>>>> > >> > > > It's
> > > >>>>> > >> > > > not clear how others are setting this up.
> > > >>>>> > >> > >
> > > >>>>> > >> > >
> > > >>>>> > >> > >
> > > >>>>> > >> > >
> > > >>>>> > >>
> > > >>>>> > >>
> > > >>>>> > >>
> > > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
> > > >>>>> > >> rds_user
> > > >>>>> rds_pwd
> > > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
> > > >>>>> rds_scripts
> > > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
CMAQAODPRODHIGH
> > > >>>>> 2020-05-12
> > > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
> > > >>>>> > >> 05-12
> > > >>>>> 20:00:00
> > > >>>>> > >> 2020-05-12
> > > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
> > > >>>>> > >> 05-13
> > > >>>>> 10:00:00
> > > >>>>> > >> 2020-05-13
> > > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
> > > >>>>> > >> 05-13
> > > >>>>> 14:00:00
> > > >>>>> > >> 2020-05-13
> > > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
> > > >>>>> > >> 05-13
> > > >>>>> 18:00:00
> > > >>>>> > >> 2020-05-13
> > > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
> > > >>>>> > >> 05-13
> > > >>>>> 22:00:00
> > > >>>>> > >> 2020-05-13
> > > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
> > > >>>>> > >> 05-14
> > > >>>>> 12:00:00
> > > >>>>> > >> true
> > > >>>>> > >> median plot_20200612_173626.data
> > > >>>>> > >> plot_20200612_173626.png
> > > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> > > >>>>> > >> false list() list() false false false false false
false
> > > >>>>> > >> true
> > > >>>>> > >> false
> > > >>>>> > >> false
> > > >>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
> > > >>>>> > >> c(1,1,0) 1 2
> > > >>>>> > >> 1.4 -2
> > > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5
3
> > > >>>>> > >> #cccccc 1
> > > >>>>> > >> listX 1
> > > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
> > > >>>>> > >> 3
> > > >>>>> > >> 0.8 1
> > > >>>>> > >> #333333
> > > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
> > > >>>>> > >> c(TRUE,TRUE)
> > > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
> > > >>>>> > >> c(1,1)
> > > >>>>> c(1,1)
> > > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > > >>>>> > >
> > > >>>>> > >
> > > >>>>> > >
> > > >>>>> > > --
> > > >>>>> > > Edward Strobach
> > > >>>>> > > EMC/NCEP/NWS/
> > > >>>>> > > IMSG Contractor
> > > >>>>> > > Cubicle#: 2029
> > > >>>>> > > 301-683-3717
> > > >>>>> > >
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>> --
> > > >>>> Edward Strobach
> > > >>>> EMC/NCEP/NWS/
> > > >>>> IMSG Contractor
> > > >>>> Cubicle#: 2029
> > > >>>> 301-683-3717
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Edward Strobach
> > > >>> EMC/NCEP/NWS/
> > > >>> IMSG Contractor
> > > >>> Cubicle#: 2029
> > > >>> 301-683-3717
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Edward Strobach
> > > >> EMC/NCEP/NWS/
> > > >> IMSG Contractor
> > > >> Cubicle#: 2029
> > > >> 301-683-3717
> > > >>
> > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> >
> >
> >
> >
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Wed Jun 17 10:00:26 2020
Ed, now you should have enough space for the data loading.
Tatiana
On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
> I was basically able to build my xml file but it says I have no
> available
> space. I'm not sure how that can be since I just started adding
data
> to my
> account
>
> scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space
left
> on
> device
> ERROR: Command returned with non-zero status (1): scp -r ./*
> edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
> 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
> TABLE_SCHEMA USER SIZE_MB
> mv_met_g2o_gefs binbin.zhou 140027.31
> mv_met_system_hrefv2_v3 binbin.zhou 71986.11
>
> On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Thanks Tatiana. Yes, I've created a database which seems
different
> > from
> > mv_group. How can one create their own group so that I can store
and
> > access
> > the database. The database exists but cannot be viewed on the
> > metviewer
> > GUI. The list starts with EMC HREFv3 and ends with vhagerty. I
> > named my
> > database Ed_Strobach, but that was really intended for the group
> > name.
> >
> > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> >> We don't have database and METviewer running on wcoss.
> >> Instead, users use METviewer-wcoss scripts that help to connect
to
> >> and
> >> use METviewer that is running on AWS. That is why XML connection
and
> >> folder
> >> sections look somewhat different than once from the Docs page.
> >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
> >> mv_create_db_on_aws.sh <user_name> <database_name>
> >> For data loading :
> >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> >> And for batch:
> >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> >>
> >> These scripts copy XML and data files if needed to AWS, adjust
data
> >> path
> >> and users credentials, run METviewer and copy results back to
wcoss.
> >>
> >> Tatiana
> >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> >> > lastly, it does seem thatmysql -u[db_username] -p[db_password]
> >> > -e'create
> >> > database [db_name]
> >> >
> >> > would need to be used to create a database. I assume that an
sql
> >> > module is
> >> > available on wcoss; I know you don't have access to those
machines
> >> > Tatiana. If I create my own database with my own name, then I
> >> > should
> >> > be
> >> > able to proceed with project specific tasks under that tab.
> >> > Something
> >> > like
> >> > mv_EdwardStrobach.
> >> >
> >> > I can then create mv_cmaq_prod_v_para which I add data to using
> >> > load_spec.
> >> > Once that step is done, then I can run plot_spec. That's my
> >> > takeaway
> >> > currently
> >> >
> >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
> >> > edward.strobach at noaa.gov> wrote:
> >> >
> >> > > Actually, it appears that I need to run the database loading
> >> > > module
> >> > > first
> >> > > if I want to change/add a data source. From there I can run
the
> >> > > plotting
> >> > > batch module. Is that right? So if I run the database
loading
> >> > > module,
> >> > > then I can specify my dates and other related info specific
to
> >> > > the
> >> > > dataset. If I create my database with a unique name, then it
> >> > > should
> >> > > be
> >> > > recognized on the metviewer gui, presumably.
> >> > >
> >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
> >> > > <
> >> > > edward.strobach at noaa.gov> wrote:
> >> > >
> >> > >> It seems that those options must be treated separately
outside
> >> > >> of
> >> > >> plot_spec. Is it acceptable to create both a load_spec and
> >> > >> plot_spec in
> >> > >> the same xml file? Would the definitions in load_spec be
able
> >> > >> to
> >> > >> carry
> >> > >> over into plot_spec? It appears that data loading and
plotting
> >> > >> are
> >> > >> treated
> >> > >> separately.
> >> > >>
> >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
> >> > >> Affiliate <
> >> > >> edward.strobach at noaa.gov> wrote:
> >> > >>
> >> > >>> I do plan to use date_range; however, in an example
provided
> >> > >>> by the
> >> > >>> link
> >> > >>> I see this:
> >> > >>>
> >> > >>> <load_spec>
> >> > >>> <connection>
> >> > >>> <host>kemosabe:3306</host>
> >> > >>> <database>metvdb_hwt</database>
> >> > >>> <user>pgoldenb</user>
> >> > >>> <password>pgoldenb</password>
> >> > >>> </connection>
> >> > >>>
> >> > >>> <date_list name="folder_dates">
> >> > >>> <start>2010051914V</start>
> >> > >>> <end><date_offset><day_offset>0</day_offset><hour>6
> >> > >>> </hour></date_offset></end>
> >> > >>> <inc>3600</inc>
> >> > >>> <format>yyyyMMddHH'V'</format>
> >> > >>> </date_list>
> >> > >>>
> >> > >>> <met_version>V3.0</met_version>
> >> > >>>
> >> > >>> <verbose>false</verbose>
> >> > >>> <insert_size>1</insert_size>
> >> > >>> <mode_header_db_check>true</mode_header_db_check>
> >> > >>> <drop_indexes>false</drop_indexes>
> >> > >>> <apply_indexes>true</apply_indexes>
> >> > >>> <group>Group name</group>
> >> > >>> <load_stat>true</load_stat>
> >> > >>> <load_mode>true</load_mode>
> >> > >>> <load_mtd>true</load_mtd>
> >> > >>> <load_mpr>false</load_mpr>
> >> > >>>
> >> > >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> >> > >>> <load_val>
> >> > >>> <field name="model">
> >> > >>> <val>arw</val>
> >> > >>> <val>nmm</val>
> >> > >>> </field>
> >> > >>>
> >> > >>> <field name="valid_time">
> >> > >>> <date_list name="folder_dates"/>
> >> > >>> </field>
> >> > >>>
> >> > >>> <field name="vx_mask">
> >> > >>> <val>FULL</val>
> >> > >>> <val>SWC</val>
> >> > >>> </field>
> >> > >>> </load_val>
> >> > >>> </load_spec>
> >> > >>>
> >> > >>> Are similar settings allowed with <plot_spec>? I see
> >> > >>> <connection>,
> >> > >>> which seems carried over. I'm looking to specify my
> >> > >>> folder_tmpl in
> >> > >>> <plot_spec>, which according to Tara, should be possible.
> >> > >>> Since my
> >> > >>> directory structure looks like this:
> >> > >>>
> >> > >>>
> >> > >>>
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> >> > >>>
> >> > >>> I would do something like this I would think:
> >> > >>>
> >> > >>> .......
> >> > >>>
> >> > >>> <date_list name="folder_dates">
> >> > >>> <start>20200612/start>
> >> > >>> <end><date_offset><day_offset>0</day_offset></end>
> >> > >>> <inc>86400</inc>
> >> > >>> <format>yyyyMMdd</format>
> >> > >>> </date_list>
> >> > >>>
> >> > >>> <folder_tmpl>
> >> > >>>
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> >> > >>> </folder_tmpl>
> >> > >>> <load_val>
> >> > >>> <field name="model">
> >> > >>> <val>prod</val>
> >> > >>> </field>
> >> > >>>
> >> > >>> <field name="valid_time">
> >> > >>> <date_list name="folder_dates"/>
> >> > >>> </field>
> >> > >>>
> >> > >>> ......
> >> > >>>
> >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
> >> > >>> Affiliate <
> >> > >>> edward.strobach at noaa.gov> wrote:
> >> > >>>
> >> > >>>> Thank you. This is a very helpful example
> >> > >>>>
> >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> >> > >>>> <met_help at ucar.edu>
> >> > >>>> wrote:
> >> > >>>>
> >> > >>>>> Hi,
> >> > >>>>> Here is some examples how to use <date_range> and
> >> > >>>>> <date_range_list> in
> >> > >>>>> batch XML.
> >> > >>>>> First, you need to defined your range. You can have more
> >> > >>>>> then one
> >> > >>>>> <date_range>:
> >> > >>>>> <date_range name="may">
> >> > >>>>> <start>2019-05-01 00:00:00</start>
> >> > >>>>> <end>2019-05-31 23:59:59</end>
> >> > >>>>> </date_range>
> >> > >>>>> <date_range name="march">
> >> > >>>>> <start>2019-03-01 00:00:00</start>
> >> > >>>>> <end>2019-03-31 23:59:59</end>
> >> > >>>>> </date_range>
> >> > >>>>>
> >> > >>>>> and then use one or all date_ranges:
> >> > >>>>> <plot_fix>
> >> > >>>>> <field name="fcst_init_beg">
> >> > >>>>> <date_range name="march"/>
> >> > >>>>> <date_range name="may"/>
> >> > >>>>> </field>...
> >> > >>>>> This configuration would include all dates for March and
May
> >> > >>>>> of
> >> > >>>>> 2019.
> >> > >>>>>
> >> > >>>>> If you need more flexible configuration you can use
> >> > >>>>> date_range_list.
> >> > >>>>> For example, you want to select first 6 hours every day
> >> > >>>>> beetween
> >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
> >> > >>>>> look
> >> > >>>>> like this:
> >> > >>>>> <date_range_list name = 'all'>
> >> > >>>>> <range_start>2020-01-01 00:00:00</range_start>
> >> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
> >> > >>>>> <range_length>360</range_length>
> >> > >>>>> <inc>1440</inc>
> >> > >>>>> </date_range_list>
> >> > >>>>> where the 1st and 2nd values are the start and end,
> >> > >>>>> range_length - the length of a range in minutes(6 houres)
> >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> >> > >>>>> This example would create following ranges:
> >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> >> > >>>>>
> >> > >>>>> Also you can add more parameters to <plot_fix> to filter
> >> > >>>>> your
> >> > >>>>> data.
> >> > >>>>> For example, if you want to select only dates with 00:00
you
> >> > >>>>> can
> >> > >>>>> specify
> >> > >>>>> <field name="valid_hour">
> >> > >>>>> <val>00</val>
> >> > >>>>> </field>
> >> > >>>>>
> >> > >>>>> Tatiana
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
> >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
> >> > >>>>> > of a
> >> > >>>>> > separate
> >> > >>>>> > script that pipes dates (and other relevant
information)
> >> > >>>>> > into
> >> > >>>>> > the XML
> >> > >>>>> > file
> >> > >>>>> > used to generate the results.
> >> > >>>>> >
> >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> >> > >>>>> > Affiliate <
> >> > >>>>> > edward.strobach at noaa.gov> wrote:
> >> > >>>>> >
> >> > >>>>> > > Thanks Tatiana. These resources are helpful.
> >> > >>>>> > >
> >> > >>>>> > > One last question. Simply typing lines like this in
can
> >> > >>>>> > > take
> >> > >>>>> > > forever:
> >> > >>>>> > >
> >> > >>>>> > > <plot_fix>
> >> > >>>>> > > <field equalize="false" name="fcst_init_beg">
> >> > >>>>> > > <set name="fcst_init_beg_0">
> >> > >>>>> > > <val>2019-05-01 12:00:00</val>
> >> > >>>>> > > <val>2019-05-02 06:00:00</val>
> >> > >>>>> > > <val>2019-05-02 12:00:00</val>
> >> > >>>>> > > <val>2019-05-03 06:00:00</val>
> >> > >>>>> > > <val>2019-05-03 12:00:00</val>
> >> > >>>>> > >
> >> > >>>>> > >
> >> > >>>>> > > Is there a better way? I see date_range,
> >> > >>>>> > > date_range_list,
> >> > >>>>> > > or
> >> > >>>>> > > date_list.
> >> > >>>>> > > Can this be used instead of typing individual values?
> >> > >>>>> > > This
> >> > >>>>> > > could
> >> > >>>>> > > prove
> >> > >>>>> > > pretty daunting if one would have to type in 700
entries
> >> > >>>>> > > each
> >> > >>>>> > > time.
> >> > >>>>> > >
> >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> >> > >>>>> > > <met_help at ucar.edu>
> >> > >>>>> > > wrote:
> >> > >>>>> > >
> >> > >>>>> > >> I found a few problems in your XML.
> >> > >>>>> > >> The database you are using doesn't have data with
> >> > >>>>> > >> vx_mask
> >> > >>>>> > >> 'abcREGabc'.
> >> > >>>>> > >> That is why METviewer returns an empty dataset and
> >> > >>>>> > >> can't
> >> > >>>>> > >> create a
> >> > >>>>> > >> plot.
> >> > >>>>> > >> The other problem is how you define fcst_lead. This
> >> > >>>>> > >> database
> >> > >>>>> > >> has
> >> > >>>>> > >> values
> >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
> >> > >>>>> > >> include
> >> > >>>>> > >> all
> >> > >>>>> > >> fcst_leads
> >> > >>>>> > >> this variable can be omitted.
> >> > >>>>> > >> To minimize problems with variables values, I would
> >> > >>>>> > >> suggest
> >> > >>>>> > >> to
> >> > >>>>> > >> create XML
> >> > >>>>> > >> using GUI and then edit it to fit batch standards.
> >> > >>>>> > >> I created a plot using you configurations with GUI
> >> > >>>>> > >> (without
> >> > >>>>> vx_mack
> >> > >>>>> > >> and
> >> > >>>>> > >> plot and axis titles) #20200612_173626. You can load
it
> >> > >>>>> > >> back
> >> > >>>>> > >> to
> >> > >>>>> the
> >> > >>>>> > >> page by
> >> > >>>>> > >> using the "History" tab (left bar on GUI). Find the
> >> > >>>>> > >> plot,
> >> > >>>>> > >> open XML
> >> > >>>>> > >> and
> >> > >>>>> > >> click on "Load to page". After that you can recreate
a
> >> > >>>>> > >> plot
> >> > >>>>> > >> and
> >> > >>>>> edit
> >> > >>>>> > >> the
> >> > >>>>> > >> configurations. You can also download a new XML and
> >> > >>>>> > >> edit it
> >> > >>>>> > >> for
> >> > >>>>> > >> batch.
> >> > >>>>> > >> I attached the example of load XML to this message.
> >> > >>>>> > >>
> >> > >>>>> > >> Description of rds parameters:
> >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
> >> > >>>>> > >> scripts
> >> > >>>>> > >> rds_plots - temporary directory where the plots will
be
> >> > >>>>> > >> generated
> >> > >>>>> > >> rds_data - temporary directory where the data
retrieved
> >> > >>>>> > >> from
> >> > >>>>> > >> the
> >> > >>>>> > >> database
> >> > >>>>> > >> and other intermediate files will be stored
> >> > >>>>> > >> rds_scripts - temporary directory where R scripts
for
> >> > >>>>> > >> this
> >> > >>>>> > >> run
> >> > >>>>> will
> >> > >>>>> > >> be
> >> > >>>>> > >> stored
> >> > >>>>> > >> After the execution is done all the files from
> >> > >>>>> > >> temporary
> >> > >>>>> directories
> >> > >>>>> > >> would be removed.
> >> > >>>>> > >>
> >> > >>>>> > >>
> >> > >>>>> > >> We don't have the offitial user guide for METviewer.
> >> > >>>>> > >> There
> >> > >>>>> > >> are
> >> > >>>>> some
> >> > >>>>> > >> instructions on how to crate XML config files:
> >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> >> > >>>>> > >>
> >> > >>>>> > >> Tatiana
> >> > >>>>> > >>
> >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
> >> > >>>>> > >> wrote:
> >> > >>>>> > >> > Hi,
> >> > >>>>> > >> >
> >> > >>>>> > >> > I'm attaching the xml below, but am having hard
time
> >> > >>>>> > >> > following
> >> > >>>>> how
> >> > >>>>> > >> > to
> >> > >>>>> > >> > go
> >> > >>>>> > >> > about the other steps since I have zero experience
on
> >> > >>>>> > >> > how
> >> > >>>>> > >> > to do
> >> > >>>>> > >> > these
> >> > >>>>> > >> > things.
> >> > >>>>> > >> >
> >> > >>>>> > >> > #2. All parameters in <folders> section should NOT
be
> >> > >>>>> > >> > changed
> >> > >>>>> and
> >> > >>>>> > >> > will
> >> > >>>>> > >> > be
> >> > >>>>> > >> > replaced by other scripts during the batch
execution.
> >> > >>>>> > >> > The
> >> > >>>>> > >> > data
> >> > >>>>> for
> >> > >>>>> > >> > the
> >> > >>>>> > >> > batch mode is coming from the database and not
from
> >> > >>>>> > >> > files.
> >> > >>>>> > >> > The
> >> > >>>>> > >> > name of
> >> > >>>>> > >> > the
> >> > >>>>> > >> > database is in <database> tag.
> >> > >>>>> > >> >
> >> > >>>>> > >> > I understand that nothing should be changed in the
> >> > >>>>> > >> > folders
> >> > >>>>> > >> > section,
> >> > >>>>> > >> > but
> >> > >>>>> > >> > there is no indication how and where these
variables
> >> > >>>>> > >> > are
> >> > >>>>> > >> > being
> >> > >>>>> > >> > defined.
> >> > >>>>> > >> > Grepping these names of the files available does
not
> >> > >>>>> > >> > reveal
> >> > >>>>> formal
> >> > >>>>> > >> > definitions of these variables. If I understood
> >> > >>>>> > >> > exactly
> >> > >>>>> > >> > how
> >> > >>>>> these
> >> > >>>>> > >> > were
> >> > >>>>> > >> > being defined, then I would be able connect this
> >> > >>>>> > >> > better.
> >> > >>>>> > >> >
> >> > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
> >> > >>>>> > >> > use
> >> > >>>>> METviewer
> >> > >>>>> > >> > load
> >> > >>>>> > >> > mode to load these files to the specific database.
In
> >> > >>>>> > >> > the
> >> > >>>>> > >> > config
> >> > >>>>> > >> > file
> >> > >>>>> > >> > for
> >> > >>>>> > >> > this process they also can specify the database
> >> > >>>>> > >> > group. For
> >> > >>>>> example
> >> > >>>>> > >> > database
> >> > >>>>> > >> > mv_dev is in Test12 group.
> >> > >>>>> > >> > After the data has been loaded to the database it
is
> >> > >>>>> > >> > visible and
> >> > >>>>> > >> > available
> >> > >>>>> > >> > in the GUI and also can be used by the batch mode.
> >> > >>>>> > >> > You can always add more data to the database using
> >> > >>>>> > >> > the
> >> > >>>>> > >> > same load
> >> > >>>>> > >> > mode.
> >> > >>>>> > >> >
> >> > >>>>> > >> > Is there a guide for me to understand the process
> >> > >>>>> > >> > that you
> >> > >>>>> > >> > are
> >> > >>>>> > >> > referring to?
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via
RT
> >> > >>>>> > >> > <met_help at ucar.edu>
> >> > >>>>> > >> > wrote:
> >> > >>>>> > >> >
> >> > >>>>> > >> > > Hi Edward,
> >> > >>>>> > >> > > #1. I don't have an access to NOAA servers and
> >> > >>>>> > >> > > can't
> >> > >>>>> > >> > > view your
> >> > >>>>> > >> > > XML
> >> > >>>>> > >> > > for the
> >> > >>>>> > >> > > batch mode. Please, attache it to this issue and
I
> >> > >>>>> > >> > > will
> >> > >>>>> > >> > > take a
> >> > >>>>> > >> > > look.
> >> > >>>>> > >> > >
> >> > >>>>> > >> > > #2. All parameters in <folders> section should
NOT
> >> > >>>>> > >> > > be
> >> > >>>>> > >> > > changed
> >> > >>>>> > >> > > and
> >> > >>>>> > >> > > will be
> >> > >>>>> > >> > > replaced by other scripts during the batch
> >> > >>>>> > >> > > execution.
> >> > >>>>> > >> > > The data
> >> > >>>>> > >> > > for
> >> > >>>>> > >> > > the
> >> > >>>>> > >> > > batch mode is coming from the database and not
from
> >> > >>>>> > >> > > files. The
> >> > >>>>> > >> > > name
> >> > >>>>> > >> > > of the
> >> > >>>>> > >> > > database is in <database> tag.
> >> > >>>>> > >> > >
> >> > >>>>> > >> > > #3. Users rum MET and produce MET output files.
> >> > >>>>> > >> > > They use
> >> > >>>>> > >> > > METviewer
> >> > >>>>> > >> > > load
> >> > >>>>> > >> > > mode to load these files to the specific
database.
> >> > >>>>> > >> > > In
> >> > >>>>> > >> > > the
> >> > >>>>> config
> >> > >>>>> > >> > > file
> >> > >>>>> > >> > > for
> >> > >>>>> > >> > > this process they also can specify the database
> >> > >>>>> > >> > > group.
> >> > >>>>> > >> > > For
> >> > >>>>> > >> > > example
> >> > >>>>> > >> > > database
> >> > >>>>> > >> > > mv_dev is in Test12 group.
> >> > >>>>> > >> > > After the data has been loaded to the database
it
> >> > >>>>> > >> > > is
> >> > >>>>> > >> > > visible
> >> > >>>>> and
> >> > >>>>> > >> > > available
> >> > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
> >> > >>>>> > >> > > You can always add more data to the database
using
> >> > >>>>> > >> > > the
> >> > >>>>> > >> > > same
> >> > >>>>> load
> >> > >>>>> > >> > > mode.
> >> > >>>>> > >> > >
> >> > >>>>> > >> > > Tatiana
> >> > >>>>> > >> > >
> >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> >> > >>>>> > >> > > edward.strobach at noaa.gov
> >> > >>>>> > >> > > wrote:
> >> > >>>>> > >> > > > Good morning,
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Over the last two weeks I've made an effort to
> >> > >>>>> > >> > > > understand
> >> > >>>>> > >> > > > metviewer,
> >> > >>>>> > >> > > > test
> >> > >>>>> > >> > > > different cases using the history tab,
changing
> >> > >>>>> > >> > > > information
> >> > >>>>> to
> >> > >>>>> > >> > > > see
> >> > >>>>> > >> > > > what
> >> > >>>>> > >> > > > works and what does not, and make connections
> >> > >>>>> > >> > > > between
> >> > >>>>> > >> > > > the
> >> > >>>>> GUI
> >> > >>>>> > >> > > > settings
> >> > >>>>> > >> > > > and
> >> > >>>>> > >> > > > the XML file. Although my comfort level with
> >> > >>>>> > >> > > > metviewer has
> >> > >>>>> > >> > > > significantly
> >> > >>>>> > >> > > > increased during this time, there are some
> >> > >>>>> > >> > > > remaining
> >> > >>>>> > >> > > > issues
> >> > >>>>> > >> > > > that
> >> > >>>>> > >> > > > I'm
> >> > >>>>> > >> > > > hoping
> >> > >>>>> > >> > > > to clear up.
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > 1. Problem changing XML file and running with
> >> > >>>>> > >> > > > different
> >> > >>>>> > >> > > > statistic
> >> > >>>>> > >> > > > and
> >> > >>>>> > >> > > > plot
> >> > >>>>> > >> > > > type:
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me
> >> > >>>>> > >> > > > the
> >> > >>>>> > >> > > > working
> >> > >>>>> > >> > > > directory
> >> > >>>>> > >> > > > to get
> >> > >>>>> > >> > > > started which I copied over:
> >> > >>>>> > >> > > >
> >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
> >> > >>>>> > >> > > > to
> >> > >>>>> > >> > > > run his
> >> > >>>>> > >> > > > example.
> >> > >>>>> > >> > > > running of the script "bash test.sh" produced
a
> >> > >>>>> > >> > > > CSI
> >> > >>>>> > >> > > > plot
> >> > >>>>> that
> >> > >>>>> > >> > > > was
> >> > >>>>> > >> > > > stored
> >> > >>>>> > >> > > > here:
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > >
> >> > >>>>> > >>
> >> > >>>>>
> >>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Running the script produced the expected
result.
> >> > >>>>> > >> > > > The
> >> > >>>>> > >> > > > next
> >> > >>>>> > >> > > > natural
> >> > >>>>> > >> > > > step was
> >> > >>>>> > >> > > > to copy hochum_xml.template and create a new
xml
> >> > >>>>> > >> > > > file
> >> > >>>>> > >> > > > called
> >> > >>>>> > >> > > > Strobach_xml.template in order to create a
time
> >> > >>>>> > >> > > > series
> >> > >>>>> > >> > > > where
> >> > >>>>> > >> > > > the
> >> > >>>>> > >> > > > independent time is fcst_valid_beg instead of
> >> > >>>>> > >> > > > obs_thresh.
> >> > >>>>> > >> > > > Accordingly,
> >> > >>>>> > >> > > > I
> >> > >>>>> > >> > > > made changes to the fixed fields (removing
> >> > >>>>> > >> > > > fcst_valid_beg)
> >> > >>>>> and
> >> > >>>>> > >> > > > changed
> >> > >>>>> > >> > > > the
> >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
> >> > >>>>> > >> > > > tried).
> >> > >>>>> > >> > > > The
> >> > >>>>> error
> >> > >>>>> > >> > > > that
> >> > >>>>> > >> > > > resulted
> >> > >>>>> > >> > > > is as follows:
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-01
> >> > >>>>> > >> > > > 13:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 01
> >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> >> > >>>>> > >> > > > 16:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 01
> >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> >> > >>>>> > >> > > > 19:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 01
> >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> >> > >>>>> > >> > > > 22:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 01
> >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> >> > >>>>> > >> > > > 01:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 02
> >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> >> > >>>>> > >> > > > 04:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 02
> >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> >> > >>>>> > >> > > > 07:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 02
> >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> >> > >>>>> > >> > > > 10:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 02
> >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> >> > >>>>> > >> > > > AND ld.fmean != -9999;
> >> > >>>>> > >> > > > No data was returned from database
> >> > >>>>> > >> > > > 2020-06-09
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > >
> >> > >>>>> > >>
> >> > >>>>>
> >>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >> > >>>>> > >> > > > Source) -1 printFormattedTable - ** ERROR:
> >> > >>>>> > >> > > > Caught
> >> > >>>>> > >> > > > class
> >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> >> > >>>>> > >> > > > printFormattedTable(ResultSet
> >> > >>>>> > >> > > > res): result set contained no data
> >> > >>>>> > >> > > > ** ERROR: Caught class
> >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >> > >>>>> > >> > > > in
> >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> >> > >>>>> > >> > > > contained no
> >> > >>>>> > >> > > > data
> >> > >>>>> > >> > > > ** ERROR: Caught class
> >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >> > >>>>> > >> > > > in
> >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> >> > >>>>> > >> > > > contained no
> >> > >>>>> > >> > > > data
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Apparently there was no data during this time.
I
> >> > >>>>> > >> > > > tried
> >> > >>>>> > >> > > > changing
> >> > >>>>> > >> > > > the
> >> > >>>>> > >> > > > range
> >> > >>>>> > >> > > > of valid times and changed the valid time to
> >> > >>>>> > >> > > > different
> >> > >>>>> > >> > > > days
> >> > >>>>> > >> > > > just in
> >> > >>>>> > >> > > > case
> >> > >>>>> > >> > > > data was missing. The issue was the same. To
> >> > >>>>> > >> > > > confirm
> >> > >>>>> > >> > > > my
> >> > >>>>> > >> > > > logic, I
> >> > >>>>> > >> > > > reproduced my attempted approach on metviewer,
> >> > >>>>> > >> > > > which I
> >> > >>>>> > >> > > > found
> >> > >>>>> > >> > > > to
> >> > >>>>> > >> > > > work.
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > 2. Running in batch mode
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > I think I lack an understanding of how to set
up
> >> > >>>>> > >> > > > batch
> >> > >>>>> > >> > > > mode
> >> > >>>>> > >> > > > because
> >> > >>>>> > >> > > > I
> >> > >>>>> > >> > > > don't
> >> > >>>>> > >> > > > see where these definitions are being
explicitly
> >> > >>>>> > >> > > > defined:
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > <plot_spec>
> >> > >>>>> > >> > > > <connection>
> >> > >>>>> > >> > > > <host>rds_host:3306</host>
> >> > >>>>> > >> > > >
> >> > >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >> > >>>>> > >> > > > <user>rds_user</user>
> >> > >>>>> > >> > > > <password>rds_pwd</password>
> >> > >>>>> > >> > > > </connection>
> >> > >>>>> > >> > > > <rscript>Rscript</rscript>
> >> > >>>>> > >> > > > <folders>
> >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> >> > >>>>> > >> > > > <plots>rds_plots</plots>
> >> > >>>>> > >> > > > <data>rds_data</data>
> >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> >> > >>>>> > >> > > > </folders>
> >> > >>>>> > >> > > > <plot>
> >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > I see the <folders> section, but grepping all
> >> > >>>>> > >> > > > available
> >> > >>>>> files
> >> > >>>>> > >> > > > does
> >> > >>>>> > >> > > > not
> >> > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well
> >> > >>>>> > >> > > > as
> >> > >>>>> > >> > > > other
> >> > >>>>> > >> > > > similarly
> >> > >>>>> > >> > > > defined
> >> > >>>>> > >> > > > variables. I think if I know how these are
being
> >> > >>>>> > >> > > > defined,
> >> > >>>>> > >> > > > then
> >> > >>>>> > >> > > > I'll
> >> > >>>>> > >> > > > be
> >> > >>>>> > >> > > > able to pipe in my information and process the
> >> > >>>>> > >> > > > data
> >> > >>>>> > >> > > > using
> >> > >>>>> met
> >> > >>>>> > >> > > > and
> >> > >>>>> > >> > > > produce
> >> > >>>>> > >> > > > results in batch mode. Am I right on that?
> >> > >>>>> > >> > > > Also, if
> >> > >>>>> > >> > > > I want
> >> > >>>>> > >> > > > to use
> >> > >>>>> > >> > > > obs for
> >> > >>>>> > >> > > > verification, do I add another line in the XML
> >> > >>>>> > >> > > > file
> >> > >>>>> > >> > > > that
> >> > >>>>> > >> > > > designates
> >> > >>>>> > >> > > > file
> >> > >>>>> > >> > > > location of obs? I assume that other steps
have
> >> > >>>>> > >> > > > to be
> >> > >>>>> > >> > > > run
> >> > >>>>> > >> > > > before
> >> > >>>>> > >> > > > all
> >> > >>>>> > >> > > > this. If data is in prepbufr form, for
example,
> >> > >>>>> > >> > > > I
> >> > >>>>> > >> > > > would
> >> > >>>>> have
> >> > >>>>> > >> > > > to
> >> > >>>>> > >> > > > separately
> >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I would
want
> >> > >>>>> > >> > > > to
> >> > >>>>> > >> > > > do that
> >> > >>>>> > >> > > > as
> >> > >>>>> > >> > > > well.
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > 3. Communicating results to metviewer online
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
> >> > >>>>> > >> > > > here -
> >> > >>>>> > >> > > > see
> >> > >>>>> > >> > > > attachment
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > How do I get the results I generate to be
> >> > >>>>> > >> > > > uploaded to
> >> > >>>>> > >> > > > this
> >> > >>>>> > >> > > > location?
> >> > >>>>> > >> > > > It's
> >> > >>>>> > >> > > > not clear how others are setting this up.
> >> > >>>>> > >> > >
> >> > >>>>> > >> > >
> >> > >>>>> > >> > >
> >> > >>>>> > >> > >
> >> > >>>>> > >>
> >> > >>>>> > >>
> >> > >>>>> > >>
> >> > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
> >> > >>>>> > >> rds_user
> >> > >>>>> rds_pwd
> >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
> >> > >>>>> rds_scripts
> >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
> >> > >>>>> > >> CMAQAODPRODHIGH
> >> > >>>>> 2020-05-12
> >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
> >> > >>>>> > >> 05-12
> >> > >>>>> 20:00:00
> >> > >>>>> > >> 2020-05-12
> >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
> >> > >>>>> > >> 05-13
> >> > >>>>> 10:00:00
> >> > >>>>> > >> 2020-05-13
> >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
> >> > >>>>> > >> 05-13
> >> > >>>>> 14:00:00
> >> > >>>>> > >> 2020-05-13
> >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
> >> > >>>>> > >> 05-13
> >> > >>>>> 18:00:00
> >> > >>>>> > >> 2020-05-13
> >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
> >> > >>>>> > >> 05-13
> >> > >>>>> 22:00:00
> >> > >>>>> > >> 2020-05-13
> >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
> >> > >>>>> > >> 05-14
> >> > >>>>> 12:00:00
> >> > >>>>> > >> true
> >> > >>>>> > >> median plot_20200612_173626.data
> >> > >>>>> > >> plot_20200612_173626.png
> >> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> >> > >>>>> > >> false list() list() false false false false false
false
> >> > >>>>> > >> true
> >> > >>>>> > >> false
> >> > >>>>> > >> false
> >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
> >> > >>>>> > >> c(1,1,0) 1 2
> >> > >>>>> > >> 1.4 -2
> >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5
3
> >> > >>>>> > >> #cccccc 1
> >> > >>>>> > >> listX 1
> >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
> >> > >>>>> > >> 3
> >> > >>>>> > >> 0.8 1
> >> > >>>>> > >> #333333
> >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
> >> > >>>>> > >> c(TRUE,TRUE)
> >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
> >> > >>>>> > >> c(1,1)
> >> > >>>>> c(1,1)
> >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> >> > >>>>> > >
> >> > >>>>> > >
> >> > >>>>> > >
> >> > >>>>> > > --
> >> > >>>>> > > Edward Strobach
> >> > >>>>> > > EMC/NCEP/NWS/
> >> > >>>>> > > IMSG Contractor
> >> > >>>>> > > Cubicle#: 2029
> >> > >>>>> > > 301-683-3717
> >> > >>>>> > >
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>
> >> > >>>> --
> >> > >>>> Edward Strobach
> >> > >>>> EMC/NCEP/NWS/
> >> > >>>> IMSG Contractor
> >> > >>>> Cubicle#: 2029
> >> > >>>> 301-683-3717
> >> > >>>>
> >> > >>>
> >> > >>>
> >> > >>> --
> >> > >>> Edward Strobach
> >> > >>> EMC/NCEP/NWS/
> >> > >>> IMSG Contractor
> >> > >>> Cubicle#: 2029
> >> > >>> 301-683-3717
> >> > >>>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Edward Strobach
> >> > >> EMC/NCEP/NWS/
> >> > >> IMSG Contractor
> >> > >> Cubicle#: 2029
> >> > >> 301-683-3717
> >> > >>
> >> > >
> >> > >
> >> > > --
> >> > > Edward Strobach
> >> > > EMC/NCEP/NWS/
> >> > > IMSG Contractor
> >> > > Cubicle#: 2029
> >> > > 301-683-3717
> >> > >
> >>
> >>
> >>
> >>
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 10:00:27 2020
I understand, but I made mv_cmaq_para_metplus, and that is not listed.
I
ran a script called check_my_db.sh and it was part of the list. I see
that
many people have their own groups, and with that group they have
separate
databases. As you can see, none of the groups have mv_ in front of
them.
This is reserved for the database. The group does not need mv_.
Please
see attached example.
On Wed, Jun 17, 2020 at 11:41 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Only databases that have a name that starts with 'mv_'are visible in
GUI.
> That is why 'Ed_Strobach' is not on the list.
> If database doesn't have a group it will end up in "NO GROUP" group.
> You specify the name for the group you want to see your database in
when
> you load data.
> There is a tag <group> where you put the group name.
>
> Tatiana
>
> On Tue Jun 16 15:40:50 2020, edward.strobach at noaa.gov wrote:
> > Thanks Tatiana. Yes, I've created a database which seems
different
> > from
> > mv_group. How can one create their own group so that I can store
and
> > access
> > the database. The database exists but cannot be viewed on the
> > metviewer
> > GUI. The list starts with EMC HREFv3 and ends with vhagerty. I
named
> > my
> > database Ed_Strobach, but that was really intended for the group
name.
> >
> > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > We don't have database and METviewer running on wcoss.
> > > Instead, users use METviewer-wcoss scripts that help to connect
to
> > > and use
> > > METviewer that is running on AWS. That is why XML connection and
> > > folder
> > > sections look somewhat different than once from the Docs page.
> > > To create a database on AWS you need to run
mv_create_db_on_aws.sh
> > > mv_create_db_on_aws.sh <user_name> <database_name>
> > > For data loading :
> > > mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> > > And for batch:
> > > mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> > >
> > > These scripts copy XML and data files if needed to AWS, adjust
data
> > > path
> > > and users credentials, run METviewer and copy results back to
wcoss.
> > >
> > > Tatiana
> > > On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> > > > lastly, it does seem thatmysql -u[db_username] -p[db_password]
> > > > -e'create
> > > > database [db_name]
> > > >
> > > > would need to be used to create a database. I assume that an
sql
> > > > module is
> > > > available on wcoss; I know you don't have access to those
machines
> > > > Tatiana. If I create my own database with my own name, then I
> > > > should
> > > > be
> > > > able to proceed with project specific tasks under that tab.
> > > > Something
> > > > like
> > > > mv_EdwardStrobach.
> > > >
> > > > I can then create mv_cmaq_prod_v_para which I add data to
using
> > > > load_spec.
> > > > Once that step is done, then I can run plot_spec. That's my
> > > > takeaway
> > > > currently
> > > >
> > > > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
> > > > edward.strobach at noaa.gov> wrote:
> > > >
> > > > > Actually, it appears that I need to run the database loading
> > > > > module
> > > > > first
> > > > > if I want to change/add a data source. From there I can run
the
> > > > > plotting
> > > > > batch module. Is that right? So if I run the database
loading
> > > > > module,
> > > > > then I can specify my dates and other related info specific
to
> > > > > the
> > > > > dataset. If I create my database with a unique name, then
it
> > > > > should
> > > > > be
> > > > > recognized on the metviewer gui, presumably.
> > > > >
> > > > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
> > > > > <
> > > > > edward.strobach at noaa.gov> wrote:
> > > > >
> > > > >> It seems that those options must be treated separately
outside
> > > > >> of
> > > > >> plot_spec. Is it acceptable to create both a load_spec and
> > > > >> plot_spec in
> > > > >> the same xml file? Would the definitions in load_spec be
able
> > > > >> to
> > > > >> carry
> > > > >> over into plot_spec? It appears that data loading and
plotting
> > > > >> are
> > > > >> treated
> > > > >> separately.
> > > > >>
> > > > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
Affiliate
> > > > >> <
> > > > >> edward.strobach at noaa.gov> wrote:
> > > > >>
> > > > >>> I do plan to use date_range; however, in an example
provided by
> > > > >>> the
> > > > >>> link
> > > > >>> I see this:
> > > > >>>
> > > > >>> <load_spec>
> > > > >>> <connection>
> > > > >>> <host>kemosabe:3306</host>
> > > > >>> <database>metvdb_hwt</database>
> > > > >>> <user>pgoldenb</user>
> > > > >>> <password>pgoldenb</password>
> > > > >>> </connection>
> > > > >>>
> > > > >>> <date_list name="folder_dates">
> > > > >>> <start>2010051914V</start>
> > > > >>> <end><date_offset><day_offset>0</day_offset><hour>6
> > > > >>> </hour></date_offset></end>
> > > > >>> <inc>3600</inc>
> > > > >>> <format>yyyyMMddHH'V'</format>
> > > > >>> </date_list>
> > > > >>>
> > > > >>> <met_version>V3.0</met_version>
> > > > >>>
> > > > >>> <verbose>false</verbose>
> > > > >>> <insert_size>1</insert_size>
> > > > >>> <mode_header_db_check>true</mode_header_db_check>
> > > > >>> <drop_indexes>false</drop_indexes>
> > > > >>> <apply_indexes>true</apply_indexes>
> > > > >>> <group>Group name</group>
> > > > >>> <load_stat>true</load_stat>
> > > > >>> <load_mode>true</load_mode>
> > > > >>> <load_mtd>true</load_mtd>
> > > > >>> <load_mpr>false</load_mpr>
> > > > >>>
> > > > >>>
> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > > > >>> <load_val>
> > > > >>> <field name="model">
> > > > >>> <val>arw</val>
> > > > >>> <val>nmm</val>
> > > > >>> </field>
> > > > >>>
> > > > >>> <field name="valid_time">
> > > > >>> <date_list name="folder_dates"/>
> > > > >>> </field>
> > > > >>>
> > > > >>> <field name="vx_mask">
> > > > >>> <val>FULL</val>
> > > > >>> <val>SWC</val>
> > > > >>> </field>
> > > > >>> </load_val>
> > > > >>> </load_spec>
> > > > >>>
> > > > >>> Are similar settings allowed with <plot_spec>? I see
> > > > >>> <connection>,
> > > > >>> which seems carried over. I'm looking to specify my
> > > > >>> folder_tmpl in
> > > > >>> <plot_spec>, which according to Tara, should be possible.
> > > > >>> Since my
> > > > >>> directory structure looks like this:
> > > > >>>
> > > > >>>
> > > > >>>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > > > >>>
> > > > >>> I would do something like this I would think:
> > > > >>>
> > > > >>> .......
> > > > >>>
> > > > >>> <date_list name="folder_dates">
> > > > >>> <start>20200612/start>
> > > > >>> <end><date_offset><day_offset>0</day_offset></end>
> > > > >>> <inc>86400</inc>
> > > > >>> <format>yyyyMMdd</format>
> > > > >>> </date_list>
> > > > >>>
> > > > >>> <folder_tmpl>
> > > > >>>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > > > >>> </folder_tmpl>
> > > > >>> <load_val>
> > > > >>> <field name="model">
> > > > >>> <val>prod</val>
> > > > >>> </field>
> > > > >>>
> > > > >>> <field name="valid_time">
> > > > >>> <date_list name="folder_dates"/>
> > > > >>> </field>
> > > > >>>
> > > > >>> ......
> > > > >>>
> > > > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
> > > > >>> Affiliate <
> > > > >>> edward.strobach at noaa.gov> wrote:
> > > > >>>
> > > > >>>> Thank you. This is a very helpful example
> > > > >>>>
> > > > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> > > > >>>> <met_help at ucar.edu>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Hi,
> > > > >>>>> Here is some examples how to use <date_range> and
> > > > >>>>> <date_range_list> in
> > > > >>>>> batch XML.
> > > > >>>>> First, you need to defined your range. You can have more
then
> > > > >>>>> one
> > > > >>>>> <date_range>:
> > > > >>>>> <date_range name="may">
> > > > >>>>> <start>2019-05-01 00:00:00</start>
> > > > >>>>> <end>2019-05-31 23:59:59</end>
> > > > >>>>> </date_range>
> > > > >>>>> <date_range name="march">
> > > > >>>>> <start>2019-03-01 00:00:00</start>
> > > > >>>>> <end>2019-03-31 23:59:59</end>
> > > > >>>>> </date_range>
> > > > >>>>>
> > > > >>>>> and then use one or all date_ranges:
> > > > >>>>> <plot_fix>
> > > > >>>>> <field name="fcst_init_beg">
> > > > >>>>> <date_range name="march"/>
> > > > >>>>> <date_range name="may"/>
> > > > >>>>> </field>...
> > > > >>>>> This configuration would include all dates for March and
May
> > > > >>>>> of
> > > > >>>>> 2019.
> > > > >>>>>
> > > > >>>>> If you need more flexible configuration you can use
> > > > >>>>> date_range_list.
> > > > >>>>> For example, you want to select first 6 hours every day
> > > > >>>>> beetween
> > > > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
> > > > >>>>> look
> > > > >>>>> like this:
> > > > >>>>> <date_range_list name = 'all'>
> > > > >>>>> <range_start>2020-01-01 00:00:00</range_start>
> > > > >>>>> <range_end>2020-01-05 00:00:00</range_end>
> > > > >>>>> <range_length>360</range_length>
> > > > >>>>> <inc>1440</inc>
> > > > >>>>> </date_range_list>
> > > > >>>>> where the 1st and 2nd values are the start and end,
> > > > >>>>> range_length - the length of a range in minutes(6
houres)
> > > > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> > > > >>>>> This example would create following ranges:
> > > > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> > > > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> > > > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> > > > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> > > > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> > > > >>>>>
> > > > >>>>> Also you can add more parameters to <plot_fix> to filter
your
> > > > >>>>> data.
> > > > >>>>> For example, if you want to select only dates with 00:00
you
> > > > >>>>> can
> > > > >>>>> specify
> > > > >>>>> <field name="valid_hour">
> > > > >>>>> <val>00</val>
> > > > >>>>> </field>
> > > > >>>>>
> > > > >>>>> Tatiana
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
> > > > >>>>> > and if there isn't a better way, I wonder the
whereabouts
> > > > >>>>> > of a
> > > > >>>>> > separate
> > > > >>>>> > script that pipes dates (and other relevant
information)
> > > > >>>>> > into
> > > > >>>>> > the XML
> > > > >>>>> > file
> > > > >>>>> > used to generate the results.
> > > > >>>>> >
> > > > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
> > > > >>>>> > Affiliate <
> > > > >>>>> > edward.strobach at noaa.gov> wrote:
> > > > >>>>> >
> > > > >>>>> > > Thanks Tatiana. These resources are helpful.
> > > > >>>>> > >
> > > > >>>>> > > One last question. Simply typing lines like this in
can
> > > > >>>>> > > take
> > > > >>>>> > > forever:
> > > > >>>>> > >
> > > > >>>>> > > <plot_fix>
> > > > >>>>> > > <field equalize="false" name="fcst_init_beg">
> > > > >>>>> > > <set name="fcst_init_beg_0">
> > > > >>>>> > > <val>2019-05-01 12:00:00</val>
> > > > >>>>> > > <val>2019-05-02 06:00:00</val>
> > > > >>>>> > > <val>2019-05-02 12:00:00</val>
> > > > >>>>> > > <val>2019-05-03 06:00:00</val>
> > > > >>>>> > > <val>2019-05-03 12:00:00</val>
> > > > >>>>> > >
> > > > >>>>> > >
> > > > >>>>> > > Is there a better way? I see date_range,
> > > > >>>>> > > date_range_list,
> > > > >>>>> > > or
> > > > >>>>> > > date_list.
> > > > >>>>> > > Can this be used instead of typing individual
values?
> > > > >>>>> > > This
> > > > >>>>> > > could
> > > > >>>>> > > prove
> > > > >>>>> > > pretty daunting if one would have to type in 700
entries
> > > > >>>>> > > each
> > > > >>>>> > > time.
> > > > >>>>> > >
> > > > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > > > >>>>> > > <met_help at ucar.edu>
> > > > >>>>> > > wrote:
> > > > >>>>> > >
> > > > >>>>> > >> I found a few problems in your XML.
> > > > >>>>> > >> The database you are using doesn't have data with
> > > > >>>>> > >> vx_mask
> > > > >>>>> > >> 'abcREGabc'.
> > > > >>>>> > >> That is why METviewer returns an empty dataset and
can't
> > > > >>>>> > >> create a
> > > > >>>>> > >> plot.
> > > > >>>>> > >> The other problem is how you define fcst_lead. This
> > > > >>>>> > >> database
> > > > >>>>> > >> has
> > > > >>>>> > >> values
> > > > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
> > > > >>>>> > >> include
> > > > >>>>> > >> all
> > > > >>>>> > >> fcst_leads
> > > > >>>>> > >> this variable can be omitted.
> > > > >>>>> > >> To minimize problems with variables values, I would
> > > > >>>>> > >> suggest
> > > > >>>>> > >> to
> > > > >>>>> > >> create XML
> > > > >>>>> > >> using GUI and then edit it to fit batch standards.
> > > > >>>>> > >> I created a plot using you configurations with GUI
> > > > >>>>> > >> (without
> > > > >>>>> vx_mack
> > > > >>>>> > >> and
> > > > >>>>> > >> plot and axis titles) #20200612_173626. You can
load it
> > > > >>>>> > >> back
> > > > >>>>> > >> to
> > > > >>>>> the
> > > > >>>>> > >> page by
> > > > >>>>> > >> using the "History" tab (left bar on GUI). Find the
> > > > >>>>> > >> plot,
> > > > >>>>> > >> open XML
> > > > >>>>> > >> and
> > > > >>>>> > >> click on "Load to page". After that you can
recreate a
> > > > >>>>> > >> plot
> > > > >>>>> > >> and
> > > > >>>>> edit
> > > > >>>>> > >> the
> > > > >>>>> > >> configurations. You can also download a new XML and
edit
> > > > >>>>> > >> it
> > > > >>>>> > >> for
> > > > >>>>> > >> batch.
> > > > >>>>> > >> I attached the example of load XML to this message.
> > > > >>>>> > >>
> > > > >>>>> > >> Description of rds parameters:
> > > > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
> > > > >>>>> > >> scripts
> > > > >>>>> > >> rds_plots - temporary directory where the plots
will be
> > > > >>>>> > >> generated
> > > > >>>>> > >> rds_data - temporary directory where the data
retrieved
> > > > >>>>> > >> from
> > > > >>>>> > >> the
> > > > >>>>> > >> database
> > > > >>>>> > >> and other intermediate files will be stored
> > > > >>>>> > >> rds_scripts - temporary directory where R scripts
for
> > > > >>>>> > >> this
> > > > >>>>> > >> run
> > > > >>>>> will
> > > > >>>>> > >> be
> > > > >>>>> > >> stored
> > > > >>>>> > >> After the execution is done all the files from
temporary
> > > > >>>>> directories
> > > > >>>>> > >> would be removed.
> > > > >>>>> > >>
> > > > >>>>> > >>
> > > > >>>>> > >> We don't have the offitial user guide for
METviewer.
> > > > >>>>> > >> There
> > > > >>>>> > >> are
> > > > >>>>> some
> > > > >>>>> > >> instructions on how to crate XML config files:
> > > > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > > > >>>>> > >>
> > > > >>>>> > >> Tatiana
> > > > >>>>> > >>
> > > > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
> > > > >>>>> > >> wrote:
> > > > >>>>> > >> > Hi,
> > > > >>>>> > >> >
> > > > >>>>> > >> > I'm attaching the xml below, but am having hard
time
> > > > >>>>> > >> > following
> > > > >>>>> how
> > > > >>>>> > >> > to
> > > > >>>>> > >> > go
> > > > >>>>> > >> > about the other steps since I have zero
experience on
> > > > >>>>> > >> > how
> > > > >>>>> > >> > to do
> > > > >>>>> > >> > these
> > > > >>>>> > >> > things.
> > > > >>>>> > >> >
> > > > >>>>> > >> > #2. All parameters in <folders> section should
NOT be
> > > > >>>>> > >> > changed
> > > > >>>>> and
> > > > >>>>> > >> > will
> > > > >>>>> > >> > be
> > > > >>>>> > >> > replaced by other scripts during the batch
execution.
> > > > >>>>> > >> > The
> > > > >>>>> > >> > data
> > > > >>>>> for
> > > > >>>>> > >> > the
> > > > >>>>> > >> > batch mode is coming from the database and not
from
> > > > >>>>> > >> > files.
> > > > >>>>> > >> > The
> > > > >>>>> > >> > name of
> > > > >>>>> > >> > the
> > > > >>>>> > >> > database is in <database> tag.
> > > > >>>>> > >> >
> > > > >>>>> > >> > I understand that nothing should be changed in
the
> > > > >>>>> > >> > folders
> > > > >>>>> > >> > section,
> > > > >>>>> > >> > but
> > > > >>>>> > >> > there is no indication how and where these
variables
> > > > >>>>> > >> > are
> > > > >>>>> > >> > being
> > > > >>>>> > >> > defined.
> > > > >>>>> > >> > Grepping these names of the files available does
not
> > > > >>>>> > >> > reveal
> > > > >>>>> formal
> > > > >>>>> > >> > definitions of these variables. If I understood
> > > > >>>>> > >> > exactly
> > > > >>>>> > >> > how
> > > > >>>>> these
> > > > >>>>> > >> > were
> > > > >>>>> > >> > being defined, then I would be able connect this
> > > > >>>>> > >> > better.
> > > > >>>>> > >> >
> > > > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
> > > > >>>>> > >> > use
> > > > >>>>> METviewer
> > > > >>>>> > >> > load
> > > > >>>>> > >> > mode to load these files to the specific
database. In
> > > > >>>>> > >> > the
> > > > >>>>> > >> > config
> > > > >>>>> > >> > file
> > > > >>>>> > >> > for
> > > > >>>>> > >> > this process they also can specify the database
group.
> > > > >>>>> > >> > For
> > > > >>>>> example
> > > > >>>>> > >> > database
> > > > >>>>> > >> > mv_dev is in Test12 group.
> > > > >>>>> > >> > After the data has been loaded to the database it
is
> > > > >>>>> > >> > visible and
> > > > >>>>> > >> > available
> > > > >>>>> > >> > in the GUI and also can be used by the batch
mode.
> > > > >>>>> > >> > You can always add more data to the database
using the
> > > > >>>>> > >> > same load
> > > > >>>>> > >> > mode.
> > > > >>>>> > >> >
> > > > >>>>> > >> > Is there a guide for me to understand the process
that
> > > > >>>>> > >> > you
> > > > >>>>> > >> > are
> > > > >>>>> > >> > referring to?
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek
via RT
> > > > >>>>> > >> > <met_help at ucar.edu>
> > > > >>>>> > >> > wrote:
> > > > >>>>> > >> >
> > > > >>>>> > >> > > Hi Edward,
> > > > >>>>> > >> > > #1. I don't have an access to NOAA servers and
can't
> > > > >>>>> > >> > > view your
> > > > >>>>> > >> > > XML
> > > > >>>>> > >> > > for the
> > > > >>>>> > >> > > batch mode. Please, attache it to this issue
and I
> > > > >>>>> > >> > > will
> > > > >>>>> > >> > > take a
> > > > >>>>> > >> > > look.
> > > > >>>>> > >> > >
> > > > >>>>> > >> > > #2. All parameters in <folders> section should
NOT
> > > > >>>>> > >> > > be
> > > > >>>>> > >> > > changed
> > > > >>>>> > >> > > and
> > > > >>>>> > >> > > will be
> > > > >>>>> > >> > > replaced by other scripts during the batch
> > > > >>>>> > >> > > execution.
> > > > >>>>> > >> > > The data
> > > > >>>>> > >> > > for
> > > > >>>>> > >> > > the
> > > > >>>>> > >> > > batch mode is coming from the database and not
from
> > > > >>>>> > >> > > files. The
> > > > >>>>> > >> > > name
> > > > >>>>> > >> > > of the
> > > > >>>>> > >> > > database is in <database> tag.
> > > > >>>>> > >> > >
> > > > >>>>> > >> > > #3. Users rum MET and produce MET output files.
They
> > > > >>>>> > >> > > use
> > > > >>>>> > >> > > METviewer
> > > > >>>>> > >> > > load
> > > > >>>>> > >> > > mode to load these files to the specific
database.
> > > > >>>>> > >> > > In
> > > > >>>>> > >> > > the
> > > > >>>>> config
> > > > >>>>> > >> > > file
> > > > >>>>> > >> > > for
> > > > >>>>> > >> > > this process they also can specify the database
> > > > >>>>> > >> > > group.
> > > > >>>>> > >> > > For
> > > > >>>>> > >> > > example
> > > > >>>>> > >> > > database
> > > > >>>>> > >> > > mv_dev is in Test12 group.
> > > > >>>>> > >> > > After the data has been loaded to the database
it is
> > > > >>>>> > >> > > visible
> > > > >>>>> and
> > > > >>>>> > >> > > available
> > > > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
> > > > >>>>> > >> > > You can always add more data to the database
using
> > > > >>>>> > >> > > the
> > > > >>>>> > >> > > same
> > > > >>>>> load
> > > > >>>>> > >> > > mode.
> > > > >>>>> > >> > >
> > > > >>>>> > >> > > Tatiana
> > > > >>>>> > >> > >
> > > > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> > > > >>>>> > >> > > edward.strobach at noaa.gov
> > > > >>>>> > >> > > wrote:
> > > > >>>>> > >> > > > Good morning,
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Over the last two weeks I've made an effort
to
> > > > >>>>> > >> > > > understand
> > > > >>>>> > >> > > > metviewer,
> > > > >>>>> > >> > > > test
> > > > >>>>> > >> > > > different cases using the history tab,
changing
> > > > >>>>> > >> > > > information
> > > > >>>>> to
> > > > >>>>> > >> > > > see
> > > > >>>>> > >> > > > what
> > > > >>>>> > >> > > > works and what does not, and make connections
> > > > >>>>> > >> > > > between
> > > > >>>>> > >> > > > the
> > > > >>>>> GUI
> > > > >>>>> > >> > > > settings
> > > > >>>>> > >> > > > and
> > > > >>>>> > >> > > > the XML file. Although my comfort level with
> > > > >>>>> > >> > > > metviewer has
> > > > >>>>> > >> > > > significantly
> > > > >>>>> > >> > > > increased during this time, there are some
> > > > >>>>> > >> > > > remaining
> > > > >>>>> > >> > > > issues
> > > > >>>>> > >> > > > that
> > > > >>>>> > >> > > > I'm
> > > > >>>>> > >> > > > hoping
> > > > >>>>> > >> > > > to clear up.
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > 1. Problem changing XML file and running
with
> > > > >>>>> > >> > > > different
> > > > >>>>> > >> > > > statistic
> > > > >>>>> > >> > > > and
> > > > >>>>> > >> > > > plot
> > > > >>>>> > >> > > > type:
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided
me the
> > > > >>>>> > >> > > > working
> > > > >>>>> > >> > > > directory
> > > > >>>>> > >> > > > to get
> > > > >>>>> > >> > > > started which I copied over:
> > > > >>>>> > >> > > >
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > run his
> > > > >>>>> > >> > > > example.
> > > > >>>>> > >> > > > running of the script "bash test.sh" produced
a
> > > > >>>>> > >> > > > CSI
> > > > >>>>> > >> > > > plot
> > > > >>>>> that
> > > > >>>>> > >> > > > was
> > > > >>>>> > >> > > > stored
> > > > >>>>> > >> > > > here:
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > >
> > > > >>>>> > >>
> > > > >>>>>
> > >
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Running the script produced the expected
result.
> > > > >>>>> > >> > > > The
> > > > >>>>> > >> > > > next
> > > > >>>>> > >> > > > natural
> > > > >>>>> > >> > > > step was
> > > > >>>>> > >> > > > to copy hochum_xml.template and create a new
xml
> > > > >>>>> > >> > > > file
> > > > >>>>> > >> > > > called
> > > > >>>>> > >> > > > Strobach_xml.template in order to create a
time
> > > > >>>>> > >> > > > series
> > > > >>>>> > >> > > > where
> > > > >>>>> > >> > > > the
> > > > >>>>> > >> > > > independent time is fcst_valid_beg instead of
> > > > >>>>> > >> > > > obs_thresh.
> > > > >>>>> > >> > > > Accordingly,
> > > > >>>>> > >> > > > I
> > > > >>>>> > >> > > > made changes to the fixed fields (removing
> > > > >>>>> > >> > > > fcst_valid_beg)
> > > > >>>>> and
> > > > >>>>> > >> > > > changed
> > > > >>>>> > >> > > > the
> > > > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
tried).
> > > > >>>>> > >> > > > The
> > > > >>>>> error
> > > > >>>>> > >> > > > that
> > > > >>>>> > >> > > > resulted
> > > > >>>>> > >> > > > is as follows:
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-
01
> > > > >>>>> > >> > > > 13:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 01
> > > > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> > > > >>>>> > >> > > > 16:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 01
> > > > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> > > > >>>>> > >> > > > 19:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 01
> > > > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> > > > >>>>> > >> > > > 22:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 01
> > > > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> > > > >>>>> > >> > > > 01:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 02
> > > > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> > > > >>>>> > >> > > > 04:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 02
> > > > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> > > > >>>>> > >> > > > 07:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 02
> > > > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> > > > >>>>> > >> > > > 10:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 02
> > > > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > > > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> > > > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > > > >>>>> > >> > > > AND ld.fmean != -9999;
> > > > >>>>> > >> > > > No data was returned from database
> > > > >>>>> > >> > > > 2020-06-09
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > >
> > > > >>>>> > >>
> > > > >>>>>
> > >
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > > > >>>>> > >> > > > Source) -1 printFormattedTable - ** ERROR:
> > > > >>>>> > >> > > > Caught
> > > > >>>>> > >> > > > class
> > > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> > > > >>>>> > >> > > > printFormattedTable(ResultSet
> > > > >>>>> > >> > > > res): result set contained no data
> > > > >>>>> > >> > > > ** ERROR: Caught class
> > > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > > > >>>>> > >> > > > in
> > > > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
> > > > >>>>> > >> > > > contained no
> > > > >>>>> > >> > > > data
> > > > >>>>> > >> > > > ** ERROR: Caught class
> > > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > > > >>>>> > >> > > > in
> > > > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
> > > > >>>>> > >> > > > contained no
> > > > >>>>> > >> > > > data
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Apparently there was no data during this
time. I
> > > > >>>>> > >> > > > tried
> > > > >>>>> > >> > > > changing
> > > > >>>>> > >> > > > the
> > > > >>>>> > >> > > > range
> > > > >>>>> > >> > > > of valid times and changed the valid time to
> > > > >>>>> > >> > > > different
> > > > >>>>> > >> > > > days
> > > > >>>>> > >> > > > just in
> > > > >>>>> > >> > > > case
> > > > >>>>> > >> > > > data was missing. The issue was the same.
To
> > > > >>>>> > >> > > > confirm
> > > > >>>>> > >> > > > my
> > > > >>>>> > >> > > > logic, I
> > > > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
> > > > >>>>> > >> > > > which I
> > > > >>>>> > >> > > > found
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > work.
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > 2. Running in batch mode
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > I think I lack an understanding of how to set
up
> > > > >>>>> > >> > > > batch
> > > > >>>>> > >> > > > mode
> > > > >>>>> > >> > > > because
> > > > >>>>> > >> > > > I
> > > > >>>>> > >> > > > don't
> > > > >>>>> > >> > > > see where these definitions are being
explicitly
> > > > >>>>> > >> > > > defined:
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > <plot_spec>
> > > > >>>>> > >> > > > <connection>
> > > > >>>>> > >> > > > <host>rds_host:3306</host>
> > > > >>>>> > >> > > >
> > > > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > > > >>>>> > >> > > > <user>rds_user</user>
> > > > >>>>> > >> > > > <password>rds_pwd</password>
> > > > >>>>> > >> > > > </connection>
> > > > >>>>> > >> > > > <rscript>Rscript</rscript>
> > > > >>>>> > >> > > > <folders>
> > > > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > > > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > > > >>>>> > >> > > > <plots>rds_plots</plots>
> > > > >>>>> > >> > > > <data>rds_data</data>
> > > > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > > > >>>>> > >> > > > </folders>
> > > > >>>>> > >> > > > <plot>
> > > > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > I see the <folders> section, but grepping all
> > > > >>>>> > >> > > > available
> > > > >>>>> files
> > > > >>>>> > >> > > > does
> > > > >>>>> > >> > > > not
> > > > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well as
> > > > >>>>> > >> > > > other
> > > > >>>>> > >> > > > similarly
> > > > >>>>> > >> > > > defined
> > > > >>>>> > >> > > > variables. I think if I know how these are
being
> > > > >>>>> > >> > > > defined,
> > > > >>>>> > >> > > > then
> > > > >>>>> > >> > > > I'll
> > > > >>>>> > >> > > > be
> > > > >>>>> > >> > > > able to pipe in my information and process
the
> > > > >>>>> > >> > > > data
> > > > >>>>> > >> > > > using
> > > > >>>>> met
> > > > >>>>> > >> > > > and
> > > > >>>>> > >> > > > produce
> > > > >>>>> > >> > > > results in batch mode. Am I right on that?
Also,
> > > > >>>>> > >> > > > if
> > > > >>>>> > >> > > > I want
> > > > >>>>> > >> > > > to use
> > > > >>>>> > >> > > > obs for
> > > > >>>>> > >> > > > verification, do I add another line in the
XML
> > > > >>>>> > >> > > > file
> > > > >>>>> > >> > > > that
> > > > >>>>> > >> > > > designates
> > > > >>>>> > >> > > > file
> > > > >>>>> > >> > > > location of obs? I assume that other steps
have
> > > > >>>>> > >> > > > to be
> > > > >>>>> > >> > > > run
> > > > >>>>> > >> > > > before
> > > > >>>>> > >> > > > all
> > > > >>>>> > >> > > > this. If data is in prepbufr form, for
example, I
> > > > >>>>> > >> > > > would
> > > > >>>>> have
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > separately
> > > > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I would
want
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > do that
> > > > >>>>> > >> > > > as
> > > > >>>>> > >> > > > well.
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > 3. Communicating results to metviewer online
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
here
> > > > >>>>> > >> > > > -
> > > > >>>>> > >> > > > see
> > > > >>>>> > >> > > > attachment
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > How do I get the results I generate to be
uploaded
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > this
> > > > >>>>> > >> > > > location?
> > > > >>>>> > >> > > > It's
> > > > >>>>> > >> > > > not clear how others are setting this up.
> > > > >>>>> > >> > >
> > > > >>>>> > >> > >
> > > > >>>>> > >> > >
> > > > >>>>> > >> > >
> > > > >>>>> > >>
> > > > >>>>> > >>
> > > > >>>>> > >>
> > > > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
> > > > >>>>> > >> rds_user
> > > > >>>>> rds_pwd
> > > > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
> > > > >>>>> rds_scripts
> > > > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
CMAQAODPRODHIGH
> > > > >>>>> 2020-05-12
> > > > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
> > > > >>>>> > >> 05-12
> > > > >>>>> 20:00:00
> > > > >>>>> > >> 2020-05-12
> > > > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
> > > > >>>>> > >> 05-13
> > > > >>>>> 10:00:00
> > > > >>>>> > >> 2020-05-13
> > > > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
> > > > >>>>> > >> 05-13
> > > > >>>>> 14:00:00
> > > > >>>>> > >> 2020-05-13
> > > > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
> > > > >>>>> > >> 05-13
> > > > >>>>> 18:00:00
> > > > >>>>> > >> 2020-05-13
> > > > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
> > > > >>>>> > >> 05-13
> > > > >>>>> 22:00:00
> > > > >>>>> > >> 2020-05-13
> > > > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
> > > > >>>>> > >> 05-14
> > > > >>>>> 12:00:00
> > > > >>>>> > >> true
> > > > >>>>> > >> median plot_20200612_173626.data
> > > > >>>>> > >> plot_20200612_173626.png
> > > > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> > > > >>>>> > >> false list() list() false false false false false
false
> > > > >>>>> > >> true
> > > > >>>>> > >> false
> > > > >>>>> > >> false
> > > > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
> > > > >>>>> > >> c(1,1,0) 1 2
> > > > >>>>> > >> 1.4 -2
> > > > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2
0.5 3
> > > > >>>>> > >> #cccccc 1
> > > > >>>>> > >> listX 1
> > > > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
> > > > >>>>> > >> 3
> > > > >>>>> > >> 0.8 1
> > > > >>>>> > >> #333333
> > > > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
> > > > >>>>> > >> c(TRUE,TRUE)
> > > > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
> > > > >>>>> > >> c(1,1)
> > > > >>>>> c(1,1)
> > > > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > > > >>>>> > >
> > > > >>>>> > >
> > > > >>>>> > >
> > > > >>>>> > > --
> > > > >>>>> > > Edward Strobach
> > > > >>>>> > > EMC/NCEP/NWS/
> > > > >>>>> > > IMSG Contractor
> > > > >>>>> > > Cubicle#: 2029
> > > > >>>>> > > 301-683-3717
> > > > >>>>> > >
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Edward Strobach
> > > > >>>> EMC/NCEP/NWS/
> > > > >>>> IMSG Contractor
> > > > >>>> Cubicle#: 2029
> > > > >>>> 301-683-3717
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Edward Strobach
> > > > >>> EMC/NCEP/NWS/
> > > > >>> IMSG Contractor
> > > > >>> Cubicle#: 2029
> > > > >>> 301-683-3717
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Edward Strobach
> > > > >> EMC/NCEP/NWS/
> > > > >> IMSG Contractor
> > > > >> Cubicle#: 2029
> > > > >> 301-683-3717
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 10:05:58 2020
I'm not following you. How do I now have enough space? What changed?
On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Ed, now you should have enough space for the data loading.
>
> Tatiana
>
> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
> > I was basically able to build my xml file but it says I have no
> > available
> > space. I'm not sure how that can be since I just started adding
data
> > to my
> > account
> >
> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space
left
> > on
> > device
> > ERROR: Command returned with non-zero status (1): scp -r ./*
> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
> > 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
> > TABLE_SCHEMA USER SIZE_MB
> > mv_met_g2o_gefs binbin.zhou
140027.31
> > mv_met_system_hrefv2_v3 binbin.zhou
71986.11
> >
> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > Thanks Tatiana. Yes, I've created a database which seems
different
> > > from
> > > mv_group. How can one create their own group so that I can store
and
> > > access
> > > the database. The database exists but cannot be viewed on the
> > > metviewer
> > > GUI. The list starts with EMC HREFv3 and ends with vhagerty. I
> > > named my
> > > database Ed_Strobach, but that was really intended for the group
> > > name.
> > >
> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > >> We don't have database and METviewer running on wcoss.
> > >> Instead, users use METviewer-wcoss scripts that help to connect
to
> > >> and
> > >> use METviewer that is running on AWS. That is why XML
connection and
> > >> folder
> > >> sections look somewhat different than once from the Docs page.
> > >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
> > >> mv_create_db_on_aws.sh <user_name> <database_name>
> > >> For data loading :
> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> > >> And for batch:
> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> > >>
> > >> These scripts copy XML and data files if needed to AWS, adjust
data
> > >> path
> > >> and users credentials, run METviewer and copy results back to
wcoss.
> > >>
> > >> Tatiana
> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
> > >> > -e'create
> > >> > database [db_name]
> > >> >
> > >> > would need to be used to create a database. I assume that an
sql
> > >> > module is
> > >> > available on wcoss; I know you don't have access to those
machines
> > >> > Tatiana. If I create my own database with my own name, then
I
> > >> > should
> > >> > be
> > >> > able to proceed with project specific tasks under that tab.
> > >> > Something
> > >> > like
> > >> > mv_EdwardStrobach.
> > >> >
> > >> > I can then create mv_cmaq_prod_v_para which I add data to
using
> > >> > load_spec.
> > >> > Once that step is done, then I can run plot_spec. That's my
> > >> > takeaway
> > >> > currently
> > >> >
> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
> > >> > edward.strobach at noaa.gov> wrote:
> > >> >
> > >> > > Actually, it appears that I need to run the database
loading
> > >> > > module
> > >> > > first
> > >> > > if I want to change/add a data source. From there I can
run the
> > >> > > plotting
> > >> > > batch module. Is that right? So if I run the database
loading
> > >> > > module,
> > >> > > then I can specify my dates and other related info specific
to
> > >> > > the
> > >> > > dataset. If I create my database with a unique name, then
it
> > >> > > should
> > >> > > be
> > >> > > recognized on the metviewer gui, presumably.
> > >> > >
> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
> > >> > > <
> > >> > > edward.strobach at noaa.gov> wrote:
> > >> > >
> > >> > >> It seems that those options must be treated separately
outside
> > >> > >> of
> > >> > >> plot_spec. Is it acceptable to create both a load_spec
and
> > >> > >> plot_spec in
> > >> > >> the same xml file? Would the definitions in load_spec be
able
> > >> > >> to
> > >> > >> carry
> > >> > >> over into plot_spec? It appears that data loading and
plotting
> > >> > >> are
> > >> > >> treated
> > >> > >> separately.
> > >> > >>
> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
> > >> > >> Affiliate <
> > >> > >> edward.strobach at noaa.gov> wrote:
> > >> > >>
> > >> > >>> I do plan to use date_range; however, in an example
provided
> > >> > >>> by the
> > >> > >>> link
> > >> > >>> I see this:
> > >> > >>>
> > >> > >>> <load_spec>
> > >> > >>> <connection>
> > >> > >>> <host>kemosabe:3306</host>
> > >> > >>> <database>metvdb_hwt</database>
> > >> > >>> <user>pgoldenb</user>
> > >> > >>> <password>pgoldenb</password>
> > >> > >>> </connection>
> > >> > >>>
> > >> > >>> <date_list name="folder_dates">
> > >> > >>> <start>2010051914V</start>
> > >> > >>> <end><date_offset><day_offset>0</day_offset><hour>6
> > >> > >>> </hour></date_offset></end>
> > >> > >>> <inc>3600</inc>
> > >> > >>> <format>yyyyMMddHH'V'</format>
> > >> > >>> </date_list>
> > >> > >>>
> > >> > >>> <met_version>V3.0</met_version>
> > >> > >>>
> > >> > >>> <verbose>false</verbose>
> > >> > >>> <insert_size>1</insert_size>
> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
> > >> > >>> <drop_indexes>false</drop_indexes>
> > >> > >>> <apply_indexes>true</apply_indexes>
> > >> > >>> <group>Group name</group>
> > >> > >>> <load_stat>true</load_stat>
> > >> > >>> <load_mode>true</load_mode>
> > >> > >>> <load_mtd>true</load_mtd>
> > >> > >>> <load_mpr>false</load_mpr>
> > >> > >>>
> > >> > >>>
> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > >> > >>> <load_val>
> > >> > >>> <field name="model">
> > >> > >>> <val>arw</val>
> > >> > >>> <val>nmm</val>
> > >> > >>> </field>
> > >> > >>>
> > >> > >>> <field name="valid_time">
> > >> > >>> <date_list name="folder_dates"/>
> > >> > >>> </field>
> > >> > >>>
> > >> > >>> <field name="vx_mask">
> > >> > >>> <val>FULL</val>
> > >> > >>> <val>SWC</val>
> > >> > >>> </field>
> > >> > >>> </load_val>
> > >> > >>> </load_spec>
> > >> > >>>
> > >> > >>> Are similar settings allowed with <plot_spec>? I see
> > >> > >>> <connection>,
> > >> > >>> which seems carried over. I'm looking to specify my
> > >> > >>> folder_tmpl in
> > >> > >>> <plot_spec>, which according to Tara, should be possible.
> > >> > >>> Since my
> > >> > >>> directory structure looks like this:
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > >> > >>>
> > >> > >>> I would do something like this I would think:
> > >> > >>>
> > >> > >>> .......
> > >> > >>>
> > >> > >>> <date_list name="folder_dates">
> > >> > >>> <start>20200612/start>
> > >> > >>> <end><date_offset><day_offset>0</day_offset></end>
> > >> > >>> <inc>86400</inc>
> > >> > >>> <format>yyyyMMdd</format>
> > >> > >>> </date_list>
> > >> > >>>
> > >> > >>> <folder_tmpl>
> > >> > >>>
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > >> > >>> </folder_tmpl>
> > >> > >>> <load_val>
> > >> > >>> <field name="model">
> > >> > >>> <val>prod</val>
> > >> > >>> </field>
> > >> > >>>
> > >> > >>> <field name="valid_time">
> > >> > >>> <date_list name="folder_dates"/>
> > >> > >>> </field>
> > >> > >>>
> > >> > >>> ......
> > >> > >>>
> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
> > >> > >>> Affiliate <
> > >> > >>> edward.strobach at noaa.gov> wrote:
> > >> > >>>
> > >> > >>>> Thank you. This is a very helpful example
> > >> > >>>>
> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> > >> > >>>> <met_help at ucar.edu>
> > >> > >>>> wrote:
> > >> > >>>>
> > >> > >>>>> Hi,
> > >> > >>>>> Here is some examples how to use <date_range> and
> > >> > >>>>> <date_range_list> in
> > >> > >>>>> batch XML.
> > >> > >>>>> First, you need to defined your range. You can have
more
> > >> > >>>>> then one
> > >> > >>>>> <date_range>:
> > >> > >>>>> <date_range name="may">
> > >> > >>>>> <start>2019-05-01 00:00:00</start>
> > >> > >>>>> <end>2019-05-31 23:59:59</end>
> > >> > >>>>> </date_range>
> > >> > >>>>> <date_range name="march">
> > >> > >>>>> <start>2019-03-01 00:00:00</start>
> > >> > >>>>> <end>2019-03-31 23:59:59</end>
> > >> > >>>>> </date_range>
> > >> > >>>>>
> > >> > >>>>> and then use one or all date_ranges:
> > >> > >>>>> <plot_fix>
> > >> > >>>>> <field name="fcst_init_beg">
> > >> > >>>>> <date_range name="march"/>
> > >> > >>>>> <date_range name="may"/>
> > >> > >>>>> </field>...
> > >> > >>>>> This configuration would include all dates for March
and May
> > >> > >>>>> of
> > >> > >>>>> 2019.
> > >> > >>>>>
> > >> > >>>>> If you need more flexible configuration you can use
> > >> > >>>>> date_range_list.
> > >> > >>>>> For example, you want to select first 6 hours every day
> > >> > >>>>> beetween
> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
> > >> > >>>>> look
> > >> > >>>>> like this:
> > >> > >>>>> <date_range_list name = 'all'>
> > >> > >>>>> <range_start>2020-01-01 00:00:00</range_start>
> > >> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
> > >> > >>>>> <range_length>360</range_length>
> > >> > >>>>> <inc>1440</inc>
> > >> > >>>>> </date_range_list>
> > >> > >>>>> where the 1st and 2nd values are the start and end,
> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
> > >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> > >> > >>>>> This example would create following ranges:
> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> > >> > >>>>>
> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
> > >> > >>>>> your
> > >> > >>>>> data.
> > >> > >>>>> For example, if you want to select only dates with
00:00 you
> > >> > >>>>> can
> > >> > >>>>> specify
> > >> > >>>>> <field name="valid_hour">
> > >> > >>>>> <val>00</val>
> > >> > >>>>> </field>
> > >> > >>>>>
> > >> > >>>>> Tatiana
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
> > >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
> > >> > >>>>> > of a
> > >> > >>>>> > separate
> > >> > >>>>> > script that pipes dates (and other relevant
information)
> > >> > >>>>> > into
> > >> > >>>>> > the XML
> > >> > >>>>> > file
> > >> > >>>>> > used to generate the results.
> > >> > >>>>> >
> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
> > >> > >>>>> > Affiliate <
> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
> > >> > >>>>> >
> > >> > >>>>> > > Thanks Tatiana. These resources are helpful.
> > >> > >>>>> > >
> > >> > >>>>> > > One last question. Simply typing lines like this
in can
> > >> > >>>>> > > take
> > >> > >>>>> > > forever:
> > >> > >>>>> > >
> > >> > >>>>> > > <plot_fix>
> > >> > >>>>> > > <field equalize="false" name="fcst_init_beg">
> > >> > >>>>> > > <set name="fcst_init_beg_0">
> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
> > >> > >>>>> > >
> > >> > >>>>> > >
> > >> > >>>>> > > Is there a better way? I see date_range,
> > >> > >>>>> > > date_range_list,
> > >> > >>>>> > > or
> > >> > >>>>> > > date_list.
> > >> > >>>>> > > Can this be used instead of typing individual
values?
> > >> > >>>>> > > This
> > >> > >>>>> > > could
> > >> > >>>>> > > prove
> > >> > >>>>> > > pretty daunting if one would have to type in 700
entries
> > >> > >>>>> > > each
> > >> > >>>>> > > time.
> > >> > >>>>> > >
> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via
RT
> > >> > >>>>> > > <met_help at ucar.edu>
> > >> > >>>>> > > wrote:
> > >> > >>>>> > >
> > >> > >>>>> > >> I found a few problems in your XML.
> > >> > >>>>> > >> The database you are using doesn't have data with
> > >> > >>>>> > >> vx_mask
> > >> > >>>>> > >> 'abcREGabc'.
> > >> > >>>>> > >> That is why METviewer returns an empty dataset and
> > >> > >>>>> > >> can't
> > >> > >>>>> > >> create a
> > >> > >>>>> > >> plot.
> > >> > >>>>> > >> The other problem is how you define fcst_lead.
This
> > >> > >>>>> > >> database
> > >> > >>>>> > >> has
> > >> > >>>>> > >> values
> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since
you
> > >> > >>>>> > >> include
> > >> > >>>>> > >> all
> > >> > >>>>> > >> fcst_leads
> > >> > >>>>> > >> this variable can be omitted.
> > >> > >>>>> > >> To minimize problems with variables values, I
would
> > >> > >>>>> > >> suggest
> > >> > >>>>> > >> to
> > >> > >>>>> > >> create XML
> > >> > >>>>> > >> using GUI and then edit it to fit batch standards.
> > >> > >>>>> > >> I created a plot using you configurations with GUI
> > >> > >>>>> > >> (without
> > >> > >>>>> vx_mack
> > >> > >>>>> > >> and
> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You can
load it
> > >> > >>>>> > >> back
> > >> > >>>>> > >> to
> > >> > >>>>> the
> > >> > >>>>> > >> page by
> > >> > >>>>> > >> using the "History" tab (left bar on GUI). Find
the
> > >> > >>>>> > >> plot,
> > >> > >>>>> > >> open XML
> > >> > >>>>> > >> and
> > >> > >>>>> > >> click on "Load to page". After that you can
recreate a
> > >> > >>>>> > >> plot
> > >> > >>>>> > >> and
> > >> > >>>>> edit
> > >> > >>>>> > >> the
> > >> > >>>>> > >> configurations. You can also download a new XML
and
> > >> > >>>>> > >> edit it
> > >> > >>>>> > >> for
> > >> > >>>>> > >> batch.
> > >> > >>>>> > >> I attached the example of load XML to this
message.
> > >> > >>>>> > >>
> > >> > >>>>> > >> Description of rds parameters:
> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer
R
> > >> > >>>>> > >> scripts
> > >> > >>>>> > >> rds_plots - temporary directory where the plots
will be
> > >> > >>>>> > >> generated
> > >> > >>>>> > >> rds_data - temporary directory where the data
retrieved
> > >> > >>>>> > >> from
> > >> > >>>>> > >> the
> > >> > >>>>> > >> database
> > >> > >>>>> > >> and other intermediate files will be stored
> > >> > >>>>> > >> rds_scripts - temporary directory where R scripts
for
> > >> > >>>>> > >> this
> > >> > >>>>> > >> run
> > >> > >>>>> will
> > >> > >>>>> > >> be
> > >> > >>>>> > >> stored
> > >> > >>>>> > >> After the execution is done all the files from
> > >> > >>>>> > >> temporary
> > >> > >>>>> directories
> > >> > >>>>> > >> would be removed.
> > >> > >>>>> > >>
> > >> > >>>>> > >>
> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
> > >> > >>>>> > >> There
> > >> > >>>>> > >> are
> > >> > >>>>> some
> > >> > >>>>> > >> instructions on how to crate XML config files:
> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > >> > >>>>> > >>
> > >> > >>>>> > >> Tatiana
> > >> > >>>>> > >>
> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
> > >> > >>>>> > >> wrote:
> > >> > >>>>> > >> > Hi,
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > I'm attaching the xml below, but am having hard
time
> > >> > >>>>> > >> > following
> > >> > >>>>> how
> > >> > >>>>> > >> > to
> > >> > >>>>> > >> > go
> > >> > >>>>> > >> > about the other steps since I have zero
experience on
> > >> > >>>>> > >> > how
> > >> > >>>>> > >> > to do
> > >> > >>>>> > >> > these
> > >> > >>>>> > >> > things.
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > #2. All parameters in <folders> section should
NOT be
> > >> > >>>>> > >> > changed
> > >> > >>>>> and
> > >> > >>>>> > >> > will
> > >> > >>>>> > >> > be
> > >> > >>>>> > >> > replaced by other scripts during the batch
execution.
> > >> > >>>>> > >> > The
> > >> > >>>>> > >> > data
> > >> > >>>>> for
> > >> > >>>>> > >> > the
> > >> > >>>>> > >> > batch mode is coming from the database and not
from
> > >> > >>>>> > >> > files.
> > >> > >>>>> > >> > The
> > >> > >>>>> > >> > name of
> > >> > >>>>> > >> > the
> > >> > >>>>> > >> > database is in <database> tag.
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > I understand that nothing should be changed in
the
> > >> > >>>>> > >> > folders
> > >> > >>>>> > >> > section,
> > >> > >>>>> > >> > but
> > >> > >>>>> > >> > there is no indication how and where these
variables
> > >> > >>>>> > >> > are
> > >> > >>>>> > >> > being
> > >> > >>>>> > >> > defined.
> > >> > >>>>> > >> > Grepping these names of the files available does
not
> > >> > >>>>> > >> > reveal
> > >> > >>>>> formal
> > >> > >>>>> > >> > definitions of these variables. If I understood
> > >> > >>>>> > >> > exactly
> > >> > >>>>> > >> > how
> > >> > >>>>> these
> > >> > >>>>> > >> > were
> > >> > >>>>> > >> > being defined, then I would be able connect this
> > >> > >>>>> > >> > better.
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
> > >> > >>>>> > >> > use
> > >> > >>>>> METviewer
> > >> > >>>>> > >> > load
> > >> > >>>>> > >> > mode to load these files to the specific
database. In
> > >> > >>>>> > >> > the
> > >> > >>>>> > >> > config
> > >> > >>>>> > >> > file
> > >> > >>>>> > >> > for
> > >> > >>>>> > >> > this process they also can specify the database
> > >> > >>>>> > >> > group. For
> > >> > >>>>> example
> > >> > >>>>> > >> > database
> > >> > >>>>> > >> > mv_dev is in Test12 group.
> > >> > >>>>> > >> > After the data has been loaded to the database
it is
> > >> > >>>>> > >> > visible and
> > >> > >>>>> > >> > available
> > >> > >>>>> > >> > in the GUI and also can be used by the batch
mode.
> > >> > >>>>> > >> > You can always add more data to the database
using
> > >> > >>>>> > >> > the
> > >> > >>>>> > >> > same load
> > >> > >>>>> > >> > mode.
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > Is there a guide for me to understand the
process
> > >> > >>>>> > >> > that you
> > >> > >>>>> > >> > are
> > >> > >>>>> > >> > referring to?
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek
via RT
> > >> > >>>>> > >> > <met_help at ucar.edu>
> > >> > >>>>> > >> > wrote:
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > > Hi Edward,
> > >> > >>>>> > >> > > #1. I don't have an access to NOAA servers and
> > >> > >>>>> > >> > > can't
> > >> > >>>>> > >> > > view your
> > >> > >>>>> > >> > > XML
> > >> > >>>>> > >> > > for the
> > >> > >>>>> > >> > > batch mode. Please, attache it to this issue
and I
> > >> > >>>>> > >> > > will
> > >> > >>>>> > >> > > take a
> > >> > >>>>> > >> > > look.
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > > #2. All parameters in <folders> section should
NOT
> > >> > >>>>> > >> > > be
> > >> > >>>>> > >> > > changed
> > >> > >>>>> > >> > > and
> > >> > >>>>> > >> > > will be
> > >> > >>>>> > >> > > replaced by other scripts during the batch
> > >> > >>>>> > >> > > execution.
> > >> > >>>>> > >> > > The data
> > >> > >>>>> > >> > > for
> > >> > >>>>> > >> > > the
> > >> > >>>>> > >> > > batch mode is coming from the database and not
from
> > >> > >>>>> > >> > > files. The
> > >> > >>>>> > >> > > name
> > >> > >>>>> > >> > > of the
> > >> > >>>>> > >> > > database is in <database> tag.
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
> > >> > >>>>> > >> > > They use
> > >> > >>>>> > >> > > METviewer
> > >> > >>>>> > >> > > load
> > >> > >>>>> > >> > > mode to load these files to the specific
database.
> > >> > >>>>> > >> > > In
> > >> > >>>>> > >> > > the
> > >> > >>>>> config
> > >> > >>>>> > >> > > file
> > >> > >>>>> > >> > > for
> > >> > >>>>> > >> > > this process they also can specify the
database
> > >> > >>>>> > >> > > group.
> > >> > >>>>> > >> > > For
> > >> > >>>>> > >> > > example
> > >> > >>>>> > >> > > database
> > >> > >>>>> > >> > > mv_dev is in Test12 group.
> > >> > >>>>> > >> > > After the data has been loaded to the database
it
> > >> > >>>>> > >> > > is
> > >> > >>>>> > >> > > visible
> > >> > >>>>> and
> > >> > >>>>> > >> > > available
> > >> > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
> > >> > >>>>> > >> > > You can always add more data to the database
using
> > >> > >>>>> > >> > > the
> > >> > >>>>> > >> > > same
> > >> > >>>>> load
> > >> > >>>>> > >> > > mode.
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > > Tatiana
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> > >> > >>>>> > >> > > edward.strobach at noaa.gov
> > >> > >>>>> > >> > > wrote:
> > >> > >>>>> > >> > > > Good morning,
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Over the last two weeks I've made an effort
to
> > >> > >>>>> > >> > > > understand
> > >> > >>>>> > >> > > > metviewer,
> > >> > >>>>> > >> > > > test
> > >> > >>>>> > >> > > > different cases using the history tab,
changing
> > >> > >>>>> > >> > > > information
> > >> > >>>>> to
> > >> > >>>>> > >> > > > see
> > >> > >>>>> > >> > > > what
> > >> > >>>>> > >> > > > works and what does not, and make
connections
> > >> > >>>>> > >> > > > between
> > >> > >>>>> > >> > > > the
> > >> > >>>>> GUI
> > >> > >>>>> > >> > > > settings
> > >> > >>>>> > >> > > > and
> > >> > >>>>> > >> > > > the XML file. Although my comfort level
with
> > >> > >>>>> > >> > > > metviewer has
> > >> > >>>>> > >> > > > significantly
> > >> > >>>>> > >> > > > increased during this time, there are some
> > >> > >>>>> > >> > > > remaining
> > >> > >>>>> > >> > > > issues
> > >> > >>>>> > >> > > > that
> > >> > >>>>> > >> > > > I'm
> > >> > >>>>> > >> > > > hoping
> > >> > >>>>> > >> > > > to clear up.
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > 1. Problem changing XML file and running
with
> > >> > >>>>> > >> > > > different
> > >> > >>>>> > >> > > > statistic
> > >> > >>>>> > >> > > > and
> > >> > >>>>> > >> > > > plot
> > >> > >>>>> > >> > > > type:
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided
me
> > >> > >>>>> > >> > > > the
> > >> > >>>>> > >> > > > working
> > >> > >>>>> > >> > > > directory
> > >> > >>>>> > >> > > > to get
> > >> > >>>>> > >> > > > started which I copied over:
> > >> > >>>>> > >> > > >
> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
> > >> > >>>>> > >> > > > to
> > >> > >>>>> > >> > > > run his
> > >> > >>>>> > >> > > > example.
> > >> > >>>>> > >> > > > running of the script "bash test.sh"
produced a
> > >> > >>>>> > >> > > > CSI
> > >> > >>>>> > >> > > > plot
> > >> > >>>>> that
> > >> > >>>>> > >> > > > was
> > >> > >>>>> > >> > > > stored
> > >> > >>>>> > >> > > > here:
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >>
> > >> > >>>>>
> > >>
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Running the script produced the expected
result.
> > >> > >>>>> > >> > > > The
> > >> > >>>>> > >> > > > next
> > >> > >>>>> > >> > > > natural
> > >> > >>>>> > >> > > > step was
> > >> > >>>>> > >> > > > to copy hochum_xml.template and create a new
xml
> > >> > >>>>> > >> > > > file
> > >> > >>>>> > >> > > > called
> > >> > >>>>> > >> > > > Strobach_xml.template in order to create a
time
> > >> > >>>>> > >> > > > series
> > >> > >>>>> > >> > > > where
> > >> > >>>>> > >> > > > the
> > >> > >>>>> > >> > > > independent time is fcst_valid_beg instead
of
> > >> > >>>>> > >> > > > obs_thresh.
> > >> > >>>>> > >> > > > Accordingly,
> > >> > >>>>> > >> > > > I
> > >> > >>>>> > >> > > > made changes to the fixed fields (removing
> > >> > >>>>> > >> > > > fcst_valid_beg)
> > >> > >>>>> and
> > >> > >>>>> > >> > > > changed
> > >> > >>>>> > >> > > > the
> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
> > >> > >>>>> > >> > > > tried).
> > >> > >>>>> > >> > > > The
> > >> > >>>>> error
> > >> > >>>>> > >> > > > that
> > >> > >>>>> > >> > > > resulted
> > >> > >>>>> > >> > > > is as follows:
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-05-
01
> > >> > >>>>> > >> > > > 13:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 01
> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-
01
> > >> > >>>>> > >> > > > 16:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 01
> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-
01
> > >> > >>>>> > >> > > > 19:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 01
> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-
01
> > >> > >>>>> > >> > > > 22:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 01
> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-
02
> > >> > >>>>> > >> > > > 01:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 02
> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-
02
> > >> > >>>>> > >> > > > 04:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 02
> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-
02
> > >> > >>>>> > >> > > > 07:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 02
> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-
02
> > >> > >>>>> > >> > > > 10:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 02
> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> > >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
> > >> > >>>>> > >> > > > No data was returned from database
> > >> > >>>>> > >> > > > 2020-06-09
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >>
> > >> > >>>>>
> > >>
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
ERROR:
> > >> > >>>>> > >> > > > Caught
> > >> > >>>>> > >> > > > class
> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
in
> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
> > >> > >>>>> > >> > > > res): result set contained no data
> > >> > >>>>> > >> > > > ** ERROR: Caught class
> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >> > >>>>> > >> > > > in
> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
> > >> > >>>>> > >> > > > contained no
> > >> > >>>>> > >> > > > data
> > >> > >>>>> > >> > > > ** ERROR: Caught class
> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >> > >>>>> > >> > > > in
> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
> > >> > >>>>> > >> > > > contained no
> > >> > >>>>> > >> > > > data
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Apparently there was no data during this
time. I
> > >> > >>>>> > >> > > > tried
> > >> > >>>>> > >> > > > changing
> > >> > >>>>> > >> > > > the
> > >> > >>>>> > >> > > > range
> > >> > >>>>> > >> > > > of valid times and changed the valid time to
> > >> > >>>>> > >> > > > different
> > >> > >>>>> > >> > > > days
> > >> > >>>>> > >> > > > just in
> > >> > >>>>> > >> > > > case
> > >> > >>>>> > >> > > > data was missing. The issue was the same.
To
> > >> > >>>>> > >> > > > confirm
> > >> > >>>>> > >> > > > my
> > >> > >>>>> > >> > > > logic, I
> > >> > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
> > >> > >>>>> > >> > > > which I
> > >> > >>>>> > >> > > > found
> > >> > >>>>> > >> > > > to
> > >> > >>>>> > >> > > > work.
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > 2. Running in batch mode
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > I think I lack an understanding of how to
set up
> > >> > >>>>> > >> > > > batch
> > >> > >>>>> > >> > > > mode
> > >> > >>>>> > >> > > > because
> > >> > >>>>> > >> > > > I
> > >> > >>>>> > >> > > > don't
> > >> > >>>>> > >> > > > see where these definitions are being
explicitly
> > >> > >>>>> > >> > > > defined:
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > <plot_spec>
> > >> > >>>>> > >> > > > <connection>
> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
> > >> > >>>>> > >> > > >
> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > >> > >>>>> > >> > > > <user>rds_user</user>
> > >> > >>>>> > >> > > > <password>rds_pwd</password>
> > >> > >>>>> > >> > > > </connection>
> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
> > >> > >>>>> > >> > > > <folders>
> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
> > >> > >>>>> > >> > > > <data>rds_data</data>
> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > >> > >>>>> > >> > > > </folders>
> > >> > >>>>> > >> > > > <plot>
> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > I see the <folders> section, but grepping
all
> > >> > >>>>> > >> > > > available
> > >> > >>>>> files
> > >> > >>>>> > >> > > > does
> > >> > >>>>> > >> > > > not
> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well
> > >> > >>>>> > >> > > > as
> > >> > >>>>> > >> > > > other
> > >> > >>>>> > >> > > > similarly
> > >> > >>>>> > >> > > > defined
> > >> > >>>>> > >> > > > variables. I think if I know how these are
being
> > >> > >>>>> > >> > > > defined,
> > >> > >>>>> > >> > > > then
> > >> > >>>>> > >> > > > I'll
> > >> > >>>>> > >> > > > be
> > >> > >>>>> > >> > > > able to pipe in my information and process
the
> > >> > >>>>> > >> > > > data
> > >> > >>>>> > >> > > > using
> > >> > >>>>> met
> > >> > >>>>> > >> > > > and
> > >> > >>>>> > >> > > > produce
> > >> > >>>>> > >> > > > results in batch mode. Am I right on that?
> > >> > >>>>> > >> > > > Also, if
> > >> > >>>>> > >> > > > I want
> > >> > >>>>> > >> > > > to use
> > >> > >>>>> > >> > > > obs for
> > >> > >>>>> > >> > > > verification, do I add another line in the
XML
> > >> > >>>>> > >> > > > file
> > >> > >>>>> > >> > > > that
> > >> > >>>>> > >> > > > designates
> > >> > >>>>> > >> > > > file
> > >> > >>>>> > >> > > > location of obs? I assume that other steps
have
> > >> > >>>>> > >> > > > to be
> > >> > >>>>> > >> > > > run
> > >> > >>>>> > >> > > > before
> > >> > >>>>> > >> > > > all
> > >> > >>>>> > >> > > > this. If data is in prepbufr form, for
example,
> > >> > >>>>> > >> > > > I
> > >> > >>>>> > >> > > > would
> > >> > >>>>> have
> > >> > >>>>> > >> > > > to
> > >> > >>>>> > >> > > > separately
> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I would
want
> > >> > >>>>> > >> > > > to
> > >> > >>>>> > >> > > > do that
> > >> > >>>>> > >> > > > as
> > >> > >>>>> > >> > > > well.
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > 3. Communicating results to metviewer online
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
> > >> > >>>>> > >> > > > here -
> > >> > >>>>> > >> > > > see
> > >> > >>>>> > >> > > > attachment
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > How do I get the results I generate to be
> > >> > >>>>> > >> > > > uploaded to
> > >> > >>>>> > >> > > > this
> > >> > >>>>> > >> > > > location?
> > >> > >>>>> > >> > > > It's
> > >> > >>>>> > >> > > > not clear how others are setting this up.
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >>
> > >> > >>>>> > >>
> > >> > >>>>> > >>
> > >> > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
> > >> > >>>>> > >> rds_user
> > >> > >>>>> rds_pwd
> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
> > >> > >>>>> rds_scripts
> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
> > >> > >>>>> > >> CMAQAODPRODHIGH
> > >> > >>>>> 2020-05-12
> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
> > >> > >>>>> > >> 05-12
> > >> > >>>>> 20:00:00
> > >> > >>>>> > >> 2020-05-12
> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
> > >> > >>>>> > >> 05-13
> > >> > >>>>> 10:00:00
> > >> > >>>>> > >> 2020-05-13
> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
> > >> > >>>>> > >> 05-13
> > >> > >>>>> 14:00:00
> > >> > >>>>> > >> 2020-05-13
> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
> > >> > >>>>> > >> 05-13
> > >> > >>>>> 18:00:00
> > >> > >>>>> > >> 2020-05-13
> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
> > >> > >>>>> > >> 05-13
> > >> > >>>>> 22:00:00
> > >> > >>>>> > >> 2020-05-13
> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
> > >> > >>>>> > >> 05-14
> > >> > >>>>> 12:00:00
> > >> > >>>>> > >> true
> > >> > >>>>> > >> median plot_20200612_173626.data
> > >> > >>>>> > >> plot_20200612_173626.png
> > >> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> > >> > >>>>> > >> false list() list() false false false false false
false
> > >> > >>>>> > >> true
> > >> > >>>>> > >> false
> > >> > >>>>> > >> false
> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
> > >> > >>>>> > >> c(1,1,0) 1 2
> > >> > >>>>> > >> 1.4 -2
> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2
0.5 3
> > >> > >>>>> > >> #cccccc 1
> > >> > >>>>> > >> listX 1
> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
> > >> > >>>>> > >> 3
> > >> > >>>>> > >> 0.8 1
> > >> > >>>>> > >> #333333
> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
> > >> > >>>>> > >> c(TRUE,TRUE)
> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
> > >> > >>>>> > >> c(1,1)
> > >> > >>>>> c(1,1)
> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > >> > >>>>> > >
> > >> > >>>>> > >
> > >> > >>>>> > >
> > >> > >>>>> > > --
> > >> > >>>>> > > Edward Strobach
> > >> > >>>>> > > EMC/NCEP/NWS/
> > >> > >>>>> > > IMSG Contractor
> > >> > >>>>> > > Cubicle#: 2029
> > >> > >>>>> > > 301-683-3717
> > >> > >>>>> > >
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>
> > >> > >>>> --
> > >> > >>>> Edward Strobach
> > >> > >>>> EMC/NCEP/NWS/
> > >> > >>>> IMSG Contractor
> > >> > >>>> Cubicle#: 2029
> > >> > >>>> 301-683-3717
> > >> > >>>>
> > >> > >>>
> > >> > >>>
> > >> > >>> --
> > >> > >>> Edward Strobach
> > >> > >>> EMC/NCEP/NWS/
> > >> > >>> IMSG Contractor
> > >> > >>> Cubicle#: 2029
> > >> > >>> 301-683-3717
> > >> > >>>
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> Edward Strobach
> > >> > >> EMC/NCEP/NWS/
> > >> > >> IMSG Contractor
> > >> > >> Cubicle#: 2029
> > >> > >> 301-683-3717
> > >> > >>
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Edward Strobach
> > >> > > EMC/NCEP/NWS/
> > >> > > IMSG Contractor
> > >> > > Cubicle#: 2029
> > >> > > 301-683-3717
> > >> > >
> > >>
> > >>
> > >>
> > >>
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 10:19:02 2020
you're right. I didn't have the storage problem this time. I'm not
sure
what changed. All I know is that I have now created my load xml which
now
should allow me to move on to the next step of plotting
On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> I'm not following you. How do I now have enough space? What
changed?
>
> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> Ed, now you should have enough space for the data loading.
>>
>> Tatiana
>>
>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>> > I was basically able to build my xml file but it says I have no
>> > available
>> > space. I'm not sure how that can be since I just started adding
data
>> > to my
>> > account
>> >
>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space
left
>> > on
>> > device
>> > ERROR: Command returned with non-zero status (1): scp -r ./*
>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>> > 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
>> > TABLE_SCHEMA USER
SIZE_MB
>> > mv_met_g2o_gefs binbin.zhou
140027.31
>> > mv_met_system_hrefv2_v3 binbin.zhou
71986.11
>> >
>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate
<
>> > edward.strobach at noaa.gov> wrote:
>> >
>> > > Thanks Tatiana. Yes, I've created a database which seems
different
>> > > from
>> > > mv_group. How can one create their own group so that I can
store and
>> > > access
>> > > the database. The database exists but cannot be viewed on the
>> > > metviewer
>> > > GUI. The list starts with EMC HREFv3 and ends with vhagerty.
I
>> > > named my
>> > > database Ed_Strobach, but that was really intended for the
group
>> > > name.
>> > >
>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>> > > <met_help at ucar.edu>
>> > > wrote:
>> > >
>> > >> We don't have database and METviewer running on wcoss.
>> > >> Instead, users use METviewer-wcoss scripts that help to
connect to
>> > >> and
>> > >> use METviewer that is running on AWS. That is why XML
connection and
>> > >> folder
>> > >> sections look somewhat different than once from the Docs page.
>> > >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>> > >> For data loading :
>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>> > >> And for batch:
>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>> > >>
>> > >> These scripts copy XML and data files if needed to AWS, adjust
data
>> > >> path
>> > >> and users credentials, run METviewer and copy results back to
wcoss.
>> > >>
>> > >> Tatiana
>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
>> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
>> > >> > -e'create
>> > >> > database [db_name]
>> > >> >
>> > >> > would need to be used to create a database. I assume that
an sql
>> > >> > module is
>> > >> > available on wcoss; I know you don't have access to those
machines
>> > >> > Tatiana. If I create my own database with my own name, then
I
>> > >> > should
>> > >> > be
>> > >> > able to proceed with project specific tasks under that tab.
>> > >> > Something
>> > >> > like
>> > >> > mv_EdwardStrobach.
>> > >> >
>> > >> > I can then create mv_cmaq_prod_v_para which I add data to
using
>> > >> > load_spec.
>> > >> > Once that step is done, then I can run plot_spec. That's my
>> > >> > takeaway
>> > >> > currently
>> > >> >
>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
>> > >> > edward.strobach at noaa.gov> wrote:
>> > >> >
>> > >> > > Actually, it appears that I need to run the database
loading
>> > >> > > module
>> > >> > > first
>> > >> > > if I want to change/add a data source. From there I can
run the
>> > >> > > plotting
>> > >> > > batch module. Is that right? So if I run the database
loading
>> > >> > > module,
>> > >> > > then I can specify my dates and other related info
specific to
>> > >> > > the
>> > >> > > dataset. If I create my database with a unique name, then
it
>> > >> > > should
>> > >> > > be
>> > >> > > recognized on the metviewer gui, presumably.
>> > >> > >
>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
>> > >> > > <
>> > >> > > edward.strobach at noaa.gov> wrote:
>> > >> > >
>> > >> > >> It seems that those options must be treated separately
outside
>> > >> > >> of
>> > >> > >> plot_spec. Is it acceptable to create both a load_spec
and
>> > >> > >> plot_spec in
>> > >> > >> the same xml file? Would the definitions in load_spec be
able
>> > >> > >> to
>> > >> > >> carry
>> > >> > >> over into plot_spec? It appears that data loading and
plotting
>> > >> > >> are
>> > >> > >> treated
>> > >> > >> separately.
>> > >> > >>
>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
>> > >> > >> Affiliate <
>> > >> > >> edward.strobach at noaa.gov> wrote:
>> > >> > >>
>> > >> > >>> I do plan to use date_range; however, in an example
provided
>> > >> > >>> by the
>> > >> > >>> link
>> > >> > >>> I see this:
>> > >> > >>>
>> > >> > >>> <load_spec>
>> > >> > >>> <connection>
>> > >> > >>> <host>kemosabe:3306</host>
>> > >> > >>> <database>metvdb_hwt</database>
>> > >> > >>> <user>pgoldenb</user>
>> > >> > >>> <password>pgoldenb</password>
>> > >> > >>> </connection>
>> > >> > >>>
>> > >> > >>> <date_list name="folder_dates">
>> > >> > >>> <start>2010051914V</start>
>> > >> > >>> <end><date_offset><day_offset>0</day_offset><hour>6
>> > >> > >>> </hour></date_offset></end>
>> > >> > >>> <inc>3600</inc>
>> > >> > >>> <format>yyyyMMddHH'V'</format>
>> > >> > >>> </date_list>
>> > >> > >>>
>> > >> > >>> <met_version>V3.0</met_version>
>> > >> > >>>
>> > >> > >>> <verbose>false</verbose>
>> > >> > >>> <insert_size>1</insert_size>
>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>> > >> > >>> <drop_indexes>false</drop_indexes>
>> > >> > >>> <apply_indexes>true</apply_indexes>
>> > >> > >>> <group>Group name</group>
>> > >> > >>> <load_stat>true</load_stat>
>> > >> > >>> <load_mode>true</load_mode>
>> > >> > >>> <load_mtd>true</load_mtd>
>> > >> > >>> <load_mpr>false</load_mpr>
>> > >> > >>>
>> > >> > >>>
>> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>> > >> > >>> <load_val>
>> > >> > >>> <field name="model">
>> > >> > >>> <val>arw</val>
>> > >> > >>> <val>nmm</val>
>> > >> > >>> </field>
>> > >> > >>>
>> > >> > >>> <field name="valid_time">
>> > >> > >>> <date_list name="folder_dates"/>
>> > >> > >>> </field>
>> > >> > >>>
>> > >> > >>> <field name="vx_mask">
>> > >> > >>> <val>FULL</val>
>> > >> > >>> <val>SWC</val>
>> > >> > >>> </field>
>> > >> > >>> </load_val>
>> > >> > >>> </load_spec>
>> > >> > >>>
>> > >> > >>> Are similar settings allowed with <plot_spec>? I see
>> > >> > >>> <connection>,
>> > >> > >>> which seems carried over. I'm looking to specify my
>> > >> > >>> folder_tmpl in
>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>> > >> > >>> Since my
>> > >> > >>> directory structure looks like this:
>> > >> > >>>
>> > >> > >>>
>> > >> > >>>
>> > >>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>> > >> > >>>
>> > >> > >>> I would do something like this I would think:
>> > >> > >>>
>> > >> > >>> .......
>> > >> > >>>
>> > >> > >>> <date_list name="folder_dates">
>> > >> > >>> <start>20200612/start>
>> > >> > >>> <end><date_offset><day_offset>0</day_offset></end>
>> > >> > >>> <inc>86400</inc>
>> > >> > >>> <format>yyyyMMdd</format>
>> > >> > >>> </date_list>
>> > >> > >>>
>> > >> > >>> <folder_tmpl>
>> > >> > >>>
>> > >>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>> > >> > >>> </folder_tmpl>
>> > >> > >>> <load_val>
>> > >> > >>> <field name="model">
>> > >> > >>> <val>prod</val>
>> > >> > >>> </field>
>> > >> > >>>
>> > >> > >>> <field name="valid_time">
>> > >> > >>> <date_list name="folder_dates"/>
>> > >> > >>> </field>
>> > >> > >>>
>> > >> > >>> ......
>> > >> > >>>
>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
>> > >> > >>> Affiliate <
>> > >> > >>> edward.strobach at noaa.gov> wrote:
>> > >> > >>>
>> > >> > >>>> Thank you. This is a very helpful example
>> > >> > >>>>
>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
>> > >> > >>>> <met_help at ucar.edu>
>> > >> > >>>> wrote:
>> > >> > >>>>
>> > >> > >>>>> Hi,
>> > >> > >>>>> Here is some examples how to use <date_range> and
>> > >> > >>>>> <date_range_list> in
>> > >> > >>>>> batch XML.
>> > >> > >>>>> First, you need to defined your range. You can have
more
>> > >> > >>>>> then one
>> > >> > >>>>> <date_range>:
>> > >> > >>>>> <date_range name="may">
>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>> > >> > >>>>> </date_range>
>> > >> > >>>>> <date_range name="march">
>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>> > >> > >>>>> </date_range>
>> > >> > >>>>>
>> > >> > >>>>> and then use one or all date_ranges:
>> > >> > >>>>> <plot_fix>
>> > >> > >>>>> <field name="fcst_init_beg">
>> > >> > >>>>> <date_range name="march"/>
>> > >> > >>>>> <date_range name="may"/>
>> > >> > >>>>> </field>...
>> > >> > >>>>> This configuration would include all dates for March
and May
>> > >> > >>>>> of
>> > >> > >>>>> 2019.
>> > >> > >>>>>
>> > >> > >>>>> If you need more flexible configuration you can use
>> > >> > >>>>> date_range_list.
>> > >> > >>>>> For example, you want to select first 6 hours every
day
>> > >> > >>>>> beetween
>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
>> > >> > >>>>> look
>> > >> > >>>>> like this:
>> > >> > >>>>> <date_range_list name = 'all'>
>> > >> > >>>>> <range_start>2020-01-01 00:00:00</range_start>
>> > >> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
>> > >> > >>>>> <range_length>360</range_length>
>> > >> > >>>>> <inc>1440</inc>
>> > >> > >>>>> </date_range_list>
>> > >> > >>>>> where the 1st and 2nd values are the start and end,
>> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
>> > >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
>> > >> > >>>>> This example would create following ranges:
>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>> > >> > >>>>>
>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
>> > >> > >>>>> your
>> > >> > >>>>> data.
>> > >> > >>>>> For example, if you want to select only dates with
00:00 you
>> > >> > >>>>> can
>> > >> > >>>>> specify
>> > >> > >>>>> <field name="valid_hour">
>> > >> > >>>>> <val>00</val>
>> > >> > >>>>> </field>
>> > >> > >>>>>
>> > >> > >>>>> Tatiana
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
>> > >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
>> > >> > >>>>> > of a
>> > >> > >>>>> > separate
>> > >> > >>>>> > script that pipes dates (and other relevant
information)
>> > >> > >>>>> > into
>> > >> > >>>>> > the XML
>> > >> > >>>>> > file
>> > >> > >>>>> > used to generate the results.
>> > >> > >>>>> >
>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
>> > >> > >>>>> > Affiliate <
>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>> > >> > >>>>> >
>> > >> > >>>>> > > Thanks Tatiana. These resources are helpful.
>> > >> > >>>>> > >
>> > >> > >>>>> > > One last question. Simply typing lines like this
in can
>> > >> > >>>>> > > take
>> > >> > >>>>> > > forever:
>> > >> > >>>>> > >
>> > >> > >>>>> > > <plot_fix>
>> > >> > >>>>> > > <field equalize="false" name="fcst_init_beg">
>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>> > >> > >>>>> > >
>> > >> > >>>>> > >
>> > >> > >>>>> > > Is there a better way? I see date_range,
>> > >> > >>>>> > > date_range_list,
>> > >> > >>>>> > > or
>> > >> > >>>>> > > date_list.
>> > >> > >>>>> > > Can this be used instead of typing individual
values?
>> > >> > >>>>> > > This
>> > >> > >>>>> > > could
>> > >> > >>>>> > > prove
>> > >> > >>>>> > > pretty daunting if one would have to type in 700
entries
>> > >> > >>>>> > > each
>> > >> > >>>>> > > time.
>> > >> > >>>>> > >
>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via
RT
>> > >> > >>>>> > > <met_help at ucar.edu>
>> > >> > >>>>> > > wrote:
>> > >> > >>>>> > >
>> > >> > >>>>> > >> I found a few problems in your XML.
>> > >> > >>>>> > >> The database you are using doesn't have data with
>> > >> > >>>>> > >> vx_mask
>> > >> > >>>>> > >> 'abcREGabc'.
>> > >> > >>>>> > >> That is why METviewer returns an empty dataset
and
>> > >> > >>>>> > >> can't
>> > >> > >>>>> > >> create a
>> > >> > >>>>> > >> plot.
>> > >> > >>>>> > >> The other problem is how you define fcst_lead.
This
>> > >> > >>>>> > >> database
>> > >> > >>>>> > >> has
>> > >> > >>>>> > >> values
>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since
you
>> > >> > >>>>> > >> include
>> > >> > >>>>> > >> all
>> > >> > >>>>> > >> fcst_leads
>> > >> > >>>>> > >> this variable can be omitted.
>> > >> > >>>>> > >> To minimize problems with variables values, I
would
>> > >> > >>>>> > >> suggest
>> > >> > >>>>> > >> to
>> > >> > >>>>> > >> create XML
>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>> > >> > >>>>> > >> I created a plot using you configurations with
GUI
>> > >> > >>>>> > >> (without
>> > >> > >>>>> vx_mack
>> > >> > >>>>> > >> and
>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You can
load it
>> > >> > >>>>> > >> back
>> > >> > >>>>> > >> to
>> > >> > >>>>> the
>> > >> > >>>>> > >> page by
>> > >> > >>>>> > >> using the "History" tab (left bar on GUI). Find
the
>> > >> > >>>>> > >> plot,
>> > >> > >>>>> > >> open XML
>> > >> > >>>>> > >> and
>> > >> > >>>>> > >> click on "Load to page". After that you can
recreate a
>> > >> > >>>>> > >> plot
>> > >> > >>>>> > >> and
>> > >> > >>>>> edit
>> > >> > >>>>> > >> the
>> > >> > >>>>> > >> configurations. You can also download a new XML
and
>> > >> > >>>>> > >> edit it
>> > >> > >>>>> > >> for
>> > >> > >>>>> > >> batch.
>> > >> > >>>>> > >> I attached the example of load XML to this
message.
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> Description of rds parameters:
>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer
R
>> > >> > >>>>> > >> scripts
>> > >> > >>>>> > >> rds_plots - temporary directory where the plots
will be
>> > >> > >>>>> > >> generated
>> > >> > >>>>> > >> rds_data - temporary directory where the data
retrieved
>> > >> > >>>>> > >> from
>> > >> > >>>>> > >> the
>> > >> > >>>>> > >> database
>> > >> > >>>>> > >> and other intermediate files will be stored
>> > >> > >>>>> > >> rds_scripts - temporary directory where R scripts
for
>> > >> > >>>>> > >> this
>> > >> > >>>>> > >> run
>> > >> > >>>>> will
>> > >> > >>>>> > >> be
>> > >> > >>>>> > >> stored
>> > >> > >>>>> > >> After the execution is done all the files from
>> > >> > >>>>> > >> temporary
>> > >> > >>>>> directories
>> > >> > >>>>> > >> would be removed.
>> > >> > >>>>> > >>
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
>> > >> > >>>>> > >> There
>> > >> > >>>>> > >> are
>> > >> > >>>>> some
>> > >> > >>>>> > >> instructions on how to crate XML config files:
>> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> Tatiana
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
>> > >> > >>>>> > >> wrote:
>> > >> > >>>>> > >> > Hi,
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > I'm attaching the xml below, but am having hard
time
>> > >> > >>>>> > >> > following
>> > >> > >>>>> how
>> > >> > >>>>> > >> > to
>> > >> > >>>>> > >> > go
>> > >> > >>>>> > >> > about the other steps since I have zero
experience on
>> > >> > >>>>> > >> > how
>> > >> > >>>>> > >> > to do
>> > >> > >>>>> > >> > these
>> > >> > >>>>> > >> > things.
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > #2. All parameters in <folders> section should
NOT be
>> > >> > >>>>> > >> > changed
>> > >> > >>>>> and
>> > >> > >>>>> > >> > will
>> > >> > >>>>> > >> > be
>> > >> > >>>>> > >> > replaced by other scripts during the batch
execution.
>> > >> > >>>>> > >> > The
>> > >> > >>>>> > >> > data
>> > >> > >>>>> for
>> > >> > >>>>> > >> > the
>> > >> > >>>>> > >> > batch mode is coming from the database and not
from
>> > >> > >>>>> > >> > files.
>> > >> > >>>>> > >> > The
>> > >> > >>>>> > >> > name of
>> > >> > >>>>> > >> > the
>> > >> > >>>>> > >> > database is in <database> tag.
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > I understand that nothing should be changed in
the
>> > >> > >>>>> > >> > folders
>> > >> > >>>>> > >> > section,
>> > >> > >>>>> > >> > but
>> > >> > >>>>> > >> > there is no indication how and where these
variables
>> > >> > >>>>> > >> > are
>> > >> > >>>>> > >> > being
>> > >> > >>>>> > >> > defined.
>> > >> > >>>>> > >> > Grepping these names of the files available
does not
>> > >> > >>>>> > >> > reveal
>> > >> > >>>>> formal
>> > >> > >>>>> > >> > definitions of these variables. If I
understood
>> > >> > >>>>> > >> > exactly
>> > >> > >>>>> > >> > how
>> > >> > >>>>> these
>> > >> > >>>>> > >> > were
>> > >> > >>>>> > >> > being defined, then I would be able connect
this
>> > >> > >>>>> > >> > better.
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
>> > >> > >>>>> > >> > use
>> > >> > >>>>> METviewer
>> > >> > >>>>> > >> > load
>> > >> > >>>>> > >> > mode to load these files to the specific
database. In
>> > >> > >>>>> > >> > the
>> > >> > >>>>> > >> > config
>> > >> > >>>>> > >> > file
>> > >> > >>>>> > >> > for
>> > >> > >>>>> > >> > this process they also can specify the database
>> > >> > >>>>> > >> > group. For
>> > >> > >>>>> example
>> > >> > >>>>> > >> > database
>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>> > >> > >>>>> > >> > After the data has been loaded to the database
it is
>> > >> > >>>>> > >> > visible and
>> > >> > >>>>> > >> > available
>> > >> > >>>>> > >> > in the GUI and also can be used by the batch
mode.
>> > >> > >>>>> > >> > You can always add more data to the database
using
>> > >> > >>>>> > >> > the
>> > >> > >>>>> > >> > same load
>> > >> > >>>>> > >> > mode.
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > Is there a guide for me to understand the
process
>> > >> > >>>>> > >> > that you
>> > >> > >>>>> > >> > are
>> > >> > >>>>> > >> > referring to?
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek
via RT
>> > >> > >>>>> > >> > <met_help at ucar.edu>
>> > >> > >>>>> > >> > wrote:
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > > Hi Edward,
>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA servers
and
>> > >> > >>>>> > >> > > can't
>> > >> > >>>>> > >> > > view your
>> > >> > >>>>> > >> > > XML
>> > >> > >>>>> > >> > > for the
>> > >> > >>>>> > >> > > batch mode. Please, attache it to this issue
and I
>> > >> > >>>>> > >> > > will
>> > >> > >>>>> > >> > > take a
>> > >> > >>>>> > >> > > look.
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
should NOT
>> > >> > >>>>> > >> > > be
>> > >> > >>>>> > >> > > changed
>> > >> > >>>>> > >> > > and
>> > >> > >>>>> > >> > > will be
>> > >> > >>>>> > >> > > replaced by other scripts during the batch
>> > >> > >>>>> > >> > > execution.
>> > >> > >>>>> > >> > > The data
>> > >> > >>>>> > >> > > for
>> > >> > >>>>> > >> > > the
>> > >> > >>>>> > >> > > batch mode is coming from the database and
not from
>> > >> > >>>>> > >> > > files. The
>> > >> > >>>>> > >> > > name
>> > >> > >>>>> > >> > > of the
>> > >> > >>>>> > >> > > database is in <database> tag.
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
>> > >> > >>>>> > >> > > They use
>> > >> > >>>>> > >> > > METviewer
>> > >> > >>>>> > >> > > load
>> > >> > >>>>> > >> > > mode to load these files to the specific
database.
>> > >> > >>>>> > >> > > In
>> > >> > >>>>> > >> > > the
>> > >> > >>>>> config
>> > >> > >>>>> > >> > > file
>> > >> > >>>>> > >> > > for
>> > >> > >>>>> > >> > > this process they also can specify the
database
>> > >> > >>>>> > >> > > group.
>> > >> > >>>>> > >> > > For
>> > >> > >>>>> > >> > > example
>> > >> > >>>>> > >> > > database
>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>> > >> > >>>>> > >> > > After the data has been loaded to the
database it
>> > >> > >>>>> > >> > > is
>> > >> > >>>>> > >> > > visible
>> > >> > >>>>> and
>> > >> > >>>>> > >> > > available
>> > >> > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
>> > >> > >>>>> > >> > > You can always add more data to the database
using
>> > >> > >>>>> > >> > > the
>> > >> > >>>>> > >> > > same
>> > >> > >>>>> load
>> > >> > >>>>> > >> > > mode.
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > > Tatiana
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>> > >> > >>>>> > >> > > wrote:
>> > >> > >>>>> > >> > > > Good morning,
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Over the last two weeks I've made an effort
to
>> > >> > >>>>> > >> > > > understand
>> > >> > >>>>> > >> > > > metviewer,
>> > >> > >>>>> > >> > > > test
>> > >> > >>>>> > >> > > > different cases using the history tab,
changing
>> > >> > >>>>> > >> > > > information
>> > >> > >>>>> to
>> > >> > >>>>> > >> > > > see
>> > >> > >>>>> > >> > > > what
>> > >> > >>>>> > >> > > > works and what does not, and make
connections
>> > >> > >>>>> > >> > > > between
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> GUI
>> > >> > >>>>> > >> > > > settings
>> > >> > >>>>> > >> > > > and
>> > >> > >>>>> > >> > > > the XML file. Although my comfort level
with
>> > >> > >>>>> > >> > > > metviewer has
>> > >> > >>>>> > >> > > > significantly
>> > >> > >>>>> > >> > > > increased during this time, there are some
>> > >> > >>>>> > >> > > > remaining
>> > >> > >>>>> > >> > > > issues
>> > >> > >>>>> > >> > > > that
>> > >> > >>>>> > >> > > > I'm
>> > >> > >>>>> > >> > > > hoping
>> > >> > >>>>> > >> > > > to clear up.
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > 1. Problem changing XML file and running
with
>> > >> > >>>>> > >> > > > different
>> > >> > >>>>> > >> > > > statistic
>> > >> > >>>>> > >> > > > and
>> > >> > >>>>> > >> > > > plot
>> > >> > >>>>> > >> > > > type:
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided
me
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> > >> > > > working
>> > >> > >>>>> > >> > > > directory
>> > >> > >>>>> > >> > > > to get
>> > >> > >>>>> > >> > > > started which I copied over:
>> > >> > >>>>> > >> > > >
>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
>> > >> > >>>>> > >> > > > to
>> > >> > >>>>> > >> > > > run his
>> > >> > >>>>> > >> > > > example.
>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
produced a
>> > >> > >>>>> > >> > > > CSI
>> > >> > >>>>> > >> > > > plot
>> > >> > >>>>> that
>> > >> > >>>>> > >> > > > was
>> > >> > >>>>> > >> > > > stored
>> > >> > >>>>> > >> > > > here:
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >>
>> > >> > >>>>>
>> > >>
>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Running the script produced the expected
result.
>> > >> > >>>>> > >> > > > The
>> > >> > >>>>> > >> > > > next
>> > >> > >>>>> > >> > > > natural
>> > >> > >>>>> > >> > > > step was
>> > >> > >>>>> > >> > > > to copy hochum_xml.template and create a
new xml
>> > >> > >>>>> > >> > > > file
>> > >> > >>>>> > >> > > > called
>> > >> > >>>>> > >> > > > Strobach_xml.template in order to create a
time
>> > >> > >>>>> > >> > > > series
>> > >> > >>>>> > >> > > > where
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg instead
of
>> > >> > >>>>> > >> > > > obs_thresh.
>> > >> > >>>>> > >> > > > Accordingly,
>> > >> > >>>>> > >> > > > I
>> > >> > >>>>> > >> > > > made changes to the fixed fields (removing
>> > >> > >>>>> > >> > > > fcst_valid_beg)
>> > >> > >>>>> and
>> > >> > >>>>> > >> > > > changed
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
>> > >> > >>>>> > >> > > > tried).
>> > >> > >>>>> > >> > > > The
>> > >> > >>>>> error
>> > >> > >>>>> > >> > > > that
>> > >> > >>>>> > >> > > > resulted
>> > >> > >>>>> > >> > > > is as follows:
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-
05-01
>> > >> > >>>>> > >> > > > 13:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 01
>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-
01
>> > >> > >>>>> > >> > > > 16:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 01
>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-
01
>> > >> > >>>>> > >> > > > 19:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 01
>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-
01
>> > >> > >>>>> > >> > > > 22:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 01
>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-
02
>> > >> > >>>>> > >> > > > 01:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 02
>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-
02
>> > >> > >>>>> > >> > > > 04:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 02
>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-
02
>> > >> > >>>>> > >> > > > 07:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 02
>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-
02
>> > >> > >>>>> > >> > > > 10:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 02
>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>> > >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>> > >> > >>>>> > >> > > > No data was returned from database
>> > >> > >>>>> > >> > > > 2020-06-09
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >>
>> > >> > >>>>>
>> > >>
>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
ERROR:
>> > >> > >>>>> > >> > > > Caught
>> > >> > >>>>> > >> > > > class
>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
in
>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>> > >> > >>>>> > >> > > > res): result set contained no data
>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >> > >>>>> > >> > > > in
>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
>> > >> > >>>>> > >> > > > contained no
>> > >> > >>>>> > >> > > > data
>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >> > >>>>> > >> > > > in
>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
>> > >> > >>>>> > >> > > > contained no
>> > >> > >>>>> > >> > > > data
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Apparently there was no data during this
time. I
>> > >> > >>>>> > >> > > > tried
>> > >> > >>>>> > >> > > > changing
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> > >> > > > range
>> > >> > >>>>> > >> > > > of valid times and changed the valid time
to
>> > >> > >>>>> > >> > > > different
>> > >> > >>>>> > >> > > > days
>> > >> > >>>>> > >> > > > just in
>> > >> > >>>>> > >> > > > case
>> > >> > >>>>> > >> > > > data was missing. The issue was the same.
To
>> > >> > >>>>> > >> > > > confirm
>> > >> > >>>>> > >> > > > my
>> > >> > >>>>> > >> > > > logic, I
>> > >> > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
>> > >> > >>>>> > >> > > > which I
>> > >> > >>>>> > >> > > > found
>> > >> > >>>>> > >> > > > to
>> > >> > >>>>> > >> > > > work.
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > 2. Running in batch mode
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > I think I lack an understanding of how to
set up
>> > >> > >>>>> > >> > > > batch
>> > >> > >>>>> > >> > > > mode
>> > >> > >>>>> > >> > > > because
>> > >> > >>>>> > >> > > > I
>> > >> > >>>>> > >> > > > don't
>> > >> > >>>>> > >> > > > see where these definitions are being
explicitly
>> > >> > >>>>> > >> > > > defined:
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > <plot_spec>
>> > >> > >>>>> > >> > > > <connection>
>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>> > >> > >>>>> > >> > > >
>> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>> > >> > >>>>> > >> > > > <user>rds_user</user>
>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>> > >> > >>>>> > >> > > > </connection>
>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>> > >> > >>>>> > >> > > > <folders>
>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>> > >> > >>>>> > >> > > > <data>rds_data</data>
>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>> > >> > >>>>> > >> > > > </folders>
>> > >> > >>>>> > >> > > > <plot>
>> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > I see the <folders> section, but grepping
all
>> > >> > >>>>> > >> > > > available
>> > >> > >>>>> files
>> > >> > >>>>> > >> > > > does
>> > >> > >>>>> > >> > > > not
>> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well
>> > >> > >>>>> > >> > > > as
>> > >> > >>>>> > >> > > > other
>> > >> > >>>>> > >> > > > similarly
>> > >> > >>>>> > >> > > > defined
>> > >> > >>>>> > >> > > > variables. I think if I know how these are
being
>> > >> > >>>>> > >> > > > defined,
>> > >> > >>>>> > >> > > > then
>> > >> > >>>>> > >> > > > I'll
>> > >> > >>>>> > >> > > > be
>> > >> > >>>>> > >> > > > able to pipe in my information and process
the
>> > >> > >>>>> > >> > > > data
>> > >> > >>>>> > >> > > > using
>> > >> > >>>>> met
>> > >> > >>>>> > >> > > > and
>> > >> > >>>>> > >> > > > produce
>> > >> > >>>>> > >> > > > results in batch mode. Am I right on that?
>> > >> > >>>>> > >> > > > Also, if
>> > >> > >>>>> > >> > > > I want
>> > >> > >>>>> > >> > > > to use
>> > >> > >>>>> > >> > > > obs for
>> > >> > >>>>> > >> > > > verification, do I add another line in the
XML
>> > >> > >>>>> > >> > > > file
>> > >> > >>>>> > >> > > > that
>> > >> > >>>>> > >> > > > designates
>> > >> > >>>>> > >> > > > file
>> > >> > >>>>> > >> > > > location of obs? I assume that other steps
have
>> > >> > >>>>> > >> > > > to be
>> > >> > >>>>> > >> > > > run
>> > >> > >>>>> > >> > > > before
>> > >> > >>>>> > >> > > > all
>> > >> > >>>>> > >> > > > this. If data is in prepbufr form, for
example,
>> > >> > >>>>> > >> > > > I
>> > >> > >>>>> > >> > > > would
>> > >> > >>>>> have
>> > >> > >>>>> > >> > > > to
>> > >> > >>>>> > >> > > > separately
>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I
would want
>> > >> > >>>>> > >> > > > to
>> > >> > >>>>> > >> > > > do that
>> > >> > >>>>> > >> > > > as
>> > >> > >>>>> > >> > > > well.
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
>> > >> > >>>>> > >> > > > here -
>> > >> > >>>>> > >> > > > see
>> > >> > >>>>> > >> > > > attachment
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > How do I get the results I generate to be
>> > >> > >>>>> > >> > > > uploaded to
>> > >> > >>>>> > >> > > > this
>> > >> > >>>>> > >> > > > location?
>> > >> > >>>>> > >> > > > It's
>> > >> > >>>>> > >> > > > not clear how others are setting this up.
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >>
>> > >> > >>>>> > >>
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>> > >> > >>>>> > >> rds_user
>> > >> > >>>>> rds_pwd
>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
>> > >> > >>>>> rds_scripts
>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>> > >> > >>>>> > >> CMAQAODPRODHIGH
>> > >> > >>>>> 2020-05-12
>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
>> > >> > >>>>> > >> 05-12
>> > >> > >>>>> 20:00:00
>> > >> > >>>>> > >> 2020-05-12
>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
>> > >> > >>>>> > >> 05-13
>> > >> > >>>>> 10:00:00
>> > >> > >>>>> > >> 2020-05-13
>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
>> > >> > >>>>> > >> 05-13
>> > >> > >>>>> 14:00:00
>> > >> > >>>>> > >> 2020-05-13
>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
>> > >> > >>>>> > >> 05-13
>> > >> > >>>>> 18:00:00
>> > >> > >>>>> > >> 2020-05-13
>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
>> > >> > >>>>> > >> 05-13
>> > >> > >>>>> 22:00:00
>> > >> > >>>>> > >> 2020-05-13
>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
>> > >> > >>>>> > >> 05-14
>> > >> > >>>>> 12:00:00
>> > >> > >>>>> > >> true
>> > >> > >>>>> > >> median plot_20200612_173626.data
>> > >> > >>>>> > >> plot_20200612_173626.png
>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
>> > >> > >>>>> > >> false list() list() false false false false false
false
>> > >> > >>>>> > >> true
>> > >> > >>>>> > >> false
>> > >> > >>>>> > >> false
>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
>> > >> > >>>>> > >> c(1,1,0) 1 2
>> > >> > >>>>> > >> 1.4 -2
>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2
0.5 3
>> > >> > >>>>> > >> #cccccc 1
>> > >> > >>>>> > >> listX 1
>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
>> > >> > >>>>> > >> 3
>> > >> > >>>>> > >> 0.8 1
>> > >> > >>>>> > >> #333333
>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
>> > >> > >>>>> > >> c(TRUE,TRUE)
>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
>> > >> > >>>>> > >> c(1,1)
>> > >> > >>>>> c(1,1)
>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>> > >> > >>>>> > >
>> > >> > >>>>> > >
>> > >> > >>>>> > >
>> > >> > >>>>> > > --
>> > >> > >>>>> > > Edward Strobach
>> > >> > >>>>> > > EMC/NCEP/NWS/
>> > >> > >>>>> > > IMSG Contractor
>> > >> > >>>>> > > Cubicle#: 2029
>> > >> > >>>>> > > 301-683-3717
>> > >> > >>>>> > >
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>
>> > >> > >>>> --
>> > >> > >>>> Edward Strobach
>> > >> > >>>> EMC/NCEP/NWS/
>> > >> > >>>> IMSG Contractor
>> > >> > >>>> Cubicle#: 2029
>> > >> > >>>> 301-683-3717
>> > >> > >>>>
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> --
>> > >> > >>> Edward Strobach
>> > >> > >>> EMC/NCEP/NWS/
>> > >> > >>> IMSG Contractor
>> > >> > >>> Cubicle#: 2029
>> > >> > >>> 301-683-3717
>> > >> > >>>
>> > >> > >>
>> > >> > >>
>> > >> > >> --
>> > >> > >> Edward Strobach
>> > >> > >> EMC/NCEP/NWS/
>> > >> > >> IMSG Contractor
>> > >> > >> Cubicle#: 2029
>> > >> > >> 301-683-3717
>> > >> > >>
>> > >> > >
>> > >> > >
>> > >> > > --
>> > >> > > Edward Strobach
>> > >> > > EMC/NCEP/NWS/
>> > >> > > IMSG Contractor
>> > >> > > Cubicle#: 2029
>> > >> > > 301-683-3717
>> > >> > >
>> > >>
>> > >>
>> > >>
>> > >>
>> > >
>> > > --
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/
>> > > IMSG Contractor
>> > > Cubicle#: 2029
>> > > 301-683-3717
>> > >
>>
>>
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Jeff.Mcqueen at noaa.gov
Time: Wed Jun 17 11:34:19 2020
Would it be ok to keep perry in the loop
On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> you're right. I didn't have the storage problem this time. I'm not
sure
> what changed. All I know is that I have now created my load xml
which now
> should allow me to move on to the next step of plotting
>
> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> I'm not following you. How do I now have enough space? What
changed?
>>
>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
<met_help at ucar.edu>
>> wrote:
>>
>>> Ed, now you should have enough space for the data loading.
>>>
>>> Tatiana
>>>
>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>> > I was basically able to build my xml file but it says I have no
>>> > available
>>> > space. I'm not sure how that can be since I just started adding
data
>>> > to my
>>> > account
>>> >
>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space
left
>>> > on
>>> > device
>>> > ERROR: Command returned with non-zero status (1): scp -r ./*
>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>> > 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
>>> > TABLE_SCHEMA USER
SIZE_MB
>>> > mv_met_g2o_gefs binbin.zhou
140027.31
>>> > mv_met_system_hrefv2_v3 binbin.zhou
71986.11
>>> >
>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate
<
>>> > edward.strobach at noaa.gov> wrote:
>>> >
>>> > > Thanks Tatiana. Yes, I've created a database which seems
different
>>> > > from
>>> > > mv_group. How can one create their own group so that I can
store and
>>> > > access
>>> > > the database. The database exists but cannot be viewed on the
>>> > > metviewer
>>> > > GUI. The list starts with EMC HREFv3 and ends with vhagerty.
I
>>> > > named my
>>> > > database Ed_Strobach, but that was really intended for the
group
>>> > > name.
>>> > >
>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>> > > <met_help at ucar.edu>
>>> > > wrote:
>>> > >
>>> > >> We don't have database and METviewer running on wcoss.
>>> > >> Instead, users use METviewer-wcoss scripts that help to
connect to
>>> > >> and
>>> > >> use METviewer that is running on AWS. That is why XML
connection and
>>> > >> folder
>>> > >> sections look somewhat different than once from the Docs
page.
>>> > >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>> > >> For data loading :
>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>> > >> And for batch:
>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>> > >>
>>> > >> These scripts copy XML and data files if needed to AWS,
adjust data
>>> > >> path
>>> > >> and users credentials, run METviewer and copy results back to
wcoss.
>>> > >>
>>> > >> Tatiana
>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
>>> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
>>> > >> > -e'create
>>> > >> > database [db_name]
>>> > >> >
>>> > >> > would need to be used to create a database. I assume that
an sql
>>> > >> > module is
>>> > >> > available on wcoss; I know you don't have access to those
machines
>>> > >> > Tatiana. If I create my own database with my own name,
then I
>>> > >> > should
>>> > >> > be
>>> > >> > able to proceed with project specific tasks under that tab.
>>> > >> > Something
>>> > >> > like
>>> > >> > mv_EdwardStrobach.
>>> > >> >
>>> > >> > I can then create mv_cmaq_prod_v_para which I add data to
using
>>> > >> > load_spec.
>>> > >> > Once that step is done, then I can run plot_spec. That's
my
>>> > >> > takeaway
>>> > >> > currently
>>> > >> >
>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
>>> > >> > edward.strobach at noaa.gov> wrote:
>>> > >> >
>>> > >> > > Actually, it appears that I need to run the database
loading
>>> > >> > > module
>>> > >> > > first
>>> > >> > > if I want to change/add a data source. From there I can
run the
>>> > >> > > plotting
>>> > >> > > batch module. Is that right? So if I run the database
loading
>>> > >> > > module,
>>> > >> > > then I can specify my dates and other related info
specific to
>>> > >> > > the
>>> > >> > > dataset. If I create my database with a unique name,
then it
>>> > >> > > should
>>> > >> > > be
>>> > >> > > recognized on the metviewer gui, presumably.
>>> > >> > >
>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
>>> > >> > > <
>>> > >> > > edward.strobach at noaa.gov> wrote:
>>> > >> > >
>>> > >> > >> It seems that those options must be treated separately
outside
>>> > >> > >> of
>>> > >> > >> plot_spec. Is it acceptable to create both a load_spec
and
>>> > >> > >> plot_spec in
>>> > >> > >> the same xml file? Would the definitions in load_spec
be able
>>> > >> > >> to
>>> > >> > >> carry
>>> > >> > >> over into plot_spec? It appears that data loading and
plotting
>>> > >> > >> are
>>> > >> > >> treated
>>> > >> > >> separately.
>>> > >> > >>
>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
>>> > >> > >> Affiliate <
>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>> > >> > >>
>>> > >> > >>> I do plan to use date_range; however, in an example
provided
>>> > >> > >>> by the
>>> > >> > >>> link
>>> > >> > >>> I see this:
>>> > >> > >>>
>>> > >> > >>> <load_spec>
>>> > >> > >>> <connection>
>>> > >> > >>> <host>kemosabe:3306</host>
>>> > >> > >>> <database>metvdb_hwt</database>
>>> > >> > >>> <user>pgoldenb</user>
>>> > >> > >>> <password>pgoldenb</password>
>>> > >> > >>> </connection>
>>> > >> > >>>
>>> > >> > >>> <date_list name="folder_dates">
>>> > >> > >>> <start>2010051914V</start>
>>> > >> > >>> <end><date_offset><day_offset>0</day_offset><hour>6
>>> > >> > >>> </hour></date_offset></end>
>>> > >> > >>> <inc>3600</inc>
>>> > >> > >>> <format>yyyyMMddHH'V'</format>
>>> > >> > >>> </date_list>
>>> > >> > >>>
>>> > >> > >>> <met_version>V3.0</met_version>
>>> > >> > >>>
>>> > >> > >>> <verbose>false</verbose>
>>> > >> > >>> <insert_size>1</insert_size>
>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>> > >> > >>> <group>Group name</group>
>>> > >> > >>> <load_stat>true</load_stat>
>>> > >> > >>> <load_mode>true</load_mode>
>>> > >> > >>> <load_mtd>true</load_mtd>
>>> > >> > >>> <load_mpr>false</load_mpr>
>>> > >> > >>>
>>> > >> > >>>
>>> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>> > >> > >>> <load_val>
>>> > >> > >>> <field name="model">
>>> > >> > >>> <val>arw</val>
>>> > >> > >>> <val>nmm</val>
>>> > >> > >>> </field>
>>> > >> > >>>
>>> > >> > >>> <field name="valid_time">
>>> > >> > >>> <date_list name="folder_dates"/>
>>> > >> > >>> </field>
>>> > >> > >>>
>>> > >> > >>> <field name="vx_mask">
>>> > >> > >>> <val>FULL</val>
>>> > >> > >>> <val>SWC</val>
>>> > >> > >>> </field>
>>> > >> > >>> </load_val>
>>> > >> > >>> </load_spec>
>>> > >> > >>>
>>> > >> > >>> Are similar settings allowed with <plot_spec>? I see
>>> > >> > >>> <connection>,
>>> > >> > >>> which seems carried over. I'm looking to specify my
>>> > >> > >>> folder_tmpl in
>>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>>> > >> > >>> Since my
>>> > >> > >>> directory structure looks like this:
>>> > >> > >>>
>>> > >> > >>>
>>> > >> > >>>
>>> > >>
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>> > >> > >>>
>>> > >> > >>> I would do something like this I would think:
>>> > >> > >>>
>>> > >> > >>> .......
>>> > >> > >>>
>>> > >> > >>> <date_list name="folder_dates">
>>> > >> > >>> <start>20200612/start>
>>> > >> > >>> <end><date_offset><day_offset>0</day_offset></end>
>>> > >> > >>> <inc>86400</inc>
>>> > >> > >>> <format>yyyyMMdd</format>
>>> > >> > >>> </date_list>
>>> > >> > >>>
>>> > >> > >>> <folder_tmpl>
>>> > >> > >>>
>>> > >>
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>> > >> > >>> </folder_tmpl>
>>> > >> > >>> <load_val>
>>> > >> > >>> <field name="model">
>>> > >> > >>> <val>prod</val>
>>> > >> > >>> </field>
>>> > >> > >>>
>>> > >> > >>> <field name="valid_time">
>>> > >> > >>> <date_list name="folder_dates"/>
>>> > >> > >>> </field>
>>> > >> > >>>
>>> > >> > >>> ......
>>> > >> > >>>
>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
>>> > >> > >>> Affiliate <
>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>> > >> > >>>
>>> > >> > >>>> Thank you. This is a very helpful example
>>> > >> > >>>>
>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
>>> > >> > >>>> <met_help at ucar.edu>
>>> > >> > >>>> wrote:
>>> > >> > >>>>
>>> > >> > >>>>> Hi,
>>> > >> > >>>>> Here is some examples how to use <date_range> and
>>> > >> > >>>>> <date_range_list> in
>>> > >> > >>>>> batch XML.
>>> > >> > >>>>> First, you need to defined your range. You can have
more
>>> > >> > >>>>> then one
>>> > >> > >>>>> <date_range>:
>>> > >> > >>>>> <date_range name="may">
>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>>> > >> > >>>>> </date_range>
>>> > >> > >>>>> <date_range name="march">
>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>>> > >> > >>>>> </date_range>
>>> > >> > >>>>>
>>> > >> > >>>>> and then use one or all date_ranges:
>>> > >> > >>>>> <plot_fix>
>>> > >> > >>>>> <field name="fcst_init_beg">
>>> > >> > >>>>> <date_range name="march"/>
>>> > >> > >>>>> <date_range name="may"/>
>>> > >> > >>>>> </field>...
>>> > >> > >>>>> This configuration would include all dates for March
and May
>>> > >> > >>>>> of
>>> > >> > >>>>> 2019.
>>> > >> > >>>>>
>>> > >> > >>>>> If you need more flexible configuration you can use
>>> > >> > >>>>> date_range_list.
>>> > >> > >>>>> For example, you want to select first 6 hours every
day
>>> > >> > >>>>> beetween
>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
>>> > >> > >>>>> look
>>> > >> > >>>>> like this:
>>> > >> > >>>>> <date_range_list name = 'all'>
>>> > >> > >>>>> <range_start>2020-01-01 00:00:00</range_start>
>>> > >> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
>>> > >> > >>>>> <range_length>360</range_length>
>>> > >> > >>>>> <inc>1440</inc>
>>> > >> > >>>>> </date_range_list>
>>> > >> > >>>>> where the 1st and 2nd values are the start and end,
>>> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
>>> > >> > >>>>> This example would create following ranges:
>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>>> > >> > >>>>>
>>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
>>> > >> > >>>>> your
>>> > >> > >>>>> data.
>>> > >> > >>>>> For example, if you want to select only dates with
00:00 you
>>> > >> > >>>>> can
>>> > >> > >>>>> specify
>>> > >> > >>>>> <field name="valid_hour">
>>> > >> > >>>>> <val>00</val>
>>> > >> > >>>>> </field>
>>> > >> > >>>>>
>>> > >> > >>>>> Tatiana
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
>>> wrote:
>>> > >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
>>> > >> > >>>>> > of a
>>> > >> > >>>>> > separate
>>> > >> > >>>>> > script that pipes dates (and other relevant
information)
>>> > >> > >>>>> > into
>>> > >> > >>>>> > the XML
>>> > >> > >>>>> > file
>>> > >> > >>>>> > used to generate the results.
>>> > >> > >>>>> >
>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
>>> > >> > >>>>> > Affiliate <
>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>> > >> > >>>>> >
>>> > >> > >>>>> > > Thanks Tatiana. These resources are helpful.
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > One last question. Simply typing lines like this
in can
>>> > >> > >>>>> > > take
>>> > >> > >>>>> > > forever:
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > <plot_fix>
>>> > >> > >>>>> > > <field equalize="false" name="fcst_init_beg">
>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>>> > >> > >>>>> > >
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > Is there a better way? I see date_range,
>>> > >> > >>>>> > > date_range_list,
>>> > >> > >>>>> > > or
>>> > >> > >>>>> > > date_list.
>>> > >> > >>>>> > > Can this be used instead of typing individual
values?
>>> > >> > >>>>> > > This
>>> > >> > >>>>> > > could
>>> > >> > >>>>> > > prove
>>> > >> > >>>>> > > pretty daunting if one would have to type in 700
entries
>>> > >> > >>>>> > > each
>>> > >> > >>>>> > > time.
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via
RT
>>> > >> > >>>>> > > <met_help at ucar.edu>
>>> > >> > >>>>> > > wrote:
>>> > >> > >>>>> > >
>>> > >> > >>>>> > >> I found a few problems in your XML.
>>> > >> > >>>>> > >> The database you are using doesn't have data
with
>>> > >> > >>>>> > >> vx_mask
>>> > >> > >>>>> > >> 'abcREGabc'.
>>> > >> > >>>>> > >> That is why METviewer returns an empty dataset
and
>>> > >> > >>>>> > >> can't
>>> > >> > >>>>> > >> create a
>>> > >> > >>>>> > >> plot.
>>> > >> > >>>>> > >> The other problem is how you define fcst_lead.
This
>>> > >> > >>>>> > >> database
>>> > >> > >>>>> > >> has
>>> > >> > >>>>> > >> values
>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since
you
>>> > >> > >>>>> > >> include
>>> > >> > >>>>> > >> all
>>> > >> > >>>>> > >> fcst_leads
>>> > >> > >>>>> > >> this variable can be omitted.
>>> > >> > >>>>> > >> To minimize problems with variables values, I
would
>>> > >> > >>>>> > >> suggest
>>> > >> > >>>>> > >> to
>>> > >> > >>>>> > >> create XML
>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>>> > >> > >>>>> > >> I created a plot using you configurations with
GUI
>>> > >> > >>>>> > >> (without
>>> > >> > >>>>> vx_mack
>>> > >> > >>>>> > >> and
>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You can
load it
>>> > >> > >>>>> > >> back
>>> > >> > >>>>> > >> to
>>> > >> > >>>>> the
>>> > >> > >>>>> > >> page by
>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI). Find
the
>>> > >> > >>>>> > >> plot,
>>> > >> > >>>>> > >> open XML
>>> > >> > >>>>> > >> and
>>> > >> > >>>>> > >> click on "Load to page". After that you can
recreate a
>>> > >> > >>>>> > >> plot
>>> > >> > >>>>> > >> and
>>> > >> > >>>>> edit
>>> > >> > >>>>> > >> the
>>> > >> > >>>>> > >> configurations. You can also download a new XML
and
>>> > >> > >>>>> > >> edit it
>>> > >> > >>>>> > >> for
>>> > >> > >>>>> > >> batch.
>>> > >> > >>>>> > >> I attached the example of load XML to this
message.
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> Description of rds parameters:
>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
METviewer R
>>> > >> > >>>>> > >> scripts
>>> > >> > >>>>> > >> rds_plots - temporary directory where the plots
will be
>>> > >> > >>>>> > >> generated
>>> > >> > >>>>> > >> rds_data - temporary directory where the data
retrieved
>>> > >> > >>>>> > >> from
>>> > >> > >>>>> > >> the
>>> > >> > >>>>> > >> database
>>> > >> > >>>>> > >> and other intermediate files will be stored
>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
scripts for
>>> > >> > >>>>> > >> this
>>> > >> > >>>>> > >> run
>>> > >> > >>>>> will
>>> > >> > >>>>> > >> be
>>> > >> > >>>>> > >> stored
>>> > >> > >>>>> > >> After the execution is done all the files from
>>> > >> > >>>>> > >> temporary
>>> > >> > >>>>> directories
>>> > >> > >>>>> > >> would be removed.
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
>>> > >> > >>>>> > >> There
>>> > >> > >>>>> > >> are
>>> > >> > >>>>> some
>>> > >> > >>>>> > >> instructions on how to crate XML config files:
>>> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> Tatiana
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
>>> > >> > >>>>> > >> wrote:
>>> > >> > >>>>> > >> > Hi,
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > I'm attaching the xml below, but am having
hard time
>>> > >> > >>>>> > >> > following
>>> > >> > >>>>> how
>>> > >> > >>>>> > >> > to
>>> > >> > >>>>> > >> > go
>>> > >> > >>>>> > >> > about the other steps since I have zero
experience on
>>> > >> > >>>>> > >> > how
>>> > >> > >>>>> > >> > to do
>>> > >> > >>>>> > >> > these
>>> > >> > >>>>> > >> > things.
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > #2. All parameters in <folders> section should
NOT be
>>> > >> > >>>>> > >> > changed
>>> > >> > >>>>> and
>>> > >> > >>>>> > >> > will
>>> > >> > >>>>> > >> > be
>>> > >> > >>>>> > >> > replaced by other scripts during the batch
execution.
>>> > >> > >>>>> > >> > The
>>> > >> > >>>>> > >> > data
>>> > >> > >>>>> for
>>> > >> > >>>>> > >> > the
>>> > >> > >>>>> > >> > batch mode is coming from the database and not
from
>>> > >> > >>>>> > >> > files.
>>> > >> > >>>>> > >> > The
>>> > >> > >>>>> > >> > name of
>>> > >> > >>>>> > >> > the
>>> > >> > >>>>> > >> > database is in <database> tag.
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > I understand that nothing should be changed in
the
>>> > >> > >>>>> > >> > folders
>>> > >> > >>>>> > >> > section,
>>> > >> > >>>>> > >> > but
>>> > >> > >>>>> > >> > there is no indication how and where these
variables
>>> > >> > >>>>> > >> > are
>>> > >> > >>>>> > >> > being
>>> > >> > >>>>> > >> > defined.
>>> > >> > >>>>> > >> > Grepping these names of the files available
does not
>>> > >> > >>>>> > >> > reveal
>>> > >> > >>>>> formal
>>> > >> > >>>>> > >> > definitions of these variables. If I
understood
>>> > >> > >>>>> > >> > exactly
>>> > >> > >>>>> > >> > how
>>> > >> > >>>>> these
>>> > >> > >>>>> > >> > were
>>> > >> > >>>>> > >> > being defined, then I would be able connect
this
>>> > >> > >>>>> > >> > better.
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
files. They
>>> > >> > >>>>> > >> > use
>>> > >> > >>>>> METviewer
>>> > >> > >>>>> > >> > load
>>> > >> > >>>>> > >> > mode to load these files to the specific
database. In
>>> > >> > >>>>> > >> > the
>>> > >> > >>>>> > >> > config
>>> > >> > >>>>> > >> > file
>>> > >> > >>>>> > >> > for
>>> > >> > >>>>> > >> > this process they also can specify the
database
>>> > >> > >>>>> > >> > group. For
>>> > >> > >>>>> example
>>> > >> > >>>>> > >> > database
>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>> > >> > >>>>> > >> > After the data has been loaded to the database
it is
>>> > >> > >>>>> > >> > visible and
>>> > >> > >>>>> > >> > available
>>> > >> > >>>>> > >> > in the GUI and also can be used by the batch
mode.
>>> > >> > >>>>> > >> > You can always add more data to the database
using
>>> > >> > >>>>> > >> > the
>>> > >> > >>>>> > >> > same load
>>> > >> > >>>>> > >> > mode.
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > Is there a guide for me to understand the
process
>>> > >> > >>>>> > >> > that you
>>> > >> > >>>>> > >> > are
>>> > >> > >>>>> > >> > referring to?
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek
via RT
>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>> > >> > >>>>> > >> > wrote:
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > > Hi Edward,
>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA servers
and
>>> > >> > >>>>> > >> > > can't
>>> > >> > >>>>> > >> > > view your
>>> > >> > >>>>> > >> > > XML
>>> > >> > >>>>> > >> > > for the
>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this issue
and I
>>> > >> > >>>>> > >> > > will
>>> > >> > >>>>> > >> > > take a
>>> > >> > >>>>> > >> > > look.
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
should NOT
>>> > >> > >>>>> > >> > > be
>>> > >> > >>>>> > >> > > changed
>>> > >> > >>>>> > >> > > and
>>> > >> > >>>>> > >> > > will be
>>> > >> > >>>>> > >> > > replaced by other scripts during the batch
>>> > >> > >>>>> > >> > > execution.
>>> > >> > >>>>> > >> > > The data
>>> > >> > >>>>> > >> > > for
>>> > >> > >>>>> > >> > > the
>>> > >> > >>>>> > >> > > batch mode is coming from the database and
not from
>>> > >> > >>>>> > >> > > files. The
>>> > >> > >>>>> > >> > > name
>>> > >> > >>>>> > >> > > of the
>>> > >> > >>>>> > >> > > database is in <database> tag.
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
>>> > >> > >>>>> > >> > > They use
>>> > >> > >>>>> > >> > > METviewer
>>> > >> > >>>>> > >> > > load
>>> > >> > >>>>> > >> > > mode to load these files to the specific
database.
>>> > >> > >>>>> > >> > > In
>>> > >> > >>>>> > >> > > the
>>> > >> > >>>>> config
>>> > >> > >>>>> > >> > > file
>>> > >> > >>>>> > >> > > for
>>> > >> > >>>>> > >> > > this process they also can specify the
database
>>> > >> > >>>>> > >> > > group.
>>> > >> > >>>>> > >> > > For
>>> > >> > >>>>> > >> > > example
>>> > >> > >>>>> > >> > > database
>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>> > >> > >>>>> > >> > > After the data has been loaded to the
database it
>>> > >> > >>>>> > >> > > is
>>> > >> > >>>>> > >> > > visible
>>> > >> > >>>>> and
>>> > >> > >>>>> > >> > > available
>>> > >> > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
>>> > >> > >>>>> > >> > > You can always add more data to the database
using
>>> > >> > >>>>> > >> > > the
>>> > >> > >>>>> > >> > > same
>>> > >> > >>>>> load
>>> > >> > >>>>> > >> > > mode.
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > > Tatiana
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>> > >> > >>>>> > >> > > wrote:
>>> > >> > >>>>> > >> > > > Good morning,
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
effort to
>>> > >> > >>>>> > >> > > > understand
>>> > >> > >>>>> > >> > > > metviewer,
>>> > >> > >>>>> > >> > > > test
>>> > >> > >>>>> > >> > > > different cases using the history tab,
changing
>>> > >> > >>>>> > >> > > > information
>>> > >> > >>>>> to
>>> > >> > >>>>> > >> > > > see
>>> > >> > >>>>> > >> > > > what
>>> > >> > >>>>> > >> > > > works and what does not, and make
connections
>>> > >> > >>>>> > >> > > > between
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> GUI
>>> > >> > >>>>> > >> > > > settings
>>> > >> > >>>>> > >> > > > and
>>> > >> > >>>>> > >> > > > the XML file. Although my comfort level
with
>>> > >> > >>>>> > >> > > > metviewer has
>>> > >> > >>>>> > >> > > > significantly
>>> > >> > >>>>> > >> > > > increased during this time, there are some
>>> > >> > >>>>> > >> > > > remaining
>>> > >> > >>>>> > >> > > > issues
>>> > >> > >>>>> > >> > > > that
>>> > >> > >>>>> > >> > > > I'm
>>> > >> > >>>>> > >> > > > hoping
>>> > >> > >>>>> > >> > > > to clear up.
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and running
with
>>> > >> > >>>>> > >> > > > different
>>> > >> > >>>>> > >> > > > statistic
>>> > >> > >>>>> > >> > > > and
>>> > >> > >>>>> > >> > > > plot
>>> > >> > >>>>> > >> > > > type:
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
provided me
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> > >> > > > working
>>> > >> > >>>>> > >> > > > directory
>>> > >> > >>>>> > >> > > > to get
>>> > >> > >>>>> > >> > > > started which I copied over:
>>> > >> > >>>>> > >> > > >
>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted
and used
>>> > >> > >>>>> > >> > > > to
>>> > >> > >>>>> > >> > > > run his
>>> > >> > >>>>> > >> > > > example.
>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
produced a
>>> > >> > >>>>> > >> > > > CSI
>>> > >> > >>>>> > >> > > > plot
>>> > >> > >>>>> that
>>> > >> > >>>>> > >> > > > was
>>> > >> > >>>>> > >> > > > stored
>>> > >> > >>>>> > >> > > > here:
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >>
>>> > >> > >>>>>
>>> > >>
>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Running the script produced the expected
result.
>>> > >> > >>>>> > >> > > > The
>>> > >> > >>>>> > >> > > > next
>>> > >> > >>>>> > >> > > > natural
>>> > >> > >>>>> > >> > > > step was
>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and create a
new xml
>>> > >> > >>>>> > >> > > > file
>>> > >> > >>>>> > >> > > > called
>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to create a
time
>>> > >> > >>>>> > >> > > > series
>>> > >> > >>>>> > >> > > > where
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg instead
of
>>> > >> > >>>>> > >> > > > obs_thresh.
>>> > >> > >>>>> > >> > > > Accordingly,
>>> > >> > >>>>> > >> > > > I
>>> > >> > >>>>> > >> > > > made changes to the fixed fields (removing
>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>> > >> > >>>>> and
>>> > >> > >>>>> > >> > > > changed
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
>>> > >> > >>>>> > >> > > > tried).
>>> > >> > >>>>> > >> > > > The
>>> > >> > >>>>> error
>>> > >> > >>>>> > >> > > > that
>>> > >> > >>>>> > >> > > > resulted
>>> > >> > >>>>> > >> > > > is as follows:
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-
05-01
>>> > >> > >>>>> > >> > > > 13:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 01
>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-
05-01
>>> > >> > >>>>> > >> > > > 16:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 01
>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-
05-01
>>> > >> > >>>>> > >> > > > 19:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 01
>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-
05-01
>>> > >> > >>>>> > >> > > > 22:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 01
>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-
05-02
>>> > >> > >>>>> > >> > > > 01:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 02
>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-
05-02
>>> > >> > >>>>> > >> > > > 04:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 02
>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-
05-02
>>> > >> > >>>>> > >> > > > 07:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 02
>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-
05-02
>>> > >> > >>>>> > >> > > > 10:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 02
>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>> > >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>> > >> > >>>>> > >> > > > No data was returned from database
>>> > >> > >>>>> > >> > > > 2020-06-09
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >>
>>> > >> > >>>>>
>>> > >>
>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
ERROR:
>>> > >> > >>>>> > >> > > > Caught
>>> > >> > >>>>> > >> > > > class
>>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
in
>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>> > >> > >>>>> > >> > > > res): result set contained no data
>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>> > >> > >>>>> > >> > > > in
>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
>>> > >> > >>>>> > >> > > > contained no
>>> > >> > >>>>> > >> > > > data
>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>> > >> > >>>>> > >> > > > in
>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
>>> > >> > >>>>> > >> > > > contained no
>>> > >> > >>>>> > >> > > > data
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Apparently there was no data during this
time. I
>>> > >> > >>>>> > >> > > > tried
>>> > >> > >>>>> > >> > > > changing
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> > >> > > > range
>>> > >> > >>>>> > >> > > > of valid times and changed the valid time
to
>>> > >> > >>>>> > >> > > > different
>>> > >> > >>>>> > >> > > > days
>>> > >> > >>>>> > >> > > > just in
>>> > >> > >>>>> > >> > > > case
>>> > >> > >>>>> > >> > > > data was missing. The issue was the same.
To
>>> > >> > >>>>> > >> > > > confirm
>>> > >> > >>>>> > >> > > > my
>>> > >> > >>>>> > >> > > > logic, I
>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
>>> > >> > >>>>> > >> > > > which I
>>> > >> > >>>>> > >> > > > found
>>> > >> > >>>>> > >> > > > to
>>> > >> > >>>>> > >> > > > work.
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > 2. Running in batch mode
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > I think I lack an understanding of how to
set up
>>> > >> > >>>>> > >> > > > batch
>>> > >> > >>>>> > >> > > > mode
>>> > >> > >>>>> > >> > > > because
>>> > >> > >>>>> > >> > > > I
>>> > >> > >>>>> > >> > > > don't
>>> > >> > >>>>> > >> > > > see where these definitions are being
explicitly
>>> > >> > >>>>> > >> > > > defined:
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > <plot_spec>
>>> > >> > >>>>> > >> > > > <connection>
>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>> > >> > >>>>> > >> > > > </connection>
>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>> > >> > >>>>> > >> > > > <folders>
>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>> > >> > >>>>> > >> > > > </folders>
>>> > >> > >>>>> > >> > > > <plot>
>>> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > I see the <folders> section, but grepping
all
>>> > >> > >>>>> > >> > > > available
>>> > >> > >>>>> files
>>> > >> > >>>>> > >> > > > does
>>> > >> > >>>>> > >> > > > not
>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well
>>> > >> > >>>>> > >> > > > as
>>> > >> > >>>>> > >> > > > other
>>> > >> > >>>>> > >> > > > similarly
>>> > >> > >>>>> > >> > > > defined
>>> > >> > >>>>> > >> > > > variables. I think if I know how these
are being
>>> > >> > >>>>> > >> > > > defined,
>>> > >> > >>>>> > >> > > > then
>>> > >> > >>>>> > >> > > > I'll
>>> > >> > >>>>> > >> > > > be
>>> > >> > >>>>> > >> > > > able to pipe in my information and process
the
>>> > >> > >>>>> > >> > > > data
>>> > >> > >>>>> > >> > > > using
>>> > >> > >>>>> met
>>> > >> > >>>>> > >> > > > and
>>> > >> > >>>>> > >> > > > produce
>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
that?
>>> > >> > >>>>> > >> > > > Also, if
>>> > >> > >>>>> > >> > > > I want
>>> > >> > >>>>> > >> > > > to use
>>> > >> > >>>>> > >> > > > obs for
>>> > >> > >>>>> > >> > > > verification, do I add another line in the
XML
>>> > >> > >>>>> > >> > > > file
>>> > >> > >>>>> > >> > > > that
>>> > >> > >>>>> > >> > > > designates
>>> > >> > >>>>> > >> > > > file
>>> > >> > >>>>> > >> > > > location of obs? I assume that other
steps have
>>> > >> > >>>>> > >> > > > to be
>>> > >> > >>>>> > >> > > > run
>>> > >> > >>>>> > >> > > > before
>>> > >> > >>>>> > >> > > > all
>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form, for
example,
>>> > >> > >>>>> > >> > > > I
>>> > >> > >>>>> > >> > > > would
>>> > >> > >>>>> have
>>> > >> > >>>>> > >> > > > to
>>> > >> > >>>>> > >> > > > separately
>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I
would want
>>> > >> > >>>>> > >> > > > to
>>> > >> > >>>>> > >> > > > do that
>>> > >> > >>>>> > >> > > > as
>>> > >> > >>>>> > >> > > > well.
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
>>> > >> > >>>>> > >> > > > here -
>>> > >> > >>>>> > >> > > > see
>>> > >> > >>>>> > >> > > > attachment
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > How do I get the results I generate to be
>>> > >> > >>>>> > >> > > > uploaded to
>>> > >> > >>>>> > >> > > > this
>>> > >> > >>>>> > >> > > > location?
>>> > >> > >>>>> > >> > > > It's
>>> > >> > >>>>> > >> > > > not clear how others are setting this up.
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>>> > >> > >>>>> > >> rds_user
>>> > >> > >>>>> rds_pwd
>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
>>> > >> > >>>>> rds_scripts
>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>> > >> > >>>>> 2020-05-12
>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
>>> > >> > >>>>> > >> 05-12
>>> > >> > >>>>> 20:00:00
>>> > >> > >>>>> > >> 2020-05-12
>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
>>> > >> > >>>>> > >> 05-13
>>> > >> > >>>>> 10:00:00
>>> > >> > >>>>> > >> 2020-05-13
>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
>>> > >> > >>>>> > >> 05-13
>>> > >> > >>>>> 14:00:00
>>> > >> > >>>>> > >> 2020-05-13
>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
>>> > >> > >>>>> > >> 05-13
>>> > >> > >>>>> 18:00:00
>>> > >> > >>>>> > >> 2020-05-13
>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
>>> > >> > >>>>> > >> 05-13
>>> > >> > >>>>> 22:00:00
>>> > >> > >>>>> > >> 2020-05-13
>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
>>> > >> > >>>>> > >> 05-14
>>> > >> > >>>>> 12:00:00
>>> > >> > >>>>> > >> true
>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>> > >> > >>>>> > >> plot_20200612_173626.png
>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
>>> > >> > >>>>> > >> false list() list() false false false false
false false
>>> > >> > >>>>> > >> true
>>> > >> > >>>>> > >> false
>>> > >> > >>>>> > >> false
>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>> > >> > >>>>> > >> 1.4 -2
>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2
0.5 3
>>> > >> > >>>>> > >> #cccccc 1
>>> > >> > >>>>> > >> listX 1
>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o
c(0, -.25)
>>> > >> > >>>>> > >> 3
>>> > >> > >>>>> > >> 0.8 1
>>> > >> > >>>>> > >> #333333
>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
>>> > >> > >>>>> > >> c(1,1)
>>> > >> > >>>>> c(1,1)
>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>> > >> > >>>>> > >
>>> > >> > >>>>> > >
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > --
>>> > >> > >>>>> > > Edward Strobach
>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>> > >> > >>>>> > > IMSG Contractor
>>> > >> > >>>>> > > Cubicle#: 2029
>>> > >> > >>>>> > > 301-683-3717
>>> > >> > >>>>> > >
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>
>>> > >> > >>>> --
>>> > >> > >>>> Edward Strobach
>>> > >> > >>>> EMC/NCEP/NWS/
>>> > >> > >>>> IMSG Contractor
>>> > >> > >>>> Cubicle#: 2029
>>> > >> > >>>> 301-683-3717
>>> > >> > >>>>
>>> > >> > >>>
>>> > >> > >>>
>>> > >> > >>> --
>>> > >> > >>> Edward Strobach
>>> > >> > >>> EMC/NCEP/NWS/
>>> > >> > >>> IMSG Contractor
>>> > >> > >>> Cubicle#: 2029
>>> > >> > >>> 301-683-3717
>>> > >> > >>>
>>> > >> > >>
>>> > >> > >>
>>> > >> > >> --
>>> > >> > >> Edward Strobach
>>> > >> > >> EMC/NCEP/NWS/
>>> > >> > >> IMSG Contractor
>>> > >> > >> Cubicle#: 2029
>>> > >> > >> 301-683-3717
>>> > >> > >>
>>> > >> > >
>>> > >> > >
>>> > >> > > --
>>> > >> > > Edward Strobach
>>> > >> > > EMC/NCEP/NWS/
>>> > >> > > IMSG Contractor
>>> > >> > > Cubicle#: 2029
>>> > >> > > 301-683-3717
>>> > >> > >
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >
>>> > > --
>>> > > Edward Strobach
>>> > > EMC/NCEP/NWS/
>>> > > IMSG Contractor
>>> > > Cubicle#: 2029
>>> > > 301-683-3717
>>> > >
>>>
>>>
>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Jeffery T. McQueen
Meteorologist
National Center for Weather and Climate Prediction
Room 2095
5830 University Research Court
College Park, MD 20741
301-683-3736
Fax: 301-683-3703
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 12:28:11 2020
Just adding more detail to elaborate my concerns...
Not too long ago I was given a test set-up by Ho-Chun to generate a
plot
for aerosol optical depth. The result was successful and produced a
CSI
plot for AOD, comparing the production run and experimental run.
Although
I created a database, I do not seem to have a group where I can access
specific databases. I recently created a load_xml file, which should
have
loaded the data based on what I specified into the group. It seemed
successful based on the log output that was generated. Additionally,
all
information that I specified in this step was create in the load_*xml.
The next step is to generate plots since the data should be loaded.
Here's
what happens when I run Ho-Chun's - the successful case
HO-CHUN:
*+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
hochun_xml.template+ bash
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # # # #
# # #
# # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT model,
ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
This is early on in the script.
ED_STROBACH:
*+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed -e
's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20 PM
org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
create
initial connections of pool.java.sql.SQLException: Communications link
failure with primary. No active connection found for master. at
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
at
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
at
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
at
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
at
org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
at
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
My interpretation of this is that it fails to bridge a connection to
the
group or database; therefore, it's unable to proceed with running the
XML
file to generate the plots based on the specified content. If I move
past
this step, then I'm golden.
On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
jeff.mcqueen at noaa.gov> wrote:
> Would it be ok to keep perry in the loop
>
> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> you're right. I didn't have the storage problem this time. I'm
not sure
>> what changed. All I know is that I have now created my load xml
which now
>> should allow me to move on to the next step of plotting
>>
>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> I'm not following you. How do I now have enough space? What
changed?
>>>
>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
<met_help at ucar.edu>
>>> wrote:
>>>
>>>> Ed, now you should have enough space for the data loading.
>>>>
>>>> Tatiana
>>>>
>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>>> > I was basically able to build my xml file but it says I have no
>>>> > available
>>>> > space. I'm not sure how that can be since I just started
adding data
>>>> > to my
>>>> > account
>>>> >
>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space left
>>>> > on
>>>> > device
>>>> > ERROR: Command returned with non-zero status (1): scp -r ./*
>>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>>> > 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
>>>> > TABLE_SCHEMA USER
SIZE_MB
>>>> > mv_met_g2o_gefs binbin.zhou
140027.31
>>>> > mv_met_system_hrefv2_v3 binbin.zhou
71986.11
>>>> >
>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate <
>>>> > edward.strobach at noaa.gov> wrote:
>>>> >
>>>> > > Thanks Tatiana. Yes, I've created a database which seems
different
>>>> > > from
>>>> > > mv_group. How can one create their own group so that I can
store and
>>>> > > access
>>>> > > the database. The database exists but cannot be viewed on
the
>>>> > > metviewer
>>>> > > GUI. The list starts with EMC HREFv3 and ends with vhagerty.
I
>>>> > > named my
>>>> > > database Ed_Strobach, but that was really intended for the
group
>>>> > > name.
>>>> > >
>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>>> > > <met_help at ucar.edu>
>>>> > > wrote:
>>>> > >
>>>> > >> We don't have database and METviewer running on wcoss.
>>>> > >> Instead, users use METviewer-wcoss scripts that help to
connect to
>>>> > >> and
>>>> > >> use METviewer that is running on AWS. That is why XML
connection
>>>> and
>>>> > >> folder
>>>> > >> sections look somewhat different than once from the Docs
page.
>>>> > >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>>> > >> For data loading :
>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>>> > >> And for batch:
>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>>> > >>
>>>> > >> These scripts copy XML and data files if needed to AWS,
adjust data
>>>> > >> path
>>>> > >> and users credentials, run METviewer and copy results back
to
>>>> wcoss.
>>>> > >>
>>>> > >> Tatiana
>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
>>>> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
>>>> > >> > -e'create
>>>> > >> > database [db_name]
>>>> > >> >
>>>> > >> > would need to be used to create a database. I assume that
an sql
>>>> > >> > module is
>>>> > >> > available on wcoss; I know you don't have access to those
>>>> machines
>>>> > >> > Tatiana. If I create my own database with my own name,
then I
>>>> > >> > should
>>>> > >> > be
>>>> > >> > able to proceed with project specific tasks under that
tab.
>>>> > >> > Something
>>>> > >> > like
>>>> > >> > mv_EdwardStrobach.
>>>> > >> >
>>>> > >> > I can then create mv_cmaq_prod_v_para which I add data to
using
>>>> > >> > load_spec.
>>>> > >> > Once that step is done, then I can run plot_spec. That's
my
>>>> > >> > takeaway
>>>> > >> > currently
>>>> > >> >
>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate
>>>> <
>>>> > >> > edward.strobach at noaa.gov> wrote:
>>>> > >> >
>>>> > >> > > Actually, it appears that I need to run the database
loading
>>>> > >> > > module
>>>> > >> > > first
>>>> > >> > > if I want to change/add a data source. From there I can
run
>>>> the
>>>> > >> > > plotting
>>>> > >> > > batch module. Is that right? So if I run the database
>>>> loading
>>>> > >> > > module,
>>>> > >> > > then I can specify my dates and other related info
specific to
>>>> > >> > > the
>>>> > >> > > dataset. If I create my database with a unique name,
then it
>>>> > >> > > should
>>>> > >> > > be
>>>> > >> > > recognized on the metviewer gui, presumably.
>>>> > >> > >
>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
>>>> Affiliate
>>>> > >> > > <
>>>> > >> > > edward.strobach at noaa.gov> wrote:
>>>> > >> > >
>>>> > >> > >> It seems that those options must be treated separately
outside
>>>> > >> > >> of
>>>> > >> > >> plot_spec. Is it acceptable to create both a load_spec
and
>>>> > >> > >> plot_spec in
>>>> > >> > >> the same xml file? Would the definitions in load_spec
be able
>>>> > >> > >> to
>>>> > >> > >> carry
>>>> > >> > >> over into plot_spec? It appears that data loading and
>>>> plotting
>>>> > >> > >> are
>>>> > >> > >> treated
>>>> > >> > >> separately.
>>>> > >> > >>
>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
>>>> > >> > >> Affiliate <
>>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>>> > >> > >>
>>>> > >> > >>> I do plan to use date_range; however, in an example
provided
>>>> > >> > >>> by the
>>>> > >> > >>> link
>>>> > >> > >>> I see this:
>>>> > >> > >>>
>>>> > >> > >>> <load_spec>
>>>> > >> > >>> <connection>
>>>> > >> > >>> <host>kemosabe:3306</host>
>>>> > >> > >>> <database>metvdb_hwt</database>
>>>> > >> > >>> <user>pgoldenb</user>
>>>> > >> > >>> <password>pgoldenb</password>
>>>> > >> > >>> </connection>
>>>> > >> > >>>
>>>> > >> > >>> <date_list name="folder_dates">
>>>> > >> > >>> <start>2010051914V</start>
>>>> > >> > >>> <end><date_offset><day_offset>0</day_offset><hour>6
>>>> > >> > >>> </hour></date_offset></end>
>>>> > >> > >>> <inc>3600</inc>
>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
>>>> > >> > >>> </date_list>
>>>> > >> > >>>
>>>> > >> > >>> <met_version>V3.0</met_version>
>>>> > >> > >>>
>>>> > >> > >>> <verbose>false</verbose>
>>>> > >> > >>> <insert_size>1</insert_size>
>>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>>> > >> > >>> <group>Group name</group>
>>>> > >> > >>> <load_stat>true</load_stat>
>>>> > >> > >>> <load_mode>true</load_mode>
>>>> > >> > >>> <load_mtd>true</load_mtd>
>>>> > >> > >>> <load_mpr>false</load_mpr>
>>>> > >> > >>>
>>>> > >> > >>>
>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>> > >> > >>> <load_val>
>>>> > >> > >>> <field name="model">
>>>> > >> > >>> <val>arw</val>
>>>> > >> > >>> <val>nmm</val>
>>>> > >> > >>> </field>
>>>> > >> > >>>
>>>> > >> > >>> <field name="valid_time">
>>>> > >> > >>> <date_list name="folder_dates"/>
>>>> > >> > >>> </field>
>>>> > >> > >>>
>>>> > >> > >>> <field name="vx_mask">
>>>> > >> > >>> <val>FULL</val>
>>>> > >> > >>> <val>SWC</val>
>>>> > >> > >>> </field>
>>>> > >> > >>> </load_val>
>>>> > >> > >>> </load_spec>
>>>> > >> > >>>
>>>> > >> > >>> Are similar settings allowed with <plot_spec>? I see
>>>> > >> > >>> <connection>,
>>>> > >> > >>> which seems carried over. I'm looking to specify my
>>>> > >> > >>> folder_tmpl in
>>>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>>>> > >> > >>> Since my
>>>> > >> > >>> directory structure looks like this:
>>>> > >> > >>>
>>>> > >> > >>>
>>>> > >> > >>>
>>>> > >>
>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>> > >> > >>>
>>>> > >> > >>> I would do something like this I would think:
>>>> > >> > >>>
>>>> > >> > >>> .......
>>>> > >> > >>>
>>>> > >> > >>> <date_list name="folder_dates">
>>>> > >> > >>> <start>20200612/start>
>>>> > >> > >>> <end><date_offset><day_offset>0</day_offset></end>
>>>> > >> > >>> <inc>86400</inc>
>>>> > >> > >>> <format>yyyyMMdd</format>
>>>> > >> > >>> </date_list>
>>>> > >> > >>>
>>>> > >> > >>> <folder_tmpl>
>>>> > >> > >>>
>>>> > >>
>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>>> > >> > >>> </folder_tmpl>
>>>> > >> > >>> <load_val>
>>>> > >> > >>> <field name="model">
>>>> > >> > >>> <val>prod</val>
>>>> > >> > >>> </field>
>>>> > >> > >>>
>>>> > >> > >>> <field name="valid_time">
>>>> > >> > >>> <date_list name="folder_dates"/>
>>>> > >> > >>> </field>
>>>> > >> > >>>
>>>> > >> > >>> ......
>>>> > >> > >>>
>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
>>>> > >> > >>> Affiliate <
>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>>> > >> > >>>
>>>> > >> > >>>> Thank you. This is a very helpful example
>>>> > >> > >>>>
>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
>>>> > >> > >>>> <met_help at ucar.edu>
>>>> > >> > >>>> wrote:
>>>> > >> > >>>>
>>>> > >> > >>>>> Hi,
>>>> > >> > >>>>> Here is some examples how to use <date_range> and
>>>> > >> > >>>>> <date_range_list> in
>>>> > >> > >>>>> batch XML.
>>>> > >> > >>>>> First, you need to defined your range. You can have
more
>>>> > >> > >>>>> then one
>>>> > >> > >>>>> <date_range>:
>>>> > >> > >>>>> <date_range name="may">
>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>>>> > >> > >>>>> </date_range>
>>>> > >> > >>>>> <date_range name="march">
>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>>>> > >> > >>>>> </date_range>
>>>> > >> > >>>>>
>>>> > >> > >>>>> and then use one or all date_ranges:
>>>> > >> > >>>>> <plot_fix>
>>>> > >> > >>>>> <field name="fcst_init_beg">
>>>> > >> > >>>>> <date_range name="march"/>
>>>> > >> > >>>>> <date_range name="may"/>
>>>> > >> > >>>>> </field>...
>>>> > >> > >>>>> This configuration would include all dates for March
and
>>>> May
>>>> > >> > >>>>> of
>>>> > >> > >>>>> 2019.
>>>> > >> > >>>>>
>>>> > >> > >>>>> If you need more flexible configuration you can use
>>>> > >> > >>>>> date_range_list.
>>>> > >> > >>>>> For example, you want to select first 6 hours every
day
>>>> > >> > >>>>> beetween
>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your
XML would
>>>> > >> > >>>>> look
>>>> > >> > >>>>> like this:
>>>> > >> > >>>>> <date_range_list name = 'all'>
>>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
>>>> > >> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
>>>> > >> > >>>>> <range_length>360</range_length>
>>>> > >> > >>>>> <inc>1440</inc>
>>>> > >> > >>>>> </date_range_list>
>>>> > >> > >>>>> where the 1st and 2nd values are the start and end,
>>>> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
>>>> > >> > >>>>> This example would create following ranges:
>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>>>> > >> > >>>>>
>>>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
>>>> > >> > >>>>> your
>>>> > >> > >>>>> data.
>>>> > >> > >>>>> For example, if you want to select only dates with
00:00
>>>> you
>>>> > >> > >>>>> can
>>>> > >> > >>>>> specify
>>>> > >> > >>>>> <field name="valid_hour">
>>>> > >> > >>>>> <val>00</val>
>>>> > >> > >>>>> </field>
>>>> > >> > >>>>>
>>>> > >> > >>>>> Tatiana
>>>> > >> > >>>>>
>>>> > >> > >>>>>
>>>> > >> > >>>>>
>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
>>>> wrote:
>>>> > >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
>>>> > >> > >>>>> > of a
>>>> > >> > >>>>> > separate
>>>> > >> > >>>>> > script that pipes dates (and other relevant
information)
>>>> > >> > >>>>> > into
>>>> > >> > >>>>> > the XML
>>>> > >> > >>>>> > file
>>>> > >> > >>>>> > used to generate the results.
>>>> > >> > >>>>> >
>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
>>>> > >> > >>>>> > Affiliate <
>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>>> > >> > >>>>> >
>>>> > >> > >>>>> > > Thanks Tatiana. These resources are helpful.
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > > One last question. Simply typing lines like
this in
>>>> can
>>>> > >> > >>>>> > > take
>>>> > >> > >>>>> > > forever:
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > > <plot_fix>
>>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > > Is there a better way? I see date_range,
>>>> > >> > >>>>> > > date_range_list,
>>>> > >> > >>>>> > > or
>>>> > >> > >>>>> > > date_list.
>>>> > >> > >>>>> > > Can this be used instead of typing individual
values?
>>>> > >> > >>>>> > > This
>>>> > >> > >>>>> > > could
>>>> > >> > >>>>> > > prove
>>>> > >> > >>>>> > > pretty daunting if one would have to type in 700
>>>> entries
>>>> > >> > >>>>> > > each
>>>> > >> > >>>>> > > time.
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek
via RT
>>>> > >> > >>>>> > > <met_help at ucar.edu>
>>>> > >> > >>>>> > > wrote:
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > >> I found a few problems in your XML.
>>>> > >> > >>>>> > >> The database you are using doesn't have data
with
>>>> > >> > >>>>> > >> vx_mask
>>>> > >> > >>>>> > >> 'abcREGabc'.
>>>> > >> > >>>>> > >> That is why METviewer returns an empty dataset
and
>>>> > >> > >>>>> > >> can't
>>>> > >> > >>>>> > >> create a
>>>> > >> > >>>>> > >> plot.
>>>> > >> > >>>>> > >> The other problem is how you define fcst_lead.
This
>>>> > >> > >>>>> > >> database
>>>> > >> > >>>>> > >> has
>>>> > >> > >>>>> > >> values
>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since
you
>>>> > >> > >>>>> > >> include
>>>> > >> > >>>>> > >> all
>>>> > >> > >>>>> > >> fcst_leads
>>>> > >> > >>>>> > >> this variable can be omitted.
>>>> > >> > >>>>> > >> To minimize problems with variables values, I
would
>>>> > >> > >>>>> > >> suggest
>>>> > >> > >>>>> > >> to
>>>> > >> > >>>>> > >> create XML
>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>>>> > >> > >>>>> > >> I created a plot using you configurations with
GUI
>>>> > >> > >>>>> > >> (without
>>>> > >> > >>>>> vx_mack
>>>> > >> > >>>>> > >> and
>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You can
load
>>>> it
>>>> > >> > >>>>> > >> back
>>>> > >> > >>>>> > >> to
>>>> > >> > >>>>> the
>>>> > >> > >>>>> > >> page by
>>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI). Find
the
>>>> > >> > >>>>> > >> plot,
>>>> > >> > >>>>> > >> open XML
>>>> > >> > >>>>> > >> and
>>>> > >> > >>>>> > >> click on "Load to page". After that you can
recreate a
>>>> > >> > >>>>> > >> plot
>>>> > >> > >>>>> > >> and
>>>> > >> > >>>>> edit
>>>> > >> > >>>>> > >> the
>>>> > >> > >>>>> > >> configurations. You can also download a new XML
and
>>>> > >> > >>>>> > >> edit it
>>>> > >> > >>>>> > >> for
>>>> > >> > >>>>> > >> batch.
>>>> > >> > >>>>> > >> I attached the example of load XML to this
message.
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >> Description of rds parameters:
>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
METviewer R
>>>> > >> > >>>>> > >> scripts
>>>> > >> > >>>>> > >> rds_plots - temporary directory where the plots
will
>>>> be
>>>> > >> > >>>>> > >> generated
>>>> > >> > >>>>> > >> rds_data - temporary directory where the data
>>>> retrieved
>>>> > >> > >>>>> > >> from
>>>> > >> > >>>>> > >> the
>>>> > >> > >>>>> > >> database
>>>> > >> > >>>>> > >> and other intermediate files will be stored
>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
scripts for
>>>> > >> > >>>>> > >> this
>>>> > >> > >>>>> > >> run
>>>> > >> > >>>>> will
>>>> > >> > >>>>> > >> be
>>>> > >> > >>>>> > >> stored
>>>> > >> > >>>>> > >> After the execution is done all the files from
>>>> > >> > >>>>> > >> temporary
>>>> > >> > >>>>> directories
>>>> > >> > >>>>> > >> would be removed.
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
>>>> > >> > >>>>> > >> There
>>>> > >> > >>>>> > >> are
>>>> > >> > >>>>> some
>>>> > >> > >>>>> > >> instructions on how to crate XML config files:
>>>> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >> Tatiana
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
>>>> > >> > >>>>> > >> wrote:
>>>> > >> > >>>>> > >> > Hi,
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am having
hard time
>>>> > >> > >>>>> > >> > following
>>>> > >> > >>>>> how
>>>> > >> > >>>>> > >> > to
>>>> > >> > >>>>> > >> > go
>>>> > >> > >>>>> > >> > about the other steps since I have zero
experience
>>>> on
>>>> > >> > >>>>> > >> > how
>>>> > >> > >>>>> > >> > to do
>>>> > >> > >>>>> > >> > these
>>>> > >> > >>>>> > >> > things.
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should NOT
>>>> be
>>>> > >> > >>>>> > >> > changed
>>>> > >> > >>>>> and
>>>> > >> > >>>>> > >> > will
>>>> > >> > >>>>> > >> > be
>>>> > >> > >>>>> > >> > replaced by other scripts during the batch
>>>> execution.
>>>> > >> > >>>>> > >> > The
>>>> > >> > >>>>> > >> > data
>>>> > >> > >>>>> for
>>>> > >> > >>>>> > >> > the
>>>> > >> > >>>>> > >> > batch mode is coming from the database and
not from
>>>> > >> > >>>>> > >> > files.
>>>> > >> > >>>>> > >> > The
>>>> > >> > >>>>> > >> > name of
>>>> > >> > >>>>> > >> > the
>>>> > >> > >>>>> > >> > database is in <database> tag.
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > I understand that nothing should be changed
in the
>>>> > >> > >>>>> > >> > folders
>>>> > >> > >>>>> > >> > section,
>>>> > >> > >>>>> > >> > but
>>>> > >> > >>>>> > >> > there is no indication how and where these
variables
>>>> > >> > >>>>> > >> > are
>>>> > >> > >>>>> > >> > being
>>>> > >> > >>>>> > >> > defined.
>>>> > >> > >>>>> > >> > Grepping these names of the files available
does not
>>>> > >> > >>>>> > >> > reveal
>>>> > >> > >>>>> formal
>>>> > >> > >>>>> > >> > definitions of these variables. If I
understood
>>>> > >> > >>>>> > >> > exactly
>>>> > >> > >>>>> > >> > how
>>>> > >> > >>>>> these
>>>> > >> > >>>>> > >> > were
>>>> > >> > >>>>> > >> > being defined, then I would be able connect
this
>>>> > >> > >>>>> > >> > better.
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
files. They
>>>> > >> > >>>>> > >> > use
>>>> > >> > >>>>> METviewer
>>>> > >> > >>>>> > >> > load
>>>> > >> > >>>>> > >> > mode to load these files to the specific
database.
>>>> In
>>>> > >> > >>>>> > >> > the
>>>> > >> > >>>>> > >> > config
>>>> > >> > >>>>> > >> > file
>>>> > >> > >>>>> > >> > for
>>>> > >> > >>>>> > >> > this process they also can specify the
database
>>>> > >> > >>>>> > >> > group. For
>>>> > >> > >>>>> example
>>>> > >> > >>>>> > >> > database
>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>>> > >> > >>>>> > >> > After the data has been loaded to the
database it is
>>>> > >> > >>>>> > >> > visible and
>>>> > >> > >>>>> > >> > available
>>>> > >> > >>>>> > >> > in the GUI and also can be used by the batch
mode.
>>>> > >> > >>>>> > >> > You can always add more data to the database
using
>>>> > >> > >>>>> > >> > the
>>>> > >> > >>>>> > >> > same load
>>>> > >> > >>>>> > >> > mode.
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > Is there a guide for me to understand the
process
>>>> > >> > >>>>> > >> > that you
>>>> > >> > >>>>> > >> > are
>>>> > >> > >>>>> > >> > referring to?
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek via
>>>> RT
>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>>> > >> > >>>>> > >> > wrote:
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > > Hi Edward,
>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA servers
and
>>>> > >> > >>>>> > >> > > can't
>>>> > >> > >>>>> > >> > > view your
>>>> > >> > >>>>> > >> > > XML
>>>> > >> > >>>>> > >> > > for the
>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue and I
>>>> > >> > >>>>> > >> > > will
>>>> > >> > >>>>> > >> > > take a
>>>> > >> > >>>>> > >> > > look.
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
should NOT
>>>> > >> > >>>>> > >> > > be
>>>> > >> > >>>>> > >> > > changed
>>>> > >> > >>>>> > >> > > and
>>>> > >> > >>>>> > >> > > will be
>>>> > >> > >>>>> > >> > > replaced by other scripts during the batch
>>>> > >> > >>>>> > >> > > execution.
>>>> > >> > >>>>> > >> > > The data
>>>> > >> > >>>>> > >> > > for
>>>> > >> > >>>>> > >> > > the
>>>> > >> > >>>>> > >> > > batch mode is coming from the database and
not
>>>> from
>>>> > >> > >>>>> > >> > > files. The
>>>> > >> > >>>>> > >> > > name
>>>> > >> > >>>>> > >> > > of the
>>>> > >> > >>>>> > >> > > database is in <database> tag.
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
>>>> > >> > >>>>> > >> > > They use
>>>> > >> > >>>>> > >> > > METviewer
>>>> > >> > >>>>> > >> > > load
>>>> > >> > >>>>> > >> > > mode to load these files to the specific
database.
>>>> > >> > >>>>> > >> > > In
>>>> > >> > >>>>> > >> > > the
>>>> > >> > >>>>> config
>>>> > >> > >>>>> > >> > > file
>>>> > >> > >>>>> > >> > > for
>>>> > >> > >>>>> > >> > > this process they also can specify the
database
>>>> > >> > >>>>> > >> > > group.
>>>> > >> > >>>>> > >> > > For
>>>> > >> > >>>>> > >> > > example
>>>> > >> > >>>>> > >> > > database
>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>>> > >> > >>>>> > >> > > After the data has been loaded to the
database it
>>>> > >> > >>>>> > >> > > is
>>>> > >> > >>>>> > >> > > visible
>>>> > >> > >>>>> and
>>>> > >> > >>>>> > >> > > available
>>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch mode.
>>>> > >> > >>>>> > >> > > You can always add more data to the
database using
>>>> > >> > >>>>> > >> > > the
>>>> > >> > >>>>> > >> > > same
>>>> > >> > >>>>> load
>>>> > >> > >>>>> > >> > > mode.
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > > Tatiana
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>>> > >> > >>>>> > >> > > wrote:
>>>> > >> > >>>>> > >> > > > Good morning,
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
effort to
>>>> > >> > >>>>> > >> > > > understand
>>>> > >> > >>>>> > >> > > > metviewer,
>>>> > >> > >>>>> > >> > > > test
>>>> > >> > >>>>> > >> > > > different cases using the history tab,
changing
>>>> > >> > >>>>> > >> > > > information
>>>> > >> > >>>>> to
>>>> > >> > >>>>> > >> > > > see
>>>> > >> > >>>>> > >> > > > what
>>>> > >> > >>>>> > >> > > > works and what does not, and make
connections
>>>> > >> > >>>>> > >> > > > between
>>>> > >> > >>>>> > >> > > > the
>>>> > >> > >>>>> GUI
>>>> > >> > >>>>> > >> > > > settings
>>>> > >> > >>>>> > >> > > > and
>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort level
with
>>>> > >> > >>>>> > >> > > > metviewer has
>>>> > >> > >>>>> > >> > > > significantly
>>>> > >> > >>>>> > >> > > > increased during this time, there are
some
>>>> > >> > >>>>> > >> > > > remaining
>>>> > >> > >>>>> > >> > > > issues
>>>> > >> > >>>>> > >> > > > that
>>>> > >> > >>>>> > >> > > > I'm
>>>> > >> > >>>>> > >> > > > hoping
>>>> > >> > >>>>> > >> > > > to clear up.
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and running
with
>>>> > >> > >>>>> > >> > > > different
>>>> > >> > >>>>> > >> > > > statistic
>>>> > >> > >>>>> > >> > > > and
>>>> > >> > >>>>> > >> > > > plot
>>>> > >> > >>>>> > >> > > > type:
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
provided me
>>>> > >> > >>>>> > >> > > > the
>>>> > >> > >>>>> > >> > > > working
>>>> > >> > >>>>> > >> > > > directory
>>>> > >> > >>>>> > >> > > > to get
>>>> > >> > >>>>> > >> > > > started which I copied over:
>>>> > >> > >>>>> > >> > > >
>>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted
and used
>>>> > >> > >>>>> > >> > > > to
>>>> > >> > >>>>> > >> > > > run his
>>>> > >> > >>>>> > >> > > > example.
>>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
produced a
>>>> > >> > >>>>> > >> > > > CSI
>>>> > >> > >>>>> > >> > > > plot
>>>> > >> > >>>>> that
>>>> > >> > >>>>> > >> > > > was
>>>> > >> > >>>>> > >> > > > stored
>>>> > >> > >>>>> > >> > > > here:
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>>
>>>> > >>
>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > Running the script produced the expected
result.
>>>> > >> > >>>>> > >> > > > The
>>>> > >> > >>>>> > >> > > > next
>>>> > >> > >>>>> > >> > > > natural
>>>> > >> > >>>>> > >> > > > step was
>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and create a
new xml
>>>> > >> > >>>>> > >> > > > file
>>>> > >> > >>>>> > >> > > > called
>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to create
a time
>>>> > >> > >>>>> > >> > > > series
>>>> > >> > >>>>> > >> > > > where
>>>> > >> > >>>>> > >> > > > the
>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
instead of
>>>> > >> > >>>>> > >> > > > obs_thresh.
>>>> > >> > >>>>> > >> > > > Accordingly,
>>>> > >> > >>>>> > >> > > > I
>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>>> > >> > >>>>> and
>>>> > >> > >>>>> > >> > > > changed
>>>> > >> > >>>>> > >> > > > the
>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was
also
>>>> > >> > >>>>> > >> > > > tried).
>>>> > >> > >>>>> > >> > > > The
>>>> > >> > >>>>> error
>>>> > >> > >>>>> > >> > > > that
>>>> > >> > >>>>> > >> > > > resulted
>>>> > >> > >>>>> > >> > > > is as follows:
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN ('2020-
05-01
>>>> > >> > >>>>> > >> > > > 13:00:00',
>>>> > >> > >>>>> > >> > > > '2020-05-
>>>> > >> > >>>>> > >> > > > 01
>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-
05-01
>>>> > >> > >>>>> > >> > > > 16:00:00',
>>>> > >> > >>>>> > >> > > > '2020-05-
>>>> > >> > >>>>> > >> > > > 01
>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-
05-01
>>>> > >> > >>>>> > >> > > > 19:00:00',
>>>> > >> > >>>>> > >> > > > '2020-05-
>>>> > >> > >>>>> > >> > > > 01
>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-
05-01
>>>> > >> > >>>>> > >> > > > 22:00:00',
>>>> > >> > >>>>> > >> > > > '2020-05-
>>>> > >> > >>>>> > >> > > > 01
>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-
05-02
>>>> > >> > >>>>> > >> > > > 01:00:00',
>>>> > >> > >>>>> > >> > > > '2020-05-
>>>> > >> > >>>>> > >> > > > 02
>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-
05-02
>>>> > >> > >>>>> > >> > > > 04:00:00',
>>>> > >> > >>>>> > >> > > > '2020-05-
>>>> > >> > >>>>> > >> > > > 02
>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-
05-02
>>>> > >> > >>>>> > >> > > > 07:00:00',
>>>> > >> > >>>>> > >> > > > '2020-05-
>>>> > >> > >>>>> > >> > > > 02
>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-
05-02
>>>> > >> > >>>>> > >> > > > 10:00:00',
>>>> > >> > >>>>> > >> > > > '2020-05-
>>>> > >> > >>>>> > >> > > > 02
>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>>> > >> > >>>>> > >> > > > No data was returned from database
>>>> > >> > >>>>> > >> > > > 2020-06-09
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>>
>>>> > >>
>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
ERROR:
>>>> > >> > >>>>> > >> > > > Caught
>>>> > >> > >>>>> > >> > > > class
>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException in
>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>>> > >> > >>>>> > >> > > > res): result set contained no data
>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>> > >> > >>>>> > >> > > > in
>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result set
>>>> > >> > >>>>> > >> > > > contained no
>>>> > >> > >>>>> > >> > > > data
>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>> > >> > >>>>> > >> > > > in
>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result set
>>>> > >> > >>>>> > >> > > > contained no
>>>> > >> > >>>>> > >> > > > data
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > Apparently there was no data during this
time.
>>>> I
>>>> > >> > >>>>> > >> > > > tried
>>>> > >> > >>>>> > >> > > > changing
>>>> > >> > >>>>> > >> > > > the
>>>> > >> > >>>>> > >> > > > range
>>>> > >> > >>>>> > >> > > > of valid times and changed the valid time
to
>>>> > >> > >>>>> > >> > > > different
>>>> > >> > >>>>> > >> > > > days
>>>> > >> > >>>>> > >> > > > just in
>>>> > >> > >>>>> > >> > > > case
>>>> > >> > >>>>> > >> > > > data was missing. The issue was the
same. To
>>>> > >> > >>>>> > >> > > > confirm
>>>> > >> > >>>>> > >> > > > my
>>>> > >> > >>>>> > >> > > > logic, I
>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
>>>> > >> > >>>>> > >> > > > which I
>>>> > >> > >>>>> > >> > > > found
>>>> > >> > >>>>> > >> > > > to
>>>> > >> > >>>>> > >> > > > work.
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > I think I lack an understanding of how to
set up
>>>> > >> > >>>>> > >> > > > batch
>>>> > >> > >>>>> > >> > > > mode
>>>> > >> > >>>>> > >> > > > because
>>>> > >> > >>>>> > >> > > > I
>>>> > >> > >>>>> > >> > > > don't
>>>> > >> > >>>>> > >> > > > see where these definitions are being
explicitly
>>>> > >> > >>>>> > >> > > > defined:
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > <plot_spec>
>>>> > >> > >>>>> > >> > > > <connection>
>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>>> > >> > >>>>> > >> > > > </connection>
>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>>> > >> > >>>>> > >> > > > <folders>
>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>> > >> > >>>>> > >> > > > </folders>
>>>> > >> > >>>>> > >> > > > <plot>
>>>> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > I see the <folders> section, but grepping
all
>>>> > >> > >>>>> > >> > > > available
>>>> > >> > >>>>> files
>>>> > >> > >>>>> > >> > > > does
>>>> > >> > >>>>> > >> > > > not
>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined,
as well
>>>> > >> > >>>>> > >> > > > as
>>>> > >> > >>>>> > >> > > > other
>>>> > >> > >>>>> > >> > > > similarly
>>>> > >> > >>>>> > >> > > > defined
>>>> > >> > >>>>> > >> > > > variables. I think if I know how these
are
>>>> being
>>>> > >> > >>>>> > >> > > > defined,
>>>> > >> > >>>>> > >> > > > then
>>>> > >> > >>>>> > >> > > > I'll
>>>> > >> > >>>>> > >> > > > be
>>>> > >> > >>>>> > >> > > > able to pipe in my information and
process the
>>>> > >> > >>>>> > >> > > > data
>>>> > >> > >>>>> > >> > > > using
>>>> > >> > >>>>> met
>>>> > >> > >>>>> > >> > > > and
>>>> > >> > >>>>> > >> > > > produce
>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
that?
>>>> > >> > >>>>> > >> > > > Also, if
>>>> > >> > >>>>> > >> > > > I want
>>>> > >> > >>>>> > >> > > > to use
>>>> > >> > >>>>> > >> > > > obs for
>>>> > >> > >>>>> > >> > > > verification, do I add another line in
the XML
>>>> > >> > >>>>> > >> > > > file
>>>> > >> > >>>>> > >> > > > that
>>>> > >> > >>>>> > >> > > > designates
>>>> > >> > >>>>> > >> > > > file
>>>> > >> > >>>>> > >> > > > location of obs? I assume that other
steps have
>>>> > >> > >>>>> > >> > > > to be
>>>> > >> > >>>>> > >> > > > run
>>>> > >> > >>>>> > >> > > > before
>>>> > >> > >>>>> > >> > > > all
>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form, for
example,
>>>> > >> > >>>>> > >> > > > I
>>>> > >> > >>>>> > >> > > > would
>>>> > >> > >>>>> have
>>>> > >> > >>>>> > >> > > > to
>>>> > >> > >>>>> > >> > > > separately
>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I
would
>>>> want
>>>> > >> > >>>>> > >> > > > to
>>>> > >> > >>>>> > >> > > > do that
>>>> > >> > >>>>> > >> > > > as
>>>> > >> > >>>>> > >> > > > well.
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
>>>> > >> > >>>>> > >> > > > here -
>>>> > >> > >>>>> > >> > > > see
>>>> > >> > >>>>> > >> > > > attachment
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > How do I get the results I generate to be
>>>> > >> > >>>>> > >> > > > uploaded to
>>>> > >> > >>>>> > >> > > > this
>>>> > >> > >>>>> > >> > > > location?
>>>> > >> > >>>>> > >> > > > It's
>>>> > >> > >>>>> > >> > > > not clear how others are setting this up.
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>>>> > >> > >>>>> > >> rds_user
>>>> > >> > >>>>> rds_pwd
>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
>>>> rds_data
>>>> > >> > >>>>> rds_scripts
>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>>> > >> > >>>>> 2020-05-12
>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
19:00:00 2020-
>>>> > >> > >>>>> > >> 05-12
>>>> > >> > >>>>> 20:00:00
>>>> > >> > >>>>> > >> 2020-05-12
>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
23:00:00 2020-
>>>> > >> > >>>>> > >> 05-13
>>>> > >> > >>>>> 10:00:00
>>>> > >> > >>>>> > >> 2020-05-13
>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
13:00:00 2020-
>>>> > >> > >>>>> > >> 05-13
>>>> > >> > >>>>> 14:00:00
>>>> > >> > >>>>> > >> 2020-05-13
>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
17:00:00 2020-
>>>> > >> > >>>>> > >> 05-13
>>>> > >> > >>>>> 18:00:00
>>>> > >> > >>>>> > >> 2020-05-13
>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
21:00:00 2020-
>>>> > >> > >>>>> > >> 05-13
>>>> > >> > >>>>> 22:00:00
>>>> > >> > >>>>> > >> 2020-05-13
>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
11:00:00 2020-
>>>> > >> > >>>>> > >> 05-14
>>>> > >> > >>>>> 12:00:00
>>>> > >> > >>>>> > >> true
>>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>>> > >> > >>>>> > >> plot_20200612_173626.png
>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
>>>> > >> > >>>>> > >> false list() list() false false false false
false
>>>> false
>>>> > >> > >>>>> > >> true
>>>> > >> > >>>>> > >> false
>>>> > >> > >>>>> > >> false
>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
>>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>>> > >> > >>>>> > >> 1.4 -2
>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1
-2 0.5 3
>>>> > >> > >>>>> > >> #cccccc 1
>>>> > >> > >>>>> > >> listX 1
>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o
c(0,
>>>> -.25)
>>>> > >> > >>>>> > >> 3
>>>> > >> > >>>>> > >> 0.8 1
>>>> > >> > >>>>> > >> #333333
>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
>>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
>>>> > >> > >>>>> > >> c(1,1)
>>>> > >> > >>>>> c(1,1)
>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > > --
>>>> > >> > >>>>> > > Edward Strobach
>>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>>> > >> > >>>>> > > IMSG Contractor
>>>> > >> > >>>>> > > Cubicle#: 2029
>>>> > >> > >>>>> > > 301-683-3717
>>>> > >> > >>>>> > >
>>>> > >> > >>>>>
>>>> > >> > >>>>>
>>>> > >> > >>>>>
>>>> > >> > >>>>>
>>>> > >> > >>>>
>>>> > >> > >>>> --
>>>> > >> > >>>> Edward Strobach
>>>> > >> > >>>> EMC/NCEP/NWS/
>>>> > >> > >>>> IMSG Contractor
>>>> > >> > >>>> Cubicle#: 2029
>>>> > >> > >>>> 301-683-3717
>>>> > >> > >>>>
>>>> > >> > >>>
>>>> > >> > >>>
>>>> > >> > >>> --
>>>> > >> > >>> Edward Strobach
>>>> > >> > >>> EMC/NCEP/NWS/
>>>> > >> > >>> IMSG Contractor
>>>> > >> > >>> Cubicle#: 2029
>>>> > >> > >>> 301-683-3717
>>>> > >> > >>>
>>>> > >> > >>
>>>> > >> > >>
>>>> > >> > >> --
>>>> > >> > >> Edward Strobach
>>>> > >> > >> EMC/NCEP/NWS/
>>>> > >> > >> IMSG Contractor
>>>> > >> > >> Cubicle#: 2029
>>>> > >> > >> 301-683-3717
>>>> > >> > >>
>>>> > >> > >
>>>> > >> > >
>>>> > >> > > --
>>>> > >> > > Edward Strobach
>>>> > >> > > EMC/NCEP/NWS/
>>>> > >> > > IMSG Contractor
>>>> > >> > > Cubicle#: 2029
>>>> > >> > > 301-683-3717
>>>> > >> > >
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >
>>>> > > --
>>>> > > Edward Strobach
>>>> > > EMC/NCEP/NWS/
>>>> > > IMSG Contractor
>>>> > > Cubicle#: 2029
>>>> > > 301-683-3717
>>>> > >
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
> --
> Jeffery T. McQueen
> Meteorologist
> National Center for Weather and Climate Prediction
> Room 2095
> 5830 University Research Court
> College Park, MD 20741
> 301-683-3736
> Fax: 301-683-3703
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Wed Jun 17 12:42:15 2020
Ed,
Could you please attache your XML to this ticket.
Tatiana
On Wed Jun 17 12:28:11 2020, edward.strobach at noaa.gov wrote:
> Just adding more detail to elaborate my concerns...
>
> Not too long ago I was given a test set-up by Ho-Chun to generate a
> plot
> for aerosol optical depth. The result was successful and produced a
> CSI
> plot for AOD, comparing the production run and experimental run.
> Although
> I created a database, I do not seem to have a group where I can
access
> specific databases. I recently created a load_xml file, which
should
> have
> loaded the data based on what I specified into the group. It seemed
> successful based on the log output that was generated.
Additionally,
> all
> information that I specified in this step was create in the
load_*xml.
>
> The next step is to generate plots since the data should be loaded.
> Here's
> what happens when I run Ho-Chun's - the successful case
>
> HO-CHUN:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
> hochun_xml.template+ bash
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
> MVBatch
> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # # #
#
> # # #
> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT model,
> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
>
> This is early on in the script.
>
> ED_STROBACH:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed -e
> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
> MVBatch
> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20 PM
> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
> create
> initial connections of pool.java.sql.SQLException: Communications
link
> failure with primary. No active connection found for master.
at
>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
> at
>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
> at
>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
> at
>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
> at
> org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
> at
>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
>
> My interpretation of this is that it fails to bridge a connection to
> the
> group or database; therefore, it's unable to proceed with running
the
> XML
> file to generate the plots based on the specified content. If I
move
> past
> this step, then I'm golden.
>
> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
> jeff.mcqueen at noaa.gov> wrote:
>
> > Would it be ok to keep perry in the loop
> >
> > On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA Affiliate
<
> > edward.strobach at noaa.gov> wrote:
> >
> >> you're right. I didn't have the storage problem this time. I'm
not
> >> sure
> >> what changed. All I know is that I have now created my load xml
> >> which now
> >> should allow me to move on to the next step of plotting
> >>
> >> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA Affiliate
<
> >> edward.strobach at noaa.gov> wrote:
> >>
> >>> I'm not following you. How do I now have enough space? What
> >>> changed?
> >>>
> >>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
> >>> <met_help at ucar.edu>
> >>> wrote:
> >>>
> >>>> Ed, now you should have enough space for the data loading.
> >>>>
> >>>> Tatiana
> >>>>
> >>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
> >>>> > I was basically able to build my xml file but it says I have
no
> >>>> > available
> >>>> > space. I'm not sure how that can be since I just started
adding
> >>>> > data
> >>>> > to my
> >>>> > account
> >>>> >
> >>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space
> >>>> > left
> >>>> > on
> >>>> > device
> >>>> > ERROR: Command returned with non-zero status (1): scp -r ./*
> >>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
> >>>> > 4) Check METviewer AWS database list using
mv_db_size_on_aws.sh
> >>>> > TABLE_SCHEMA USER
> >>>> > SIZE_MB
> >>>> > mv_met_g2o_gefs binbin.zhou
> >>>> > 140027.31
> >>>> > mv_met_system_hrefv2_v3 binbin.zhou
> >>>> > 71986.11
> >>>> >
> >>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate
> >>>> > <
> >>>> > edward.strobach at noaa.gov> wrote:
> >>>> >
> >>>> > > Thanks Tatiana. Yes, I've created a database which seems
> >>>> > > different
> >>>> > > from
> >>>> > > mv_group. How can one create their own group so that I can
> >>>> > > store and
> >>>> > > access
> >>>> > > the database. The database exists but cannot be viewed on
the
> >>>> > > metviewer
> >>>> > > GUI. The list starts with EMC HREFv3 and ends with
vhagerty.
> >>>> > > I
> >>>> > > named my
> >>>> > > database Ed_Strobach, but that was really intended for the
> >>>> > > group
> >>>> > > name.
> >>>> > >
> >>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> >>>> > > <met_help at ucar.edu>
> >>>> > > wrote:
> >>>> > >
> >>>> > >> We don't have database and METviewer running on wcoss.
> >>>> > >> Instead, users use METviewer-wcoss scripts that help to
> >>>> > >> connect to
> >>>> > >> and
> >>>> > >> use METviewer that is running on AWS. That is why XML
> >>>> > >> connection
> >>>> and
> >>>> > >> folder
> >>>> > >> sections look somewhat different than once from the Docs
> >>>> > >> page.
> >>>> > >> To create a database on AWS you need to run
> >>>> > >> mv_create_db_on_aws.sh
> >>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
> >>>> > >> For data loading :
> >>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> >>>> > >> And for batch:
> >>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> >>>> > >>
> >>>> > >> These scripts copy XML and data files if needed to AWS,
> >>>> > >> adjust data
> >>>> > >> path
> >>>> > >> and users credentials, run METviewer and copy results back
to
> >>>> wcoss.
> >>>> > >>
> >>>> > >> Tatiana
> >>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
wrote:
> >>>> > >> > lastly, it does seem thatmysql -u[db_username]
> >>>> > >> > -p[db_password]
> >>>> > >> > -e'create
> >>>> > >> > database [db_name]
> >>>> > >> >
> >>>> > >> > would need to be used to create a database. I assume
that
> >>>> > >> > an sql
> >>>> > >> > module is
> >>>> > >> > available on wcoss; I know you don't have access to
those
> >>>> machines
> >>>> > >> > Tatiana. If I create my own database with my own name,
> >>>> > >> > then I
> >>>> > >> > should
> >>>> > >> > be
> >>>> > >> > able to proceed with project specific tasks under that
tab.
> >>>> > >> > Something
> >>>> > >> > like
> >>>> > >> > mv_EdwardStrobach.
> >>>> > >> >
> >>>> > >> > I can then create mv_cmaq_prod_v_para which I add data
to
> >>>> > >> > using
> >>>> > >> > load_spec.
> >>>> > >> > Once that step is done, then I can run plot_spec.
That's
> >>>> > >> > my
> >>>> > >> > takeaway
> >>>> > >> > currently
> >>>> > >> >
> >>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
> >>>> > >> > Affiliate
> >>>> <
> >>>> > >> > edward.strobach at noaa.gov> wrote:
> >>>> > >> >
> >>>> > >> > > Actually, it appears that I need to run the database
> >>>> > >> > > loading
> >>>> > >> > > module
> >>>> > >> > > first
> >>>> > >> > > if I want to change/add a data source. From there I
can
> >>>> > >> > > run
> >>>> the
> >>>> > >> > > plotting
> >>>> > >> > > batch module. Is that right? So if I run the
database
> >>>> loading
> >>>> > >> > > module,
> >>>> > >> > > then I can specify my dates and other related info
> >>>> > >> > > specific to
> >>>> > >> > > the
> >>>> > >> > > dataset. If I create my database with a unique name,
> >>>> > >> > > then it
> >>>> > >> > > should
> >>>> > >> > > be
> >>>> > >> > > recognized on the metviewer gui, presumably.
> >>>> > >> > >
> >>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
> >>>> Affiliate
> >>>> > >> > > <
> >>>> > >> > > edward.strobach at noaa.gov> wrote:
> >>>> > >> > >
> >>>> > >> > >> It seems that those options must be treated
separately
> >>>> > >> > >> outside
> >>>> > >> > >> of
> >>>> > >> > >> plot_spec. Is it acceptable to create both a
load_spec
> >>>> > >> > >> and
> >>>> > >> > >> plot_spec in
> >>>> > >> > >> the same xml file? Would the definitions in
load_spec
> >>>> > >> > >> be able
> >>>> > >> > >> to
> >>>> > >> > >> carry
> >>>> > >> > >> over into plot_spec? It appears that data loading
and
> >>>> plotting
> >>>> > >> > >> are
> >>>> > >> > >> treated
> >>>> > >> > >> separately.
> >>>> > >> > >>
> >>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
NOAA
> >>>> > >> > >> Affiliate <
> >>>> > >> > >> edward.strobach at noaa.gov> wrote:
> >>>> > >> > >>
> >>>> > >> > >>> I do plan to use date_range; however, in an example
> >>>> > >> > >>> provided
> >>>> > >> > >>> by the
> >>>> > >> > >>> link
> >>>> > >> > >>> I see this:
> >>>> > >> > >>>
> >>>> > >> > >>> <load_spec>
> >>>> > >> > >>> <connection>
> >>>> > >> > >>> <host>kemosabe:3306</host>
> >>>> > >> > >>> <database>metvdb_hwt</database>
> >>>> > >> > >>> <user>pgoldenb</user>
> >>>> > >> > >>> <password>pgoldenb</password>
> >>>> > >> > >>> </connection>
> >>>> > >> > >>>
> >>>> > >> > >>> <date_list name="folder_dates">
> >>>> > >> > >>> <start>2010051914V</start>
> >>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
> >>>> > >> > >>> </hour></date_offset></end>
> >>>> > >> > >>> <inc>3600</inc>
> >>>> > >> > >>> <format>yyyyMMddHH'V'</format>
> >>>> > >> > >>> </date_list>
> >>>> > >> > >>>
> >>>> > >> > >>> <met_version>V3.0</met_version>
> >>>> > >> > >>>
> >>>> > >> > >>> <verbose>false</verbose>
> >>>> > >> > >>> <insert_size>1</insert_size>
> >>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
> >>>> > >> > >>> <drop_indexes>false</drop_indexes>
> >>>> > >> > >>> <apply_indexes>true</apply_indexes>
> >>>> > >> > >>> <group>Group name</group>
> >>>> > >> > >>> <load_stat>true</load_stat>
> >>>> > >> > >>> <load_mode>true</load_mode>
> >>>> > >> > >>> <load_mtd>true</load_mtd>
> >>>> > >> > >>> <load_mpr>false</load_mpr>
> >>>> > >> > >>>
> >>>> > >> > >>>
> >>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> >>>> > >> > >>> <load_val>
> >>>> > >> > >>> <field name="model">
> >>>> > >> > >>> <val>arw</val>
> >>>> > >> > >>> <val>nmm</val>
> >>>> > >> > >>> </field>
> >>>> > >> > >>>
> >>>> > >> > >>> <field name="valid_time">
> >>>> > >> > >>> <date_list name="folder_dates"/>
> >>>> > >> > >>> </field>
> >>>> > >> > >>>
> >>>> > >> > >>> <field name="vx_mask">
> >>>> > >> > >>> <val>FULL</val>
> >>>> > >> > >>> <val>SWC</val>
> >>>> > >> > >>> </field>
> >>>> > >> > >>> </load_val>
> >>>> > >> > >>> </load_spec>
> >>>> > >> > >>>
> >>>> > >> > >>> Are similar settings allowed with <plot_spec>? I
see
> >>>> > >> > >>> <connection>,
> >>>> > >> > >>> which seems carried over. I'm looking to specify my
> >>>> > >> > >>> folder_tmpl in
> >>>> > >> > >>> <plot_spec>, which according to Tara, should be
> >>>> > >> > >>> possible.
> >>>> > >> > >>> Since my
> >>>> > >> > >>> directory structure looks like this:
> >>>> > >> > >>>
> >>>> > >> > >>>
> >>>> > >> > >>>
> >>>> > >>
> >>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> >>>> > >> > >>>
> >>>> > >> > >>> I would do something like this I would think:
> >>>> > >> > >>>
> >>>> > >> > >>> .......
> >>>> > >> > >>>
> >>>> > >> > >>> <date_list name="folder_dates">
> >>>> > >> > >>> <start>20200612/start>
> >>>> > >> > >>> <end><date_offset><day_offset>0</day_offset></end>
> >>>> > >> > >>> <inc>86400</inc>
> >>>> > >> > >>> <format>yyyyMMdd</format>
> >>>> > >> > >>> </date_list>
> >>>> > >> > >>>
> >>>> > >> > >>> <folder_tmpl>
> >>>> > >> > >>>
> >>>> > >>
> >>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> >>>> > >> > >>> </folder_tmpl>
> >>>> > >> > >>> <load_val>
> >>>> > >> > >>> <field name="model">
> >>>> > >> > >>> <val>prod</val>
> >>>> > >> > >>> </field>
> >>>> > >> > >>>
> >>>> > >> > >>> <field name="valid_time">
> >>>> > >> > >>> <date_list name="folder_dates"/>
> >>>> > >> > >>> </field>
> >>>> > >> > >>>
> >>>> > >> > >>> ......
> >>>> > >> > >>>
> >>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach -
NOAA
> >>>> > >> > >>> Affiliate <
> >>>> > >> > >>> edward.strobach at noaa.gov> wrote:
> >>>> > >> > >>>
> >>>> > >> > >>>> Thank you. This is a very helpful example
> >>>> > >> > >>>>
> >>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via
RT
> >>>> > >> > >>>> <met_help at ucar.edu>
> >>>> > >> > >>>> wrote:
> >>>> > >> > >>>>
> >>>> > >> > >>>>> Hi,
> >>>> > >> > >>>>> Here is some examples how to use <date_range> and
> >>>> > >> > >>>>> <date_range_list> in
> >>>> > >> > >>>>> batch XML.
> >>>> > >> > >>>>> First, you need to defined your range. You can
have
> >>>> > >> > >>>>> more
> >>>> > >> > >>>>> then one
> >>>> > >> > >>>>> <date_range>:
> >>>> > >> > >>>>> <date_range name="may">
> >>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
> >>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
> >>>> > >> > >>>>> </date_range>
> >>>> > >> > >>>>> <date_range name="march">
> >>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
> >>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
> >>>> > >> > >>>>> </date_range>
> >>>> > >> > >>>>>
> >>>> > >> > >>>>> and then use one or all date_ranges:
> >>>> > >> > >>>>> <plot_fix>
> >>>> > >> > >>>>> <field name="fcst_init_beg">
> >>>> > >> > >>>>> <date_range name="march"/>
> >>>> > >> > >>>>> <date_range name="may"/>
> >>>> > >> > >>>>> </field>...
> >>>> > >> > >>>>> This configuration would include all dates for
March
> >>>> > >> > >>>>> and
> >>>> May
> >>>> > >> > >>>>> of
> >>>> > >> > >>>>> 2019.
> >>>> > >> > >>>>>
> >>>> > >> > >>>>> If you need more flexible configuration you can
use
> >>>> > >> > >>>>> date_range_list.
> >>>> > >> > >>>>> For example, you want to select first 6 hours
every
> >>>> > >> > >>>>> day
> >>>> > >> > >>>>> beetween
> >>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your
XML
> >>>> > >> > >>>>> would
> >>>> > >> > >>>>> look
> >>>> > >> > >>>>> like this:
> >>>> > >> > >>>>> <date_range_list name = 'all'>
> >>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
> >>>> > >> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
> >>>> > >> > >>>>> <range_length>360</range_length>
> >>>> > >> > >>>>> <inc>1440</inc>
> >>>> > >> > >>>>> </date_range_list>
> >>>> > >> > >>>>> where the 1st and 2nd values are the start and
end,
> >>>> > >> > >>>>> range_length - the length of a range in minutes(6
> >>>> > >> > >>>>> houres)
> >>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
houres)
> >>>> > >> > >>>>> This example would create following ranges:
> >>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
> >>>> > >> > >>>>> 06:00:00'
> >>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
> >>>> > >> > >>>>> 06:00:00'
> >>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
> >>>> > >> > >>>>> 06:00:00'
> >>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
> >>>> > >> > >>>>> 06:00:00'
> >>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
> >>>> > >> > >>>>> 06:00:00'
> >>>> > >> > >>>>>
> >>>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
> >>>> > >> > >>>>> filter
> >>>> > >> > >>>>> your
> >>>> > >> > >>>>> data.
> >>>> > >> > >>>>> For example, if you want to select only dates with
> >>>> > >> > >>>>> 00:00
> >>>> you
> >>>> > >> > >>>>> can
> >>>> > >> > >>>>> specify
> >>>> > >> > >>>>> <field name="valid_hour">
> >>>> > >> > >>>>> <val>00</val>
> >>>> > >> > >>>>> </field>
> >>>> > >> > >>>>>
> >>>> > >> > >>>>> Tatiana
> >>>> > >> > >>>>>
> >>>> > >> > >>>>>
> >>>> > >> > >>>>>
> >>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
> >>>> wrote:
> >>>> > >> > >>>>> > and if there isn't a better way, I wonder the
> >>>> > >> > >>>>> > whereabouts
> >>>> > >> > >>>>> > of a
> >>>> > >> > >>>>> > separate
> >>>> > >> > >>>>> > script that pipes dates (and other relevant
> >>>> > >> > >>>>> > information)
> >>>> > >> > >>>>> > into
> >>>> > >> > >>>>> > the XML
> >>>> > >> > >>>>> > file
> >>>> > >> > >>>>> > used to generate the results.
> >>>> > >> > >>>>> >
> >>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach
-
> >>>> > >> > >>>>> > NOAA
> >>>> > >> > >>>>> > Affiliate <
> >>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
> >>>> > >> > >>>>> >
> >>>> > >> > >>>>> > > Thanks Tatiana. These resources are helpful.
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>> > > One last question. Simply typing lines like
this
> >>>> > >> > >>>>> > > in
> >>>> can
> >>>> > >> > >>>>> > > take
> >>>> > >> > >>>>> > > forever:
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>> > > <plot_fix>
> >>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
> >>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
> >>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
> >>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
> >>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
> >>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
> >>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>> > > Is there a better way? I see date_range,
> >>>> > >> > >>>>> > > date_range_list,
> >>>> > >> > >>>>> > > or
> >>>> > >> > >>>>> > > date_list.
> >>>> > >> > >>>>> > > Can this be used instead of typing individual
> >>>> > >> > >>>>> > > values?
> >>>> > >> > >>>>> > > This
> >>>> > >> > >>>>> > > could
> >>>> > >> > >>>>> > > prove
> >>>> > >> > >>>>> > > pretty daunting if one would have to type in
700
> >>>> entries
> >>>> > >> > >>>>> > > each
> >>>> > >> > >>>>> > > time.
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek
via
> >>>> > >> > >>>>> > > RT
> >>>> > >> > >>>>> > > <met_help at ucar.edu>
> >>>> > >> > >>>>> > > wrote:
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>> > >> I found a few problems in your XML.
> >>>> > >> > >>>>> > >> The database you are using doesn't have data
> >>>> > >> > >>>>> > >> with
> >>>> > >> > >>>>> > >> vx_mask
> >>>> > >> > >>>>> > >> 'abcREGabc'.
> >>>> > >> > >>>>> > >> That is why METviewer returns an empty
dataset
> >>>> > >> > >>>>> > >> and
> >>>> > >> > >>>>> > >> can't
> >>>> > >> > >>>>> > >> create a
> >>>> > >> > >>>>> > >> plot.
> >>>> > >> > >>>>> > >> The other problem is how you define
fcst_lead.
> >>>> > >> > >>>>> > >> This
> >>>> > >> > >>>>> > >> database
> >>>> > >> > >>>>> > >> has
> >>>> > >> > >>>>> > >> values
> >>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
Since
> >>>> > >> > >>>>> > >> you
> >>>> > >> > >>>>> > >> include
> >>>> > >> > >>>>> > >> all
> >>>> > >> > >>>>> > >> fcst_leads
> >>>> > >> > >>>>> > >> this variable can be omitted.
> >>>> > >> > >>>>> > >> To minimize problems with variables values, I
> >>>> > >> > >>>>> > >> would
> >>>> > >> > >>>>> > >> suggest
> >>>> > >> > >>>>> > >> to
> >>>> > >> > >>>>> > >> create XML
> >>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
> >>>> > >> > >>>>> > >> standards.
> >>>> > >> > >>>>> > >> I created a plot using you configurations
with
> >>>> > >> > >>>>> > >> GUI
> >>>> > >> > >>>>> > >> (without
> >>>> > >> > >>>>> vx_mack
> >>>> > >> > >>>>> > >> and
> >>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You
can
> >>>> > >> > >>>>> > >> load
> >>>> it
> >>>> > >> > >>>>> > >> back
> >>>> > >> > >>>>> > >> to
> >>>> > >> > >>>>> the
> >>>> > >> > >>>>> > >> page by
> >>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI).
Find
> >>>> > >> > >>>>> > >> the
> >>>> > >> > >>>>> > >> plot,
> >>>> > >> > >>>>> > >> open XML
> >>>> > >> > >>>>> > >> and
> >>>> > >> > >>>>> > >> click on "Load to page". After that you can
> >>>> > >> > >>>>> > >> recreate a
> >>>> > >> > >>>>> > >> plot
> >>>> > >> > >>>>> > >> and
> >>>> > >> > >>>>> edit
> >>>> > >> > >>>>> > >> the
> >>>> > >> > >>>>> > >> configurations. You can also download a new
XML
> >>>> > >> > >>>>> > >> and
> >>>> > >> > >>>>> > >> edit it
> >>>> > >> > >>>>> > >> for
> >>>> > >> > >>>>> > >> batch.
> >>>> > >> > >>>>> > >> I attached the example of load XML to this
> >>>> > >> > >>>>> > >> message.
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>> > >> Description of rds parameters:
> >>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
> >>>> > >> > >>>>> > >> METviewer R
> >>>> > >> > >>>>> > >> scripts
> >>>> > >> > >>>>> > >> rds_plots - temporary directory where the
plots
> >>>> > >> > >>>>> > >> will
> >>>> be
> >>>> > >> > >>>>> > >> generated
> >>>> > >> > >>>>> > >> rds_data - temporary directory where the data
> >>>> retrieved
> >>>> > >> > >>>>> > >> from
> >>>> > >> > >>>>> > >> the
> >>>> > >> > >>>>> > >> database
> >>>> > >> > >>>>> > >> and other intermediate files will be stored
> >>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
> >>>> > >> > >>>>> > >> scripts for
> >>>> > >> > >>>>> > >> this
> >>>> > >> > >>>>> > >> run
> >>>> > >> > >>>>> will
> >>>> > >> > >>>>> > >> be
> >>>> > >> > >>>>> > >> stored
> >>>> > >> > >>>>> > >> After the execution is done all the files
from
> >>>> > >> > >>>>> > >> temporary
> >>>> > >> > >>>>> directories
> >>>> > >> > >>>>> > >> would be removed.
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>> > >> We don't have the offitial user guide for
> >>>> > >> > >>>>> > >> METviewer.
> >>>> > >> > >>>>> > >> There
> >>>> > >> > >>>>> > >> are
> >>>> > >> > >>>>> some
> >>>> > >> > >>>>> > >> instructions on how to crate XML config
files:
> >>>> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>> > >> Tatiana
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
> >>>> > >> > >>>>> > >> edward.strobach at noaa.gov
> >>>> > >> > >>>>> > >> wrote:
> >>>> > >> > >>>>> > >> > Hi,
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> > I'm attaching the xml below, but am having
> >>>> > >> > >>>>> > >> > hard time
> >>>> > >> > >>>>> > >> > following
> >>>> > >> > >>>>> how
> >>>> > >> > >>>>> > >> > to
> >>>> > >> > >>>>> > >> > go
> >>>> > >> > >>>>> > >> > about the other steps since I have zero
> >>>> > >> > >>>>> > >> > experience
> >>>> on
> >>>> > >> > >>>>> > >> > how
> >>>> > >> > >>>>> > >> > to do
> >>>> > >> > >>>>> > >> > these
> >>>> > >> > >>>>> > >> > things.
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should
> >>>> > >> > >>>>> > >> > NOT
> >>>> be
> >>>> > >> > >>>>> > >> > changed
> >>>> > >> > >>>>> and
> >>>> > >> > >>>>> > >> > will
> >>>> > >> > >>>>> > >> > be
> >>>> > >> > >>>>> > >> > replaced by other scripts during the batch
> >>>> execution.
> >>>> > >> > >>>>> > >> > The
> >>>> > >> > >>>>> > >> > data
> >>>> > >> > >>>>> for
> >>>> > >> > >>>>> > >> > the
> >>>> > >> > >>>>> > >> > batch mode is coming from the database and
not
> >>>> > >> > >>>>> > >> > from
> >>>> > >> > >>>>> > >> > files.
> >>>> > >> > >>>>> > >> > The
> >>>> > >> > >>>>> > >> > name of
> >>>> > >> > >>>>> > >> > the
> >>>> > >> > >>>>> > >> > database is in <database> tag.
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> > I understand that nothing should be changed
in
> >>>> > >> > >>>>> > >> > the
> >>>> > >> > >>>>> > >> > folders
> >>>> > >> > >>>>> > >> > section,
> >>>> > >> > >>>>> > >> > but
> >>>> > >> > >>>>> > >> > there is no indication how and where these
> >>>> > >> > >>>>> > >> > variables
> >>>> > >> > >>>>> > >> > are
> >>>> > >> > >>>>> > >> > being
> >>>> > >> > >>>>> > >> > defined.
> >>>> > >> > >>>>> > >> > Grepping these names of the files available
> >>>> > >> > >>>>> > >> > does not
> >>>> > >> > >>>>> > >> > reveal
> >>>> > >> > >>>>> formal
> >>>> > >> > >>>>> > >> > definitions of these variables. If I
> >>>> > >> > >>>>> > >> > understood
> >>>> > >> > >>>>> > >> > exactly
> >>>> > >> > >>>>> > >> > how
> >>>> > >> > >>>>> these
> >>>> > >> > >>>>> > >> > were
> >>>> > >> > >>>>> > >> > being defined, then I would be able connect
> >>>> > >> > >>>>> > >> > this
> >>>> > >> > >>>>> > >> > better.
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
> >>>> > >> > >>>>> > >> > files. They
> >>>> > >> > >>>>> > >> > use
> >>>> > >> > >>>>> METviewer
> >>>> > >> > >>>>> > >> > load
> >>>> > >> > >>>>> > >> > mode to load these files to the specific
> >>>> > >> > >>>>> > >> > database.
> >>>> In
> >>>> > >> > >>>>> > >> > the
> >>>> > >> > >>>>> > >> > config
> >>>> > >> > >>>>> > >> > file
> >>>> > >> > >>>>> > >> > for
> >>>> > >> > >>>>> > >> > this process they also can specify the
> >>>> > >> > >>>>> > >> > database
> >>>> > >> > >>>>> > >> > group. For
> >>>> > >> > >>>>> example
> >>>> > >> > >>>>> > >> > database
> >>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
> >>>> > >> > >>>>> > >> > After the data has been loaded to the
database
> >>>> > >> > >>>>> > >> > it is
> >>>> > >> > >>>>> > >> > visible and
> >>>> > >> > >>>>> > >> > available
> >>>> > >> > >>>>> > >> > in the GUI and also can be used by the
batch
> >>>> > >> > >>>>> > >> > mode.
> >>>> > >> > >>>>> > >> > You can always add more data to the
database
> >>>> > >> > >>>>> > >> > using
> >>>> > >> > >>>>> > >> > the
> >>>> > >> > >>>>> > >> > same load
> >>>> > >> > >>>>> > >> > mode.
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> > Is there a guide for me to understand the
> >>>> > >> > >>>>> > >> > process
> >>>> > >> > >>>>> > >> > that you
> >>>> > >> > >>>>> > >> > are
> >>>> > >> > >>>>> > >> > referring to?
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek
> >>>> > >> > >>>>> > >> > via
> >>>> RT
> >>>> > >> > >>>>> > >> > <met_help at ucar.edu>
> >>>> > >> > >>>>> > >> > wrote:
> >>>> > >> > >>>>> > >> >
> >>>> > >> > >>>>> > >> > > Hi Edward,
> >>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
servers
> >>>> > >> > >>>>> > >> > > and
> >>>> > >> > >>>>> > >> > > can't
> >>>> > >> > >>>>> > >> > > view your
> >>>> > >> > >>>>> > >> > > XML
> >>>> > >> > >>>>> > >> > > for the
> >>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue
> >>>> > >> > >>>>> > >> > > and I
> >>>> > >> > >>>>> > >> > > will
> >>>> > >> > >>>>> > >> > > take a
> >>>> > >> > >>>>> > >> > > look.
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
> >>>> > >> > >>>>> > >> > > should NOT
> >>>> > >> > >>>>> > >> > > be
> >>>> > >> > >>>>> > >> > > changed
> >>>> > >> > >>>>> > >> > > and
> >>>> > >> > >>>>> > >> > > will be
> >>>> > >> > >>>>> > >> > > replaced by other scripts during the
batch
> >>>> > >> > >>>>> > >> > > execution.
> >>>> > >> > >>>>> > >> > > The data
> >>>> > >> > >>>>> > >> > > for
> >>>> > >> > >>>>> > >> > > the
> >>>> > >> > >>>>> > >> > > batch mode is coming from the database
and
> >>>> > >> > >>>>> > >> > > not
> >>>> from
> >>>> > >> > >>>>> > >> > > files. The
> >>>> > >> > >>>>> > >> > > name
> >>>> > >> > >>>>> > >> > > of the
> >>>> > >> > >>>>> > >> > > database is in <database> tag.
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
> >>>> > >> > >>>>> > >> > > files.
> >>>> > >> > >>>>> > >> > > They use
> >>>> > >> > >>>>> > >> > > METviewer
> >>>> > >> > >>>>> > >> > > load
> >>>> > >> > >>>>> > >> > > mode to load these files to the specific
> >>>> > >> > >>>>> > >> > > database.
> >>>> > >> > >>>>> > >> > > In
> >>>> > >> > >>>>> > >> > > the
> >>>> > >> > >>>>> config
> >>>> > >> > >>>>> > >> > > file
> >>>> > >> > >>>>> > >> > > for
> >>>> > >> > >>>>> > >> > > this process they also can specify the
> >>>> > >> > >>>>> > >> > > database
> >>>> > >> > >>>>> > >> > > group.
> >>>> > >> > >>>>> > >> > > For
> >>>> > >> > >>>>> > >> > > example
> >>>> > >> > >>>>> > >> > > database
> >>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
> >>>> > >> > >>>>> > >> > > After the data has been loaded to the
> >>>> > >> > >>>>> > >> > > database it
> >>>> > >> > >>>>> > >> > > is
> >>>> > >> > >>>>> > >> > > visible
> >>>> > >> > >>>>> and
> >>>> > >> > >>>>> > >> > > available
> >>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch
> >>>> > >> > >>>>> > >> > > mode.
> >>>> > >> > >>>>> > >> > > You can always add more data to the
database
> >>>> > >> > >>>>> > >> > > using
> >>>> > >> > >>>>> > >> > > the
> >>>> > >> > >>>>> > >> > > same
> >>>> > >> > >>>>> load
> >>>> > >> > >>>>> > >> > > mode.
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >> > > Tatiana
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> >>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
> >>>> > >> > >>>>> > >> > > wrote:
> >>>> > >> > >>>>> > >> > > > Good morning,
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
> >>>> > >> > >>>>> > >> > > > effort to
> >>>> > >> > >>>>> > >> > > > understand
> >>>> > >> > >>>>> > >> > > > metviewer,
> >>>> > >> > >>>>> > >> > > > test
> >>>> > >> > >>>>> > >> > > > different cases using the history tab,
> >>>> > >> > >>>>> > >> > > > changing
> >>>> > >> > >>>>> > >> > > > information
> >>>> > >> > >>>>> to
> >>>> > >> > >>>>> > >> > > > see
> >>>> > >> > >>>>> > >> > > > what
> >>>> > >> > >>>>> > >> > > > works and what does not, and make
> >>>> > >> > >>>>> > >> > > > connections
> >>>> > >> > >>>>> > >> > > > between
> >>>> > >> > >>>>> > >> > > > the
> >>>> > >> > >>>>> GUI
> >>>> > >> > >>>>> > >> > > > settings
> >>>> > >> > >>>>> > >> > > > and
> >>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
level
> >>>> > >> > >>>>> > >> > > > with
> >>>> > >> > >>>>> > >> > > > metviewer has
> >>>> > >> > >>>>> > >> > > > significantly
> >>>> > >> > >>>>> > >> > > > increased during this time, there are
some
> >>>> > >> > >>>>> > >> > > > remaining
> >>>> > >> > >>>>> > >> > > > issues
> >>>> > >> > >>>>> > >> > > > that
> >>>> > >> > >>>>> > >> > > > I'm
> >>>> > >> > >>>>> > >> > > > hoping
> >>>> > >> > >>>>> > >> > > > to clear up.
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
running
> >>>> > >> > >>>>> > >> > > > with
> >>>> > >> > >>>>> > >> > > > different
> >>>> > >> > >>>>> > >> > > > statistic
> >>>> > >> > >>>>> > >> > > > and
> >>>> > >> > >>>>> > >> > > > plot
> >>>> > >> > >>>>> > >> > > > type:
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
> >>>> > >> > >>>>> > >> > > > provided me
> >>>> > >> > >>>>> > >> > > > the
> >>>> > >> > >>>>> > >> > > > working
> >>>> > >> > >>>>> > >> > > > directory
> >>>> > >> > >>>>> > >> > > > to get
> >>>> > >> > >>>>> > >> > > > started which I copied over:
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted
> >>>> > >> > >>>>> > >> > > > and used
> >>>> > >> > >>>>> > >> > > > to
> >>>> > >> > >>>>> > >> > > > run his
> >>>> > >> > >>>>> > >> > > > example.
> >>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
> >>>> > >> > >>>>> > >> > > > produced a
> >>>> > >> > >>>>> > >> > > > CSI
> >>>> > >> > >>>>> > >> > > > plot
> >>>> > >> > >>>>> that
> >>>> > >> > >>>>> > >> > > > was
> >>>> > >> > >>>>> > >> > > > stored
> >>>> > >> > >>>>> > >> > > > here:
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>>
> >>>> > >>
> >>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > Running the script produced the
expected
> >>>> > >> > >>>>> > >> > > > result.
> >>>> > >> > >>>>> > >> > > > The
> >>>> > >> > >>>>> > >> > > > next
> >>>> > >> > >>>>> > >> > > > natural
> >>>> > >> > >>>>> > >> > > > step was
> >>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and create
a
> >>>> > >> > >>>>> > >> > > > new xml
> >>>> > >> > >>>>> > >> > > > file
> >>>> > >> > >>>>> > >> > > > called
> >>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
create a
> >>>> > >> > >>>>> > >> > > > time
> >>>> > >> > >>>>> > >> > > > series
> >>>> > >> > >>>>> > >> > > > where
> >>>> > >> > >>>>> > >> > > > the
> >>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
instead
> >>>> > >> > >>>>> > >> > > > of
> >>>> > >> > >>>>> > >> > > > obs_thresh.
> >>>> > >> > >>>>> > >> > > > Accordingly,
> >>>> > >> > >>>>> > >> > > > I
> >>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
> >>>> > >> > >>>>> > >> > > > fcst_valid_beg)
> >>>> > >> > >>>>> and
> >>>> > >> > >>>>> > >> > > > changed
> >>>> > >> > >>>>> > >> > > > the
> >>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was
also
> >>>> > >> > >>>>> > >> > > > tried).
> >>>> > >> > >>>>> > >> > > > The
> >>>> > >> > >>>>> error
> >>>> > >> > >>>>> > >> > > > that
> >>>> > >> > >>>>> > >> > > > resulted
> >>>> > >> > >>>>> > >> > > > is as follows:
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
('2020-
> >>>> > >> > >>>>> > >> > > > 05-01
> >>>> > >> > >>>>> > >> > > > 13:00:00',
> >>>> > >> > >>>>> > >> > > > '2020-05-
> >>>> > >> > >>>>> > >> > > > 01
> >>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
'2020-
> >>>> > >> > >>>>> > >> > > > 05-01
> >>>> > >> > >>>>> > >> > > > 16:00:00',
> >>>> > >> > >>>>> > >> > > > '2020-05-
> >>>> > >> > >>>>> > >> > > > 01
> >>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
'2020-
> >>>> > >> > >>>>> > >> > > > 05-01
> >>>> > >> > >>>>> > >> > > > 19:00:00',
> >>>> > >> > >>>>> > >> > > > '2020-05-
> >>>> > >> > >>>>> > >> > > > 01
> >>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
'2020-
> >>>> > >> > >>>>> > >> > > > 05-01
> >>>> > >> > >>>>> > >> > > > 22:00:00',
> >>>> > >> > >>>>> > >> > > > '2020-05-
> >>>> > >> > >>>>> > >> > > > 01
> >>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
'2020-
> >>>> > >> > >>>>> > >> > > > 05-02
> >>>> > >> > >>>>> > >> > > > 01:00:00',
> >>>> > >> > >>>>> > >> > > > '2020-05-
> >>>> > >> > >>>>> > >> > > > 02
> >>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
'2020-
> >>>> > >> > >>>>> > >> > > > 05-02
> >>>> > >> > >>>>> > >> > > > 04:00:00',
> >>>> > >> > >>>>> > >> > > > '2020-05-
> >>>> > >> > >>>>> > >> > > > 02
> >>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
'2020-
> >>>> > >> > >>>>> > >> > > > 05-02
> >>>> > >> > >>>>> > >> > > > 07:00:00',
> >>>> > >> > >>>>> > >> > > > '2020-05-
> >>>> > >> > >>>>> > >> > > > 02
> >>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
'2020-
> >>>> > >> > >>>>> > >> > > > 05-02
> >>>> > >> > >>>>> > >> > > > 10:00:00',
> >>>> > >> > >>>>> > >> > > > '2020-05-
> >>>> > >> > >>>>> > >> > > > 02
> >>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> >>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> >>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
h.stat_header_id
> >>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
> >>>> > >> > >>>>> > >> > > > No data was returned from database
> >>>> > >> > >>>>> > >> > > > 2020-06-09
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>>
> >>>> > >>
> >>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
> >>>> > >> > >>>>> > >> > > > ERROR:
> >>>> > >> > >>>>> > >> > > > Caught
> >>>> > >> > >>>>> > >> > > > class
> >>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> >>>> > >> > >>>>> > >> > > > in
> >>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
> >>>> > >> > >>>>> > >> > > > res): result set contained no data
> >>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> >>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> >>>> > >> > >>>>> > >> > > > in
> >>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
> >>>> > >> > >>>>> > >> > > > set
> >>>> > >> > >>>>> > >> > > > contained no
> >>>> > >> > >>>>> > >> > > > data
> >>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> >>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> >>>> > >> > >>>>> > >> > > > in
> >>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
> >>>> > >> > >>>>> > >> > > > set
> >>>> > >> > >>>>> > >> > > > contained no
> >>>> > >> > >>>>> > >> > > > data
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > Apparently there was no data during
this
> >>>> > >> > >>>>> > >> > > > time.
> >>>> I
> >>>> > >> > >>>>> > >> > > > tried
> >>>> > >> > >>>>> > >> > > > changing
> >>>> > >> > >>>>> > >> > > > the
> >>>> > >> > >>>>> > >> > > > range
> >>>> > >> > >>>>> > >> > > > of valid times and changed the valid
time
> >>>> > >> > >>>>> > >> > > > to
> >>>> > >> > >>>>> > >> > > > different
> >>>> > >> > >>>>> > >> > > > days
> >>>> > >> > >>>>> > >> > > > just in
> >>>> > >> > >>>>> > >> > > > case
> >>>> > >> > >>>>> > >> > > > data was missing. The issue was the
same.
> >>>> > >> > >>>>> > >> > > > To
> >>>> > >> > >>>>> > >> > > > confirm
> >>>> > >> > >>>>> > >> > > > my
> >>>> > >> > >>>>> > >> > > > logic, I
> >>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
> >>>> > >> > >>>>> > >> > > > metviewer,
> >>>> > >> > >>>>> > >> > > > which I
> >>>> > >> > >>>>> > >> > > > found
> >>>> > >> > >>>>> > >> > > > to
> >>>> > >> > >>>>> > >> > > > work.
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > 2. Running in batch mode
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > I think I lack an understanding of how
to
> >>>> > >> > >>>>> > >> > > > set up
> >>>> > >> > >>>>> > >> > > > batch
> >>>> > >> > >>>>> > >> > > > mode
> >>>> > >> > >>>>> > >> > > > because
> >>>> > >> > >>>>> > >> > > > I
> >>>> > >> > >>>>> > >> > > > don't
> >>>> > >> > >>>>> > >> > > > see where these definitions are being
> >>>> > >> > >>>>> > >> > > > explicitly
> >>>> > >> > >>>>> > >> > > > defined:
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > <plot_spec>
> >>>> > >> > >>>>> > >> > > > <connection>
> >>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >>>> > >> > >>>>> > >> > > > <user>rds_user</user>
> >>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
> >>>> > >> > >>>>> > >> > > > </connection>
> >>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
> >>>> > >> > >>>>> > >> > > > <folders>
> >>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> >>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
> >>>> > >> > >>>>> > >> > > > <data>rds_data</data>
> >>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> >>>> > >> > >>>>> > >> > > > </folders>
> >>>> > >> > >>>>> > >> > > > <plot>
> >>>> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > I see the <folders> section, but
grepping
> >>>> > >> > >>>>> > >> > > > all
> >>>> > >> > >>>>> > >> > > > available
> >>>> > >> > >>>>> files
> >>>> > >> > >>>>> > >> > > > does
> >>>> > >> > >>>>> > >> > > > not
> >>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined,
as
> >>>> > >> > >>>>> > >> > > > well
> >>>> > >> > >>>>> > >> > > > as
> >>>> > >> > >>>>> > >> > > > other
> >>>> > >> > >>>>> > >> > > > similarly
> >>>> > >> > >>>>> > >> > > > defined
> >>>> > >> > >>>>> > >> > > > variables. I think if I know how these
> >>>> > >> > >>>>> > >> > > > are
> >>>> being
> >>>> > >> > >>>>> > >> > > > defined,
> >>>> > >> > >>>>> > >> > > > then
> >>>> > >> > >>>>> > >> > > > I'll
> >>>> > >> > >>>>> > >> > > > be
> >>>> > >> > >>>>> > >> > > > able to pipe in my information and
process
> >>>> > >> > >>>>> > >> > > > the
> >>>> > >> > >>>>> > >> > > > data
> >>>> > >> > >>>>> > >> > > > using
> >>>> > >> > >>>>> met
> >>>> > >> > >>>>> > >> > > > and
> >>>> > >> > >>>>> > >> > > > produce
> >>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
> >>>> > >> > >>>>> > >> > > > that?
> >>>> > >> > >>>>> > >> > > > Also, if
> >>>> > >> > >>>>> > >> > > > I want
> >>>> > >> > >>>>> > >> > > > to use
> >>>> > >> > >>>>> > >> > > > obs for
> >>>> > >> > >>>>> > >> > > > verification, do I add another line in
the
> >>>> > >> > >>>>> > >> > > > XML
> >>>> > >> > >>>>> > >> > > > file
> >>>> > >> > >>>>> > >> > > > that
> >>>> > >> > >>>>> > >> > > > designates
> >>>> > >> > >>>>> > >> > > > file
> >>>> > >> > >>>>> > >> > > > location of obs? I assume that other
> >>>> > >> > >>>>> > >> > > > steps have
> >>>> > >> > >>>>> > >> > > > to be
> >>>> > >> > >>>>> > >> > > > run
> >>>> > >> > >>>>> > >> > > > before
> >>>> > >> > >>>>> > >> > > > all
> >>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form, for
> >>>> > >> > >>>>> > >> > > > example,
> >>>> > >> > >>>>> > >> > > > I
> >>>> > >> > >>>>> > >> > > > would
> >>>> > >> > >>>>> have
> >>>> > >> > >>>>> > >> > > > to
> >>>> > >> > >>>>> > >> > > > separately
> >>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I
> >>>> > >> > >>>>> > >> > > > would
> >>>> want
> >>>> > >> > >>>>> > >> > > > to
> >>>> > >> > >>>>> > >> > > > do that
> >>>> > >> > >>>>> > >> > > > as
> >>>> > >> > >>>>> > >> > > > well.
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
> >>>> > >> > >>>>> > >> > > > online
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
> >>>> > >> > >>>>> > >> > > > listed
> >>>> > >> > >>>>> > >> > > > here -
> >>>> > >> > >>>>> > >> > > > see
> >>>> > >> > >>>>> > >> > > > attachment
> >>>> > >> > >>>>> > >> > > >
> >>>> > >> > >>>>> > >> > > > How do I get the results I generate to
be
> >>>> > >> > >>>>> > >> > > > uploaded to
> >>>> > >> > >>>>> > >> > > > this
> >>>> > >> > >>>>> > >> > > > location?
> >>>> > >> > >>>>> > >> > > > It's
> >>>> > >> > >>>>> > >> > > > not clear how others are setting this
up.
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >> > >
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>> > >>
> >>>> > >> > >>>>> > >> rds_host:3306
> >>>> > >> > >>>>> > >> mv_cmaqv5para8_aod_grid2grid_metplus
> >>>> > >> > >>>>> > >> rds_user
> >>>> > >> > >>>>> rds_pwd
> >>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
rds_plots
> >>>> rds_data
> >>>> > >> > >>>>> rds_scripts
> >>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
> >>>> > >> > >>>>> > >> CMAQAODPRODHIGH
> >>>> > >> > >>>>> 2020-05-12
> >>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
19:00:00
> >>>> > >> > >>>>> > >> 2020-
> >>>> > >> > >>>>> > >> 05-12
> >>>> > >> > >>>>> 20:00:00
> >>>> > >> > >>>>> > >> 2020-05-12
> >>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
23:00:00
> >>>> > >> > >>>>> > >> 2020-
> >>>> > >> > >>>>> > >> 05-13
> >>>> > >> > >>>>> 10:00:00
> >>>> > >> > >>>>> > >> 2020-05-13
> >>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
13:00:00
> >>>> > >> > >>>>> > >> 2020-
> >>>> > >> > >>>>> > >> 05-13
> >>>> > >> > >>>>> 14:00:00
> >>>> > >> > >>>>> > >> 2020-05-13
> >>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
17:00:00
> >>>> > >> > >>>>> > >> 2020-
> >>>> > >> > >>>>> > >> 05-13
> >>>> > >> > >>>>> 18:00:00
> >>>> > >> > >>>>> > >> 2020-05-13
> >>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
21:00:00
> >>>> > >> > >>>>> > >> 2020-
> >>>> > >> > >>>>> > >> 05-13
> >>>> > >> > >>>>> 22:00:00
> >>>> > >> > >>>>> > >> 2020-05-13
> >>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
11:00:00
> >>>> > >> > >>>>> > >> 2020-
> >>>> > >> > >>>>> > >> 05-14
> >>>> > >> > >>>>> 12:00:00
> >>>> > >> > >>>>> > >> true
> >>>> > >> > >>>>> > >> median plot_20200612_173626.data
> >>>> > >> > >>>>> > >> plot_20200612_173626.png
> >>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
> >>>> > >> > >>>>> > >> false list() list() false false false false
> >>>> > >> > >>>>> > >> false
> >>>> false
> >>>> > >> > >>>>> > >> true
> >>>> > >> > >>>>> > >> false
> >>>> > >> > >>>>> > >> false
> >>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
> >>>> > >> > >>>>> > >> c(8,4,5,4)
> >>>> > >> > >>>>> > >> c(1,1,0) 1 2
> >>>> > >> > >>>>> > >> 1.4 -2
> >>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1
-2
> >>>> > >> > >>>>> > >> 0.5 3
> >>>> > >> > >>>>> > >> #cccccc 1
> >>>> > >> > >>>>> > >> listX 1
> >>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o
> >>>> > >> > >>>>> > >> c(0,
> >>>> -.25)
> >>>> > >> > >>>>> > >> 3
> >>>> > >> > >>>>> > >> 0.8 1
> >>>> > >> > >>>>> > >> #333333
> >>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
> >>>> > >> > >>>>> > >> c(TRUE,TRUE)
> >>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
c("b","b")
> >>>> > >> > >>>>> > >> c(1,1)
> >>>> > >> > >>>>> > >> c(1,1)
> >>>> > >> > >>>>> c(1,1)
> >>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>> > > --
> >>>> > >> > >>>>> > > Edward Strobach
> >>>> > >> > >>>>> > > EMC/NCEP/NWS/
> >>>> > >> > >>>>> > > IMSG Contractor
> >>>> > >> > >>>>> > > Cubicle#: 2029
> >>>> > >> > >>>>> > > 301-683-3717
> >>>> > >> > >>>>> > >
> >>>> > >> > >>>>>
> >>>> > >> > >>>>>
> >>>> > >> > >>>>>
> >>>> > >> > >>>>>
> >>>> > >> > >>>>
> >>>> > >> > >>>> --
> >>>> > >> > >>>> Edward Strobach
> >>>> > >> > >>>> EMC/NCEP/NWS/
> >>>> > >> > >>>> IMSG Contractor
> >>>> > >> > >>>> Cubicle#: 2029
> >>>> > >> > >>>> 301-683-3717
> >>>> > >> > >>>>
> >>>> > >> > >>>
> >>>> > >> > >>>
> >>>> > >> > >>> --
> >>>> > >> > >>> Edward Strobach
> >>>> > >> > >>> EMC/NCEP/NWS/
> >>>> > >> > >>> IMSG Contractor
> >>>> > >> > >>> Cubicle#: 2029
> >>>> > >> > >>> 301-683-3717
> >>>> > >> > >>>
> >>>> > >> > >>
> >>>> > >> > >>
> >>>> > >> > >> --
> >>>> > >> > >> Edward Strobach
> >>>> > >> > >> EMC/NCEP/NWS/
> >>>> > >> > >> IMSG Contractor
> >>>> > >> > >> Cubicle#: 2029
> >>>> > >> > >> 301-683-3717
> >>>> > >> > >>
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > > --
> >>>> > >> > > Edward Strobach
> >>>> > >> > > EMC/NCEP/NWS/
> >>>> > >> > > IMSG Contractor
> >>>> > >> > > Cubicle#: 2029
> >>>> > >> > > 301-683-3717
> >>>> > >> > >
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >
> >>>> > > --
> >>>> > > Edward Strobach
> >>>> > > EMC/NCEP/NWS/
> >>>> > > IMSG Contractor
> >>>> > > Cubicle#: 2029
> >>>> > > 301-683-3717
> >>>> > >
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> Edward Strobach
> >>> EMC/NCEP/NWS/
> >>> IMSG Contractor
> >>> Cubicle#: 2029
> >>> 301-683-3717
> >>>
> >>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> > --
> > Jeffery T. McQueen
> > Meteorologist
> > National Center for Weather and Climate Prediction
> > Room 2095
> > 5830 University Research Court
> > College Park, MD 20741
> > 301-683-3736
> > Fax: 301-683-3703
> >
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Ho-Chun Huang - NOAA Affiliate
Time: Wed Jun 17 13:16:40 2020
Hi, Ed:
You should go back to read 1README again from the directory I sent
about
replacing the top portion of your metviewer xml to produce a batch
xml.
I check your
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
Please update the section from <connection> to </folder> as those in
hochun_xml.template. I also do not know whether AWS can handle
comments
<!-- to !--> in batch xml , better make your batch xml clean.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Just adding more detail to elaborate my concerns...
>
> Not too long ago I was given a test set-up by Ho-Chun to generate a
plot
> for aerosol optical depth. The result was successful and produced a
CSI
> plot for AOD, comparing the production run and experimental run.
Although
> I created a database, I do not seem to have a group where I can
access
> specific databases. I recently created a load_xml file, which
should have
> loaded the data based on what I specified into the group. It seemed
> successful based on the log output that was generated.
Additionally, all
> information that I specified in this step was create in the
load_*xml.
>
> The next step is to generate plots since the data should be loaded.
> Here's what happens when I run Ho-Chun's - the successful case
>
> HO-CHUN:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
> hochun_xml.template+ bash
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # # #
# # # #
> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT model,
> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
>
> This is early on in the script.
>
> ED_STROBACH:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed -e
> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20 PM
> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
create
> initial connections of pool.java.sql.SQLException: Communications
link
> failure with primary. No active connection found for master.
at
>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
> at
>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
> at
>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
> at
>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
> at
> org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
> at
>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
>
> My interpretation of this is that it fails to bridge a connection to
the
> group or database; therefore, it's unable to proceed with running
the XML
> file to generate the plots based on the specified content. If I
move past
> this step, then I'm golden.
>
> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
> jeff.mcqueen at noaa.gov> wrote:
>
>> Would it be ok to keep perry in the loop
>>
>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> you're right. I didn't have the storage problem this time. I'm
not
>>> sure what changed. All I know is that I have now created my load
xml which
>>> now should allow me to move on to the next step of plotting
>>>
>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA Affiliate
<
>>> edward.strobach at noaa.gov> wrote:
>>>
>>>> I'm not following you. How do I now have enough space? What
changed?
>>>>
>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>> Ed, now you should have enough space for the data loading.
>>>>>
>>>>> Tatiana
>>>>>
>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>>>> > I was basically able to build my xml file but it says I have
no
>>>>> > available
>>>>> > space. I'm not sure how that can be since I just started
adding data
>>>>> > to my
>>>>> > account
>>>>> >
>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space left
>>>>> > on
>>>>> > device
>>>>> > ERROR: Command returned with non-zero status (1): scp -r ./*
>>>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>>>> > 4) Check METviewer AWS database list using
mv_db_size_on_aws.sh
>>>>> > TABLE_SCHEMA USER
SIZE_MB
>>>>> > mv_met_g2o_gefs binbin.zhou
140027.31
>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
71986.11
>>>>> >
>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate <
>>>>> > edward.strobach at noaa.gov> wrote:
>>>>> >
>>>>> > > Thanks Tatiana. Yes, I've created a database which seems
different
>>>>> > > from
>>>>> > > mv_group. How can one create their own group so that I can
store
>>>>> and
>>>>> > > access
>>>>> > > the database. The database exists but cannot be viewed on
the
>>>>> > > metviewer
>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
vhagerty. I
>>>>> > > named my
>>>>> > > database Ed_Strobach, but that was really intended for the
group
>>>>> > > name.
>>>>> > >
>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>>>> > > <met_help at ucar.edu>
>>>>> > > wrote:
>>>>> > >
>>>>> > >> We don't have database and METviewer running on wcoss.
>>>>> > >> Instead, users use METviewer-wcoss scripts that help to
connect to
>>>>> > >> and
>>>>> > >> use METviewer that is running on AWS. That is why XML
connection
>>>>> and
>>>>> > >> folder
>>>>> > >> sections look somewhat different than once from the Docs
page.
>>>>> > >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>>>> > >> For data loading :
>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>>>> > >> And for batch:
>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>>>> > >>
>>>>> > >> These scripts copy XML and data files if needed to AWS,
adjust
>>>>> data
>>>>> > >> path
>>>>> > >> and users credentials, run METviewer and copy results back
to
>>>>> wcoss.
>>>>> > >>
>>>>> > >> Tatiana
>>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
wrote:
>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
>>>>> > >> > -e'create
>>>>> > >> > database [db_name]
>>>>> > >> >
>>>>> > >> > would need to be used to create a database. I assume
that an
>>>>> sql
>>>>> > >> > module is
>>>>> > >> > available on wcoss; I know you don't have access to those
>>>>> machines
>>>>> > >> > Tatiana. If I create my own database with my own name,
then I
>>>>> > >> > should
>>>>> > >> > be
>>>>> > >> > able to proceed with project specific tasks under that
tab.
>>>>> > >> > Something
>>>>> > >> > like
>>>>> > >> > mv_EdwardStrobach.
>>>>> > >> >
>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add data to
using
>>>>> > >> > load_spec.
>>>>> > >> > Once that step is done, then I can run plot_spec. That's
my
>>>>> > >> > takeaway
>>>>> > >> > currently
>>>>> > >> >
>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
>>>>> Affiliate <
>>>>> > >> > edward.strobach at noaa.gov> wrote:
>>>>> > >> >
>>>>> > >> > > Actually, it appears that I need to run the database
loading
>>>>> > >> > > module
>>>>> > >> > > first
>>>>> > >> > > if I want to change/add a data source. From there I
can run
>>>>> the
>>>>> > >> > > plotting
>>>>> > >> > > batch module. Is that right? So if I run the
database
>>>>> loading
>>>>> > >> > > module,
>>>>> > >> > > then I can specify my dates and other related info
specific to
>>>>> > >> > > the
>>>>> > >> > > dataset. If I create my database with a unique name,
then it
>>>>> > >> > > should
>>>>> > >> > > be
>>>>> > >> > > recognized on the metviewer gui, presumably.
>>>>> > >> > >
>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
>>>>> Affiliate
>>>>> > >> > > <
>>>>> > >> > > edward.strobach at noaa.gov> wrote:
>>>>> > >> > >
>>>>> > >> > >> It seems that those options must be treated separately
>>>>> outside
>>>>> > >> > >> of
>>>>> > >> > >> plot_spec. Is it acceptable to create both a
load_spec and
>>>>> > >> > >> plot_spec in
>>>>> > >> > >> the same xml file? Would the definitions in load_spec
be
>>>>> able
>>>>> > >> > >> to
>>>>> > >> > >> carry
>>>>> > >> > >> over into plot_spec? It appears that data loading and
>>>>> plotting
>>>>> > >> > >> are
>>>>> > >> > >> treated
>>>>> > >> > >> separately.
>>>>> > >> > >>
>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
>>>>> > >> > >> Affiliate <
>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>>>> > >> > >>
>>>>> > >> > >>> I do plan to use date_range; however, in an example
provided
>>>>> > >> > >>> by the
>>>>> > >> > >>> link
>>>>> > >> > >>> I see this:
>>>>> > >> > >>>
>>>>> > >> > >>> <load_spec>
>>>>> > >> > >>> <connection>
>>>>> > >> > >>> <host>kemosabe:3306</host>
>>>>> > >> > >>> <database>metvdb_hwt</database>
>>>>> > >> > >>> <user>pgoldenb</user>
>>>>> > >> > >>> <password>pgoldenb</password>
>>>>> > >> > >>> </connection>
>>>>> > >> > >>>
>>>>> > >> > >>> <date_list name="folder_dates">
>>>>> > >> > >>> <start>2010051914V</start>
>>>>> > >> > >>> <end><date_offset><day_offset>0</day_offset><hour>6
>>>>> > >> > >>> </hour></date_offset></end>
>>>>> > >> > >>> <inc>3600</inc>
>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
>>>>> > >> > >>> </date_list>
>>>>> > >> > >>>
>>>>> > >> > >>> <met_version>V3.0</met_version>
>>>>> > >> > >>>
>>>>> > >> > >>> <verbose>false</verbose>
>>>>> > >> > >>> <insert_size>1</insert_size>
>>>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>>>> > >> > >>> <group>Group name</group>
>>>>> > >> > >>> <load_stat>true</load_stat>
>>>>> > >> > >>> <load_mode>true</load_mode>
>>>>> > >> > >>> <load_mtd>true</load_mtd>
>>>>> > >> > >>> <load_mpr>false</load_mpr>
>>>>> > >> > >>>
>>>>> > >> > >>>
>>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>>> > >> > >>> <load_val>
>>>>> > >> > >>> <field name="model">
>>>>> > >> > >>> <val>arw</val>
>>>>> > >> > >>> <val>nmm</val>
>>>>> > >> > >>> </field>
>>>>> > >> > >>>
>>>>> > >> > >>> <field name="valid_time">
>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>> > >> > >>> </field>
>>>>> > >> > >>>
>>>>> > >> > >>> <field name="vx_mask">
>>>>> > >> > >>> <val>FULL</val>
>>>>> > >> > >>> <val>SWC</val>
>>>>> > >> > >>> </field>
>>>>> > >> > >>> </load_val>
>>>>> > >> > >>> </load_spec>
>>>>> > >> > >>>
>>>>> > >> > >>> Are similar settings allowed with <plot_spec>? I see
>>>>> > >> > >>> <connection>,
>>>>> > >> > >>> which seems carried over. I'm looking to specify my
>>>>> > >> > >>> folder_tmpl in
>>>>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>>>>> > >> > >>> Since my
>>>>> > >> > >>> directory structure looks like this:
>>>>> > >> > >>>
>>>>> > >> > >>>
>>>>> > >> > >>>
>>>>> > >>
>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>>> > >> > >>>
>>>>> > >> > >>> I would do something like this I would think:
>>>>> > >> > >>>
>>>>> > >> > >>> .......
>>>>> > >> > >>>
>>>>> > >> > >>> <date_list name="folder_dates">
>>>>> > >> > >>> <start>20200612/start>
>>>>> > >> > >>> <end><date_offset><day_offset>0</day_offset></end>
>>>>> > >> > >>> <inc>86400</inc>
>>>>> > >> > >>> <format>yyyyMMdd</format>
>>>>> > >> > >>> </date_list>
>>>>> > >> > >>>
>>>>> > >> > >>> <folder_tmpl>
>>>>> > >> > >>>
>>>>> > >>
>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>>>> > >> > >>> </folder_tmpl>
>>>>> > >> > >>> <load_val>
>>>>> > >> > >>> <field name="model">
>>>>> > >> > >>> <val>prod</val>
>>>>> > >> > >>> </field>
>>>>> > >> > >>>
>>>>> > >> > >>> <field name="valid_time">
>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>> > >> > >>> </field>
>>>>> > >> > >>>
>>>>> > >> > >>> ......
>>>>> > >> > >>>
>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach -
NOAA
>>>>> > >> > >>> Affiliate <
>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>>>> > >> > >>>
>>>>> > >> > >>>> Thank you. This is a very helpful example
>>>>> > >> > >>>>
>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
>>>>> > >> > >>>> <met_help at ucar.edu>
>>>>> > >> > >>>> wrote:
>>>>> > >> > >>>>
>>>>> > >> > >>>>> Hi,
>>>>> > >> > >>>>> Here is some examples how to use <date_range> and
>>>>> > >> > >>>>> <date_range_list> in
>>>>> > >> > >>>>> batch XML.
>>>>> > >> > >>>>> First, you need to defined your range. You can have
more
>>>>> > >> > >>>>> then one
>>>>> > >> > >>>>> <date_range>:
>>>>> > >> > >>>>> <date_range name="may">
>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>>>>> > >> > >>>>> </date_range>
>>>>> > >> > >>>>> <date_range name="march">
>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>>>>> > >> > >>>>> </date_range>
>>>>> > >> > >>>>>
>>>>> > >> > >>>>> and then use one or all date_ranges:
>>>>> > >> > >>>>> <plot_fix>
>>>>> > >> > >>>>> <field name="fcst_init_beg">
>>>>> > >> > >>>>> <date_range name="march"/>
>>>>> > >> > >>>>> <date_range name="may"/>
>>>>> > >> > >>>>> </field>...
>>>>> > >> > >>>>> This configuration would include all dates for
March and
>>>>> May
>>>>> > >> > >>>>> of
>>>>> > >> > >>>>> 2019.
>>>>> > >> > >>>>>
>>>>> > >> > >>>>> If you need more flexible configuration you can use
>>>>> > >> > >>>>> date_range_list.
>>>>> > >> > >>>>> For example, you want to select first 6 hours every
day
>>>>> > >> > >>>>> beetween
>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your
XML
>>>>> would
>>>>> > >> > >>>>> look
>>>>> > >> > >>>>> like this:
>>>>> > >> > >>>>> <date_range_list name = 'all'>
>>>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
>>>>> > >> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
>>>>> > >> > >>>>> <range_length>360</range_length>
>>>>> > >> > >>>>> <inc>1440</inc>
>>>>> > >> > >>>>> </date_range_list>
>>>>> > >> > >>>>> where the 1st and 2nd values are the start and end,
>>>>> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
>>>>> > >> > >>>>> This example would create following ranges:
>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>>>>> > >> > >>>>>
>>>>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
>>>>> > >> > >>>>> your
>>>>> > >> > >>>>> data.
>>>>> > >> > >>>>> For example, if you want to select only dates with
00:00
>>>>> you
>>>>> > >> > >>>>> can
>>>>> > >> > >>>>> specify
>>>>> > >> > >>>>> <field name="valid_hour">
>>>>> > >> > >>>>> <val>00</val>
>>>>> > >> > >>>>> </field>
>>>>> > >> > >>>>>
>>>>> > >> > >>>>> Tatiana
>>>>> > >> > >>>>>
>>>>> > >> > >>>>>
>>>>> > >> > >>>>>
>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
>>>>> wrote:
>>>>> > >> > >>>>> > and if there isn't a better way, I wonder the
>>>>> whereabouts
>>>>> > >> > >>>>> > of a
>>>>> > >> > >>>>> > separate
>>>>> > >> > >>>>> > script that pipes dates (and other relevant
information)
>>>>> > >> > >>>>> > into
>>>>> > >> > >>>>> > the XML
>>>>> > >> > >>>>> > file
>>>>> > >> > >>>>> > used to generate the results.
>>>>> > >> > >>>>> >
>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach
- NOAA
>>>>> > >> > >>>>> > Affiliate <
>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>>>> > >> > >>>>> >
>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are helpful.
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>> > > One last question. Simply typing lines like
this in
>>>>> can
>>>>> > >> > >>>>> > > take
>>>>> > >> > >>>>> > > forever:
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>> > > <plot_fix>
>>>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>> > > Is there a better way? I see date_range,
>>>>> > >> > >>>>> > > date_range_list,
>>>>> > >> > >>>>> > > or
>>>>> > >> > >>>>> > > date_list.
>>>>> > >> > >>>>> > > Can this be used instead of typing individual
values?
>>>>> > >> > >>>>> > > This
>>>>> > >> > >>>>> > > could
>>>>> > >> > >>>>> > > prove
>>>>> > >> > >>>>> > > pretty daunting if one would have to type in
700
>>>>> entries
>>>>> > >> > >>>>> > > each
>>>>> > >> > >>>>> > > time.
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek
via RT
>>>>> > >> > >>>>> > > <met_help at ucar.edu>
>>>>> > >> > >>>>> > > wrote:
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>> > >> I found a few problems in your XML.
>>>>> > >> > >>>>> > >> The database you are using doesn't have data
with
>>>>> > >> > >>>>> > >> vx_mask
>>>>> > >> > >>>>> > >> 'abcREGabc'.
>>>>> > >> > >>>>> > >> That is why METviewer returns an empty dataset
and
>>>>> > >> > >>>>> > >> can't
>>>>> > >> > >>>>> > >> create a
>>>>> > >> > >>>>> > >> plot.
>>>>> > >> > >>>>> > >> The other problem is how you define fcst_lead.
This
>>>>> > >> > >>>>> > >> database
>>>>> > >> > >>>>> > >> has
>>>>> > >> > >>>>> > >> values
>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
Since you
>>>>> > >> > >>>>> > >> include
>>>>> > >> > >>>>> > >> all
>>>>> > >> > >>>>> > >> fcst_leads
>>>>> > >> > >>>>> > >> this variable can be omitted.
>>>>> > >> > >>>>> > >> To minimize problems with variables values, I
would
>>>>> > >> > >>>>> > >> suggest
>>>>> > >> > >>>>> > >> to
>>>>> > >> > >>>>> > >> create XML
>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>>>>> > >> > >>>>> > >> I created a plot using you configurations with
GUI
>>>>> > >> > >>>>> > >> (without
>>>>> > >> > >>>>> vx_mack
>>>>> > >> > >>>>> > >> and
>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You
can load
>>>>> it
>>>>> > >> > >>>>> > >> back
>>>>> > >> > >>>>> > >> to
>>>>> > >> > >>>>> the
>>>>> > >> > >>>>> > >> page by
>>>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI).
Find the
>>>>> > >> > >>>>> > >> plot,
>>>>> > >> > >>>>> > >> open XML
>>>>> > >> > >>>>> > >> and
>>>>> > >> > >>>>> > >> click on "Load to page". After that you can
recreate
>>>>> a
>>>>> > >> > >>>>> > >> plot
>>>>> > >> > >>>>> > >> and
>>>>> > >> > >>>>> edit
>>>>> > >> > >>>>> > >> the
>>>>> > >> > >>>>> > >> configurations. You can also download a new
XML and
>>>>> > >> > >>>>> > >> edit it
>>>>> > >> > >>>>> > >> for
>>>>> > >> > >>>>> > >> batch.
>>>>> > >> > >>>>> > >> I attached the example of load XML to this
message.
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>> > >> Description of rds parameters:
>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
METviewer R
>>>>> > >> > >>>>> > >> scripts
>>>>> > >> > >>>>> > >> rds_plots - temporary directory where the
plots will
>>>>> be
>>>>> > >> > >>>>> > >> generated
>>>>> > >> > >>>>> > >> rds_data - temporary directory where the data
>>>>> retrieved
>>>>> > >> > >>>>> > >> from
>>>>> > >> > >>>>> > >> the
>>>>> > >> > >>>>> > >> database
>>>>> > >> > >>>>> > >> and other intermediate files will be stored
>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
scripts for
>>>>> > >> > >>>>> > >> this
>>>>> > >> > >>>>> > >> run
>>>>> > >> > >>>>> will
>>>>> > >> > >>>>> > >> be
>>>>> > >> > >>>>> > >> stored
>>>>> > >> > >>>>> > >> After the execution is done all the files from
>>>>> > >> > >>>>> > >> temporary
>>>>> > >> > >>>>> directories
>>>>> > >> > >>>>> > >> would be removed.
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
>>>>> > >> > >>>>> > >> There
>>>>> > >> > >>>>> > >> are
>>>>> > >> > >>>>> some
>>>>> > >> > >>>>> > >> instructions on how to crate XML config files:
>>>>> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>> > >> Tatiana
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
>>>>> edward.strobach at noaa.gov
>>>>> > >> > >>>>> > >> wrote:
>>>>> > >> > >>>>> > >> > Hi,
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am having
hard
>>>>> time
>>>>> > >> > >>>>> > >> > following
>>>>> > >> > >>>>> how
>>>>> > >> > >>>>> > >> > to
>>>>> > >> > >>>>> > >> > go
>>>>> > >> > >>>>> > >> > about the other steps since I have zero
experience
>>>>> on
>>>>> > >> > >>>>> > >> > how
>>>>> > >> > >>>>> > >> > to do
>>>>> > >> > >>>>> > >> > these
>>>>> > >> > >>>>> > >> > things.
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should NOT
>>>>> be
>>>>> > >> > >>>>> > >> > changed
>>>>> > >> > >>>>> and
>>>>> > >> > >>>>> > >> > will
>>>>> > >> > >>>>> > >> > be
>>>>> > >> > >>>>> > >> > replaced by other scripts during the batch
>>>>> execution.
>>>>> > >> > >>>>> > >> > The
>>>>> > >> > >>>>> > >> > data
>>>>> > >> > >>>>> for
>>>>> > >> > >>>>> > >> > the
>>>>> > >> > >>>>> > >> > batch mode is coming from the database and
not from
>>>>> > >> > >>>>> > >> > files.
>>>>> > >> > >>>>> > >> > The
>>>>> > >> > >>>>> > >> > name of
>>>>> > >> > >>>>> > >> > the
>>>>> > >> > >>>>> > >> > database is in <database> tag.
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> > I understand that nothing should be changed
in the
>>>>> > >> > >>>>> > >> > folders
>>>>> > >> > >>>>> > >> > section,
>>>>> > >> > >>>>> > >> > but
>>>>> > >> > >>>>> > >> > there is no indication how and where these
>>>>> variables
>>>>> > >> > >>>>> > >> > are
>>>>> > >> > >>>>> > >> > being
>>>>> > >> > >>>>> > >> > defined.
>>>>> > >> > >>>>> > >> > Grepping these names of the files available
does
>>>>> not
>>>>> > >> > >>>>> > >> > reveal
>>>>> > >> > >>>>> formal
>>>>> > >> > >>>>> > >> > definitions of these variables. If I
understood
>>>>> > >> > >>>>> > >> > exactly
>>>>> > >> > >>>>> > >> > how
>>>>> > >> > >>>>> these
>>>>> > >> > >>>>> > >> > were
>>>>> > >> > >>>>> > >> > being defined, then I would be able connect
this
>>>>> > >> > >>>>> > >> > better.
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
files.
>>>>> They
>>>>> > >> > >>>>> > >> > use
>>>>> > >> > >>>>> METviewer
>>>>> > >> > >>>>> > >> > load
>>>>> > >> > >>>>> > >> > mode to load these files to the specific
database.
>>>>> In
>>>>> > >> > >>>>> > >> > the
>>>>> > >> > >>>>> > >> > config
>>>>> > >> > >>>>> > >> > file
>>>>> > >> > >>>>> > >> > for
>>>>> > >> > >>>>> > >> > this process they also can specify the
database
>>>>> > >> > >>>>> > >> > group. For
>>>>> > >> > >>>>> example
>>>>> > >> > >>>>> > >> > database
>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>>>> > >> > >>>>> > >> > After the data has been loaded to the
database it
>>>>> is
>>>>> > >> > >>>>> > >> > visible and
>>>>> > >> > >>>>> > >> > available
>>>>> > >> > >>>>> > >> > in the GUI and also can be used by the batch
mode.
>>>>> > >> > >>>>> > >> > You can always add more data to the database
using
>>>>> > >> > >>>>> > >> > the
>>>>> > >> > >>>>> > >> > same load
>>>>> > >> > >>>>> > >> > mode.
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> > Is there a guide for me to understand the
process
>>>>> > >> > >>>>> > >> > that you
>>>>> > >> > >>>>> > >> > are
>>>>> > >> > >>>>> > >> > referring to?
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek via
>>>>> RT
>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>>>> > >> > >>>>> > >> > wrote:
>>>>> > >> > >>>>> > >> >
>>>>> > >> > >>>>> > >> > > Hi Edward,
>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA servers
and
>>>>> > >> > >>>>> > >> > > can't
>>>>> > >> > >>>>> > >> > > view your
>>>>> > >> > >>>>> > >> > > XML
>>>>> > >> > >>>>> > >> > > for the
>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue and
>>>>> I
>>>>> > >> > >>>>> > >> > > will
>>>>> > >> > >>>>> > >> > > take a
>>>>> > >> > >>>>> > >> > > look.
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
should
>>>>> NOT
>>>>> > >> > >>>>> > >> > > be
>>>>> > >> > >>>>> > >> > > changed
>>>>> > >> > >>>>> > >> > > and
>>>>> > >> > >>>>> > >> > > will be
>>>>> > >> > >>>>> > >> > > replaced by other scripts during the batch
>>>>> > >> > >>>>> > >> > > execution.
>>>>> > >> > >>>>> > >> > > The data
>>>>> > >> > >>>>> > >> > > for
>>>>> > >> > >>>>> > >> > > the
>>>>> > >> > >>>>> > >> > > batch mode is coming from the database and
not
>>>>> from
>>>>> > >> > >>>>> > >> > > files. The
>>>>> > >> > >>>>> > >> > > name
>>>>> > >> > >>>>> > >> > > of the
>>>>> > >> > >>>>> > >> > > database is in <database> tag.
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
>>>>> > >> > >>>>> > >> > > They use
>>>>> > >> > >>>>> > >> > > METviewer
>>>>> > >> > >>>>> > >> > > load
>>>>> > >> > >>>>> > >> > > mode to load these files to the specific
>>>>> database.
>>>>> > >> > >>>>> > >> > > In
>>>>> > >> > >>>>> > >> > > the
>>>>> > >> > >>>>> config
>>>>> > >> > >>>>> > >> > > file
>>>>> > >> > >>>>> > >> > > for
>>>>> > >> > >>>>> > >> > > this process they also can specify the
database
>>>>> > >> > >>>>> > >> > > group.
>>>>> > >> > >>>>> > >> > > For
>>>>> > >> > >>>>> > >> > > example
>>>>> > >> > >>>>> > >> > > database
>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>>>> > >> > >>>>> > >> > > After the data has been loaded to the
database it
>>>>> > >> > >>>>> > >> > > is
>>>>> > >> > >>>>> > >> > > visible
>>>>> > >> > >>>>> and
>>>>> > >> > >>>>> > >> > > available
>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch
>>>>> mode.
>>>>> > >> > >>>>> > >> > > You can always add more data to the
database
>>>>> using
>>>>> > >> > >>>>> > >> > > the
>>>>> > >> > >>>>> > >> > > same
>>>>> > >> > >>>>> load
>>>>> > >> > >>>>> > >> > > mode.
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >> > > Tatiana
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>>>> > >> > >>>>> > >> > > wrote:
>>>>> > >> > >>>>> > >> > > > Good morning,
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
effort to
>>>>> > >> > >>>>> > >> > > > understand
>>>>> > >> > >>>>> > >> > > > metviewer,
>>>>> > >> > >>>>> > >> > > > test
>>>>> > >> > >>>>> > >> > > > different cases using the history tab,
changing
>>>>> > >> > >>>>> > >> > > > information
>>>>> > >> > >>>>> to
>>>>> > >> > >>>>> > >> > > > see
>>>>> > >> > >>>>> > >> > > > what
>>>>> > >> > >>>>> > >> > > > works and what does not, and make
connections
>>>>> > >> > >>>>> > >> > > > between
>>>>> > >> > >>>>> > >> > > > the
>>>>> > >> > >>>>> GUI
>>>>> > >> > >>>>> > >> > > > settings
>>>>> > >> > >>>>> > >> > > > and
>>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort level
with
>>>>> > >> > >>>>> > >> > > > metviewer has
>>>>> > >> > >>>>> > >> > > > significantly
>>>>> > >> > >>>>> > >> > > > increased during this time, there are
some
>>>>> > >> > >>>>> > >> > > > remaining
>>>>> > >> > >>>>> > >> > > > issues
>>>>> > >> > >>>>> > >> > > > that
>>>>> > >> > >>>>> > >> > > > I'm
>>>>> > >> > >>>>> > >> > > > hoping
>>>>> > >> > >>>>> > >> > > > to clear up.
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
running with
>>>>> > >> > >>>>> > >> > > > different
>>>>> > >> > >>>>> > >> > > > statistic
>>>>> > >> > >>>>> > >> > > > and
>>>>> > >> > >>>>> > >> > > > plot
>>>>> > >> > >>>>> > >> > > > type:
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
provided me
>>>>> > >> > >>>>> > >> > > > the
>>>>> > >> > >>>>> > >> > > > working
>>>>> > >> > >>>>> > >> > > > directory
>>>>> > >> > >>>>> > >> > > > to get
>>>>> > >> > >>>>> > >> > > > started which I copied over:
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted
and
>>>>> used
>>>>> > >> > >>>>> > >> > > > to
>>>>> > >> > >>>>> > >> > > > run his
>>>>> > >> > >>>>> > >> > > > example.
>>>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
produced a
>>>>> > >> > >>>>> > >> > > > CSI
>>>>> > >> > >>>>> > >> > > > plot
>>>>> > >> > >>>>> that
>>>>> > >> > >>>>> > >> > > > was
>>>>> > >> > >>>>> > >> > > > stored
>>>>> > >> > >>>>> > >> > > > here:
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>>
>>>>> > >>
>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > Running the script produced the expected
>>>>> result.
>>>>> > >> > >>>>> > >> > > > The
>>>>> > >> > >>>>> > >> > > > next
>>>>> > >> > >>>>> > >> > > > natural
>>>>> > >> > >>>>> > >> > > > step was
>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and create a
new
>>>>> xml
>>>>> > >> > >>>>> > >> > > > file
>>>>> > >> > >>>>> > >> > > > called
>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to create
a time
>>>>> > >> > >>>>> > >> > > > series
>>>>> > >> > >>>>> > >> > > > where
>>>>> > >> > >>>>> > >> > > > the
>>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
instead of
>>>>> > >> > >>>>> > >> > > > obs_thresh.
>>>>> > >> > >>>>> > >> > > > Accordingly,
>>>>> > >> > >>>>> > >> > > > I
>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>>>> > >> > >>>>> and
>>>>> > >> > >>>>> > >> > > > changed
>>>>> > >> > >>>>> > >> > > > the
>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was
also
>>>>> > >> > >>>>> > >> > > > tried).
>>>>> > >> > >>>>> > >> > > > The
>>>>> > >> > >>>>> error
>>>>> > >> > >>>>> > >> > > > that
>>>>> > >> > >>>>> > >> > > > resulted
>>>>> > >> > >>>>> > >> > > > is as follows:
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
('2020-05-01
>>>>> > >> > >>>>> > >> > > > 13:00:00',
>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>> > >> > >>>>> > >> > > > 01
>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-
05-01
>>>>> > >> > >>>>> > >> > > > 16:00:00',
>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>> > >> > >>>>> > >> > > > 01
>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-
05-01
>>>>> > >> > >>>>> > >> > > > 19:00:00',
>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>> > >> > >>>>> > >> > > > 01
>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-
05-01
>>>>> > >> > >>>>> > >> > > > 22:00:00',
>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>> > >> > >>>>> > >> > > > 01
>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-
05-02
>>>>> > >> > >>>>> > >> > > > 01:00:00',
>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>> > >> > >>>>> > >> > > > 02
>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-
05-02
>>>>> > >> > >>>>> > >> > > > 04:00:00',
>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>> > >> > >>>>> > >> > > > 02
>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-
05-02
>>>>> > >> > >>>>> > >> > > > 07:00:00',
>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>> > >> > >>>>> > >> > > > 02
>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-
05-02
>>>>> > >> > >>>>> > >> > > > 10:00:00',
>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>> > >> > >>>>> > >> > > > 02
>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>>>> > >> > >>>>> > >> > > > No data was returned from database
>>>>> > >> > >>>>> > >> > > > 2020-06-09
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>>
>>>>> > >>
>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
ERROR:
>>>>> > >> > >>>>> > >> > > > Caught
>>>>> > >> > >>>>> > >> > > > class
>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException in
>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>>>> > >> > >>>>> > >> > > > res): result set contained no data
>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>> > >> > >>>>> > >> > > > in
>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result set
>>>>> > >> > >>>>> > >> > > > contained no
>>>>> > >> > >>>>> > >> > > > data
>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>> > >> > >>>>> > >> > > > in
>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result set
>>>>> > >> > >>>>> > >> > > > contained no
>>>>> > >> > >>>>> > >> > > > data
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > Apparently there was no data during this
>>>>> time. I
>>>>> > >> > >>>>> > >> > > > tried
>>>>> > >> > >>>>> > >> > > > changing
>>>>> > >> > >>>>> > >> > > > the
>>>>> > >> > >>>>> > >> > > > range
>>>>> > >> > >>>>> > >> > > > of valid times and changed the valid
time to
>>>>> > >> > >>>>> > >> > > > different
>>>>> > >> > >>>>> > >> > > > days
>>>>> > >> > >>>>> > >> > > > just in
>>>>> > >> > >>>>> > >> > > > case
>>>>> > >> > >>>>> > >> > > > data was missing. The issue was the
same. To
>>>>> > >> > >>>>> > >> > > > confirm
>>>>> > >> > >>>>> > >> > > > my
>>>>> > >> > >>>>> > >> > > > logic, I
>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
>>>>> > >> > >>>>> > >> > > > which I
>>>>> > >> > >>>>> > >> > > > found
>>>>> > >> > >>>>> > >> > > > to
>>>>> > >> > >>>>> > >> > > > work.
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > I think I lack an understanding of how
to set
>>>>> up
>>>>> > >> > >>>>> > >> > > > batch
>>>>> > >> > >>>>> > >> > > > mode
>>>>> > >> > >>>>> > >> > > > because
>>>>> > >> > >>>>> > >> > > > I
>>>>> > >> > >>>>> > >> > > > don't
>>>>> > >> > >>>>> > >> > > > see where these definitions are being
>>>>> explicitly
>>>>> > >> > >>>>> > >> > > > defined:
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > <plot_spec>
>>>>> > >> > >>>>> > >> > > > <connection>
>>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>>>> > >> > >>>>> > >> > > > </connection>
>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>>>> > >> > >>>>> > >> > > > <folders>
>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>>> > >> > >>>>> > >> > > > </folders>
>>>>> > >> > >>>>> > >> > > > <plot>
>>>>> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
grepping all
>>>>> > >> > >>>>> > >> > > > available
>>>>> > >> > >>>>> files
>>>>> > >> > >>>>> > >> > > > does
>>>>> > >> > >>>>> > >> > > > not
>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined,
as well
>>>>> > >> > >>>>> > >> > > > as
>>>>> > >> > >>>>> > >> > > > other
>>>>> > >> > >>>>> > >> > > > similarly
>>>>> > >> > >>>>> > >> > > > defined
>>>>> > >> > >>>>> > >> > > > variables. I think if I know how these
are
>>>>> being
>>>>> > >> > >>>>> > >> > > > defined,
>>>>> > >> > >>>>> > >> > > > then
>>>>> > >> > >>>>> > >> > > > I'll
>>>>> > >> > >>>>> > >> > > > be
>>>>> > >> > >>>>> > >> > > > able to pipe in my information and
process the
>>>>> > >> > >>>>> > >> > > > data
>>>>> > >> > >>>>> > >> > > > using
>>>>> > >> > >>>>> met
>>>>> > >> > >>>>> > >> > > > and
>>>>> > >> > >>>>> > >> > > > produce
>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
that?
>>>>> > >> > >>>>> > >> > > > Also, if
>>>>> > >> > >>>>> > >> > > > I want
>>>>> > >> > >>>>> > >> > > > to use
>>>>> > >> > >>>>> > >> > > > obs for
>>>>> > >> > >>>>> > >> > > > verification, do I add another line in
the XML
>>>>> > >> > >>>>> > >> > > > file
>>>>> > >> > >>>>> > >> > > > that
>>>>> > >> > >>>>> > >> > > > designates
>>>>> > >> > >>>>> > >> > > > file
>>>>> > >> > >>>>> > >> > > > location of obs? I assume that other
steps
>>>>> have
>>>>> > >> > >>>>> > >> > > > to be
>>>>> > >> > >>>>> > >> > > > run
>>>>> > >> > >>>>> > >> > > > before
>>>>> > >> > >>>>> > >> > > > all
>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form, for
>>>>> example,
>>>>> > >> > >>>>> > >> > > > I
>>>>> > >> > >>>>> > >> > > > would
>>>>> > >> > >>>>> have
>>>>> > >> > >>>>> > >> > > > to
>>>>> > >> > >>>>> > >> > > > separately
>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I
would
>>>>> want
>>>>> > >> > >>>>> > >> > > > to
>>>>> > >> > >>>>> > >> > > > do that
>>>>> > >> > >>>>> > >> > > > as
>>>>> > >> > >>>>> > >> > > > well.
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
>>>>> > >> > >>>>> > >> > > > here -
>>>>> > >> > >>>>> > >> > > > see
>>>>> > >> > >>>>> > >> > > > attachment
>>>>> > >> > >>>>> > >> > > >
>>>>> > >> > >>>>> > >> > > > How do I get the results I generate to
be
>>>>> > >> > >>>>> > >> > > > uploaded to
>>>>> > >> > >>>>> > >> > > > this
>>>>> > >> > >>>>> > >> > > > location?
>>>>> > >> > >>>>> > >> > > > It's
>>>>> > >> > >>>>> > >> > > > not clear how others are setting this
up.
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >> > >
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>> > >>
>>>>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>>>>> > >> > >>>>> > >> rds_user
>>>>> > >> > >>>>> rds_pwd
>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
>>>>> rds_data
>>>>> > >> > >>>>> rds_scripts
>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>>>> > >> > >>>>> 2020-05-12
>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
19:00:00
>>>>> 2020-
>>>>> > >> > >>>>> > >> 05-12
>>>>> > >> > >>>>> 20:00:00
>>>>> > >> > >>>>> > >> 2020-05-12
>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
23:00:00
>>>>> 2020-
>>>>> > >> > >>>>> > >> 05-13
>>>>> > >> > >>>>> 10:00:00
>>>>> > >> > >>>>> > >> 2020-05-13
>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
13:00:00
>>>>> 2020-
>>>>> > >> > >>>>> > >> 05-13
>>>>> > >> > >>>>> 14:00:00
>>>>> > >> > >>>>> > >> 2020-05-13
>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
17:00:00
>>>>> 2020-
>>>>> > >> > >>>>> > >> 05-13
>>>>> > >> > >>>>> 18:00:00
>>>>> > >> > >>>>> > >> 2020-05-13
>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
21:00:00
>>>>> 2020-
>>>>> > >> > >>>>> > >> 05-13
>>>>> > >> > >>>>> 22:00:00
>>>>> > >> > >>>>> > >> 2020-05-13
>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
11:00:00
>>>>> 2020-
>>>>> > >> > >>>>> > >> 05-14
>>>>> > >> > >>>>> 12:00:00
>>>>> > >> > >>>>> > >> true
>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>>>> > >> > >>>>> > >> plot_20200612_173626.png
>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
>>>>> > >> > >>>>> > >> false list() list() false false false false
false
>>>>> false
>>>>> > >> > >>>>> > >> true
>>>>> > >> > >>>>> > >> false
>>>>> > >> > >>>>> > >> false
>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>>>> > >> > >>>>> > >> 1.4 -2
>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1
-2 0.5
>>>>> 3
>>>>> > >> > >>>>> > >> #cccccc 1
>>>>> > >> > >>>>> > >> listX 1
>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o
c(0,
>>>>> -.25)
>>>>> > >> > >>>>> > >> 3
>>>>> > >> > >>>>> > >> 0.8 1
>>>>> > >> > >>>>> > >> #333333
>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
>>>>> > >> > >>>>> > >> c(1,1)
>>>>> > >> > >>>>> c(1,1)
>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>> > > --
>>>>> > >> > >>>>> > > Edward Strobach
>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>>>> > >> > >>>>> > > IMSG Contractor
>>>>> > >> > >>>>> > > Cubicle#: 2029
>>>>> > >> > >>>>> > > 301-683-3717
>>>>> > >> > >>>>> > >
>>>>> > >> > >>>>>
>>>>> > >> > >>>>>
>>>>> > >> > >>>>>
>>>>> > >> > >>>>>
>>>>> > >> > >>>>
>>>>> > >> > >>>> --
>>>>> > >> > >>>> Edward Strobach
>>>>> > >> > >>>> EMC/NCEP/NWS/
>>>>> > >> > >>>> IMSG Contractor
>>>>> > >> > >>>> Cubicle#: 2029
>>>>> > >> > >>>> 301-683-3717
>>>>> > >> > >>>>
>>>>> > >> > >>>
>>>>> > >> > >>>
>>>>> > >> > >>> --
>>>>> > >> > >>> Edward Strobach
>>>>> > >> > >>> EMC/NCEP/NWS/
>>>>> > >> > >>> IMSG Contractor
>>>>> > >> > >>> Cubicle#: 2029
>>>>> > >> > >>> 301-683-3717
>>>>> > >> > >>>
>>>>> > >> > >>
>>>>> > >> > >>
>>>>> > >> > >> --
>>>>> > >> > >> Edward Strobach
>>>>> > >> > >> EMC/NCEP/NWS/
>>>>> > >> > >> IMSG Contractor
>>>>> > >> > >> Cubicle#: 2029
>>>>> > >> > >> 301-683-3717
>>>>> > >> > >>
>>>>> > >> > >
>>>>> > >> > >
>>>>> > >> > > --
>>>>> > >> > > Edward Strobach
>>>>> > >> > > EMC/NCEP/NWS/
>>>>> > >> > > IMSG Contractor
>>>>> > >> > > Cubicle#: 2029
>>>>> > >> > > 301-683-3717
>>>>> > >> > >
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >
>>>>> > > --
>>>>> > > Edward Strobach
>>>>> > > EMC/NCEP/NWS/
>>>>> > > IMSG Contractor
>>>>> > > Cubicle#: 2029
>>>>> > > 301-683-3717
>>>>> > >
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Edward Strobach
>>>> EMC/NCEP/NWS/
>>>> IMSG Contractor
>>>> Cubicle#: 2029
>>>> 301-683-3717
>>>>
>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>> --
>> Jeffery T. McQueen
>> Meteorologist
>> National Center for Weather and Climate Prediction
>> Room 2095
>> 5830 University Research Court
>> College Park, MD 20741
>> 301-683-3736
>> Fax: 301-683-3703
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 13:18:38 2020
Sure
I attached both the load database xml and the plotting xml
On Wed, Jun 17, 2020 at 2:42 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Ed,
> Could you please attache your XML to this ticket.
>
> Tatiana
>
> On Wed Jun 17 12:28:11 2020, edward.strobach at noaa.gov wrote:
> > Just adding more detail to elaborate my concerns...
> >
> > Not too long ago I was given a test set-up by Ho-Chun to generate
a
> > plot
> > for aerosol optical depth. The result was successful and produced
a
> > CSI
> > plot for AOD, comparing the production run and experimental run.
> > Although
> > I created a database, I do not seem to have a group where I can
access
> > specific databases. I recently created a load_xml file, which
should
> > have
> > loaded the data based on what I specified into the group. It
seemed
> > successful based on the log output that was generated.
Additionally,
> > all
> > information that I specified in this step was create in the
load_*xml.
> >
> > The next step is to generate plots since the data should be
loaded.
> > Here's
> > what happens when I run Ho-Chun's - the successful case
> >
> > HO-CHUN:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
> > hochun_xml.template+ bash
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> > edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> > plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
> > MVBatch
> > ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # #
# #
> > # # #
> > # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT
model,
> > ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
> >
> > This is early on in the script.
> >
> > ED_STROBACH:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed
-e
> > 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> > edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> > plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
> > MVBatch
> > ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20 PM
> > org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
> > create
> > initial connections of pool.java.sql.SQLException: Communications
link
> > failure with primary. No active connection found for master.
at
> >
>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
> > at
> >
>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
> > at
> >
>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
> > at
> >
>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
> > at
> > org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
> > at
> >
>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
> > at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
> >
> > My interpretation of this is that it fails to bridge a connection
to
> > the
> > group or database; therefore, it's unable to proceed with running
the
> > XML
> > file to generate the plots based on the specified content. If I
move
> > past
> > this step, then I'm golden.
> >
> > On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
> > jeff.mcqueen at noaa.gov> wrote:
> >
> > > Would it be ok to keep perry in the loop
> > >
> > > On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA
Affiliate <
> > > edward.strobach at noaa.gov> wrote:
> > >
> > >> you're right. I didn't have the storage problem this time.
I'm not
> > >> sure
> > >> what changed. All I know is that I have now created my load
xml
> > >> which now
> > >> should allow me to move on to the next step of plotting
> > >>
> > >> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA
Affiliate <
> > >> edward.strobach at noaa.gov> wrote:
> > >>
> > >>> I'm not following you. How do I now have enough space? What
> > >>> changed?
> > >>>
> > >>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
> > >>> <met_help at ucar.edu>
> > >>> wrote:
> > >>>
> > >>>> Ed, now you should have enough space for the data loading.
> > >>>>
> > >>>> Tatiana
> > >>>>
> > >>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
> > >>>> > I was basically able to build my xml file but it says I
have no
> > >>>> > available
> > >>>> > space. I'm not sure how that can be since I just started
adding
> > >>>> > data
> > >>>> > to my
> > >>>> > account
> > >>>> >
> > >>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space
> > >>>> > left
> > >>>> > on
> > >>>> > device
> > >>>> > ERROR: Command returned with non-zero status (1): scp -r
./*
> > >>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
> > >>>> > 4) Check METviewer AWS database list using
mv_db_size_on_aws.sh
> > >>>> > TABLE_SCHEMA USER
> > >>>> > SIZE_MB
> > >>>> > mv_met_g2o_gefs binbin.zhou
> > >>>> > 140027.31
> > >>>> > mv_met_system_hrefv2_v3 binbin.zhou
> > >>>> > 71986.11
> > >>>> >
> > >>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate
> > >>>> > <
> > >>>> > edward.strobach at noaa.gov> wrote:
> > >>>> >
> > >>>> > > Thanks Tatiana. Yes, I've created a database which seems
> > >>>> > > different
> > >>>> > > from
> > >>>> > > mv_group. How can one create their own group so that I
can
> > >>>> > > store and
> > >>>> > > access
> > >>>> > > the database. The database exists but cannot be viewed
on the
> > >>>> > > metviewer
> > >>>> > > GUI. The list starts with EMC HREFv3 and ends with
vhagerty.
> > >>>> > > I
> > >>>> > > named my
> > >>>> > > database Ed_Strobach, but that was really intended for
the
> > >>>> > > group
> > >>>> > > name.
> > >>>> > >
> > >>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> > >>>> > > <met_help at ucar.edu>
> > >>>> > > wrote:
> > >>>> > >
> > >>>> > >> We don't have database and METviewer running on wcoss.
> > >>>> > >> Instead, users use METviewer-wcoss scripts that help to
> > >>>> > >> connect to
> > >>>> > >> and
> > >>>> > >> use METviewer that is running on AWS. That is why XML
> > >>>> > >> connection
> > >>>> and
> > >>>> > >> folder
> > >>>> > >> sections look somewhat different than once from the Docs
> > >>>> > >> page.
> > >>>> > >> To create a database on AWS you need to run
> > >>>> > >> mv_create_db_on_aws.sh
> > >>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
> > >>>> > >> For data loading :
> > >>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> > >>>> > >> And for batch:
> > >>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> > >>>> > >>
> > >>>> > >> These scripts copy XML and data files if needed to AWS,
> > >>>> > >> adjust data
> > >>>> > >> path
> > >>>> > >> and users credentials, run METviewer and copy results
back to
> > >>>> wcoss.
> > >>>> > >>
> > >>>> > >> Tatiana
> > >>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
wrote:
> > >>>> > >> > lastly, it does seem thatmysql -u[db_username]
> > >>>> > >> > -p[db_password]
> > >>>> > >> > -e'create
> > >>>> > >> > database [db_name]
> > >>>> > >> >
> > >>>> > >> > would need to be used to create a database. I assume
that
> > >>>> > >> > an sql
> > >>>> > >> > module is
> > >>>> > >> > available on wcoss; I know you don't have access to
those
> > >>>> machines
> > >>>> > >> > Tatiana. If I create my own database with my own
name,
> > >>>> > >> > then I
> > >>>> > >> > should
> > >>>> > >> > be
> > >>>> > >> > able to proceed with project specific tasks under that
tab.
> > >>>> > >> > Something
> > >>>> > >> > like
> > >>>> > >> > mv_EdwardStrobach.
> > >>>> > >> >
> > >>>> > >> > I can then create mv_cmaq_prod_v_para which I add data
to
> > >>>> > >> > using
> > >>>> > >> > load_spec.
> > >>>> > >> > Once that step is done, then I can run plot_spec.
That's
> > >>>> > >> > my
> > >>>> > >> > takeaway
> > >>>> > >> > currently
> > >>>> > >> >
> > >>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
> > >>>> > >> > Affiliate
> > >>>> <
> > >>>> > >> > edward.strobach at noaa.gov> wrote:
> > >>>> > >> >
> > >>>> > >> > > Actually, it appears that I need to run the database
> > >>>> > >> > > loading
> > >>>> > >> > > module
> > >>>> > >> > > first
> > >>>> > >> > > if I want to change/add a data source. From there I
can
> > >>>> > >> > > run
> > >>>> the
> > >>>> > >> > > plotting
> > >>>> > >> > > batch module. Is that right? So if I run the
database
> > >>>> loading
> > >>>> > >> > > module,
> > >>>> > >> > > then I can specify my dates and other related info
> > >>>> > >> > > specific to
> > >>>> > >> > > the
> > >>>> > >> > > dataset. If I create my database with a unique
name,
> > >>>> > >> > > then it
> > >>>> > >> > > should
> > >>>> > >> > > be
> > >>>> > >> > > recognized on the metviewer gui, presumably.
> > >>>> > >> > >
> > >>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach -
NOAA
> > >>>> Affiliate
> > >>>> > >> > > <
> > >>>> > >> > > edward.strobach at noaa.gov> wrote:
> > >>>> > >> > >
> > >>>> > >> > >> It seems that those options must be treated
separately
> > >>>> > >> > >> outside
> > >>>> > >> > >> of
> > >>>> > >> > >> plot_spec. Is it acceptable to create both a
load_spec
> > >>>> > >> > >> and
> > >>>> > >> > >> plot_spec in
> > >>>> > >> > >> the same xml file? Would the definitions in
load_spec
> > >>>> > >> > >> be able
> > >>>> > >> > >> to
> > >>>> > >> > >> carry
> > >>>> > >> > >> over into plot_spec? It appears that data loading
and
> > >>>> plotting
> > >>>> > >> > >> are
> > >>>> > >> > >> treated
> > >>>> > >> > >> separately.
> > >>>> > >> > >>
> > >>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
NOAA
> > >>>> > >> > >> Affiliate <
> > >>>> > >> > >> edward.strobach at noaa.gov> wrote:
> > >>>> > >> > >>
> > >>>> > >> > >>> I do plan to use date_range; however, in an
example
> > >>>> > >> > >>> provided
> > >>>> > >> > >>> by the
> > >>>> > >> > >>> link
> > >>>> > >> > >>> I see this:
> > >>>> > >> > >>>
> > >>>> > >> > >>> <load_spec>
> > >>>> > >> > >>> <connection>
> > >>>> > >> > >>> <host>kemosabe:3306</host>
> > >>>> > >> > >>> <database>metvdb_hwt</database>
> > >>>> > >> > >>> <user>pgoldenb</user>
> > >>>> > >> > >>> <password>pgoldenb</password>
> > >>>> > >> > >>> </connection>
> > >>>> > >> > >>>
> > >>>> > >> > >>> <date_list name="folder_dates">
> > >>>> > >> > >>> <start>2010051914V</start>
> > >>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
> > >>>> > >> > >>> </hour></date_offset></end>
> > >>>> > >> > >>> <inc>3600</inc>
> > >>>> > >> > >>> <format>yyyyMMddHH'V'</format>
> > >>>> > >> > >>> </date_list>
> > >>>> > >> > >>>
> > >>>> > >> > >>> <met_version>V3.0</met_version>
> > >>>> > >> > >>>
> > >>>> > >> > >>> <verbose>false</verbose>
> > >>>> > >> > >>> <insert_size>1</insert_size>
> > >>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
> > >>>> > >> > >>> <drop_indexes>false</drop_indexes>
> > >>>> > >> > >>> <apply_indexes>true</apply_indexes>
> > >>>> > >> > >>> <group>Group name</group>
> > >>>> > >> > >>> <load_stat>true</load_stat>
> > >>>> > >> > >>> <load_mode>true</load_mode>
> > >>>> > >> > >>> <load_mtd>true</load_mtd>
> > >>>> > >> > >>> <load_mpr>false</load_mpr>
> > >>>> > >> > >>>
> > >>>> > >> > >>>
> > >>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > >>>> > >> > >>> <load_val>
> > >>>> > >> > >>> <field name="model">
> > >>>> > >> > >>> <val>arw</val>
> > >>>> > >> > >>> <val>nmm</val>
> > >>>> > >> > >>> </field>
> > >>>> > >> > >>>
> > >>>> > >> > >>> <field name="valid_time">
> > >>>> > >> > >>> <date_list name="folder_dates"/>
> > >>>> > >> > >>> </field>
> > >>>> > >> > >>>
> > >>>> > >> > >>> <field name="vx_mask">
> > >>>> > >> > >>> <val>FULL</val>
> > >>>> > >> > >>> <val>SWC</val>
> > >>>> > >> > >>> </field>
> > >>>> > >> > >>> </load_val>
> > >>>> > >> > >>> </load_spec>
> > >>>> > >> > >>>
> > >>>> > >> > >>> Are similar settings allowed with <plot_spec>? I
see
> > >>>> > >> > >>> <connection>,
> > >>>> > >> > >>> which seems carried over. I'm looking to specify
my
> > >>>> > >> > >>> folder_tmpl in
> > >>>> > >> > >>> <plot_spec>, which according to Tara, should be
> > >>>> > >> > >>> possible.
> > >>>> > >> > >>> Since my
> > >>>> > >> > >>> directory structure looks like this:
> > >>>> > >> > >>>
> > >>>> > >> > >>>
> > >>>> > >> > >>>
> > >>>> > >>
> > >>>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > >>>> > >> > >>>
> > >>>> > >> > >>> I would do something like this I would think:
> > >>>> > >> > >>>
> > >>>> > >> > >>> .......
> > >>>> > >> > >>>
> > >>>> > >> > >>> <date_list name="folder_dates">
> > >>>> > >> > >>> <start>20200612/start>
> > >>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset></end>
> > >>>> > >> > >>> <inc>86400</inc>
> > >>>> > >> > >>> <format>yyyyMMdd</format>
> > >>>> > >> > >>> </date_list>
> > >>>> > >> > >>>
> > >>>> > >> > >>> <folder_tmpl>
> > >>>> > >> > >>>
> > >>>> > >>
> > >>>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > >>>> > >> > >>> </folder_tmpl>
> > >>>> > >> > >>> <load_val>
> > >>>> > >> > >>> <field name="model">
> > >>>> > >> > >>> <val>prod</val>
> > >>>> > >> > >>> </field>
> > >>>> > >> > >>>
> > >>>> > >> > >>> <field name="valid_time">
> > >>>> > >> > >>> <date_list name="folder_dates"/>
> > >>>> > >> > >>> </field>
> > >>>> > >> > >>>
> > >>>> > >> > >>> ......
> > >>>> > >> > >>>
> > >>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach -
NOAA
> > >>>> > >> > >>> Affiliate <
> > >>>> > >> > >>> edward.strobach at noaa.gov> wrote:
> > >>>> > >> > >>>
> > >>>> > >> > >>>> Thank you. This is a very helpful example
> > >>>> > >> > >>>>
> > >>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via
RT
> > >>>> > >> > >>>> <met_help at ucar.edu>
> > >>>> > >> > >>>> wrote:
> > >>>> > >> > >>>>
> > >>>> > >> > >>>>> Hi,
> > >>>> > >> > >>>>> Here is some examples how to use <date_range>
and
> > >>>> > >> > >>>>> <date_range_list> in
> > >>>> > >> > >>>>> batch XML.
> > >>>> > >> > >>>>> First, you need to defined your range. You can
have
> > >>>> > >> > >>>>> more
> > >>>> > >> > >>>>> then one
> > >>>> > >> > >>>>> <date_range>:
> > >>>> > >> > >>>>> <date_range name="may">
> > >>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
> > >>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
> > >>>> > >> > >>>>> </date_range>
> > >>>> > >> > >>>>> <date_range name="march">
> > >>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
> > >>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
> > >>>> > >> > >>>>> </date_range>
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>> and then use one or all date_ranges:
> > >>>> > >> > >>>>> <plot_fix>
> > >>>> > >> > >>>>> <field name="fcst_init_beg">
> > >>>> > >> > >>>>> <date_range name="march"/>
> > >>>> > >> > >>>>> <date_range name="may"/>
> > >>>> > >> > >>>>> </field>...
> > >>>> > >> > >>>>> This configuration would include all dates for
March
> > >>>> > >> > >>>>> and
> > >>>> May
> > >>>> > >> > >>>>> of
> > >>>> > >> > >>>>> 2019.
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>> If you need more flexible configuration you can
use
> > >>>> > >> > >>>>> date_range_list.
> > >>>> > >> > >>>>> For example, you want to select first 6 hours
every
> > >>>> > >> > >>>>> day
> > >>>> > >> > >>>>> beetween
> > >>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00.
Your XML
> > >>>> > >> > >>>>> would
> > >>>> > >> > >>>>> look
> > >>>> > >> > >>>>> like this:
> > >>>> > >> > >>>>> <date_range_list name = 'all'>
> > >>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
> > >>>> > >> > >>>>> <range_end>2020-01-05
00:00:00</range_end>
> > >>>> > >> > >>>>> <range_length>360</range_length>
> > >>>> > >> > >>>>> <inc>1440</inc>
> > >>>> > >> > >>>>> </date_range_list>
> > >>>> > >> > >>>>> where the 1st and 2nd values are the start and
end,
> > >>>> > >> > >>>>> range_length - the length of a range in
minutes(6
> > >>>> > >> > >>>>> houres)
> > >>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
houres)
> > >>>> > >> > >>>>> This example would create following ranges:
> > >>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
> > >>>> > >> > >>>>> 06:00:00'
> > >>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
> > >>>> > >> > >>>>> 06:00:00'
> > >>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
> > >>>> > >> > >>>>> 06:00:00'
> > >>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
> > >>>> > >> > >>>>> 06:00:00'
> > >>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
> > >>>> > >> > >>>>> 06:00:00'
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>> Also you can add more parameters to <plot_fix>
to
> > >>>> > >> > >>>>> filter
> > >>>> > >> > >>>>> your
> > >>>> > >> > >>>>> data.
> > >>>> > >> > >>>>> For example, if you want to select only dates
with
> > >>>> > >> > >>>>> 00:00
> > >>>> you
> > >>>> > >> > >>>>> can
> > >>>> > >> > >>>>> specify
> > >>>> > >> > >>>>> <field name="valid_hour">
> > >>>> > >> > >>>>> <val>00</val>
> > >>>> > >> > >>>>> </field>
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>> Tatiana
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
> > >>>> wrote:
> > >>>> > >> > >>>>> > and if there isn't a better way, I wonder the
> > >>>> > >> > >>>>> > whereabouts
> > >>>> > >> > >>>>> > of a
> > >>>> > >> > >>>>> > separate
> > >>>> > >> > >>>>> > script that pipes dates (and other relevant
> > >>>> > >> > >>>>> > information)
> > >>>> > >> > >>>>> > into
> > >>>> > >> > >>>>> > the XML
> > >>>> > >> > >>>>> > file
> > >>>> > >> > >>>>> > used to generate the results.
> > >>>> > >> > >>>>> >
> > >>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward
Strobach -
> > >>>> > >> > >>>>> > NOAA
> > >>>> > >> > >>>>> > Affiliate <
> > >>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
> > >>>> > >> > >>>>> >
> > >>>> > >> > >>>>> > > Thanks Tatiana. These resources are
helpful.
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>> > > One last question. Simply typing lines like
this
> > >>>> > >> > >>>>> > > in
> > >>>> can
> > >>>> > >> > >>>>> > > take
> > >>>> > >> > >>>>> > > forever:
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>> > > <plot_fix>
> > >>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
> > >>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
> > >>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
> > >>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
> > >>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
> > >>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
> > >>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>> > > Is there a better way? I see date_range,
> > >>>> > >> > >>>>> > > date_range_list,
> > >>>> > >> > >>>>> > > or
> > >>>> > >> > >>>>> > > date_list.
> > >>>> > >> > >>>>> > > Can this be used instead of typing
individual
> > >>>> > >> > >>>>> > > values?
> > >>>> > >> > >>>>> > > This
> > >>>> > >> > >>>>> > > could
> > >>>> > >> > >>>>> > > prove
> > >>>> > >> > >>>>> > > pretty daunting if one would have to type in
700
> > >>>> entries
> > >>>> > >> > >>>>> > > each
> > >>>> > >> > >>>>> > > time.
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana
Burek via
> > >>>> > >> > >>>>> > > RT
> > >>>> > >> > >>>>> > > <met_help at ucar.edu>
> > >>>> > >> > >>>>> > > wrote:
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>> > >> I found a few problems in your XML.
> > >>>> > >> > >>>>> > >> The database you are using doesn't have
data
> > >>>> > >> > >>>>> > >> with
> > >>>> > >> > >>>>> > >> vx_mask
> > >>>> > >> > >>>>> > >> 'abcREGabc'.
> > >>>> > >> > >>>>> > >> That is why METviewer returns an empty
dataset
> > >>>> > >> > >>>>> > >> and
> > >>>> > >> > >>>>> > >> can't
> > >>>> > >> > >>>>> > >> create a
> > >>>> > >> > >>>>> > >> plot.
> > >>>> > >> > >>>>> > >> The other problem is how you define
fcst_lead.
> > >>>> > >> > >>>>> > >> This
> > >>>> > >> > >>>>> > >> database
> > >>>> > >> > >>>>> > >> has
> > >>>> > >> > >>>>> > >> values
> > >>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
Since
> > >>>> > >> > >>>>> > >> you
> > >>>> > >> > >>>>> > >> include
> > >>>> > >> > >>>>> > >> all
> > >>>> > >> > >>>>> > >> fcst_leads
> > >>>> > >> > >>>>> > >> this variable can be omitted.
> > >>>> > >> > >>>>> > >> To minimize problems with variables values,
I
> > >>>> > >> > >>>>> > >> would
> > >>>> > >> > >>>>> > >> suggest
> > >>>> > >> > >>>>> > >> to
> > >>>> > >> > >>>>> > >> create XML
> > >>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
> > >>>> > >> > >>>>> > >> standards.
> > >>>> > >> > >>>>> > >> I created a plot using you configurations
with
> > >>>> > >> > >>>>> > >> GUI
> > >>>> > >> > >>>>> > >> (without
> > >>>> > >> > >>>>> vx_mack
> > >>>> > >> > >>>>> > >> and
> > >>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You
can
> > >>>> > >> > >>>>> > >> load
> > >>>> it
> > >>>> > >> > >>>>> > >> back
> > >>>> > >> > >>>>> > >> to
> > >>>> > >> > >>>>> the
> > >>>> > >> > >>>>> > >> page by
> > >>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI).
Find
> > >>>> > >> > >>>>> > >> the
> > >>>> > >> > >>>>> > >> plot,
> > >>>> > >> > >>>>> > >> open XML
> > >>>> > >> > >>>>> > >> and
> > >>>> > >> > >>>>> > >> click on "Load to page". After that you can
> > >>>> > >> > >>>>> > >> recreate a
> > >>>> > >> > >>>>> > >> plot
> > >>>> > >> > >>>>> > >> and
> > >>>> > >> > >>>>> edit
> > >>>> > >> > >>>>> > >> the
> > >>>> > >> > >>>>> > >> configurations. You can also download a new
XML
> > >>>> > >> > >>>>> > >> and
> > >>>> > >> > >>>>> > >> edit it
> > >>>> > >> > >>>>> > >> for
> > >>>> > >> > >>>>> > >> batch.
> > >>>> > >> > >>>>> > >> I attached the example of load XML to this
> > >>>> > >> > >>>>> > >> message.
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>> > >> Description of rds parameters:
> > >>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
> > >>>> > >> > >>>>> > >> METviewer R
> > >>>> > >> > >>>>> > >> scripts
> > >>>> > >> > >>>>> > >> rds_plots - temporary directory where the
plots
> > >>>> > >> > >>>>> > >> will
> > >>>> be
> > >>>> > >> > >>>>> > >> generated
> > >>>> > >> > >>>>> > >> rds_data - temporary directory where the
data
> > >>>> retrieved
> > >>>> > >> > >>>>> > >> from
> > >>>> > >> > >>>>> > >> the
> > >>>> > >> > >>>>> > >> database
> > >>>> > >> > >>>>> > >> and other intermediate files will be stored
> > >>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
> > >>>> > >> > >>>>> > >> scripts for
> > >>>> > >> > >>>>> > >> this
> > >>>> > >> > >>>>> > >> run
> > >>>> > >> > >>>>> will
> > >>>> > >> > >>>>> > >> be
> > >>>> > >> > >>>>> > >> stored
> > >>>> > >> > >>>>> > >> After the execution is done all the files
from
> > >>>> > >> > >>>>> > >> temporary
> > >>>> > >> > >>>>> directories
> > >>>> > >> > >>>>> > >> would be removed.
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>> > >> We don't have the offitial user guide for
> > >>>> > >> > >>>>> > >> METviewer.
> > >>>> > >> > >>>>> > >> There
> > >>>> > >> > >>>>> > >> are
> > >>>> > >> > >>>>> some
> > >>>> > >> > >>>>> > >> instructions on how to crate XML config
files:
> > >>>> > >> > >>>>> > >>
https://metviewer.nws.noaa.gov/doc/index.html
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>> > >> Tatiana
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
> > >>>> > >> > >>>>> > >> edward.strobach at noaa.gov
> > >>>> > >> > >>>>> > >> wrote:
> > >>>> > >> > >>>>> > >> > Hi,
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> > I'm attaching the xml below, but am
having
> > >>>> > >> > >>>>> > >> > hard time
> > >>>> > >> > >>>>> > >> > following
> > >>>> > >> > >>>>> how
> > >>>> > >> > >>>>> > >> > to
> > >>>> > >> > >>>>> > >> > go
> > >>>> > >> > >>>>> > >> > about the other steps since I have zero
> > >>>> > >> > >>>>> > >> > experience
> > >>>> on
> > >>>> > >> > >>>>> > >> > how
> > >>>> > >> > >>>>> > >> > to do
> > >>>> > >> > >>>>> > >> > these
> > >>>> > >> > >>>>> > >> > things.
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should
> > >>>> > >> > >>>>> > >> > NOT
> > >>>> be
> > >>>> > >> > >>>>> > >> > changed
> > >>>> > >> > >>>>> and
> > >>>> > >> > >>>>> > >> > will
> > >>>> > >> > >>>>> > >> > be
> > >>>> > >> > >>>>> > >> > replaced by other scripts during the
batch
> > >>>> execution.
> > >>>> > >> > >>>>> > >> > The
> > >>>> > >> > >>>>> > >> > data
> > >>>> > >> > >>>>> for
> > >>>> > >> > >>>>> > >> > the
> > >>>> > >> > >>>>> > >> > batch mode is coming from the database
and not
> > >>>> > >> > >>>>> > >> > from
> > >>>> > >> > >>>>> > >> > files.
> > >>>> > >> > >>>>> > >> > The
> > >>>> > >> > >>>>> > >> > name of
> > >>>> > >> > >>>>> > >> > the
> > >>>> > >> > >>>>> > >> > database is in <database> tag.
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> > I understand that nothing should be
changed in
> > >>>> > >> > >>>>> > >> > the
> > >>>> > >> > >>>>> > >> > folders
> > >>>> > >> > >>>>> > >> > section,
> > >>>> > >> > >>>>> > >> > but
> > >>>> > >> > >>>>> > >> > there is no indication how and where
these
> > >>>> > >> > >>>>> > >> > variables
> > >>>> > >> > >>>>> > >> > are
> > >>>> > >> > >>>>> > >> > being
> > >>>> > >> > >>>>> > >> > defined.
> > >>>> > >> > >>>>> > >> > Grepping these names of the files
available
> > >>>> > >> > >>>>> > >> > does not
> > >>>> > >> > >>>>> > >> > reveal
> > >>>> > >> > >>>>> formal
> > >>>> > >> > >>>>> > >> > definitions of these variables. If I
> > >>>> > >> > >>>>> > >> > understood
> > >>>> > >> > >>>>> > >> > exactly
> > >>>> > >> > >>>>> > >> > how
> > >>>> > >> > >>>>> these
> > >>>> > >> > >>>>> > >> > were
> > >>>> > >> > >>>>> > >> > being defined, then I would be able
connect
> > >>>> > >> > >>>>> > >> > this
> > >>>> > >> > >>>>> > >> > better.
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
> > >>>> > >> > >>>>> > >> > files. They
> > >>>> > >> > >>>>> > >> > use
> > >>>> > >> > >>>>> METviewer
> > >>>> > >> > >>>>> > >> > load
> > >>>> > >> > >>>>> > >> > mode to load these files to the specific
> > >>>> > >> > >>>>> > >> > database.
> > >>>> In
> > >>>> > >> > >>>>> > >> > the
> > >>>> > >> > >>>>> > >> > config
> > >>>> > >> > >>>>> > >> > file
> > >>>> > >> > >>>>> > >> > for
> > >>>> > >> > >>>>> > >> > this process they also can specify the
> > >>>> > >> > >>>>> > >> > database
> > >>>> > >> > >>>>> > >> > group. For
> > >>>> > >> > >>>>> example
> > >>>> > >> > >>>>> > >> > database
> > >>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
> > >>>> > >> > >>>>> > >> > After the data has been loaded to the
database
> > >>>> > >> > >>>>> > >> > it is
> > >>>> > >> > >>>>> > >> > visible and
> > >>>> > >> > >>>>> > >> > available
> > >>>> > >> > >>>>> > >> > in the GUI and also can be used by the
batch
> > >>>> > >> > >>>>> > >> > mode.
> > >>>> > >> > >>>>> > >> > You can always add more data to the
database
> > >>>> > >> > >>>>> > >> > using
> > >>>> > >> > >>>>> > >> > the
> > >>>> > >> > >>>>> > >> > same load
> > >>>> > >> > >>>>> > >> > mode.
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> > Is there a guide for me to understand the
> > >>>> > >> > >>>>> > >> > process
> > >>>> > >> > >>>>> > >> > that you
> > >>>> > >> > >>>>> > >> > are
> > >>>> > >> > >>>>> > >> > referring to?
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek
> > >>>> > >> > >>>>> > >> > via
> > >>>> RT
> > >>>> > >> > >>>>> > >> > <met_help at ucar.edu>
> > >>>> > >> > >>>>> > >> > wrote:
> > >>>> > >> > >>>>> > >> >
> > >>>> > >> > >>>>> > >> > > Hi Edward,
> > >>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
servers
> > >>>> > >> > >>>>> > >> > > and
> > >>>> > >> > >>>>> > >> > > can't
> > >>>> > >> > >>>>> > >> > > view your
> > >>>> > >> > >>>>> > >> > > XML
> > >>>> > >> > >>>>> > >> > > for the
> > >>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue
> > >>>> > >> > >>>>> > >> > > and I
> > >>>> > >> > >>>>> > >> > > will
> > >>>> > >> > >>>>> > >> > > take a
> > >>>> > >> > >>>>> > >> > > look.
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
> > >>>> > >> > >>>>> > >> > > should NOT
> > >>>> > >> > >>>>> > >> > > be
> > >>>> > >> > >>>>> > >> > > changed
> > >>>> > >> > >>>>> > >> > > and
> > >>>> > >> > >>>>> > >> > > will be
> > >>>> > >> > >>>>> > >> > > replaced by other scripts during the
batch
> > >>>> > >> > >>>>> > >> > > execution.
> > >>>> > >> > >>>>> > >> > > The data
> > >>>> > >> > >>>>> > >> > > for
> > >>>> > >> > >>>>> > >> > > the
> > >>>> > >> > >>>>> > >> > > batch mode is coming from the database
and
> > >>>> > >> > >>>>> > >> > > not
> > >>>> from
> > >>>> > >> > >>>>> > >> > > files. The
> > >>>> > >> > >>>>> > >> > > name
> > >>>> > >> > >>>>> > >> > > of the
> > >>>> > >> > >>>>> > >> > > database is in <database> tag.
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET
output
> > >>>> > >> > >>>>> > >> > > files.
> > >>>> > >> > >>>>> > >> > > They use
> > >>>> > >> > >>>>> > >> > > METviewer
> > >>>> > >> > >>>>> > >> > > load
> > >>>> > >> > >>>>> > >> > > mode to load these files to the
specific
> > >>>> > >> > >>>>> > >> > > database.
> > >>>> > >> > >>>>> > >> > > In
> > >>>> > >> > >>>>> > >> > > the
> > >>>> > >> > >>>>> config
> > >>>> > >> > >>>>> > >> > > file
> > >>>> > >> > >>>>> > >> > > for
> > >>>> > >> > >>>>> > >> > > this process they also can specify the
> > >>>> > >> > >>>>> > >> > > database
> > >>>> > >> > >>>>> > >> > > group.
> > >>>> > >> > >>>>> > >> > > For
> > >>>> > >> > >>>>> > >> > > example
> > >>>> > >> > >>>>> > >> > > database
> > >>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
> > >>>> > >> > >>>>> > >> > > After the data has been loaded to the
> > >>>> > >> > >>>>> > >> > > database it
> > >>>> > >> > >>>>> > >> > > is
> > >>>> > >> > >>>>> > >> > > visible
> > >>>> > >> > >>>>> and
> > >>>> > >> > >>>>> > >> > > available
> > >>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch
> > >>>> > >> > >>>>> > >> > > mode.
> > >>>> > >> > >>>>> > >> > > You can always add more data to the
database
> > >>>> > >> > >>>>> > >> > > using
> > >>>> > >> > >>>>> > >> > > the
> > >>>> > >> > >>>>> > >> > > same
> > >>>> > >> > >>>>> load
> > >>>> > >> > >>>>> > >> > > mode.
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >> > > Tatiana
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> > >>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
> > >>>> > >> > >>>>> > >> > > wrote:
> > >>>> > >> > >>>>> > >> > > > Good morning,
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
> > >>>> > >> > >>>>> > >> > > > effort to
> > >>>> > >> > >>>>> > >> > > > understand
> > >>>> > >> > >>>>> > >> > > > metviewer,
> > >>>> > >> > >>>>> > >> > > > test
> > >>>> > >> > >>>>> > >> > > > different cases using the history
tab,
> > >>>> > >> > >>>>> > >> > > > changing
> > >>>> > >> > >>>>> > >> > > > information
> > >>>> > >> > >>>>> to
> > >>>> > >> > >>>>> > >> > > > see
> > >>>> > >> > >>>>> > >> > > > what
> > >>>> > >> > >>>>> > >> > > > works and what does not, and make
> > >>>> > >> > >>>>> > >> > > > connections
> > >>>> > >> > >>>>> > >> > > > between
> > >>>> > >> > >>>>> > >> > > > the
> > >>>> > >> > >>>>> GUI
> > >>>> > >> > >>>>> > >> > > > settings
> > >>>> > >> > >>>>> > >> > > > and
> > >>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
level
> > >>>> > >> > >>>>> > >> > > > with
> > >>>> > >> > >>>>> > >> > > > metviewer has
> > >>>> > >> > >>>>> > >> > > > significantly
> > >>>> > >> > >>>>> > >> > > > increased during this time, there are
some
> > >>>> > >> > >>>>> > >> > > > remaining
> > >>>> > >> > >>>>> > >> > > > issues
> > >>>> > >> > >>>>> > >> > > > that
> > >>>> > >> > >>>>> > >> > > > I'm
> > >>>> > >> > >>>>> > >> > > > hoping
> > >>>> > >> > >>>>> > >> > > > to clear up.
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
running
> > >>>> > >> > >>>>> > >> > > > with
> > >>>> > >> > >>>>> > >> > > > different
> > >>>> > >> > >>>>> > >> > > > statistic
> > >>>> > >> > >>>>> > >> > > > and
> > >>>> > >> > >>>>> > >> > > > plot
> > >>>> > >> > >>>>> > >> > > > type:
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
> > >>>> > >> > >>>>> > >> > > > provided me
> > >>>> > >> > >>>>> > >> > > > the
> > >>>> > >> > >>>>> > >> > > > working
> > >>>> > >> > >>>>> > >> > > > directory
> > >>>> > >> > >>>>> > >> > > > to get
> > >>>> > >> > >>>>> > >> > > > started which I copied over:
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > He has a test.sh script that I
adjusted
> > >>>> > >> > >>>>> > >> > > > and used
> > >>>> > >> > >>>>> > >> > > > to
> > >>>> > >> > >>>>> > >> > > > run his
> > >>>> > >> > >>>>> > >> > > > example.
> > >>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
> > >>>> > >> > >>>>> > >> > > > produced a
> > >>>> > >> > >>>>> > >> > > > CSI
> > >>>> > >> > >>>>> > >> > > > plot
> > >>>> > >> > >>>>> that
> > >>>> > >> > >>>>> > >> > > > was
> > >>>> > >> > >>>>> > >> > > > stored
> > >>>> > >> > >>>>> > >> > > > here:
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>>
> > >>>> > >>
> > >>>>
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > Running the script produced the
expected
> > >>>> > >> > >>>>> > >> > > > result.
> > >>>> > >> > >>>>> > >> > > > The
> > >>>> > >> > >>>>> > >> > > > next
> > >>>> > >> > >>>>> > >> > > > natural
> > >>>> > >> > >>>>> > >> > > > step was
> > >>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and
create a
> > >>>> > >> > >>>>> > >> > > > new xml
> > >>>> > >> > >>>>> > >> > > > file
> > >>>> > >> > >>>>> > >> > > > called
> > >>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
create a
> > >>>> > >> > >>>>> > >> > > > time
> > >>>> > >> > >>>>> > >> > > > series
> > >>>> > >> > >>>>> > >> > > > where
> > >>>> > >> > >>>>> > >> > > > the
> > >>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
instead
> > >>>> > >> > >>>>> > >> > > > of
> > >>>> > >> > >>>>> > >> > > > obs_thresh.
> > >>>> > >> > >>>>> > >> > > > Accordingly,
> > >>>> > >> > >>>>> > >> > > > I
> > >>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
> > >>>> > >> > >>>>> > >> > > > fcst_valid_beg)
> > >>>> > >> > >>>>> and
> > >>>> > >> > >>>>> > >> > > > changed
> > >>>> > >> > >>>>> > >> > > > the
> > >>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was
also
> > >>>> > >> > >>>>> > >> > > > tried).
> > >>>> > >> > >>>>> > >> > > > The
> > >>>> > >> > >>>>> error
> > >>>> > >> > >>>>> > >> > > > that
> > >>>> > >> > >>>>> > >> > > > resulted
> > >>>> > >> > >>>>> > >> > > > is as follows:
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
('2020-
> > >>>> > >> > >>>>> > >> > > > 05-01
> > >>>> > >> > >>>>> > >> > > > 13:00:00',
> > >>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>> > >> > >>>>> > >> > > > 01
> > >>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
'2020-
> > >>>> > >> > >>>>> > >> > > > 05-01
> > >>>> > >> > >>>>> > >> > > > 16:00:00',
> > >>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>> > >> > >>>>> > >> > > > 01
> > >>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
'2020-
> > >>>> > >> > >>>>> > >> > > > 05-01
> > >>>> > >> > >>>>> > >> > > > 19:00:00',
> > >>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>> > >> > >>>>> > >> > > > 01
> > >>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
'2020-
> > >>>> > >> > >>>>> > >> > > > 05-01
> > >>>> > >> > >>>>> > >> > > > 22:00:00',
> > >>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>> > >> > >>>>> > >> > > > 01
> > >>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
'2020-
> > >>>> > >> > >>>>> > >> > > > 05-02
> > >>>> > >> > >>>>> > >> > > > 01:00:00',
> > >>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>> > >> > >>>>> > >> > > > 02
> > >>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
'2020-
> > >>>> > >> > >>>>> > >> > > > 05-02
> > >>>> > >> > >>>>> > >> > > > 04:00:00',
> > >>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>> > >> > >>>>> > >> > > > 02
> > >>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
'2020-
> > >>>> > >> > >>>>> > >> > > > 05-02
> > >>>> > >> > >>>>> > >> > > > 07:00:00',
> > >>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>> > >> > >>>>> > >> > > > 02
> > >>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
'2020-
> > >>>> > >> > >>>>> > >> > > > 05-02
> > >>>> > >> > >>>>> > >> > > > 10:00:00',
> > >>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>> > >> > >>>>> > >> > > > 02
> > >>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > >>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> > >>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
h.stat_header_id
> > >>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
> > >>>> > >> > >>>>> > >> > > > No data was returned from database
> > >>>> > >> > >>>>> > >> > > > 2020-06-09
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>>
> > >>>> > >>
> > >>>>
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > >>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
> > >>>> > >> > >>>>> > >> > > > ERROR:
> > >>>> > >> > >>>>> > >> > > > Caught
> > >>>> > >> > >>>>> > >> > > > class
> > >>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> > >>>> > >> > >>>>> > >> > > > in
> > >>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
> > >>>> > >> > >>>>> > >> > > > res): result set contained no data
> > >>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> > >>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> > >>>> > >> > >>>>> > >> > > > in
> > >>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
> > >>>> > >> > >>>>> > >> > > > set
> > >>>> > >> > >>>>> > >> > > > contained no
> > >>>> > >> > >>>>> > >> > > > data
> > >>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> > >>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> > >>>> > >> > >>>>> > >> > > > in
> > >>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
> > >>>> > >> > >>>>> > >> > > > set
> > >>>> > >> > >>>>> > >> > > > contained no
> > >>>> > >> > >>>>> > >> > > > data
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > Apparently there was no data during
this
> > >>>> > >> > >>>>> > >> > > > time.
> > >>>> I
> > >>>> > >> > >>>>> > >> > > > tried
> > >>>> > >> > >>>>> > >> > > > changing
> > >>>> > >> > >>>>> > >> > > > the
> > >>>> > >> > >>>>> > >> > > > range
> > >>>> > >> > >>>>> > >> > > > of valid times and changed the valid
time
> > >>>> > >> > >>>>> > >> > > > to
> > >>>> > >> > >>>>> > >> > > > different
> > >>>> > >> > >>>>> > >> > > > days
> > >>>> > >> > >>>>> > >> > > > just in
> > >>>> > >> > >>>>> > >> > > > case
> > >>>> > >> > >>>>> > >> > > > data was missing. The issue was the
same.
> > >>>> > >> > >>>>> > >> > > > To
> > >>>> > >> > >>>>> > >> > > > confirm
> > >>>> > >> > >>>>> > >> > > > my
> > >>>> > >> > >>>>> > >> > > > logic, I
> > >>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
> > >>>> > >> > >>>>> > >> > > > metviewer,
> > >>>> > >> > >>>>> > >> > > > which I
> > >>>> > >> > >>>>> > >> > > > found
> > >>>> > >> > >>>>> > >> > > > to
> > >>>> > >> > >>>>> > >> > > > work.
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > 2. Running in batch mode
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > I think I lack an understanding of
how to
> > >>>> > >> > >>>>> > >> > > > set up
> > >>>> > >> > >>>>> > >> > > > batch
> > >>>> > >> > >>>>> > >> > > > mode
> > >>>> > >> > >>>>> > >> > > > because
> > >>>> > >> > >>>>> > >> > > > I
> > >>>> > >> > >>>>> > >> > > > don't
> > >>>> > >> > >>>>> > >> > > > see where these definitions are being
> > >>>> > >> > >>>>> > >> > > > explicitly
> > >>>> > >> > >>>>> > >> > > > defined:
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > <plot_spec>
> > >>>> > >> > >>>>> > >> > > > <connection>
> > >>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>>
> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > >>>> > >> > >>>>> > >> > > > <user>rds_user</user>
> > >>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
> > >>>> > >> > >>>>> > >> > > > </connection>
> > >>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
> > >>>> > >> > >>>>> > >> > > > <folders>
> > >>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > >>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > >>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
> > >>>> > >> > >>>>> > >> > > > <data>rds_data</data>
> > >>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > >>>> > >> > >>>>> > >> > > > </folders>
> > >>>> > >> > >>>>> > >> > > > <plot>
> > >>>> > >> > >>>>> > >> > > >
<template>series_plot.R_tmpl</template>
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > I see the <folders> section, but
grepping
> > >>>> > >> > >>>>> > >> > > > all
> > >>>> > >> > >>>>> > >> > > > available
> > >>>> > >> > >>>>> files
> > >>>> > >> > >>>>> > >> > > > does
> > >>>> > >> > >>>>> > >> > > > not
> > >>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being
defined, as
> > >>>> > >> > >>>>> > >> > > > well
> > >>>> > >> > >>>>> > >> > > > as
> > >>>> > >> > >>>>> > >> > > > other
> > >>>> > >> > >>>>> > >> > > > similarly
> > >>>> > >> > >>>>> > >> > > > defined
> > >>>> > >> > >>>>> > >> > > > variables. I think if I know how
these
> > >>>> > >> > >>>>> > >> > > > are
> > >>>> being
> > >>>> > >> > >>>>> > >> > > > defined,
> > >>>> > >> > >>>>> > >> > > > then
> > >>>> > >> > >>>>> > >> > > > I'll
> > >>>> > >> > >>>>> > >> > > > be
> > >>>> > >> > >>>>> > >> > > > able to pipe in my information and
process
> > >>>> > >> > >>>>> > >> > > > the
> > >>>> > >> > >>>>> > >> > > > data
> > >>>> > >> > >>>>> > >> > > > using
> > >>>> > >> > >>>>> met
> > >>>> > >> > >>>>> > >> > > > and
> > >>>> > >> > >>>>> > >> > > > produce
> > >>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
> > >>>> > >> > >>>>> > >> > > > that?
> > >>>> > >> > >>>>> > >> > > > Also, if
> > >>>> > >> > >>>>> > >> > > > I want
> > >>>> > >> > >>>>> > >> > > > to use
> > >>>> > >> > >>>>> > >> > > > obs for
> > >>>> > >> > >>>>> > >> > > > verification, do I add another line
in the
> > >>>> > >> > >>>>> > >> > > > XML
> > >>>> > >> > >>>>> > >> > > > file
> > >>>> > >> > >>>>> > >> > > > that
> > >>>> > >> > >>>>> > >> > > > designates
> > >>>> > >> > >>>>> > >> > > > file
> > >>>> > >> > >>>>> > >> > > > location of obs? I assume that other
> > >>>> > >> > >>>>> > >> > > > steps have
> > >>>> > >> > >>>>> > >> > > > to be
> > >>>> > >> > >>>>> > >> > > > run
> > >>>> > >> > >>>>> > >> > > > before
> > >>>> > >> > >>>>> > >> > > > all
> > >>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form,
for
> > >>>> > >> > >>>>> > >> > > > example,
> > >>>> > >> > >>>>> > >> > > > I
> > >>>> > >> > >>>>> > >> > > > would
> > >>>> > >> > >>>>> have
> > >>>> > >> > >>>>> > >> > > > to
> > >>>> > >> > >>>>> > >> > > > separately
> > >>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then
I
> > >>>> > >> > >>>>> > >> > > > would
> > >>>> want
> > >>>> > >> > >>>>> > >> > > > to
> > >>>> > >> > >>>>> > >> > > > do that
> > >>>> > >> > >>>>> > >> > > > as
> > >>>> > >> > >>>>> > >> > > > well.
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
> > >>>> > >> > >>>>> > >> > > > online
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > Lastly, I see a lot of
people/projects
> > >>>> > >> > >>>>> > >> > > > listed
> > >>>> > >> > >>>>> > >> > > > here -
> > >>>> > >> > >>>>> > >> > > > see
> > >>>> > >> > >>>>> > >> > > > attachment
> > >>>> > >> > >>>>> > >> > > >
> > >>>> > >> > >>>>> > >> > > > How do I get the results I generate
to be
> > >>>> > >> > >>>>> > >> > > > uploaded to
> > >>>> > >> > >>>>> > >> > > > this
> > >>>> > >> > >>>>> > >> > > > location?
> > >>>> > >> > >>>>> > >> > > > It's
> > >>>> > >> > >>>>> > >> > > > not clear how others are setting this
up.
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >> > >
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>> > >>
> > >>>> > >> > >>>>> > >> rds_host:3306
> > >>>> > >> > >>>>> > >> mv_cmaqv5para8_aod_grid2grid_metplus
> > >>>> > >> > >>>>> > >> rds_user
> > >>>> > >> > >>>>> rds_pwd
> > >>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
rds_plots
> > >>>> rds_data
> > >>>> > >> > >>>>> rds_scripts
> > >>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
> > >>>> > >> > >>>>> > >> CMAQAODPRODHIGH
> > >>>> > >> > >>>>> 2020-05-12
> > >>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
19:00:00
> > >>>> > >> > >>>>> > >> 2020-
> > >>>> > >> > >>>>> > >> 05-12
> > >>>> > >> > >>>>> 20:00:00
> > >>>> > >> > >>>>> > >> 2020-05-12
> > >>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
23:00:00
> > >>>> > >> > >>>>> > >> 2020-
> > >>>> > >> > >>>>> > >> 05-13
> > >>>> > >> > >>>>> 10:00:00
> > >>>> > >> > >>>>> > >> 2020-05-13
> > >>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
13:00:00
> > >>>> > >> > >>>>> > >> 2020-
> > >>>> > >> > >>>>> > >> 05-13
> > >>>> > >> > >>>>> 14:00:00
> > >>>> > >> > >>>>> > >> 2020-05-13
> > >>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
17:00:00
> > >>>> > >> > >>>>> > >> 2020-
> > >>>> > >> > >>>>> > >> 05-13
> > >>>> > >> > >>>>> 18:00:00
> > >>>> > >> > >>>>> > >> 2020-05-13
> > >>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
21:00:00
> > >>>> > >> > >>>>> > >> 2020-
> > >>>> > >> > >>>>> > >> 05-13
> > >>>> > >> > >>>>> 22:00:00
> > >>>> > >> > >>>>> > >> 2020-05-13
> > >>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
11:00:00
> > >>>> > >> > >>>>> > >> 2020-
> > >>>> > >> > >>>>> > >> 05-14
> > >>>> > >> > >>>>> 12:00:00
> > >>>> > >> > >>>>> > >> true
> > >>>> > >> > >>>>> > >> median plot_20200612_173626.data
> > >>>> > >> > >>>>> > >> plot_20200612_173626.png
> > >>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
> > >>>> > >> > >>>>> > >> false list() list() false false false false
> > >>>> > >> > >>>>> > >> false
> > >>>> false
> > >>>> > >> > >>>>> > >> true
> > >>>> > >> > >>>>> > >> false
> > >>>> > >> > >>>>> > >> false
> > >>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
> > >>>> > >> > >>>>> > >> c(8,4,5,4)
> > >>>> > >> > >>>>> > >> c(1,1,0) 1 2
> > >>>> > >> > >>>>> > >> 1.4 -2
> > >>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1
1 -2
> > >>>> > >> > >>>>> > >> 0.5 3
> > >>>> > >> > >>>>> > >> #cccccc 1
> > >>>> > >> > >>>>> > >> listX 1
> > >>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5
o
> > >>>> > >> > >>>>> > >> c(0,
> > >>>> -.25)
> > >>>> > >> > >>>>> > >> 3
> > >>>> > >> > >>>>> > >> 0.8 1
> > >>>> > >> > >>>>> > >> #333333
> > >>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none")
c(FALSE,FALSE)
> > >>>> > >> > >>>>> > >> c(TRUE,TRUE)
> > >>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
c("b","b")
> > >>>> > >> > >>>>> > >> c(1,1)
> > >>>> > >> > >>>>> > >> c(1,1)
> > >>>> > >> > >>>>> c(1,1)
> > >>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>> > > --
> > >>>> > >> > >>>>> > > Edward Strobach
> > >>>> > >> > >>>>> > > EMC/NCEP/NWS/
> > >>>> > >> > >>>>> > > IMSG Contractor
> > >>>> > >> > >>>>> > > Cubicle#: 2029
> > >>>> > >> > >>>>> > > 301-683-3717
> > >>>> > >> > >>>>> > >
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>>
> > >>>> > >> > >>>>
> > >>>> > >> > >>>> --
> > >>>> > >> > >>>> Edward Strobach
> > >>>> > >> > >>>> EMC/NCEP/NWS/
> > >>>> > >> > >>>> IMSG Contractor
> > >>>> > >> > >>>> Cubicle#: 2029
> > >>>> > >> > >>>> 301-683-3717
> > >>>> > >> > >>>>
> > >>>> > >> > >>>
> > >>>> > >> > >>>
> > >>>> > >> > >>> --
> > >>>> > >> > >>> Edward Strobach
> > >>>> > >> > >>> EMC/NCEP/NWS/
> > >>>> > >> > >>> IMSG Contractor
> > >>>> > >> > >>> Cubicle#: 2029
> > >>>> > >> > >>> 301-683-3717
> > >>>> > >> > >>>
> > >>>> > >> > >>
> > >>>> > >> > >>
> > >>>> > >> > >> --
> > >>>> > >> > >> Edward Strobach
> > >>>> > >> > >> EMC/NCEP/NWS/
> > >>>> > >> > >> IMSG Contractor
> > >>>> > >> > >> Cubicle#: 2029
> > >>>> > >> > >> 301-683-3717
> > >>>> > >> > >>
> > >>>> > >> > >
> > >>>> > >> > >
> > >>>> > >> > > --
> > >>>> > >> > > Edward Strobach
> > >>>> > >> > > EMC/NCEP/NWS/
> > >>>> > >> > > IMSG Contractor
> > >>>> > >> > > Cubicle#: 2029
> > >>>> > >> > > 301-683-3717
> > >>>> > >> > >
> > >>>> > >>
> > >>>> > >>
> > >>>> > >>
> > >>>> > >>
> > >>>> > >
> > >>>> > > --
> > >>>> > > Edward Strobach
> > >>>> > > EMC/NCEP/NWS/
> > >>>> > > IMSG Contractor
> > >>>> > > Cubicle#: 2029
> > >>>> > > 301-683-3717
> > >>>> > >
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>> --
> > >>> Edward Strobach
> > >>> EMC/NCEP/NWS/
> > >>> IMSG Contractor
> > >>> Cubicle#: 2029
> > >>> 301-683-3717
> > >>>
> > >>
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > > --
> > > Jeffery T. McQueen
> > > Meteorologist
> > > National Center for Weather and Climate Prediction
> > > Room 2095
> > > 5830 University Research Court
> > > College Park, MD 20741
> > > 301-683-3736
> > > Fax: 301-683-3703
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 13:21:44 2020
Hi Ho-Chun,
I've made all the changes that I would need to inside my xml file. I
was
told not to change the folders info. I instead changed my database
name,
fcst_var, stat options, series1 input, the fix fields, the indep
field, and
x_lim.
Thanks
On Wed, Jun 17, 2020 at 3:16 PM Ho-Chun Huang - NOAA Affiliate <
ho-chun.huang at noaa.gov> wrote:
> Hi, Ed:
>
> You should go back to read 1README again from the directory I sent
about
> replacing the top portion of your metviewer xml to produce a batch
xml.
>
> I check your
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
> Please update the section from <connection> to </folder> as those in
> hochun_xml.template. I also do not know whether AWS can handle
comments
> <!-- to !--> in batch xml , better make your batch xml clean.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> Just adding more detail to elaborate my concerns...
>>
>> Not too long ago I was given a test set-up by Ho-Chun to generate a
plot
>> for aerosol optical depth. The result was successful and produced
a CSI
>> plot for AOD, comparing the production run and experimental run.
Although
>> I created a database, I do not seem to have a group where I can
access
>> specific databases. I recently created a load_xml file, which
should have
>> loaded the data based on what I specified into the group. It
seemed
>> successful based on the log output that was generated.
Additionally, all
>> information that I specified in this step was create in the
load_*xml.
>>
>> The next step is to generate plots since the data should be loaded.
>> Here's what happens when I run Ho-Chun's - the successful case
>>
>> HO-CHUN:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
>> hochun_xml.template+ bash
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # # #
# # # #
>> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT model,
>> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
>>
>> This is early on in the script.
>>
>> ED_STROBACH:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed -e
>> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20 PM
>> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
create
>> initial connections of pool.java.sql.SQLException: Communications
link
>> failure with primary. No active connection found for master.
at
>>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
>> at
>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
>> at
>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
>> at
>>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
>> at
>> org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
>> at
>>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
>> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
>>
>> My interpretation of this is that it fails to bridge a connection
to the
>> group or database; therefore, it's unable to proceed with running
the XML
>> file to generate the plots based on the specified content. If I
move past
>> this step, then I'm golden.
>>
>> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
>> jeff.mcqueen at noaa.gov> wrote:
>>
>>> Would it be ok to keep perry in the loop
>>>
>>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA Affiliate
<
>>> edward.strobach at noaa.gov> wrote:
>>>
>>>> you're right. I didn't have the storage problem this time. I'm
not
>>>> sure what changed. All I know is that I have now created my load
xml which
>>>> now should allow me to move on to the next step of plotting
>>>>
>>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA Affiliate
<
>>>> edward.strobach at noaa.gov> wrote:
>>>>
>>>>> I'm not following you. How do I now have enough space? What
changed?
>>>>>
>>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
>>>>> met_help at ucar.edu> wrote:
>>>>>
>>>>>> Ed, now you should have enough space for the data loading.
>>>>>>
>>>>>> Tatiana
>>>>>>
>>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>>>>> > I was basically able to build my xml file but it says I have
no
>>>>>> > available
>>>>>> > space. I'm not sure how that can be since I just started
adding
>>>>>> data
>>>>>> > to my
>>>>>> > account
>>>>>> >
>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space
>>>>>> left
>>>>>> > on
>>>>>> > device
>>>>>> > ERROR: Command returned with non-zero status (1): scp -r ./*
>>>>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>>>>> > 4) Check METviewer AWS database list using
mv_db_size_on_aws.sh
>>>>>> > TABLE_SCHEMA USER
SIZE_MB
>>>>>> > mv_met_g2o_gefs binbin.zhou
140027.31
>>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
71986.11
>>>>>> >
>>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate <
>>>>>> > edward.strobach at noaa.gov> wrote:
>>>>>> >
>>>>>> > > Thanks Tatiana. Yes, I've created a database which seems
>>>>>> different
>>>>>> > > from
>>>>>> > > mv_group. How can one create their own group so that I can
store
>>>>>> and
>>>>>> > > access
>>>>>> > > the database. The database exists but cannot be viewed on
the
>>>>>> > > metviewer
>>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
vhagerty. I
>>>>>> > > named my
>>>>>> > > database Ed_Strobach, but that was really intended for the
group
>>>>>> > > name.
>>>>>> > >
>>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>>>>> > > <met_help at ucar.edu>
>>>>>> > > wrote:
>>>>>> > >
>>>>>> > >> We don't have database and METviewer running on wcoss.
>>>>>> > >> Instead, users use METviewer-wcoss scripts that help to
connect
>>>>>> to
>>>>>> > >> and
>>>>>> > >> use METviewer that is running on AWS. That is why XML
connection
>>>>>> and
>>>>>> > >> folder
>>>>>> > >> sections look somewhat different than once from the Docs
page.
>>>>>> > >> To create a database on AWS you need to run
>>>>>> mv_create_db_on_aws.sh
>>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>>>>> > >> For data loading :
>>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>>>>> > >> And for batch:
>>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>>>>> > >>
>>>>>> > >> These scripts copy XML and data files if needed to AWS,
adjust
>>>>>> data
>>>>>> > >> path
>>>>>> > >> and users credentials, run METviewer and copy results back
to
>>>>>> wcoss.
>>>>>> > >>
>>>>>> > >> Tatiana
>>>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
wrote:
>>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
>>>>>> > >> > -e'create
>>>>>> > >> > database [db_name]
>>>>>> > >> >
>>>>>> > >> > would need to be used to create a database. I assume
that an
>>>>>> sql
>>>>>> > >> > module is
>>>>>> > >> > available on wcoss; I know you don't have access to
those
>>>>>> machines
>>>>>> > >> > Tatiana. If I create my own database with my own name,
then I
>>>>>> > >> > should
>>>>>> > >> > be
>>>>>> > >> > able to proceed with project specific tasks under that
tab.
>>>>>> > >> > Something
>>>>>> > >> > like
>>>>>> > >> > mv_EdwardStrobach.
>>>>>> > >> >
>>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add data
to using
>>>>>> > >> > load_spec.
>>>>>> > >> > Once that step is done, then I can run plot_spec.
That's my
>>>>>> > >> > takeaway
>>>>>> > >> > currently
>>>>>> > >> >
>>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
>>>>>> Affiliate <
>>>>>> > >> > edward.strobach at noaa.gov> wrote:
>>>>>> > >> >
>>>>>> > >> > > Actually, it appears that I need to run the database
loading
>>>>>> > >> > > module
>>>>>> > >> > > first
>>>>>> > >> > > if I want to change/add a data source. From there I
can run
>>>>>> the
>>>>>> > >> > > plotting
>>>>>> > >> > > batch module. Is that right? So if I run the
database
>>>>>> loading
>>>>>> > >> > > module,
>>>>>> > >> > > then I can specify my dates and other related info
specific
>>>>>> to
>>>>>> > >> > > the
>>>>>> > >> > > dataset. If I create my database with a unique name,
then it
>>>>>> > >> > > should
>>>>>> > >> > > be
>>>>>> > >> > > recognized on the metviewer gui, presumably.
>>>>>> > >> > >
>>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
>>>>>> Affiliate
>>>>>> > >> > > <
>>>>>> > >> > > edward.strobach at noaa.gov> wrote:
>>>>>> > >> > >
>>>>>> > >> > >> It seems that those options must be treated
separately
>>>>>> outside
>>>>>> > >> > >> of
>>>>>> > >> > >> plot_spec. Is it acceptable to create both a
load_spec and
>>>>>> > >> > >> plot_spec in
>>>>>> > >> > >> the same xml file? Would the definitions in
load_spec be
>>>>>> able
>>>>>> > >> > >> to
>>>>>> > >> > >> carry
>>>>>> > >> > >> over into plot_spec? It appears that data loading
and
>>>>>> plotting
>>>>>> > >> > >> are
>>>>>> > >> > >> treated
>>>>>> > >> > >> separately.
>>>>>> > >> > >>
>>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
NOAA
>>>>>> > >> > >> Affiliate <
>>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>>>>> > >> > >>
>>>>>> > >> > >>> I do plan to use date_range; however, in an example
>>>>>> provided
>>>>>> > >> > >>> by the
>>>>>> > >> > >>> link
>>>>>> > >> > >>> I see this:
>>>>>> > >> > >>>
>>>>>> > >> > >>> <load_spec>
>>>>>> > >> > >>> <connection>
>>>>>> > >> > >>> <host>kemosabe:3306</host>
>>>>>> > >> > >>> <database>metvdb_hwt</database>
>>>>>> > >> > >>> <user>pgoldenb</user>
>>>>>> > >> > >>> <password>pgoldenb</password>
>>>>>> > >> > >>> </connection>
>>>>>> > >> > >>>
>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>> > >> > >>> <start>2010051914V</start>
>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
>>>>>> > >> > >>> </hour></date_offset></end>
>>>>>> > >> > >>> <inc>3600</inc>
>>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
>>>>>> > >> > >>> </date_list>
>>>>>> > >> > >>>
>>>>>> > >> > >>> <met_version>V3.0</met_version>
>>>>>> > >> > >>>
>>>>>> > >> > >>> <verbose>false</verbose>
>>>>>> > >> > >>> <insert_size>1</insert_size>
>>>>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>>>>> > >> > >>> <group>Group name</group>
>>>>>> > >> > >>> <load_stat>true</load_stat>
>>>>>> > >> > >>> <load_mode>true</load_mode>
>>>>>> > >> > >>> <load_mtd>true</load_mtd>
>>>>>> > >> > >>> <load_mpr>false</load_mpr>
>>>>>> > >> > >>>
>>>>>> > >> > >>>
>>>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>>>> > >> > >>> <load_val>
>>>>>> > >> > >>> <field name="model">
>>>>>> > >> > >>> <val>arw</val>
>>>>>> > >> > >>> <val>nmm</val>
>>>>>> > >> > >>> </field>
>>>>>> > >> > >>>
>>>>>> > >> > >>> <field name="valid_time">
>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>> > >> > >>> </field>
>>>>>> > >> > >>>
>>>>>> > >> > >>> <field name="vx_mask">
>>>>>> > >> > >>> <val>FULL</val>
>>>>>> > >> > >>> <val>SWC</val>
>>>>>> > >> > >>> </field>
>>>>>> > >> > >>> </load_val>
>>>>>> > >> > >>> </load_spec>
>>>>>> > >> > >>>
>>>>>> > >> > >>> Are similar settings allowed with <plot_spec>? I
see
>>>>>> > >> > >>> <connection>,
>>>>>> > >> > >>> which seems carried over. I'm looking to specify my
>>>>>> > >> > >>> folder_tmpl in
>>>>>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>>>>>> > >> > >>> Since my
>>>>>> > >> > >>> directory structure looks like this:
>>>>>> > >> > >>>
>>>>>> > >> > >>>
>>>>>> > >> > >>>
>>>>>> > >>
>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>>>> > >> > >>>
>>>>>> > >> > >>> I would do something like this I would think:
>>>>>> > >> > >>>
>>>>>> > >> > >>> .......
>>>>>> > >> > >>>
>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>> > >> > >>> <start>20200612/start>
>>>>>> > >> > >>> <end><date_offset><day_offset>0</day_offset></end>
>>>>>> > >> > >>> <inc>86400</inc>
>>>>>> > >> > >>> <format>yyyyMMdd</format>
>>>>>> > >> > >>> </date_list>
>>>>>> > >> > >>>
>>>>>> > >> > >>> <folder_tmpl>
>>>>>> > >> > >>>
>>>>>> > >>
>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>>>>> > >> > >>> </folder_tmpl>
>>>>>> > >> > >>> <load_val>
>>>>>> > >> > >>> <field name="model">
>>>>>> > >> > >>> <val>prod</val>
>>>>>> > >> > >>> </field>
>>>>>> > >> > >>>
>>>>>> > >> > >>> <field name="valid_time">
>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>> > >> > >>> </field>
>>>>>> > >> > >>>
>>>>>> > >> > >>> ......
>>>>>> > >> > >>>
>>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach -
NOAA
>>>>>> > >> > >>> Affiliate <
>>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>>>>> > >> > >>>
>>>>>> > >> > >>>> Thank you. This is a very helpful example
>>>>>> > >> > >>>>
>>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via
RT
>>>>>> > >> > >>>> <met_help at ucar.edu>
>>>>>> > >> > >>>> wrote:
>>>>>> > >> > >>>>
>>>>>> > >> > >>>>> Hi,
>>>>>> > >> > >>>>> Here is some examples how to use <date_range> and
>>>>>> > >> > >>>>> <date_range_list> in
>>>>>> > >> > >>>>> batch XML.
>>>>>> > >> > >>>>> First, you need to defined your range. You can
have more
>>>>>> > >> > >>>>> then one
>>>>>> > >> > >>>>> <date_range>:
>>>>>> > >> > >>>>> <date_range name="may">
>>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>>>>>> > >> > >>>>> </date_range>
>>>>>> > >> > >>>>> <date_range name="march">
>>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>>>>>> > >> > >>>>> </date_range>
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>> and then use one or all date_ranges:
>>>>>> > >> > >>>>> <plot_fix>
>>>>>> > >> > >>>>> <field name="fcst_init_beg">
>>>>>> > >> > >>>>> <date_range name="march"/>
>>>>>> > >> > >>>>> <date_range name="may"/>
>>>>>> > >> > >>>>> </field>...
>>>>>> > >> > >>>>> This configuration would include all dates for
March and
>>>>>> May
>>>>>> > >> > >>>>> of
>>>>>> > >> > >>>>> 2019.
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>> If you need more flexible configuration you can
use
>>>>>> > >> > >>>>> date_range_list.
>>>>>> > >> > >>>>> For example, you want to select first 6 hours
every day
>>>>>> > >> > >>>>> beetween
>>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your
XML
>>>>>> would
>>>>>> > >> > >>>>> look
>>>>>> > >> > >>>>> like this:
>>>>>> > >> > >>>>> <date_range_list name = 'all'>
>>>>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
>>>>>> > >> > >>>>> <range_end>2020-01-05 00:00:00</range_end>
>>>>>> > >> > >>>>> <range_length>360</range_length>
>>>>>> > >> > >>>>> <inc>1440</inc>
>>>>>> > >> > >>>>> </date_range_list>
>>>>>> > >> > >>>>> where the 1st and 2nd values are the start and
end,
>>>>>> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
>>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
houres)
>>>>>> > >> > >>>>> This example would create following ranges:
>>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
>>>>>> > >> > >>>>> your
>>>>>> > >> > >>>>> data.
>>>>>> > >> > >>>>> For example, if you want to select only dates with
00:00
>>>>>> you
>>>>>> > >> > >>>>> can
>>>>>> > >> > >>>>> specify
>>>>>> > >> > >>>>> <field name="valid_hour">
>>>>>> > >> > >>>>> <val>00</val>
>>>>>> > >> > >>>>> </field>
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>> Tatiana
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
>>>>>> wrote:
>>>>>> > >> > >>>>> > and if there isn't a better way, I wonder the
>>>>>> whereabouts
>>>>>> > >> > >>>>> > of a
>>>>>> > >> > >>>>> > separate
>>>>>> > >> > >>>>> > script that pipes dates (and other relevant
>>>>>> information)
>>>>>> > >> > >>>>> > into
>>>>>> > >> > >>>>> > the XML
>>>>>> > >> > >>>>> > file
>>>>>> > >> > >>>>> > used to generate the results.
>>>>>> > >> > >>>>> >
>>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach
- NOAA
>>>>>> > >> > >>>>> > Affiliate <
>>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>>>>> > >> > >>>>> >
>>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are helpful.
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>> > > One last question. Simply typing lines like
this in
>>>>>> can
>>>>>> > >> > >>>>> > > take
>>>>>> > >> > >>>>> > > forever:
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>> > > <plot_fix>
>>>>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
>>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>>>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>>>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>>>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>>>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>>>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>> > > Is there a better way? I see date_range,
>>>>>> > >> > >>>>> > > date_range_list,
>>>>>> > >> > >>>>> > > or
>>>>>> > >> > >>>>> > > date_list.
>>>>>> > >> > >>>>> > > Can this be used instead of typing individual
values?
>>>>>> > >> > >>>>> > > This
>>>>>> > >> > >>>>> > > could
>>>>>> > >> > >>>>> > > prove
>>>>>> > >> > >>>>> > > pretty daunting if one would have to type in
700
>>>>>> entries
>>>>>> > >> > >>>>> > > each
>>>>>> > >> > >>>>> > > time.
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek
via RT
>>>>>> > >> > >>>>> > > <met_help at ucar.edu>
>>>>>> > >> > >>>>> > > wrote:
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>> > >> I found a few problems in your XML.
>>>>>> > >> > >>>>> > >> The database you are using doesn't have data
with
>>>>>> > >> > >>>>> > >> vx_mask
>>>>>> > >> > >>>>> > >> 'abcREGabc'.
>>>>>> > >> > >>>>> > >> That is why METviewer returns an empty
dataset and
>>>>>> > >> > >>>>> > >> can't
>>>>>> > >> > >>>>> > >> create a
>>>>>> > >> > >>>>> > >> plot.
>>>>>> > >> > >>>>> > >> The other problem is how you define
fcst_lead. This
>>>>>> > >> > >>>>> > >> database
>>>>>> > >> > >>>>> > >> has
>>>>>> > >> > >>>>> > >> values
>>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
Since you
>>>>>> > >> > >>>>> > >> include
>>>>>> > >> > >>>>> > >> all
>>>>>> > >> > >>>>> > >> fcst_leads
>>>>>> > >> > >>>>> > >> this variable can be omitted.
>>>>>> > >> > >>>>> > >> To minimize problems with variables values, I
would
>>>>>> > >> > >>>>> > >> suggest
>>>>>> > >> > >>>>> > >> to
>>>>>> > >> > >>>>> > >> create XML
>>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>>>>>> > >> > >>>>> > >> I created a plot using you configurations
with GUI
>>>>>> > >> > >>>>> > >> (without
>>>>>> > >> > >>>>> vx_mack
>>>>>> > >> > >>>>> > >> and
>>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You
can
>>>>>> load it
>>>>>> > >> > >>>>> > >> back
>>>>>> > >> > >>>>> > >> to
>>>>>> > >> > >>>>> the
>>>>>> > >> > >>>>> > >> page by
>>>>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI).
Find the
>>>>>> > >> > >>>>> > >> plot,
>>>>>> > >> > >>>>> > >> open XML
>>>>>> > >> > >>>>> > >> and
>>>>>> > >> > >>>>> > >> click on "Load to page". After that you can
>>>>>> recreate a
>>>>>> > >> > >>>>> > >> plot
>>>>>> > >> > >>>>> > >> and
>>>>>> > >> > >>>>> edit
>>>>>> > >> > >>>>> > >> the
>>>>>> > >> > >>>>> > >> configurations. You can also download a new
XML and
>>>>>> > >> > >>>>> > >> edit it
>>>>>> > >> > >>>>> > >> for
>>>>>> > >> > >>>>> > >> batch.
>>>>>> > >> > >>>>> > >> I attached the example of load XML to this
message.
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>> > >> Description of rds parameters:
>>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
METviewer R
>>>>>> > >> > >>>>> > >> scripts
>>>>>> > >> > >>>>> > >> rds_plots - temporary directory where the
plots
>>>>>> will be
>>>>>> > >> > >>>>> > >> generated
>>>>>> > >> > >>>>> > >> rds_data - temporary directory where the data
>>>>>> retrieved
>>>>>> > >> > >>>>> > >> from
>>>>>> > >> > >>>>> > >> the
>>>>>> > >> > >>>>> > >> database
>>>>>> > >> > >>>>> > >> and other intermediate files will be stored
>>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
scripts
>>>>>> for
>>>>>> > >> > >>>>> > >> this
>>>>>> > >> > >>>>> > >> run
>>>>>> > >> > >>>>> will
>>>>>> > >> > >>>>> > >> be
>>>>>> > >> > >>>>> > >> stored
>>>>>> > >> > >>>>> > >> After the execution is done all the files
from
>>>>>> > >> > >>>>> > >> temporary
>>>>>> > >> > >>>>> directories
>>>>>> > >> > >>>>> > >> would be removed.
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
>>>>>> > >> > >>>>> > >> There
>>>>>> > >> > >>>>> > >> are
>>>>>> > >> > >>>>> some
>>>>>> > >> > >>>>> > >> instructions on how to crate XML config
files:
>>>>>> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>> > >> Tatiana
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
>>>>>> edward.strobach at noaa.gov
>>>>>> > >> > >>>>> > >> wrote:
>>>>>> > >> > >>>>> > >> > Hi,
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am having
hard
>>>>>> time
>>>>>> > >> > >>>>> > >> > following
>>>>>> > >> > >>>>> how
>>>>>> > >> > >>>>> > >> > to
>>>>>> > >> > >>>>> > >> > go
>>>>>> > >> > >>>>> > >> > about the other steps since I have zero
>>>>>> experience on
>>>>>> > >> > >>>>> > >> > how
>>>>>> > >> > >>>>> > >> > to do
>>>>>> > >> > >>>>> > >> > these
>>>>>> > >> > >>>>> > >> > things.
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should
>>>>>> NOT be
>>>>>> > >> > >>>>> > >> > changed
>>>>>> > >> > >>>>> and
>>>>>> > >> > >>>>> > >> > will
>>>>>> > >> > >>>>> > >> > be
>>>>>> > >> > >>>>> > >> > replaced by other scripts during the batch
>>>>>> execution.
>>>>>> > >> > >>>>> > >> > The
>>>>>> > >> > >>>>> > >> > data
>>>>>> > >> > >>>>> for
>>>>>> > >> > >>>>> > >> > the
>>>>>> > >> > >>>>> > >> > batch mode is coming from the database and
not
>>>>>> from
>>>>>> > >> > >>>>> > >> > files.
>>>>>> > >> > >>>>> > >> > The
>>>>>> > >> > >>>>> > >> > name of
>>>>>> > >> > >>>>> > >> > the
>>>>>> > >> > >>>>> > >> > database is in <database> tag.
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> > I understand that nothing should be changed
in the
>>>>>> > >> > >>>>> > >> > folders
>>>>>> > >> > >>>>> > >> > section,
>>>>>> > >> > >>>>> > >> > but
>>>>>> > >> > >>>>> > >> > there is no indication how and where these
>>>>>> variables
>>>>>> > >> > >>>>> > >> > are
>>>>>> > >> > >>>>> > >> > being
>>>>>> > >> > >>>>> > >> > defined.
>>>>>> > >> > >>>>> > >> > Grepping these names of the files available
does
>>>>>> not
>>>>>> > >> > >>>>> > >> > reveal
>>>>>> > >> > >>>>> formal
>>>>>> > >> > >>>>> > >> > definitions of these variables. If I
understood
>>>>>> > >> > >>>>> > >> > exactly
>>>>>> > >> > >>>>> > >> > how
>>>>>> > >> > >>>>> these
>>>>>> > >> > >>>>> > >> > were
>>>>>> > >> > >>>>> > >> > being defined, then I would be able connect
this
>>>>>> > >> > >>>>> > >> > better.
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
files.
>>>>>> They
>>>>>> > >> > >>>>> > >> > use
>>>>>> > >> > >>>>> METviewer
>>>>>> > >> > >>>>> > >> > load
>>>>>> > >> > >>>>> > >> > mode to load these files to the specific
>>>>>> database. In
>>>>>> > >> > >>>>> > >> > the
>>>>>> > >> > >>>>> > >> > config
>>>>>> > >> > >>>>> > >> > file
>>>>>> > >> > >>>>> > >> > for
>>>>>> > >> > >>>>> > >> > this process they also can specify the
database
>>>>>> > >> > >>>>> > >> > group. For
>>>>>> > >> > >>>>> example
>>>>>> > >> > >>>>> > >> > database
>>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>>>>> > >> > >>>>> > >> > After the data has been loaded to the
database it
>>>>>> is
>>>>>> > >> > >>>>> > >> > visible and
>>>>>> > >> > >>>>> > >> > available
>>>>>> > >> > >>>>> > >> > in the GUI and also can be used by the
batch mode.
>>>>>> > >> > >>>>> > >> > You can always add more data to the
database using
>>>>>> > >> > >>>>> > >> > the
>>>>>> > >> > >>>>> > >> > same load
>>>>>> > >> > >>>>> > >> > mode.
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> > Is there a guide for me to understand the
process
>>>>>> > >> > >>>>> > >> > that you
>>>>>> > >> > >>>>> > >> > are
>>>>>> > >> > >>>>> > >> > referring to?
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek
>>>>>> via RT
>>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>>>>> > >> > >>>>> > >> > wrote:
>>>>>> > >> > >>>>> > >> >
>>>>>> > >> > >>>>> > >> > > Hi Edward,
>>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
servers and
>>>>>> > >> > >>>>> > >> > > can't
>>>>>> > >> > >>>>> > >> > > view your
>>>>>> > >> > >>>>> > >> > > XML
>>>>>> > >> > >>>>> > >> > > for the
>>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue
>>>>>> and I
>>>>>> > >> > >>>>> > >> > > will
>>>>>> > >> > >>>>> > >> > > take a
>>>>>> > >> > >>>>> > >> > > look.
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
should
>>>>>> NOT
>>>>>> > >> > >>>>> > >> > > be
>>>>>> > >> > >>>>> > >> > > changed
>>>>>> > >> > >>>>> > >> > > and
>>>>>> > >> > >>>>> > >> > > will be
>>>>>> > >> > >>>>> > >> > > replaced by other scripts during the
batch
>>>>>> > >> > >>>>> > >> > > execution.
>>>>>> > >> > >>>>> > >> > > The data
>>>>>> > >> > >>>>> > >> > > for
>>>>>> > >> > >>>>> > >> > > the
>>>>>> > >> > >>>>> > >> > > batch mode is coming from the database
and not
>>>>>> from
>>>>>> > >> > >>>>> > >> > > files. The
>>>>>> > >> > >>>>> > >> > > name
>>>>>> > >> > >>>>> > >> > > of the
>>>>>> > >> > >>>>> > >> > > database is in <database> tag.
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
>>>>>> > >> > >>>>> > >> > > They use
>>>>>> > >> > >>>>> > >> > > METviewer
>>>>>> > >> > >>>>> > >> > > load
>>>>>> > >> > >>>>> > >> > > mode to load these files to the specific
>>>>>> database.
>>>>>> > >> > >>>>> > >> > > In
>>>>>> > >> > >>>>> > >> > > the
>>>>>> > >> > >>>>> config
>>>>>> > >> > >>>>> > >> > > file
>>>>>> > >> > >>>>> > >> > > for
>>>>>> > >> > >>>>> > >> > > this process they also can specify the
database
>>>>>> > >> > >>>>> > >> > > group.
>>>>>> > >> > >>>>> > >> > > For
>>>>>> > >> > >>>>> > >> > > example
>>>>>> > >> > >>>>> > >> > > database
>>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>>>>> > >> > >>>>> > >> > > After the data has been loaded to the
database
>>>>>> it
>>>>>> > >> > >>>>> > >> > > is
>>>>>> > >> > >>>>> > >> > > visible
>>>>>> > >> > >>>>> and
>>>>>> > >> > >>>>> > >> > > available
>>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch
>>>>>> mode.
>>>>>> > >> > >>>>> > >> > > You can always add more data to the
database
>>>>>> using
>>>>>> > >> > >>>>> > >> > > the
>>>>>> > >> > >>>>> > >> > > same
>>>>>> > >> > >>>>> load
>>>>>> > >> > >>>>> > >> > > mode.
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >> > > Tatiana
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>>>>> > >> > >>>>> > >> > > wrote:
>>>>>> > >> > >>>>> > >> > > > Good morning,
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
effort to
>>>>>> > >> > >>>>> > >> > > > understand
>>>>>> > >> > >>>>> > >> > > > metviewer,
>>>>>> > >> > >>>>> > >> > > > test
>>>>>> > >> > >>>>> > >> > > > different cases using the history tab,
>>>>>> changing
>>>>>> > >> > >>>>> > >> > > > information
>>>>>> > >> > >>>>> to
>>>>>> > >> > >>>>> > >> > > > see
>>>>>> > >> > >>>>> > >> > > > what
>>>>>> > >> > >>>>> > >> > > > works and what does not, and make
connections
>>>>>> > >> > >>>>> > >> > > > between
>>>>>> > >> > >>>>> > >> > > > the
>>>>>> > >> > >>>>> GUI
>>>>>> > >> > >>>>> > >> > > > settings
>>>>>> > >> > >>>>> > >> > > > and
>>>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
level with
>>>>>> > >> > >>>>> > >> > > > metviewer has
>>>>>> > >> > >>>>> > >> > > > significantly
>>>>>> > >> > >>>>> > >> > > > increased during this time, there are
some
>>>>>> > >> > >>>>> > >> > > > remaining
>>>>>> > >> > >>>>> > >> > > > issues
>>>>>> > >> > >>>>> > >> > > > that
>>>>>> > >> > >>>>> > >> > > > I'm
>>>>>> > >> > >>>>> > >> > > > hoping
>>>>>> > >> > >>>>> > >> > > > to clear up.
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
running with
>>>>>> > >> > >>>>> > >> > > > different
>>>>>> > >> > >>>>> > >> > > > statistic
>>>>>> > >> > >>>>> > >> > > > and
>>>>>> > >> > >>>>> > >> > > > plot
>>>>>> > >> > >>>>> > >> > > > type:
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
provided me
>>>>>> > >> > >>>>> > >> > > > the
>>>>>> > >> > >>>>> > >> > > > working
>>>>>> > >> > >>>>> > >> > > > directory
>>>>>> > >> > >>>>> > >> > > > to get
>>>>>> > >> > >>>>> > >> > > > started which I copied over:
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted
and
>>>>>> used
>>>>>> > >> > >>>>> > >> > > > to
>>>>>> > >> > >>>>> > >> > > > run his
>>>>>> > >> > >>>>> > >> > > > example.
>>>>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
produced
>>>>>> a
>>>>>> > >> > >>>>> > >> > > > CSI
>>>>>> > >> > >>>>> > >> > > > plot
>>>>>> > >> > >>>>> that
>>>>>> > >> > >>>>> > >> > > > was
>>>>>> > >> > >>>>> > >> > > > stored
>>>>>> > >> > >>>>> > >> > > > here:
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>>
>>>>>> > >>
>>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > Running the script produced the
expected
>>>>>> result.
>>>>>> > >> > >>>>> > >> > > > The
>>>>>> > >> > >>>>> > >> > > > next
>>>>>> > >> > >>>>> > >> > > > natural
>>>>>> > >> > >>>>> > >> > > > step was
>>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and create
a new
>>>>>> xml
>>>>>> > >> > >>>>> > >> > > > file
>>>>>> > >> > >>>>> > >> > > > called
>>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
create a
>>>>>> time
>>>>>> > >> > >>>>> > >> > > > series
>>>>>> > >> > >>>>> > >> > > > where
>>>>>> > >> > >>>>> > >> > > > the
>>>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
instead of
>>>>>> > >> > >>>>> > >> > > > obs_thresh.
>>>>>> > >> > >>>>> > >> > > > Accordingly,
>>>>>> > >> > >>>>> > >> > > > I
>>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
>>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>>>>> > >> > >>>>> and
>>>>>> > >> > >>>>> > >> > > > changed
>>>>>> > >> > >>>>> > >> > > > the
>>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was
also
>>>>>> > >> > >>>>> > >> > > > tried).
>>>>>> > >> > >>>>> > >> > > > The
>>>>>> > >> > >>>>> error
>>>>>> > >> > >>>>> > >> > > > that
>>>>>> > >> > >>>>> > >> > > > resulted
>>>>>> > >> > >>>>> > >> > > > is as follows:
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
('2020-05-01
>>>>>> > >> > >>>>> > >> > > > 13:00:00',
>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>> > >> > >>>>> > >> > > > 01
>>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
'2020-05-01
>>>>>> > >> > >>>>> > >> > > > 16:00:00',
>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>> > >> > >>>>> > >> > > > 01
>>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
'2020-05-01
>>>>>> > >> > >>>>> > >> > > > 19:00:00',
>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>> > >> > >>>>> > >> > > > 01
>>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
'2020-05-01
>>>>>> > >> > >>>>> > >> > > > 22:00:00',
>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>> > >> > >>>>> > >> > > > 01
>>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
'2020-05-02
>>>>>> > >> > >>>>> > >> > > > 01:00:00',
>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>> > >> > >>>>> > >> > > > 02
>>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
'2020-05-02
>>>>>> > >> > >>>>> > >> > > > 04:00:00',
>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>> > >> > >>>>> > >> > > > 02
>>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
'2020-05-02
>>>>>> > >> > >>>>> > >> > > > 07:00:00',
>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>> > >> > >>>>> > >> > > > 02
>>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
'2020-05-02
>>>>>> > >> > >>>>> > >> > > > 10:00:00',
>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>> > >> > >>>>> > >> > > > 02
>>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
h.stat_header_id
>>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>>>>> > >> > >>>>> > >> > > > No data was returned from database
>>>>>> > >> > >>>>> > >> > > > 2020-06-09
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>>
>>>>>> > >>
>>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
ERROR:
>>>>>> > >> > >>>>> > >> > > > Caught
>>>>>> > >> > >>>>> > >> > > > class
>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException in
>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>>>>> > >> > >>>>> > >> > > > res): result set contained no data
>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>> > >> > >>>>> > >> > > > in
>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result set
>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>> > >> > >>>>> > >> > > > data
>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>> > >> > >>>>> > >> > > > in
>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result set
>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>> > >> > >>>>> > >> > > > data
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > Apparently there was no data during
this
>>>>>> time. I
>>>>>> > >> > >>>>> > >> > > > tried
>>>>>> > >> > >>>>> > >> > > > changing
>>>>>> > >> > >>>>> > >> > > > the
>>>>>> > >> > >>>>> > >> > > > range
>>>>>> > >> > >>>>> > >> > > > of valid times and changed the valid
time to
>>>>>> > >> > >>>>> > >> > > > different
>>>>>> > >> > >>>>> > >> > > > days
>>>>>> > >> > >>>>> > >> > > > just in
>>>>>> > >> > >>>>> > >> > > > case
>>>>>> > >> > >>>>> > >> > > > data was missing. The issue was the
same. To
>>>>>> > >> > >>>>> > >> > > > confirm
>>>>>> > >> > >>>>> > >> > > > my
>>>>>> > >> > >>>>> > >> > > > logic, I
>>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
>>>>>> > >> > >>>>> > >> > > > which I
>>>>>> > >> > >>>>> > >> > > > found
>>>>>> > >> > >>>>> > >> > > > to
>>>>>> > >> > >>>>> > >> > > > work.
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > I think I lack an understanding of how
to set
>>>>>> up
>>>>>> > >> > >>>>> > >> > > > batch
>>>>>> > >> > >>>>> > >> > > > mode
>>>>>> > >> > >>>>> > >> > > > because
>>>>>> > >> > >>>>> > >> > > > I
>>>>>> > >> > >>>>> > >> > > > don't
>>>>>> > >> > >>>>> > >> > > > see where these definitions are being
>>>>>> explicitly
>>>>>> > >> > >>>>> > >> > > > defined:
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > <plot_spec>
>>>>>> > >> > >>>>> > >> > > > <connection>
>>>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>>>>> > >> > >>>>> > >> > > > </connection>
>>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>>>>> > >> > >>>>> > >> > > > <folders>
>>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>>>> > >> > >>>>> > >> > > > </folders>
>>>>>> > >> > >>>>> > >> > > > <plot>
>>>>>> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
grepping all
>>>>>> > >> > >>>>> > >> > > > available
>>>>>> > >> > >>>>> files
>>>>>> > >> > >>>>> > >> > > > does
>>>>>> > >> > >>>>> > >> > > > not
>>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined,
as
>>>>>> well
>>>>>> > >> > >>>>> > >> > > > as
>>>>>> > >> > >>>>> > >> > > > other
>>>>>> > >> > >>>>> > >> > > > similarly
>>>>>> > >> > >>>>> > >> > > > defined
>>>>>> > >> > >>>>> > >> > > > variables. I think if I know how these
are
>>>>>> being
>>>>>> > >> > >>>>> > >> > > > defined,
>>>>>> > >> > >>>>> > >> > > > then
>>>>>> > >> > >>>>> > >> > > > I'll
>>>>>> > >> > >>>>> > >> > > > be
>>>>>> > >> > >>>>> > >> > > > able to pipe in my information and
process the
>>>>>> > >> > >>>>> > >> > > > data
>>>>>> > >> > >>>>> > >> > > > using
>>>>>> > >> > >>>>> met
>>>>>> > >> > >>>>> > >> > > > and
>>>>>> > >> > >>>>> > >> > > > produce
>>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
that?
>>>>>> > >> > >>>>> > >> > > > Also, if
>>>>>> > >> > >>>>> > >> > > > I want
>>>>>> > >> > >>>>> > >> > > > to use
>>>>>> > >> > >>>>> > >> > > > obs for
>>>>>> > >> > >>>>> > >> > > > verification, do I add another line in
the XML
>>>>>> > >> > >>>>> > >> > > > file
>>>>>> > >> > >>>>> > >> > > > that
>>>>>> > >> > >>>>> > >> > > > designates
>>>>>> > >> > >>>>> > >> > > > file
>>>>>> > >> > >>>>> > >> > > > location of obs? I assume that other
steps
>>>>>> have
>>>>>> > >> > >>>>> > >> > > > to be
>>>>>> > >> > >>>>> > >> > > > run
>>>>>> > >> > >>>>> > >> > > > before
>>>>>> > >> > >>>>> > >> > > > all
>>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form, for
>>>>>> example,
>>>>>> > >> > >>>>> > >> > > > I
>>>>>> > >> > >>>>> > >> > > > would
>>>>>> > >> > >>>>> have
>>>>>> > >> > >>>>> > >> > > > to
>>>>>> > >> > >>>>> > >> > > > separately
>>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I
would
>>>>>> want
>>>>>> > >> > >>>>> > >> > > > to
>>>>>> > >> > >>>>> > >> > > > do that
>>>>>> > >> > >>>>> > >> > > > as
>>>>>> > >> > >>>>> > >> > > > well.
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
>>>>>> > >> > >>>>> > >> > > > here -
>>>>>> > >> > >>>>> > >> > > > see
>>>>>> > >> > >>>>> > >> > > > attachment
>>>>>> > >> > >>>>> > >> > > >
>>>>>> > >> > >>>>> > >> > > > How do I get the results I generate to
be
>>>>>> > >> > >>>>> > >> > > > uploaded to
>>>>>> > >> > >>>>> > >> > > > this
>>>>>> > >> > >>>>> > >> > > > location?
>>>>>> > >> > >>>>> > >> > > > It's
>>>>>> > >> > >>>>> > >> > > > not clear how others are setting this
up.
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >> > >
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>> > >>
>>>>>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>>>>>> > >> > >>>>> > >> rds_user
>>>>>> > >> > >>>>> rds_pwd
>>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
rds_plots
>>>>>> rds_data
>>>>>> > >> > >>>>> rds_scripts
>>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>>>>> > >> > >>>>> 2020-05-12
>>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
19:00:00
>>>>>> 2020-
>>>>>> > >> > >>>>> > >> 05-12
>>>>>> > >> > >>>>> 20:00:00
>>>>>> > >> > >>>>> > >> 2020-05-12
>>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
23:00:00
>>>>>> 2020-
>>>>>> > >> > >>>>> > >> 05-13
>>>>>> > >> > >>>>> 10:00:00
>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
13:00:00
>>>>>> 2020-
>>>>>> > >> > >>>>> > >> 05-13
>>>>>> > >> > >>>>> 14:00:00
>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
17:00:00
>>>>>> 2020-
>>>>>> > >> > >>>>> > >> 05-13
>>>>>> > >> > >>>>> 18:00:00
>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
21:00:00
>>>>>> 2020-
>>>>>> > >> > >>>>> > >> 05-13
>>>>>> > >> > >>>>> 22:00:00
>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
11:00:00
>>>>>> 2020-
>>>>>> > >> > >>>>> > >> 05-14
>>>>>> > >> > >>>>> 12:00:00
>>>>>> > >> > >>>>> > >> true
>>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>>>>> > >> > >>>>> > >> plot_20200612_173626.png
>>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
>>>>>> > >> > >>>>> > >> false list() list() false false false false
false
>>>>>> false
>>>>>> > >> > >>>>> > >> true
>>>>>> > >> > >>>>> > >> false
>>>>>> > >> > >>>>> > >> false
>>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
>>>>>> c(8,4,5,4)
>>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>>>>> > >> > >>>>> > >> 1.4 -2
>>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1
-2
>>>>>> 0.5 3
>>>>>> > >> > >>>>> > >> #cccccc 1
>>>>>> > >> > >>>>> > >> listX 1
>>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o
c(0,
>>>>>> -.25)
>>>>>> > >> > >>>>> > >> 3
>>>>>> > >> > >>>>> > >> 0.8 1
>>>>>> > >> > >>>>> > >> #333333
>>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
>>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
c("b","b")
>>>>>> c(1,1)
>>>>>> > >> > >>>>> > >> c(1,1)
>>>>>> > >> > >>>>> c(1,1)
>>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>> > > --
>>>>>> > >> > >>>>> > > Edward Strobach
>>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>>>>> > >> > >>>>> > > IMSG Contractor
>>>>>> > >> > >>>>> > > Cubicle#: 2029
>>>>>> > >> > >>>>> > > 301-683-3717
>>>>>> > >> > >>>>> > >
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>>
>>>>>> > >> > >>>>
>>>>>> > >> > >>>> --
>>>>>> > >> > >>>> Edward Strobach
>>>>>> > >> > >>>> EMC/NCEP/NWS/
>>>>>> > >> > >>>> IMSG Contractor
>>>>>> > >> > >>>> Cubicle#: 2029
>>>>>> > >> > >>>> 301-683-3717
>>>>>> > >> > >>>>
>>>>>> > >> > >>>
>>>>>> > >> > >>>
>>>>>> > >> > >>> --
>>>>>> > >> > >>> Edward Strobach
>>>>>> > >> > >>> EMC/NCEP/NWS/
>>>>>> > >> > >>> IMSG Contractor
>>>>>> > >> > >>> Cubicle#: 2029
>>>>>> > >> > >>> 301-683-3717
>>>>>> > >> > >>>
>>>>>> > >> > >>
>>>>>> > >> > >>
>>>>>> > >> > >> --
>>>>>> > >> > >> Edward Strobach
>>>>>> > >> > >> EMC/NCEP/NWS/
>>>>>> > >> > >> IMSG Contractor
>>>>>> > >> > >> Cubicle#: 2029
>>>>>> > >> > >> 301-683-3717
>>>>>> > >> > >>
>>>>>> > >> > >
>>>>>> > >> > >
>>>>>> > >> > > --
>>>>>> > >> > > Edward Strobach
>>>>>> > >> > > EMC/NCEP/NWS/
>>>>>> > >> > > IMSG Contractor
>>>>>> > >> > > Cubicle#: 2029
>>>>>> > >> > > 301-683-3717
>>>>>> > >> > >
>>>>>> > >>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >
>>>>>> > > --
>>>>>> > > Edward Strobach
>>>>>> > > EMC/NCEP/NWS/
>>>>>> > > IMSG Contractor
>>>>>> > > Cubicle#: 2029
>>>>>> > > 301-683-3717
>>>>>> > >
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Edward Strobach
>>>>> EMC/NCEP/NWS/
>>>>> IMSG Contractor
>>>>> Cubicle#: 2029
>>>>> 301-683-3717
>>>>>
>>>>
>>>>
>>>> --
>>>> Edward Strobach
>>>> EMC/NCEP/NWS/
>>>> IMSG Contractor
>>>> Cubicle#: 2029
>>>> 301-683-3717
>>>>
>>> --
>>> Jeffery T. McQueen
>>> Meteorologist
>>> National Center for Weather and Climate Prediction
>>> Room 2095
>>> 5830 University Research Court
>>> College Park, MD 20741
>>> 301-683-3736
>>> Fax: 301-683-3703
>>>
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Ho-Chun Huang - NOAA Affiliate
Time: Wed Jun 17 13:27:38 2020
Ed:
You can try or you can keep waiting, it is your choice.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Wed, Jun 17, 2020 at 3:21 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Hi Ho-Chun,
>
> I've made all the changes that I would need to inside my xml file.
I was
> told not to change the folders info. I instead changed my database
name,
> fcst_var, stat options, series1 input, the fix fields, the indep
field, and
> x_lim.
>
> Thanks
>
> On Wed, Jun 17, 2020 at 3:16 PM Ho-Chun Huang - NOAA Affiliate <
> ho-chun.huang at noaa.gov> wrote:
>
>> Hi, Ed:
>>
>> You should go back to read 1README again from the directory I sent
about
>> replacing the top portion of your metviewer xml to produce a batch
xml.
>>
>> I check your
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
>> Please update the section from <connection> to </folder> as those
in
>> hochun_xml.template. I also do not know whether AWS can handle
comments
>> <!-- to !--> in batch xml , better make your batch xml clean.
>>
>> Ho-Chun Huang
>>
>> IMSG at NOAA/NWS/NCEP/EMC
>>
>> 5830 University Research Ct., Rm. 2792
>>
>> College Park, MD 20740
>>
>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>
>> 301-683-3958
>>
>>
>> On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> Just adding more detail to elaborate my concerns...
>>>
>>> Not too long ago I was given a test set-up by Ho-Chun to generate
a plot
>>> for aerosol optical depth. The result was successful and produced
a CSI
>>> plot for AOD, comparing the production run and experimental run.
Although
>>> I created a database, I do not seem to have a group where I can
access
>>> specific databases. I recently created a load_xml file, which
should have
>>> loaded the data based on what I specified into the group. It
seemed
>>> successful based on the log output that was generated.
Additionally, all
>>> information that I specified in this step was create in the
load_*xml.
>>>
>>> The next step is to generate plots since the data should be
loaded.
>>> Here's what happens when I run Ho-Chun's - the successful case
>>>
>>> HO-CHUN:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
>>> hochun_xml.template+ bash
>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>>> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # #
# # # # #
>>> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT
model,
>>> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
>>>
>>> This is early on in the script.
>>>
>>> ED_STROBACH:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed
-e
>>> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>>> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20 PM
>>> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
create
>>> initial connections of pool.java.sql.SQLException: Communications
link
>>> failure with primary. No active connection found for master.
at
>>>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
>>> at
>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
>>> at
>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
>>> at
>>>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
>>> at
>>> org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
>>> at
>>>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
>>> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
>>>
>>> My interpretation of this is that it fails to bridge a connection
to the
>>> group or database; therefore, it's unable to proceed with running
the XML
>>> file to generate the plots based on the specified content. If I
move past
>>> this step, then I'm golden.
>>>
>>> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
>>> jeff.mcqueen at noaa.gov> wrote:
>>>
>>>> Would it be ok to keep perry in the loop
>>>>
>>>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA Affiliate
<
>>>> edward.strobach at noaa.gov> wrote:
>>>>
>>>>> you're right. I didn't have the storage problem this time. I'm
not
>>>>> sure what changed. All I know is that I have now created my
load xml which
>>>>> now should allow me to move on to the next step of plotting
>>>>>
>>>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA
Affiliate <
>>>>> edward.strobach at noaa.gov> wrote:
>>>>>
>>>>>> I'm not following you. How do I now have enough space? What
changed?
>>>>>>
>>>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>> Ed, now you should have enough space for the data loading.
>>>>>>>
>>>>>>> Tatiana
>>>>>>>
>>>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>>>>>> > I was basically able to build my xml file but it says I have
no
>>>>>>> > available
>>>>>>> > space. I'm not sure how that can be since I just started
adding
>>>>>>> data
>>>>>>> > to my
>>>>>>> > account
>>>>>>> >
>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space
>>>>>>> left
>>>>>>> > on
>>>>>>> > device
>>>>>>> > ERROR: Command returned with non-zero status (1): scp -r ./*
>>>>>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>>>>>> > 4) Check METviewer AWS database list using
mv_db_size_on_aws.sh
>>>>>>> > TABLE_SCHEMA USER
SIZE_MB
>>>>>>> > mv_met_g2o_gefs binbin.zhou
>>>>>>> 140027.31
>>>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
71986.11
>>>>>>> >
>>>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate <
>>>>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>> >
>>>>>>> > > Thanks Tatiana. Yes, I've created a database which seems
>>>>>>> different
>>>>>>> > > from
>>>>>>> > > mv_group. How can one create their own group so that I can
store
>>>>>>> and
>>>>>>> > > access
>>>>>>> > > the database. The database exists but cannot be viewed on
the
>>>>>>> > > metviewer
>>>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
vhagerty. I
>>>>>>> > > named my
>>>>>>> > > database Ed_Strobach, but that was really intended for the
group
>>>>>>> > > name.
>>>>>>> > >
>>>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>>>>>> > > <met_help at ucar.edu>
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > >> We don't have database and METviewer running on wcoss.
>>>>>>> > >> Instead, users use METviewer-wcoss scripts that help to
connect
>>>>>>> to
>>>>>>> > >> and
>>>>>>> > >> use METviewer that is running on AWS. That is why XML
>>>>>>> connection and
>>>>>>> > >> folder
>>>>>>> > >> sections look somewhat different than once from the Docs
page.
>>>>>>> > >> To create a database on AWS you need to run
>>>>>>> mv_create_db_on_aws.sh
>>>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>>>>>> > >> For data loading :
>>>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>>>>>> > >> And for batch:
>>>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>>>>>> > >>
>>>>>>> > >> These scripts copy XML and data files if needed to AWS,
adjust
>>>>>>> data
>>>>>>> > >> path
>>>>>>> > >> and users credentials, run METviewer and copy results
back to
>>>>>>> wcoss.
>>>>>>> > >>
>>>>>>> > >> Tatiana
>>>>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
wrote:
>>>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
>>>>>>> > >> > -e'create
>>>>>>> > >> > database [db_name]
>>>>>>> > >> >
>>>>>>> > >> > would need to be used to create a database. I assume
that an
>>>>>>> sql
>>>>>>> > >> > module is
>>>>>>> > >> > available on wcoss; I know you don't have access to
those
>>>>>>> machines
>>>>>>> > >> > Tatiana. If I create my own database with my own name,
then I
>>>>>>> > >> > should
>>>>>>> > >> > be
>>>>>>> > >> > able to proceed with project specific tasks under that
tab.
>>>>>>> > >> > Something
>>>>>>> > >> > like
>>>>>>> > >> > mv_EdwardStrobach.
>>>>>>> > >> >
>>>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add data
to
>>>>>>> using
>>>>>>> > >> > load_spec.
>>>>>>> > >> > Once that step is done, then I can run plot_spec.
That's my
>>>>>>> > >> > takeaway
>>>>>>> > >> > currently
>>>>>>> > >> >
>>>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
>>>>>>> Affiliate <
>>>>>>> > >> > edward.strobach at noaa.gov> wrote:
>>>>>>> > >> >
>>>>>>> > >> > > Actually, it appears that I need to run the database
loading
>>>>>>> > >> > > module
>>>>>>> > >> > > first
>>>>>>> > >> > > if I want to change/add a data source. From there I
can
>>>>>>> run the
>>>>>>> > >> > > plotting
>>>>>>> > >> > > batch module. Is that right? So if I run the
database
>>>>>>> loading
>>>>>>> > >> > > module,
>>>>>>> > >> > > then I can specify my dates and other related info
specific
>>>>>>> to
>>>>>>> > >> > > the
>>>>>>> > >> > > dataset. If I create my database with a unique name,
then
>>>>>>> it
>>>>>>> > >> > > should
>>>>>>> > >> > > be
>>>>>>> > >> > > recognized on the metviewer gui, presumably.
>>>>>>> > >> > >
>>>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach -
NOAA
>>>>>>> Affiliate
>>>>>>> > >> > > <
>>>>>>> > >> > > edward.strobach at noaa.gov> wrote:
>>>>>>> > >> > >
>>>>>>> > >> > >> It seems that those options must be treated
separately
>>>>>>> outside
>>>>>>> > >> > >> of
>>>>>>> > >> > >> plot_spec. Is it acceptable to create both a
load_spec and
>>>>>>> > >> > >> plot_spec in
>>>>>>> > >> > >> the same xml file? Would the definitions in
load_spec be
>>>>>>> able
>>>>>>> > >> > >> to
>>>>>>> > >> > >> carry
>>>>>>> > >> > >> over into plot_spec? It appears that data loading
and
>>>>>>> plotting
>>>>>>> > >> > >> are
>>>>>>> > >> > >> treated
>>>>>>> > >> > >> separately.
>>>>>>> > >> > >>
>>>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
NOAA
>>>>>>> > >> > >> Affiliate <
>>>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>>>>>> > >> > >>
>>>>>>> > >> > >>> I do plan to use date_range; however, in an example
>>>>>>> provided
>>>>>>> > >> > >>> by the
>>>>>>> > >> > >>> link
>>>>>>> > >> > >>> I see this:
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> <load_spec>
>>>>>>> > >> > >>> <connection>
>>>>>>> > >> > >>> <host>kemosabe:3306</host>
>>>>>>> > >> > >>> <database>metvdb_hwt</database>
>>>>>>> > >> > >>> <user>pgoldenb</user>
>>>>>>> > >> > >>> <password>pgoldenb</password>
>>>>>>> > >> > >>> </connection>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>> > >> > >>> <start>2010051914V</start>
>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
>>>>>>> > >> > >>> </hour></date_offset></end>
>>>>>>> > >> > >>> <inc>3600</inc>
>>>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
>>>>>>> > >> > >>> </date_list>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> <met_version>V3.0</met_version>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> <verbose>false</verbose>
>>>>>>> > >> > >>> <insert_size>1</insert_size>
>>>>>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>>>>>> > >> > >>> <group>Group name</group>
>>>>>>> > >> > >>> <load_stat>true</load_stat>
>>>>>>> > >> > >>> <load_mode>true</load_mode>
>>>>>>> > >> > >>> <load_mtd>true</load_mtd>
>>>>>>> > >> > >>> <load_mpr>false</load_mpr>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>>
>>>>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>>>>> > >> > >>> <load_val>
>>>>>>> > >> > >>> <field name="model">
>>>>>>> > >> > >>> <val>arw</val>
>>>>>>> > >> > >>> <val>nmm</val>
>>>>>>> > >> > >>> </field>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>> > >> > >>> </field>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> <field name="vx_mask">
>>>>>>> > >> > >>> <val>FULL</val>
>>>>>>> > >> > >>> <val>SWC</val>
>>>>>>> > >> > >>> </field>
>>>>>>> > >> > >>> </load_val>
>>>>>>> > >> > >>> </load_spec>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> Are similar settings allowed with <plot_spec>? I
see
>>>>>>> > >> > >>> <connection>,
>>>>>>> > >> > >>> which seems carried over. I'm looking to specify
my
>>>>>>> > >> > >>> folder_tmpl in
>>>>>>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>>>>>>> > >> > >>> Since my
>>>>>>> > >> > >>> directory structure looks like this:
>>>>>>> > >> > >>>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>>
>>>>>>> > >>
>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> I would do something like this I would think:
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> .......
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>> > >> > >>> <start>20200612/start>
>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset></end>
>>>>>>> > >> > >>> <inc>86400</inc>
>>>>>>> > >> > >>> <format>yyyyMMdd</format>
>>>>>>> > >> > >>> </date_list>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> <folder_tmpl>
>>>>>>> > >> > >>>
>>>>>>> > >>
>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>>>>>> > >> > >>> </folder_tmpl>
>>>>>>> > >> > >>> <load_val>
>>>>>>> > >> > >>> <field name="model">
>>>>>>> > >> > >>> <val>prod</val>
>>>>>>> > >> > >>> </field>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>> > >> > >>> </field>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> ......
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach -
NOAA
>>>>>>> > >> > >>> Affiliate <
>>>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>>>>>> > >> > >>>
>>>>>>> > >> > >>>> Thank you. This is a very helpful example
>>>>>>> > >> > >>>>
>>>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via
RT
>>>>>>> > >> > >>>> <met_help at ucar.edu>
>>>>>>> > >> > >>>> wrote:
>>>>>>> > >> > >>>>
>>>>>>> > >> > >>>>> Hi,
>>>>>>> > >> > >>>>> Here is some examples how to use <date_range> and
>>>>>>> > >> > >>>>> <date_range_list> in
>>>>>>> > >> > >>>>> batch XML.
>>>>>>> > >> > >>>>> First, you need to defined your range. You can
have more
>>>>>>> > >> > >>>>> then one
>>>>>>> > >> > >>>>> <date_range>:
>>>>>>> > >> > >>>>> <date_range name="may">
>>>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>>>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>>>>>>> > >> > >>>>> </date_range>
>>>>>>> > >> > >>>>> <date_range name="march">
>>>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>>>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>>>>>>> > >> > >>>>> </date_range>
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>> and then use one or all date_ranges:
>>>>>>> > >> > >>>>> <plot_fix>
>>>>>>> > >> > >>>>> <field name="fcst_init_beg">
>>>>>>> > >> > >>>>> <date_range name="march"/>
>>>>>>> > >> > >>>>> <date_range name="may"/>
>>>>>>> > >> > >>>>> </field>...
>>>>>>> > >> > >>>>> This configuration would include all dates for
March
>>>>>>> and May
>>>>>>> > >> > >>>>> of
>>>>>>> > >> > >>>>> 2019.
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>> If you need more flexible configuration you can
use
>>>>>>> > >> > >>>>> date_range_list.
>>>>>>> > >> > >>>>> For example, you want to select first 6 hours
every day
>>>>>>> > >> > >>>>> beetween
>>>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your
XML
>>>>>>> would
>>>>>>> > >> > >>>>> look
>>>>>>> > >> > >>>>> like this:
>>>>>>> > >> > >>>>> <date_range_list name = 'all'>
>>>>>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
>>>>>>> > >> > >>>>> <range_end>2020-01-05
00:00:00</range_end>
>>>>>>> > >> > >>>>> <range_length>360</range_length>
>>>>>>> > >> > >>>>> <inc>1440</inc>
>>>>>>> > >> > >>>>> </date_range_list>
>>>>>>> > >> > >>>>> where the 1st and 2nd values are the start and
end,
>>>>>>> > >> > >>>>> range_length - the length of a range in minutes(6
>>>>>>> houres)
>>>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
houres)
>>>>>>> > >> > >>>>> This example would create following ranges:
>>>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>>>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>>>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>>>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>>>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
>>>>>>> > >> > >>>>> your
>>>>>>> > >> > >>>>> data.
>>>>>>> > >> > >>>>> For example, if you want to select only dates
with
>>>>>>> 00:00 you
>>>>>>> > >> > >>>>> can
>>>>>>> > >> > >>>>> specify
>>>>>>> > >> > >>>>> <field name="valid_hour">
>>>>>>> > >> > >>>>> <val>00</val>
>>>>>>> > >> > >>>>> </field>
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>> Tatiana
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
>>>>>>> wrote:
>>>>>>> > >> > >>>>> > and if there isn't a better way, I wonder the
>>>>>>> whereabouts
>>>>>>> > >> > >>>>> > of a
>>>>>>> > >> > >>>>> > separate
>>>>>>> > >> > >>>>> > script that pipes dates (and other relevant
>>>>>>> information)
>>>>>>> > >> > >>>>> > into
>>>>>>> > >> > >>>>> > the XML
>>>>>>> > >> > >>>>> > file
>>>>>>> > >> > >>>>> > used to generate the results.
>>>>>>> > >> > >>>>> >
>>>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward
Strobach -
>>>>>>> NOAA
>>>>>>> > >> > >>>>> > Affiliate <
>>>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>> > >> > >>>>> >
>>>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are helpful.
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>> > > One last question. Simply typing lines like
this
>>>>>>> in can
>>>>>>> > >> > >>>>> > > take
>>>>>>> > >> > >>>>> > > forever:
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>> > > <plot_fix>
>>>>>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
>>>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>>>>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>>>>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>>>>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>>>>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>>>>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>> > > Is there a better way? I see date_range,
>>>>>>> > >> > >>>>> > > date_range_list,
>>>>>>> > >> > >>>>> > > or
>>>>>>> > >> > >>>>> > > date_list.
>>>>>>> > >> > >>>>> > > Can this be used instead of typing individual
>>>>>>> values?
>>>>>>> > >> > >>>>> > > This
>>>>>>> > >> > >>>>> > > could
>>>>>>> > >> > >>>>> > > prove
>>>>>>> > >> > >>>>> > > pretty daunting if one would have to type in
700
>>>>>>> entries
>>>>>>> > >> > >>>>> > > each
>>>>>>> > >> > >>>>> > > time.
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek
via RT
>>>>>>> > >> > >>>>> > > <met_help at ucar.edu>
>>>>>>> > >> > >>>>> > > wrote:
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>> > >> I found a few problems in your XML.
>>>>>>> > >> > >>>>> > >> The database you are using doesn't have data
with
>>>>>>> > >> > >>>>> > >> vx_mask
>>>>>>> > >> > >>>>> > >> 'abcREGabc'.
>>>>>>> > >> > >>>>> > >> That is why METviewer returns an empty
dataset and
>>>>>>> > >> > >>>>> > >> can't
>>>>>>> > >> > >>>>> > >> create a
>>>>>>> > >> > >>>>> > >> plot.
>>>>>>> > >> > >>>>> > >> The other problem is how you define
fcst_lead. This
>>>>>>> > >> > >>>>> > >> database
>>>>>>> > >> > >>>>> > >> has
>>>>>>> > >> > >>>>> > >> values
>>>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
Since you
>>>>>>> > >> > >>>>> > >> include
>>>>>>> > >> > >>>>> > >> all
>>>>>>> > >> > >>>>> > >> fcst_leads
>>>>>>> > >> > >>>>> > >> this variable can be omitted.
>>>>>>> > >> > >>>>> > >> To minimize problems with variables values,
I would
>>>>>>> > >> > >>>>> > >> suggest
>>>>>>> > >> > >>>>> > >> to
>>>>>>> > >> > >>>>> > >> create XML
>>>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>>>>>>> > >> > >>>>> > >> I created a plot using you configurations
with GUI
>>>>>>> > >> > >>>>> > >> (without
>>>>>>> > >> > >>>>> vx_mack
>>>>>>> > >> > >>>>> > >> and
>>>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You
can
>>>>>>> load it
>>>>>>> > >> > >>>>> > >> back
>>>>>>> > >> > >>>>> > >> to
>>>>>>> > >> > >>>>> the
>>>>>>> > >> > >>>>> > >> page by
>>>>>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI).
Find the
>>>>>>> > >> > >>>>> > >> plot,
>>>>>>> > >> > >>>>> > >> open XML
>>>>>>> > >> > >>>>> > >> and
>>>>>>> > >> > >>>>> > >> click on "Load to page". After that you can
>>>>>>> recreate a
>>>>>>> > >> > >>>>> > >> plot
>>>>>>> > >> > >>>>> > >> and
>>>>>>> > >> > >>>>> edit
>>>>>>> > >> > >>>>> > >> the
>>>>>>> > >> > >>>>> > >> configurations. You can also download a new
XML and
>>>>>>> > >> > >>>>> > >> edit it
>>>>>>> > >> > >>>>> > >> for
>>>>>>> > >> > >>>>> > >> batch.
>>>>>>> > >> > >>>>> > >> I attached the example of load XML to this
message.
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>> > >> Description of rds parameters:
>>>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
METviewer R
>>>>>>> > >> > >>>>> > >> scripts
>>>>>>> > >> > >>>>> > >> rds_plots - temporary directory where the
plots
>>>>>>> will be
>>>>>>> > >> > >>>>> > >> generated
>>>>>>> > >> > >>>>> > >> rds_data - temporary directory where the
data
>>>>>>> retrieved
>>>>>>> > >> > >>>>> > >> from
>>>>>>> > >> > >>>>> > >> the
>>>>>>> > >> > >>>>> > >> database
>>>>>>> > >> > >>>>> > >> and other intermediate files will be stored
>>>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
scripts
>>>>>>> for
>>>>>>> > >> > >>>>> > >> this
>>>>>>> > >> > >>>>> > >> run
>>>>>>> > >> > >>>>> will
>>>>>>> > >> > >>>>> > >> be
>>>>>>> > >> > >>>>> > >> stored
>>>>>>> > >> > >>>>> > >> After the execution is done all the files
from
>>>>>>> > >> > >>>>> > >> temporary
>>>>>>> > >> > >>>>> directories
>>>>>>> > >> > >>>>> > >> would be removed.
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>> > >> We don't have the offitial user guide for
>>>>>>> METviewer.
>>>>>>> > >> > >>>>> > >> There
>>>>>>> > >> > >>>>> > >> are
>>>>>>> > >> > >>>>> some
>>>>>>> > >> > >>>>> > >> instructions on how to crate XML config
files:
>>>>>>> > >> > >>>>> > >>
https://metviewer.nws.noaa.gov/doc/index.html
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>> > >> Tatiana
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
>>>>>>> edward.strobach at noaa.gov
>>>>>>> > >> > >>>>> > >> wrote:
>>>>>>> > >> > >>>>> > >> > Hi,
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am having
hard
>>>>>>> time
>>>>>>> > >> > >>>>> > >> > following
>>>>>>> > >> > >>>>> how
>>>>>>> > >> > >>>>> > >> > to
>>>>>>> > >> > >>>>> > >> > go
>>>>>>> > >> > >>>>> > >> > about the other steps since I have zero
>>>>>>> experience on
>>>>>>> > >> > >>>>> > >> > how
>>>>>>> > >> > >>>>> > >> > to do
>>>>>>> > >> > >>>>> > >> > these
>>>>>>> > >> > >>>>> > >> > things.
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should
>>>>>>> NOT be
>>>>>>> > >> > >>>>> > >> > changed
>>>>>>> > >> > >>>>> and
>>>>>>> > >> > >>>>> > >> > will
>>>>>>> > >> > >>>>> > >> > be
>>>>>>> > >> > >>>>> > >> > replaced by other scripts during the batch
>>>>>>> execution.
>>>>>>> > >> > >>>>> > >> > The
>>>>>>> > >> > >>>>> > >> > data
>>>>>>> > >> > >>>>> for
>>>>>>> > >> > >>>>> > >> > the
>>>>>>> > >> > >>>>> > >> > batch mode is coming from the database and
not
>>>>>>> from
>>>>>>> > >> > >>>>> > >> > files.
>>>>>>> > >> > >>>>> > >> > The
>>>>>>> > >> > >>>>> > >> > name of
>>>>>>> > >> > >>>>> > >> > the
>>>>>>> > >> > >>>>> > >> > database is in <database> tag.
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> > I understand that nothing should be
changed in
>>>>>>> the
>>>>>>> > >> > >>>>> > >> > folders
>>>>>>> > >> > >>>>> > >> > section,
>>>>>>> > >> > >>>>> > >> > but
>>>>>>> > >> > >>>>> > >> > there is no indication how and where these
>>>>>>> variables
>>>>>>> > >> > >>>>> > >> > are
>>>>>>> > >> > >>>>> > >> > being
>>>>>>> > >> > >>>>> > >> > defined.
>>>>>>> > >> > >>>>> > >> > Grepping these names of the files
available does
>>>>>>> not
>>>>>>> > >> > >>>>> > >> > reveal
>>>>>>> > >> > >>>>> formal
>>>>>>> > >> > >>>>> > >> > definitions of these variables. If I
understood
>>>>>>> > >> > >>>>> > >> > exactly
>>>>>>> > >> > >>>>> > >> > how
>>>>>>> > >> > >>>>> these
>>>>>>> > >> > >>>>> > >> > were
>>>>>>> > >> > >>>>> > >> > being defined, then I would be able
connect this
>>>>>>> > >> > >>>>> > >> > better.
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
files.
>>>>>>> They
>>>>>>> > >> > >>>>> > >> > use
>>>>>>> > >> > >>>>> METviewer
>>>>>>> > >> > >>>>> > >> > load
>>>>>>> > >> > >>>>> > >> > mode to load these files to the specific
>>>>>>> database. In
>>>>>>> > >> > >>>>> > >> > the
>>>>>>> > >> > >>>>> > >> > config
>>>>>>> > >> > >>>>> > >> > file
>>>>>>> > >> > >>>>> > >> > for
>>>>>>> > >> > >>>>> > >> > this process they also can specify the
database
>>>>>>> > >> > >>>>> > >> > group. For
>>>>>>> > >> > >>>>> example
>>>>>>> > >> > >>>>> > >> > database
>>>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>>>>>> > >> > >>>>> > >> > After the data has been loaded to the
database
>>>>>>> it is
>>>>>>> > >> > >>>>> > >> > visible and
>>>>>>> > >> > >>>>> > >> > available
>>>>>>> > >> > >>>>> > >> > in the GUI and also can be used by the
batch
>>>>>>> mode.
>>>>>>> > >> > >>>>> > >> > You can always add more data to the
database
>>>>>>> using
>>>>>>> > >> > >>>>> > >> > the
>>>>>>> > >> > >>>>> > >> > same load
>>>>>>> > >> > >>>>> > >> > mode.
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> > Is there a guide for me to understand the
process
>>>>>>> > >> > >>>>> > >> > that you
>>>>>>> > >> > >>>>> > >> > are
>>>>>>> > >> > >>>>> > >> > referring to?
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek
>>>>>>> via RT
>>>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>>>>>> > >> > >>>>> > >> > wrote:
>>>>>>> > >> > >>>>> > >> >
>>>>>>> > >> > >>>>> > >> > > Hi Edward,
>>>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
servers and
>>>>>>> > >> > >>>>> > >> > > can't
>>>>>>> > >> > >>>>> > >> > > view your
>>>>>>> > >> > >>>>> > >> > > XML
>>>>>>> > >> > >>>>> > >> > > for the
>>>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue
>>>>>>> and I
>>>>>>> > >> > >>>>> > >> > > will
>>>>>>> > >> > >>>>> > >> > > take a
>>>>>>> > >> > >>>>> > >> > > look.
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
should
>>>>>>> NOT
>>>>>>> > >> > >>>>> > >> > > be
>>>>>>> > >> > >>>>> > >> > > changed
>>>>>>> > >> > >>>>> > >> > > and
>>>>>>> > >> > >>>>> > >> > > will be
>>>>>>> > >> > >>>>> > >> > > replaced by other scripts during the
batch
>>>>>>> > >> > >>>>> > >> > > execution.
>>>>>>> > >> > >>>>> > >> > > The data
>>>>>>> > >> > >>>>> > >> > > for
>>>>>>> > >> > >>>>> > >> > > the
>>>>>>> > >> > >>>>> > >> > > batch mode is coming from the database
and not
>>>>>>> from
>>>>>>> > >> > >>>>> > >> > > files. The
>>>>>>> > >> > >>>>> > >> > > name
>>>>>>> > >> > >>>>> > >> > > of the
>>>>>>> > >> > >>>>> > >> > > database is in <database> tag.
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
>>>>>>> > >> > >>>>> > >> > > They use
>>>>>>> > >> > >>>>> > >> > > METviewer
>>>>>>> > >> > >>>>> > >> > > load
>>>>>>> > >> > >>>>> > >> > > mode to load these files to the specific
>>>>>>> database.
>>>>>>> > >> > >>>>> > >> > > In
>>>>>>> > >> > >>>>> > >> > > the
>>>>>>> > >> > >>>>> config
>>>>>>> > >> > >>>>> > >> > > file
>>>>>>> > >> > >>>>> > >> > > for
>>>>>>> > >> > >>>>> > >> > > this process they also can specify the
database
>>>>>>> > >> > >>>>> > >> > > group.
>>>>>>> > >> > >>>>> > >> > > For
>>>>>>> > >> > >>>>> > >> > > example
>>>>>>> > >> > >>>>> > >> > > database
>>>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>>>>>> > >> > >>>>> > >> > > After the data has been loaded to the
database
>>>>>>> it
>>>>>>> > >> > >>>>> > >> > > is
>>>>>>> > >> > >>>>> > >> > > visible
>>>>>>> > >> > >>>>> and
>>>>>>> > >> > >>>>> > >> > > available
>>>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch
>>>>>>> mode.
>>>>>>> > >> > >>>>> > >> > > You can always add more data to the
database
>>>>>>> using
>>>>>>> > >> > >>>>> > >> > > the
>>>>>>> > >> > >>>>> > >> > > same
>>>>>>> > >> > >>>>> load
>>>>>>> > >> > >>>>> > >> > > mode.
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >> > > Tatiana
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>>>>>> > >> > >>>>> > >> > > wrote:
>>>>>>> > >> > >>>>> > >> > > > Good morning,
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
effort
>>>>>>> to
>>>>>>> > >> > >>>>> > >> > > > understand
>>>>>>> > >> > >>>>> > >> > > > metviewer,
>>>>>>> > >> > >>>>> > >> > > > test
>>>>>>> > >> > >>>>> > >> > > > different cases using the history tab,
>>>>>>> changing
>>>>>>> > >> > >>>>> > >> > > > information
>>>>>>> > >> > >>>>> to
>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>> > >> > >>>>> > >> > > > what
>>>>>>> > >> > >>>>> > >> > > > works and what does not, and make
connections
>>>>>>> > >> > >>>>> > >> > > > between
>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>> > >> > >>>>> GUI
>>>>>>> > >> > >>>>> > >> > > > settings
>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
level with
>>>>>>> > >> > >>>>> > >> > > > metviewer has
>>>>>>> > >> > >>>>> > >> > > > significantly
>>>>>>> > >> > >>>>> > >> > > > increased during this time, there are
some
>>>>>>> > >> > >>>>> > >> > > > remaining
>>>>>>> > >> > >>>>> > >> > > > issues
>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>> > >> > >>>>> > >> > > > I'm
>>>>>>> > >> > >>>>> > >> > > > hoping
>>>>>>> > >> > >>>>> > >> > > > to clear up.
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
running
>>>>>>> with
>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>> > >> > >>>>> > >> > > > statistic
>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>> > >> > >>>>> > >> > > > type:
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
provided
>>>>>>> me
>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>> > >> > >>>>> > >> > > > working
>>>>>>> > >> > >>>>> > >> > > > directory
>>>>>>> > >> > >>>>> > >> > > > to get
>>>>>>> > >> > >>>>> > >> > > > started which I copied over:
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I
adjusted and
>>>>>>> used
>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>> > >> > >>>>> > >> > > > run his
>>>>>>> > >> > >>>>> > >> > > > example.
>>>>>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
>>>>>>> produced a
>>>>>>> > >> > >>>>> > >> > > > CSI
>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>> > >> > >>>>> that
>>>>>>> > >> > >>>>> > >> > > > was
>>>>>>> > >> > >>>>> > >> > > > stored
>>>>>>> > >> > >>>>> > >> > > > here:
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>>
>>>>>>> > >>
>>>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > Running the script produced the
expected
>>>>>>> result.
>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>> > >> > >>>>> > >> > > > next
>>>>>>> > >> > >>>>> > >> > > > natural
>>>>>>> > >> > >>>>> > >> > > > step was
>>>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and create
a new
>>>>>>> xml
>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>> > >> > >>>>> > >> > > > called
>>>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
create a
>>>>>>> time
>>>>>>> > >> > >>>>> > >> > > > series
>>>>>>> > >> > >>>>> > >> > > > where
>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
instead of
>>>>>>> > >> > >>>>> > >> > > > obs_thresh.
>>>>>>> > >> > >>>>> > >> > > > Accordingly,
>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
>>>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>>>>>> > >> > >>>>> and
>>>>>>> > >> > >>>>> > >> > > > changed
>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was
also
>>>>>>> > >> > >>>>> > >> > > > tried).
>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>> > >> > >>>>> error
>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>> > >> > >>>>> > >> > > > resulted
>>>>>>> > >> > >>>>> > >> > > > is as follows:
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
>>>>>>> ('2020-05-01
>>>>>>> > >> > >>>>> > >> > > > 13:00:00',
>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
'2020-05-01
>>>>>>> > >> > >>>>> > >> > > > 16:00:00',
>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
'2020-05-01
>>>>>>> > >> > >>>>> > >> > > > 19:00:00',
>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
'2020-05-01
>>>>>>> > >> > >>>>> > >> > > > 22:00:00',
>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
'2020-05-02
>>>>>>> > >> > >>>>> > >> > > > 01:00:00',
>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
'2020-05-02
>>>>>>> > >> > >>>>> > >> > > > 04:00:00',
>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
'2020-05-02
>>>>>>> > >> > >>>>> > >> > > > 07:00:00',
>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
'2020-05-02
>>>>>>> > >> > >>>>> > >> > > > 10:00:00',
>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
h.stat_header_id
>>>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>>>>>> > >> > >>>>> > >> > > > No data was returned from database
>>>>>>> > >> > >>>>> > >> > > > 2020-06-09
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>>
>>>>>>> > >>
>>>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
ERROR:
>>>>>>> > >> > >>>>> > >> > > > Caught
>>>>>>> > >> > >>>>> > >> > > > class
>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException in
>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>>>>>> > >> > >>>>> > >> > > > res): result set contained no data
>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
>>>>>>> set
>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
>>>>>>> set
>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > Apparently there was no data during
this
>>>>>>> time. I
>>>>>>> > >> > >>>>> > >> > > > tried
>>>>>>> > >> > >>>>> > >> > > > changing
>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>> > >> > >>>>> > >> > > > range
>>>>>>> > >> > >>>>> > >> > > > of valid times and changed the valid
time to
>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>> > >> > >>>>> > >> > > > days
>>>>>>> > >> > >>>>> > >> > > > just in
>>>>>>> > >> > >>>>> > >> > > > case
>>>>>>> > >> > >>>>> > >> > > > data was missing. The issue was the
same.
>>>>>>> To
>>>>>>> > >> > >>>>> > >> > > > confirm
>>>>>>> > >> > >>>>> > >> > > > my
>>>>>>> > >> > >>>>> > >> > > > logic, I
>>>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
>>>>>>> metviewer,
>>>>>>> > >> > >>>>> > >> > > > which I
>>>>>>> > >> > >>>>> > >> > > > found
>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>> > >> > >>>>> > >> > > > work.
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > I think I lack an understanding of how
to
>>>>>>> set up
>>>>>>> > >> > >>>>> > >> > > > batch
>>>>>>> > >> > >>>>> > >> > > > mode
>>>>>>> > >> > >>>>> > >> > > > because
>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>> > >> > >>>>> > >> > > > don't
>>>>>>> > >> > >>>>> > >> > > > see where these definitions are being
>>>>>>> explicitly
>>>>>>> > >> > >>>>> > >> > > > defined:
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > <plot_spec>
>>>>>>> > >> > >>>>> > >> > > > <connection>
>>>>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>>
>>>>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>>>>>> > >> > >>>>> > >> > > > </connection>
>>>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>>>>>> > >> > >>>>> > >> > > > <folders>
>>>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>>>>> > >> > >>>>> > >> > > > </folders>
>>>>>>> > >> > >>>>> > >> > > > <plot>
>>>>>>> > >> > >>>>> > >> > > >
<template>series_plot.R_tmpl</template>
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
grepping all
>>>>>>> > >> > >>>>> > >> > > > available
>>>>>>> > >> > >>>>> files
>>>>>>> > >> > >>>>> > >> > > > does
>>>>>>> > >> > >>>>> > >> > > > not
>>>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being
defined, as
>>>>>>> well
>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>> > >> > >>>>> > >> > > > other
>>>>>>> > >> > >>>>> > >> > > > similarly
>>>>>>> > >> > >>>>> > >> > > > defined
>>>>>>> > >> > >>>>> > >> > > > variables. I think if I know how
these are
>>>>>>> being
>>>>>>> > >> > >>>>> > >> > > > defined,
>>>>>>> > >> > >>>>> > >> > > > then
>>>>>>> > >> > >>>>> > >> > > > I'll
>>>>>>> > >> > >>>>> > >> > > > be
>>>>>>> > >> > >>>>> > >> > > > able to pipe in my information and
process
>>>>>>> the
>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>> > >> > >>>>> > >> > > > using
>>>>>>> > >> > >>>>> met
>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>> > >> > >>>>> > >> > > > produce
>>>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
that?
>>>>>>> > >> > >>>>> > >> > > > Also, if
>>>>>>> > >> > >>>>> > >> > > > I want
>>>>>>> > >> > >>>>> > >> > > > to use
>>>>>>> > >> > >>>>> > >> > > > obs for
>>>>>>> > >> > >>>>> > >> > > > verification, do I add another line in
the
>>>>>>> XML
>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>> > >> > >>>>> > >> > > > designates
>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>> > >> > >>>>> > >> > > > location of obs? I assume that other
steps
>>>>>>> have
>>>>>>> > >> > >>>>> > >> > > > to be
>>>>>>> > >> > >>>>> > >> > > > run
>>>>>>> > >> > >>>>> > >> > > > before
>>>>>>> > >> > >>>>> > >> > > > all
>>>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form,
for
>>>>>>> example,
>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>> > >> > >>>>> > >> > > > would
>>>>>>> > >> > >>>>> have
>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>> > >> > >>>>> > >> > > > separately
>>>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then I
would
>>>>>>> want
>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>> > >> > >>>>> > >> > > > do that
>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>> > >> > >>>>> > >> > > > well.
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
>>>>>>> > >> > >>>>> > >> > > > here -
>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>> > >> > >>>>> > >> > > > attachment
>>>>>>> > >> > >>>>> > >> > > >
>>>>>>> > >> > >>>>> > >> > > > How do I get the results I generate to
be
>>>>>>> > >> > >>>>> > >> > > > uploaded to
>>>>>>> > >> > >>>>> > >> > > > this
>>>>>>> > >> > >>>>> > >> > > > location?
>>>>>>> > >> > >>>>> > >> > > > It's
>>>>>>> > >> > >>>>> > >> > > > not clear how others are setting this
up.
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >> > >
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>> > >>
>>>>>>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>>>>>>> > >> > >>>>> > >> rds_user
>>>>>>> > >> > >>>>> rds_pwd
>>>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
rds_plots
>>>>>>> rds_data
>>>>>>> > >> > >>>>> rds_scripts
>>>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>>>>>> > >> > >>>>> 2020-05-12
>>>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
19:00:00
>>>>>>> 2020-
>>>>>>> > >> > >>>>> > >> 05-12
>>>>>>> > >> > >>>>> 20:00:00
>>>>>>> > >> > >>>>> > >> 2020-05-12
>>>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
23:00:00
>>>>>>> 2020-
>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>> > >> > >>>>> 10:00:00
>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
13:00:00
>>>>>>> 2020-
>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>> > >> > >>>>> 14:00:00
>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
17:00:00
>>>>>>> 2020-
>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>> > >> > >>>>> 18:00:00
>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
21:00:00
>>>>>>> 2020-
>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>> > >> > >>>>> 22:00:00
>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
11:00:00
>>>>>>> 2020-
>>>>>>> > >> > >>>>> > >> 05-14
>>>>>>> > >> > >>>>> 12:00:00
>>>>>>> > >> > >>>>> > >> true
>>>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>>>>>> > >> > >>>>> > >> plot_20200612_173626.png
>>>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
>>>>>>> > >> > >>>>> > >> false list() list() false false false false
false
>>>>>>> false
>>>>>>> > >> > >>>>> > >> true
>>>>>>> > >> > >>>>> > >> false
>>>>>>> > >> > >>>>> > >> false
>>>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
>>>>>>> c(8,4,5,4)
>>>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>>>>>> > >> > >>>>> > >> 1.4 -2
>>>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1
1 -2
>>>>>>> 0.5 3
>>>>>>> > >> > >>>>> > >> #cccccc 1
>>>>>>> > >> > >>>>> > >> listX 1
>>>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o
c(0,
>>>>>>> -.25)
>>>>>>> > >> > >>>>> > >> 3
>>>>>>> > >> > >>>>> > >> 0.8 1
>>>>>>> > >> > >>>>> > >> #333333
>>>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
>>>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
c("b","b")
>>>>>>> c(1,1)
>>>>>>> > >> > >>>>> > >> c(1,1)
>>>>>>> > >> > >>>>> c(1,1)
>>>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>> > > --
>>>>>>> > >> > >>>>> > > Edward Strobach
>>>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>>>>>> > >> > >>>>> > > IMSG Contractor
>>>>>>> > >> > >>>>> > > Cubicle#: 2029
>>>>>>> > >> > >>>>> > > 301-683-3717
>>>>>>> > >> > >>>>> > >
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>>
>>>>>>> > >> > >>>>
>>>>>>> > >> > >>>> --
>>>>>>> > >> > >>>> Edward Strobach
>>>>>>> > >> > >>>> EMC/NCEP/NWS/
>>>>>>> > >> > >>>> IMSG Contractor
>>>>>>> > >> > >>>> Cubicle#: 2029
>>>>>>> > >> > >>>> 301-683-3717
>>>>>>> > >> > >>>>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>>
>>>>>>> > >> > >>> --
>>>>>>> > >> > >>> Edward Strobach
>>>>>>> > >> > >>> EMC/NCEP/NWS/
>>>>>>> > >> > >>> IMSG Contractor
>>>>>>> > >> > >>> Cubicle#: 2029
>>>>>>> > >> > >>> 301-683-3717
>>>>>>> > >> > >>>
>>>>>>> > >> > >>
>>>>>>> > >> > >>
>>>>>>> > >> > >> --
>>>>>>> > >> > >> Edward Strobach
>>>>>>> > >> > >> EMC/NCEP/NWS/
>>>>>>> > >> > >> IMSG Contractor
>>>>>>> > >> > >> Cubicle#: 2029
>>>>>>> > >> > >> 301-683-3717
>>>>>>> > >> > >>
>>>>>>> > >> > >
>>>>>>> > >> > >
>>>>>>> > >> > > --
>>>>>>> > >> > > Edward Strobach
>>>>>>> > >> > > EMC/NCEP/NWS/
>>>>>>> > >> > > IMSG Contractor
>>>>>>> > >> > > Cubicle#: 2029
>>>>>>> > >> > > 301-683-3717
>>>>>>> > >> > >
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >
>>>>>>> > > --
>>>>>>> > > Edward Strobach
>>>>>>> > > EMC/NCEP/NWS/
>>>>>>> > > IMSG Contractor
>>>>>>> > > Cubicle#: 2029
>>>>>>> > > 301-683-3717
>>>>>>> > >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Edward Strobach
>>>>>> EMC/NCEP/NWS/
>>>>>> IMSG Contractor
>>>>>> Cubicle#: 2029
>>>>>> 301-683-3717
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Edward Strobach
>>>>> EMC/NCEP/NWS/
>>>>> IMSG Contractor
>>>>> Cubicle#: 2029
>>>>> 301-683-3717
>>>>>
>>>> --
>>>> Jeffery T. McQueen
>>>> Meteorologist
>>>> National Center for Weather and Climate Prediction
>>>> Room 2095
>>>> 5830 University Research Court
>>>> College Park, MD 20741
>>>> 301-683-3736
>>>> Fax: 301-683-3703
>>>>
>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Wed Jun 17 13:30:51 2020
Ed,
Please chenge your connection and folders section in XML batch:
<connection>
<host>rds_host:3306</host>
<database>mv_cmaq_prod_metplus</database>
<user>rds_user</user>
<password>rds_pwd</password>
<management_system>aurora</management_system>
</connection>
<folders>
<r_tmpl>rds_R_tmpl</r_tmpl>
<r_work>rds_R_work</r_work>
<plots>rds_plots</plots>
<data>rds_data</data>
<scripts>rds_scripts</scripts>
</folders>
Tatiana
On Wed Jun 17 13:21:44 2020, edward.strobach at noaa.gov wrote:
> Hi Ho-Chun,
>
> I've made all the changes that I would need to inside my xml file.
I
> was
> told not to change the folders info. I instead changed my database
> name,
> fcst_var, stat options, series1 input, the fix fields, the indep
> field, and
> x_lim.
>
> Thanks
>
> On Wed, Jun 17, 2020 at 3:16 PM Ho-Chun Huang - NOAA Affiliate <
> ho-chun.huang at noaa.gov> wrote:
>
> > Hi, Ed:
> >
> > You should go back to read 1README again from the directory I sent
> > about
> > replacing the top portion of your metviewer xml to produce a batch
> > xml.
> >
> > I check your
> >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
> > Please update the section from <connection> to </folder> as those
in
> > hochun_xml.template. I also do not know whether AWS can handle
> > comments
> > <!-- to !--> in batch xml , better make your batch xml clean.
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> >> Just adding more detail to elaborate my concerns...
> >>
> >> Not too long ago I was given a test set-up by Ho-Chun to generate
a
> >> plot
> >> for aerosol optical depth. The result was successful and
produced a
> >> CSI
> >> plot for AOD, comparing the production run and experimental run.
> >> Although
> >> I created a database, I do not seem to have a group where I can
> >> access
> >> specific databases. I recently created a load_xml file, which
> >> should have
> >> loaded the data based on what I specified into the group. It
seemed
> >> successful based on the log output that was generated.
> >> Additionally, all
> >> information that I specified in this step was create in the
> >> load_*xml.
> >>
> >> The next step is to generate plots since the data should be
loaded.
> >> Here's what happens when I run Ho-Chun's - the successful case
> >>
> >> HO-CHUN:
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
> >> hochun_xml.template+ bash
> >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> >> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> >> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
> >> MVBatch
> >> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # #
#
> >> # # # #
> >> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT
model,
> >> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
> >>
> >> This is early on in the script.
> >>
> >> ED_STROBACH:
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed
-e
> >> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
> >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> >> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> >> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
> >> MVBatch
> >> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20
PM
> >> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
> >> create
> >> initial connections of pool.java.sql.SQLException: Communications
> >> link
> >> failure with primary. No active connection found for master.
> >> at
> >>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
> >> at
> >>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
> >> at
> >>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
> >> at
> >>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
> >> at
> >>
org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
> >> at
> >>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
> >> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
> >>
> >> My interpretation of this is that it fails to bridge a connection
to
> >> the
> >> group or database; therefore, it's unable to proceed with running
> >> the XML
> >> file to generate the plots based on the specified content. If I
> >> move past
> >> this step, then I'm golden.
> >>
> >> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
> >> jeff.mcqueen at noaa.gov> wrote:
> >>
> >>> Would it be ok to keep perry in the loop
> >>>
> >>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA
Affiliate <
> >>> edward.strobach at noaa.gov> wrote:
> >>>
> >>>> you're right. I didn't have the storage problem this time.
I'm
> >>>> not
> >>>> sure what changed. All I know is that I have now created my
load
> >>>> xml which
> >>>> now should allow me to move on to the next step of plotting
> >>>>
> >>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA
Affiliate
> >>>> <
> >>>> edward.strobach at noaa.gov> wrote:
> >>>>
> >>>>> I'm not following you. How do I now have enough space? What
> >>>>> changed?
> >>>>>
> >>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
> >>>>> met_help at ucar.edu> wrote:
> >>>>>
> >>>>>> Ed, now you should have enough space for the data loading.
> >>>>>>
> >>>>>> Tatiana
> >>>>>>
> >>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
> >>>>>> > I was basically able to build my xml file but it says I
have
> >>>>>> > no
> >>>>>> > available
> >>>>>> > space. I'm not sure how that can be since I just started
> >>>>>> > adding
> >>>>>> data
> >>>>>> > to my
> >>>>>> > account
> >>>>>> >
> >>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
> >>>>>> > space
> >>>>>> left
> >>>>>> > on
> >>>>>> > device
> >>>>>> > ERROR: Command returned with non-zero status (1): scp -r
./*
> >>>>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
> >>>>>> > 4) Check METviewer AWS database list using
> >>>>>> > mv_db_size_on_aws.sh
> >>>>>> > TABLE_SCHEMA USER
> >>>>>> > SIZE_MB
> >>>>>> > mv_met_g2o_gefs binbin.zhou
> >>>>>> > 140027.31
> >>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
> >>>>>> > 71986.11
> >>>>>> >
> >>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
> >>>>>> > Affiliate <
> >>>>>> > edward.strobach at noaa.gov> wrote:
> >>>>>> >
> >>>>>> > > Thanks Tatiana. Yes, I've created a database which seems
> >>>>>> different
> >>>>>> > > from
> >>>>>> > > mv_group. How can one create their own group so that I
can
> >>>>>> > > store
> >>>>>> and
> >>>>>> > > access
> >>>>>> > > the database. The database exists but cannot be viewed
on
> >>>>>> > > the
> >>>>>> > > metviewer
> >>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
> >>>>>> > > vhagerty. I
> >>>>>> > > named my
> >>>>>> > > database Ed_Strobach, but that was really intended for
the
> >>>>>> > > group
> >>>>>> > > name.
> >>>>>> > >
> >>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> >>>>>> > > <met_help at ucar.edu>
> >>>>>> > > wrote:
> >>>>>> > >
> >>>>>> > >> We don't have database and METviewer running on wcoss.
> >>>>>> > >> Instead, users use METviewer-wcoss scripts that help to
> >>>>>> > >> connect
> >>>>>> to
> >>>>>> > >> and
> >>>>>> > >> use METviewer that is running on AWS. That is why XML
> >>>>>> > >> connection
> >>>>>> and
> >>>>>> > >> folder
> >>>>>> > >> sections look somewhat different than once from the Docs
> >>>>>> > >> page.
> >>>>>> > >> To create a database on AWS you need to run
> >>>>>> mv_create_db_on_aws.sh
> >>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
> >>>>>> > >> For data loading :
> >>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> >>>>>> > >> And for batch:
> >>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> >>>>>> > >>
> >>>>>> > >> These scripts copy XML and data files if needed to AWS,
> >>>>>> > >> adjust
> >>>>>> data
> >>>>>> > >> path
> >>>>>> > >> and users credentials, run METviewer and copy results
back
> >>>>>> > >> to
> >>>>>> wcoss.
> >>>>>> > >>
> >>>>>> > >> Tatiana
> >>>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
> >>>>>> > >> wrote:
> >>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
> >>>>>> > >> > -p[db_password]
> >>>>>> > >> > -e'create
> >>>>>> > >> > database [db_name]
> >>>>>> > >> >
> >>>>>> > >> > would need to be used to create a database. I assume
> >>>>>> > >> > that an
> >>>>>> sql
> >>>>>> > >> > module is
> >>>>>> > >> > available on wcoss; I know you don't have access to
those
> >>>>>> machines
> >>>>>> > >> > Tatiana. If I create my own database with my own
name,
> >>>>>> > >> > then I
> >>>>>> > >> > should
> >>>>>> > >> > be
> >>>>>> > >> > able to proceed with project specific tasks under that
> >>>>>> > >> > tab.
> >>>>>> > >> > Something
> >>>>>> > >> > like
> >>>>>> > >> > mv_EdwardStrobach.
> >>>>>> > >> >
> >>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add data
to
> >>>>>> > >> > using
> >>>>>> > >> > load_spec.
> >>>>>> > >> > Once that step is done, then I can run plot_spec.
That's
> >>>>>> > >> > my
> >>>>>> > >> > takeaway
> >>>>>> > >> > currently
> >>>>>> > >> >
> >>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
> >>>>>> Affiliate <
> >>>>>> > >> > edward.strobach at noaa.gov> wrote:
> >>>>>> > >> >
> >>>>>> > >> > > Actually, it appears that I need to run the database
> >>>>>> > >> > > loading
> >>>>>> > >> > > module
> >>>>>> > >> > > first
> >>>>>> > >> > > if I want to change/add a data source. From there I
> >>>>>> > >> > > can run
> >>>>>> the
> >>>>>> > >> > > plotting
> >>>>>> > >> > > batch module. Is that right? So if I run the
> >>>>>> > >> > > database
> >>>>>> loading
> >>>>>> > >> > > module,
> >>>>>> > >> > > then I can specify my dates and other related info
> >>>>>> > >> > > specific
> >>>>>> to
> >>>>>> > >> > > the
> >>>>>> > >> > > dataset. If I create my database with a unique
name,
> >>>>>> > >> > > then it
> >>>>>> > >> > > should
> >>>>>> > >> > > be
> >>>>>> > >> > > recognized on the metviewer gui, presumably.
> >>>>>> > >> > >
> >>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach -
NOAA
> >>>>>> Affiliate
> >>>>>> > >> > > <
> >>>>>> > >> > > edward.strobach at noaa.gov> wrote:
> >>>>>> > >> > >
> >>>>>> > >> > >> It seems that those options must be treated
separately
> >>>>>> outside
> >>>>>> > >> > >> of
> >>>>>> > >> > >> plot_spec. Is it acceptable to create both a
> >>>>>> > >> > >> load_spec and
> >>>>>> > >> > >> plot_spec in
> >>>>>> > >> > >> the same xml file? Would the definitions in
load_spec
> >>>>>> > >> > >> be
> >>>>>> able
> >>>>>> > >> > >> to
> >>>>>> > >> > >> carry
> >>>>>> > >> > >> over into plot_spec? It appears that data loading
and
> >>>>>> plotting
> >>>>>> > >> > >> are
> >>>>>> > >> > >> treated
> >>>>>> > >> > >> separately.
> >>>>>> > >> > >>
> >>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
NOAA
> >>>>>> > >> > >> Affiliate <
> >>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
> >>>>>> > >> > >>
> >>>>>> > >> > >>> I do plan to use date_range; however, in an
example
> >>>>>> provided
> >>>>>> > >> > >>> by the
> >>>>>> > >> > >>> link
> >>>>>> > >> > >>> I see this:
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> <load_spec>
> >>>>>> > >> > >>> <connection>
> >>>>>> > >> > >>> <host>kemosabe:3306</host>
> >>>>>> > >> > >>> <database>metvdb_hwt</database>
> >>>>>> > >> > >>> <user>pgoldenb</user>
> >>>>>> > >> > >>> <password>pgoldenb</password>
> >>>>>> > >> > >>> </connection>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> <date_list name="folder_dates">
> >>>>>> > >> > >>> <start>2010051914V</start>
> >>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
> >>>>>> > >> > >>> </hour></date_offset></end>
> >>>>>> > >> > >>> <inc>3600</inc>
> >>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
> >>>>>> > >> > >>> </date_list>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> <met_version>V3.0</met_version>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> <verbose>false</verbose>
> >>>>>> > >> > >>> <insert_size>1</insert_size>
> >>>>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
> >>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
> >>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
> >>>>>> > >> > >>> <group>Group name</group>
> >>>>>> > >> > >>> <load_stat>true</load_stat>
> >>>>>> > >> > >>> <load_mode>true</load_mode>
> >>>>>> > >> > >>> <load_mtd>true</load_mtd>
> >>>>>> > >> > >>> <load_mpr>false</load_mpr>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>>
> >>>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> >>>>>> > >> > >>> <load_val>
> >>>>>> > >> > >>> <field name="model">
> >>>>>> > >> > >>> <val>arw</val>
> >>>>>> > >> > >>> <val>nmm</val>
> >>>>>> > >> > >>> </field>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> <field name="valid_time">
> >>>>>> > >> > >>> <date_list name="folder_dates"/>
> >>>>>> > >> > >>> </field>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> <field name="vx_mask">
> >>>>>> > >> > >>> <val>FULL</val>
> >>>>>> > >> > >>> <val>SWC</val>
> >>>>>> > >> > >>> </field>
> >>>>>> > >> > >>> </load_val>
> >>>>>> > >> > >>> </load_spec>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> Are similar settings allowed with <plot_spec>? I
see
> >>>>>> > >> > >>> <connection>,
> >>>>>> > >> > >>> which seems carried over. I'm looking to specify
my
> >>>>>> > >> > >>> folder_tmpl in
> >>>>>> > >> > >>> <plot_spec>, which according to Tara, should be
> >>>>>> > >> > >>> possible.
> >>>>>> > >> > >>> Since my
> >>>>>> > >> > >>> directory structure looks like this:
> >>>>>> > >> > >>>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>>
> >>>>>> > >>
> >>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> I would do something like this I would think:
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> .......
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> <date_list name="folder_dates">
> >>>>>> > >> > >>> <start>20200612/start>
> >>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset></end>
> >>>>>> > >> > >>> <inc>86400</inc>
> >>>>>> > >> > >>> <format>yyyyMMdd</format>
> >>>>>> > >> > >>> </date_list>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> <folder_tmpl>
> >>>>>> > >> > >>>
> >>>>>> > >>
> >>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> >>>>>> > >> > >>> </folder_tmpl>
> >>>>>> > >> > >>> <load_val>
> >>>>>> > >> > >>> <field name="model">
> >>>>>> > >> > >>> <val>prod</val>
> >>>>>> > >> > >>> </field>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> <field name="valid_time">
> >>>>>> > >> > >>> <date_list name="folder_dates"/>
> >>>>>> > >> > >>> </field>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> ......
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach -
> >>>>>> > >> > >>> NOAA
> >>>>>> > >> > >>> Affiliate <
> >>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
> >>>>>> > >> > >>>
> >>>>>> > >> > >>>> Thank you. This is a very helpful example
> >>>>>> > >> > >>>>
> >>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via
RT
> >>>>>> > >> > >>>> <met_help at ucar.edu>
> >>>>>> > >> > >>>> wrote:
> >>>>>> > >> > >>>>
> >>>>>> > >> > >>>>> Hi,
> >>>>>> > >> > >>>>> Here is some examples how to use <date_range>
and
> >>>>>> > >> > >>>>> <date_range_list> in
> >>>>>> > >> > >>>>> batch XML.
> >>>>>> > >> > >>>>> First, you need to defined your range. You can
have
> >>>>>> > >> > >>>>> more
> >>>>>> > >> > >>>>> then one
> >>>>>> > >> > >>>>> <date_range>:
> >>>>>> > >> > >>>>> <date_range name="may">
> >>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
> >>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
> >>>>>> > >> > >>>>> </date_range>
> >>>>>> > >> > >>>>> <date_range name="march">
> >>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
> >>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
> >>>>>> > >> > >>>>> </date_range>
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>> and then use one or all date_ranges:
> >>>>>> > >> > >>>>> <plot_fix>
> >>>>>> > >> > >>>>> <field name="fcst_init_beg">
> >>>>>> > >> > >>>>> <date_range name="march"/>
> >>>>>> > >> > >>>>> <date_range name="may"/>
> >>>>>> > >> > >>>>> </field>...
> >>>>>> > >> > >>>>> This configuration would include all dates for
> >>>>>> > >> > >>>>> March and
> >>>>>> May
> >>>>>> > >> > >>>>> of
> >>>>>> > >> > >>>>> 2019.
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>> If you need more flexible configuration you can
use
> >>>>>> > >> > >>>>> date_range_list.
> >>>>>> > >> > >>>>> For example, you want to select first 6 hours
every
> >>>>>> > >> > >>>>> day
> >>>>>> > >> > >>>>> beetween
> >>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00.
Your
> >>>>>> > >> > >>>>> XML
> >>>>>> would
> >>>>>> > >> > >>>>> look
> >>>>>> > >> > >>>>> like this:
> >>>>>> > >> > >>>>> <date_range_list name = 'all'>
> >>>>>> > >> > >>>>> <range_start>2020-01-01
> >>>>>> > >> > >>>>> 00:00:00</range_start>
> >>>>>> > >> > >>>>> <range_end>2020-01-05
00:00:00</range_end>
> >>>>>> > >> > >>>>> <range_length>360</range_length>
> >>>>>> > >> > >>>>> <inc>1440</inc>
> >>>>>> > >> > >>>>> </date_range_list>
> >>>>>> > >> > >>>>> where the 1st and 2nd values are the start and
end,
> >>>>>> > >> > >>>>> range_length - the length of a range in
minutes(6
> >>>>>> > >> > >>>>> houres)
> >>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
houres)
> >>>>>> > >> > >>>>> This example would create following ranges:
> >>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
> >>>>>> > >> > >>>>> 06:00:00'
> >>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
> >>>>>> > >> > >>>>> 06:00:00'
> >>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
> >>>>>> > >> > >>>>> 06:00:00'
> >>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
> >>>>>> > >> > >>>>> 06:00:00'
> >>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
> >>>>>> > >> > >>>>> 06:00:00'
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>> Also you can add more parameters to <plot_fix>
to
> >>>>>> > >> > >>>>> filter
> >>>>>> > >> > >>>>> your
> >>>>>> > >> > >>>>> data.
> >>>>>> > >> > >>>>> For example, if you want to select only dates
with
> >>>>>> > >> > >>>>> 00:00
> >>>>>> you
> >>>>>> > >> > >>>>> can
> >>>>>> > >> > >>>>> specify
> >>>>>> > >> > >>>>> <field name="valid_hour">
> >>>>>> > >> > >>>>> <val>00</val>
> >>>>>> > >> > >>>>> </field>
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>> Tatiana
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
> >>>>>> > >> > >>>>> edward.strobach at noaa.gov
> >>>>>> wrote:
> >>>>>> > >> > >>>>> > and if there isn't a better way, I wonder the
> >>>>>> whereabouts
> >>>>>> > >> > >>>>> > of a
> >>>>>> > >> > >>>>> > separate
> >>>>>> > >> > >>>>> > script that pipes dates (and other relevant
> >>>>>> information)
> >>>>>> > >> > >>>>> > into
> >>>>>> > >> > >>>>> > the XML
> >>>>>> > >> > >>>>> > file
> >>>>>> > >> > >>>>> > used to generate the results.
> >>>>>> > >> > >>>>> >
> >>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward
Strobach
> >>>>>> > >> > >>>>> > - NOAA
> >>>>>> > >> > >>>>> > Affiliate <
> >>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
> >>>>>> > >> > >>>>> >
> >>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are
helpful.
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>> > > One last question. Simply typing lines like
> >>>>>> > >> > >>>>> > > this in
> >>>>>> can
> >>>>>> > >> > >>>>> > > take
> >>>>>> > >> > >>>>> > > forever:
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>> > > <plot_fix>
> >>>>>> > >> > >>>>> > > <field equalize="false"
> >>>>>> > >> > >>>>> > > name="fcst_init_beg">
> >>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
> >>>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
> >>>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
> >>>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
> >>>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
> >>>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>> > > Is there a better way? I see date_range,
> >>>>>> > >> > >>>>> > > date_range_list,
> >>>>>> > >> > >>>>> > > or
> >>>>>> > >> > >>>>> > > date_list.
> >>>>>> > >> > >>>>> > > Can this be used instead of typing
individual
> >>>>>> > >> > >>>>> > > values?
> >>>>>> > >> > >>>>> > > This
> >>>>>> > >> > >>>>> > > could
> >>>>>> > >> > >>>>> > > prove
> >>>>>> > >> > >>>>> > > pretty daunting if one would have to type in
> >>>>>> > >> > >>>>> > > 700
> >>>>>> entries
> >>>>>> > >> > >>>>> > > each
> >>>>>> > >> > >>>>> > > time.
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana
Burek
> >>>>>> > >> > >>>>> > > via RT
> >>>>>> > >> > >>>>> > > <met_help at ucar.edu>
> >>>>>> > >> > >>>>> > > wrote:
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>> > >> I found a few problems in your XML.
> >>>>>> > >> > >>>>> > >> The database you are using doesn't have
data
> >>>>>> > >> > >>>>> > >> with
> >>>>>> > >> > >>>>> > >> vx_mask
> >>>>>> > >> > >>>>> > >> 'abcREGabc'.
> >>>>>> > >> > >>>>> > >> That is why METviewer returns an empty
dataset
> >>>>>> > >> > >>>>> > >> and
> >>>>>> > >> > >>>>> > >> can't
> >>>>>> > >> > >>>>> > >> create a
> >>>>>> > >> > >>>>> > >> plot.
> >>>>>> > >> > >>>>> > >> The other problem is how you define
fcst_lead.
> >>>>>> > >> > >>>>> > >> This
> >>>>>> > >> > >>>>> > >> database
> >>>>>> > >> > >>>>> > >> has
> >>>>>> > >> > >>>>> > >> values
> >>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
> >>>>>> > >> > >>>>> > >> Since you
> >>>>>> > >> > >>>>> > >> include
> >>>>>> > >> > >>>>> > >> all
> >>>>>> > >> > >>>>> > >> fcst_leads
> >>>>>> > >> > >>>>> > >> this variable can be omitted.
> >>>>>> > >> > >>>>> > >> To minimize problems with variables values,
I
> >>>>>> > >> > >>>>> > >> would
> >>>>>> > >> > >>>>> > >> suggest
> >>>>>> > >> > >>>>> > >> to
> >>>>>> > >> > >>>>> > >> create XML
> >>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
> >>>>>> > >> > >>>>> > >> standards.
> >>>>>> > >> > >>>>> > >> I created a plot using you configurations
with
> >>>>>> > >> > >>>>> > >> GUI
> >>>>>> > >> > >>>>> > >> (without
> >>>>>> > >> > >>>>> vx_mack
> >>>>>> > >> > >>>>> > >> and
> >>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You
> >>>>>> > >> > >>>>> > >> can
> >>>>>> load it
> >>>>>> > >> > >>>>> > >> back
> >>>>>> > >> > >>>>> > >> to
> >>>>>> > >> > >>>>> the
> >>>>>> > >> > >>>>> > >> page by
> >>>>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI).
> >>>>>> > >> > >>>>> > >> Find the
> >>>>>> > >> > >>>>> > >> plot,
> >>>>>> > >> > >>>>> > >> open XML
> >>>>>> > >> > >>>>> > >> and
> >>>>>> > >> > >>>>> > >> click on "Load to page". After that you can
> >>>>>> recreate a
> >>>>>> > >> > >>>>> > >> plot
> >>>>>> > >> > >>>>> > >> and
> >>>>>> > >> > >>>>> edit
> >>>>>> > >> > >>>>> > >> the
> >>>>>> > >> > >>>>> > >> configurations. You can also download a new
> >>>>>> > >> > >>>>> > >> XML and
> >>>>>> > >> > >>>>> > >> edit it
> >>>>>> > >> > >>>>> > >> for
> >>>>>> > >> > >>>>> > >> batch.
> >>>>>> > >> > >>>>> > >> I attached the example of load XML to this
> >>>>>> > >> > >>>>> > >> message.
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>> > >> Description of rds parameters:
> >>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
> >>>>>> > >> > >>>>> > >> METviewer R
> >>>>>> > >> > >>>>> > >> scripts
> >>>>>> > >> > >>>>> > >> rds_plots - temporary directory where the
> >>>>>> > >> > >>>>> > >> plots
> >>>>>> will be
> >>>>>> > >> > >>>>> > >> generated
> >>>>>> > >> > >>>>> > >> rds_data - temporary directory where the
data
> >>>>>> retrieved
> >>>>>> > >> > >>>>> > >> from
> >>>>>> > >> > >>>>> > >> the
> >>>>>> > >> > >>>>> > >> database
> >>>>>> > >> > >>>>> > >> and other intermediate files will be stored
> >>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
> >>>>>> > >> > >>>>> > >> scripts
> >>>>>> for
> >>>>>> > >> > >>>>> > >> this
> >>>>>> > >> > >>>>> > >> run
> >>>>>> > >> > >>>>> will
> >>>>>> > >> > >>>>> > >> be
> >>>>>> > >> > >>>>> > >> stored
> >>>>>> > >> > >>>>> > >> After the execution is done all the files
from
> >>>>>> > >> > >>>>> > >> temporary
> >>>>>> > >> > >>>>> directories
> >>>>>> > >> > >>>>> > >> would be removed.
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>> > >> We don't have the offitial user guide for
> >>>>>> > >> > >>>>> > >> METviewer.
> >>>>>> > >> > >>>>> > >> There
> >>>>>> > >> > >>>>> > >> are
> >>>>>> > >> > >>>>> some
> >>>>>> > >> > >>>>> > >> instructions on how to crate XML config
files:
> >>>>>> > >> > >>>>> > >>
https://metviewer.nws.noaa.gov/doc/index.html
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>> > >> Tatiana
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
> >>>>>> edward.strobach at noaa.gov
> >>>>>> > >> > >>>>> > >> wrote:
> >>>>>> > >> > >>>>> > >> > Hi,
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am
having
> >>>>>> > >> > >>>>> > >> > hard
> >>>>>> time
> >>>>>> > >> > >>>>> > >> > following
> >>>>>> > >> > >>>>> how
> >>>>>> > >> > >>>>> > >> > to
> >>>>>> > >> > >>>>> > >> > go
> >>>>>> > >> > >>>>> > >> > about the other steps since I have zero
> >>>>>> experience on
> >>>>>> > >> > >>>>> > >> > how
> >>>>>> > >> > >>>>> > >> > to do
> >>>>>> > >> > >>>>> > >> > these
> >>>>>> > >> > >>>>> > >> > things.
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
> >>>>>> > >> > >>>>> > >> > should
> >>>>>> NOT be
> >>>>>> > >> > >>>>> > >> > changed
> >>>>>> > >> > >>>>> and
> >>>>>> > >> > >>>>> > >> > will
> >>>>>> > >> > >>>>> > >> > be
> >>>>>> > >> > >>>>> > >> > replaced by other scripts during the
batch
> >>>>>> execution.
> >>>>>> > >> > >>>>> > >> > The
> >>>>>> > >> > >>>>> > >> > data
> >>>>>> > >> > >>>>> for
> >>>>>> > >> > >>>>> > >> > the
> >>>>>> > >> > >>>>> > >> > batch mode is coming from the database
and
> >>>>>> > >> > >>>>> > >> > not
> >>>>>> from
> >>>>>> > >> > >>>>> > >> > files.
> >>>>>> > >> > >>>>> > >> > The
> >>>>>> > >> > >>>>> > >> > name of
> >>>>>> > >> > >>>>> > >> > the
> >>>>>> > >> > >>>>> > >> > database is in <database> tag.
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> > I understand that nothing should be
changed
> >>>>>> > >> > >>>>> > >> > in the
> >>>>>> > >> > >>>>> > >> > folders
> >>>>>> > >> > >>>>> > >> > section,
> >>>>>> > >> > >>>>> > >> > but
> >>>>>> > >> > >>>>> > >> > there is no indication how and where
these
> >>>>>> variables
> >>>>>> > >> > >>>>> > >> > are
> >>>>>> > >> > >>>>> > >> > being
> >>>>>> > >> > >>>>> > >> > defined.
> >>>>>> > >> > >>>>> > >> > Grepping these names of the files
available
> >>>>>> > >> > >>>>> > >> > does
> >>>>>> not
> >>>>>> > >> > >>>>> > >> > reveal
> >>>>>> > >> > >>>>> formal
> >>>>>> > >> > >>>>> > >> > definitions of these variables. If I
> >>>>>> > >> > >>>>> > >> > understood
> >>>>>> > >> > >>>>> > >> > exactly
> >>>>>> > >> > >>>>> > >> > how
> >>>>>> > >> > >>>>> these
> >>>>>> > >> > >>>>> > >> > were
> >>>>>> > >> > >>>>> > >> > being defined, then I would be able
connect
> >>>>>> > >> > >>>>> > >> > this
> >>>>>> > >> > >>>>> > >> > better.
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
> >>>>>> > >> > >>>>> > >> > files.
> >>>>>> They
> >>>>>> > >> > >>>>> > >> > use
> >>>>>> > >> > >>>>> METviewer
> >>>>>> > >> > >>>>> > >> > load
> >>>>>> > >> > >>>>> > >> > mode to load these files to the specific
> >>>>>> database. In
> >>>>>> > >> > >>>>> > >> > the
> >>>>>> > >> > >>>>> > >> > config
> >>>>>> > >> > >>>>> > >> > file
> >>>>>> > >> > >>>>> > >> > for
> >>>>>> > >> > >>>>> > >> > this process they also can specify the
> >>>>>> > >> > >>>>> > >> > database
> >>>>>> > >> > >>>>> > >> > group. For
> >>>>>> > >> > >>>>> example
> >>>>>> > >> > >>>>> > >> > database
> >>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
> >>>>>> > >> > >>>>> > >> > After the data has been loaded to the
> >>>>>> > >> > >>>>> > >> > database it
> >>>>>> is
> >>>>>> > >> > >>>>> > >> > visible and
> >>>>>> > >> > >>>>> > >> > available
> >>>>>> > >> > >>>>> > >> > in the GUI and also can be used by the
batch
> >>>>>> > >> > >>>>> > >> > mode.
> >>>>>> > >> > >>>>> > >> > You can always add more data to the
database
> >>>>>> > >> > >>>>> > >> > using
> >>>>>> > >> > >>>>> > >> > the
> >>>>>> > >> > >>>>> > >> > same load
> >>>>>> > >> > >>>>> > >> > mode.
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> > Is there a guide for me to understand the
> >>>>>> > >> > >>>>> > >> > process
> >>>>>> > >> > >>>>> > >> > that you
> >>>>>> > >> > >>>>> > >> > are
> >>>>>> > >> > >>>>> > >> > referring to?
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
> >>>>>> > >> > >>>>> > >> > Burek
> >>>>>> via RT
> >>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
> >>>>>> > >> > >>>>> > >> > wrote:
> >>>>>> > >> > >>>>> > >> >
> >>>>>> > >> > >>>>> > >> > > Hi Edward,
> >>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
servers
> >>>>>> > >> > >>>>> > >> > > and
> >>>>>> > >> > >>>>> > >> > > can't
> >>>>>> > >> > >>>>> > >> > > view your
> >>>>>> > >> > >>>>> > >> > > XML
> >>>>>> > >> > >>>>> > >> > > for the
> >>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
> >>>>>> > >> > >>>>> > >> > > issue
> >>>>>> and I
> >>>>>> > >> > >>>>> > >> > > will
> >>>>>> > >> > >>>>> > >> > > take a
> >>>>>> > >> > >>>>> > >> > > look.
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
> >>>>>> > >> > >>>>> > >> > > should
> >>>>>> NOT
> >>>>>> > >> > >>>>> > >> > > be
> >>>>>> > >> > >>>>> > >> > > changed
> >>>>>> > >> > >>>>> > >> > > and
> >>>>>> > >> > >>>>> > >> > > will be
> >>>>>> > >> > >>>>> > >> > > replaced by other scripts during the
batch
> >>>>>> > >> > >>>>> > >> > > execution.
> >>>>>> > >> > >>>>> > >> > > The data
> >>>>>> > >> > >>>>> > >> > > for
> >>>>>> > >> > >>>>> > >> > > the
> >>>>>> > >> > >>>>> > >> > > batch mode is coming from the database
and
> >>>>>> > >> > >>>>> > >> > > not
> >>>>>> from
> >>>>>> > >> > >>>>> > >> > > files. The
> >>>>>> > >> > >>>>> > >> > > name
> >>>>>> > >> > >>>>> > >> > > of the
> >>>>>> > >> > >>>>> > >> > > database is in <database> tag.
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET
output
> >>>>>> > >> > >>>>> > >> > > files.
> >>>>>> > >> > >>>>> > >> > > They use
> >>>>>> > >> > >>>>> > >> > > METviewer
> >>>>>> > >> > >>>>> > >> > > load
> >>>>>> > >> > >>>>> > >> > > mode to load these files to the
specific
> >>>>>> database.
> >>>>>> > >> > >>>>> > >> > > In
> >>>>>> > >> > >>>>> > >> > > the
> >>>>>> > >> > >>>>> config
> >>>>>> > >> > >>>>> > >> > > file
> >>>>>> > >> > >>>>> > >> > > for
> >>>>>> > >> > >>>>> > >> > > this process they also can specify the
> >>>>>> > >> > >>>>> > >> > > database
> >>>>>> > >> > >>>>> > >> > > group.
> >>>>>> > >> > >>>>> > >> > > For
> >>>>>> > >> > >>>>> > >> > > example
> >>>>>> > >> > >>>>> > >> > > database
> >>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
> >>>>>> > >> > >>>>> > >> > > After the data has been loaded to the
> >>>>>> > >> > >>>>> > >> > > database
> >>>>>> it
> >>>>>> > >> > >>>>> > >> > > is
> >>>>>> > >> > >>>>> > >> > > visible
> >>>>>> > >> > >>>>> and
> >>>>>> > >> > >>>>> > >> > > available
> >>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
> >>>>>> > >> > >>>>> > >> > > batch
> >>>>>> mode.
> >>>>>> > >> > >>>>> > >> > > You can always add more data to the
> >>>>>> > >> > >>>>> > >> > > database
> >>>>>> using
> >>>>>> > >> > >>>>> > >> > > the
> >>>>>> > >> > >>>>> > >> > > same
> >>>>>> > >> > >>>>> load
> >>>>>> > >> > >>>>> > >> > > mode.
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >> > > Tatiana
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> >>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
> >>>>>> > >> > >>>>> > >> > > wrote:
> >>>>>> > >> > >>>>> > >> > > > Good morning,
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
> >>>>>> > >> > >>>>> > >> > > > effort to
> >>>>>> > >> > >>>>> > >> > > > understand
> >>>>>> > >> > >>>>> > >> > > > metviewer,
> >>>>>> > >> > >>>>> > >> > > > test
> >>>>>> > >> > >>>>> > >> > > > different cases using the history
tab,
> >>>>>> changing
> >>>>>> > >> > >>>>> > >> > > > information
> >>>>>> > >> > >>>>> to
> >>>>>> > >> > >>>>> > >> > > > see
> >>>>>> > >> > >>>>> > >> > > > what
> >>>>>> > >> > >>>>> > >> > > > works and what does not, and make
> >>>>>> > >> > >>>>> > >> > > > connections
> >>>>>> > >> > >>>>> > >> > > > between
> >>>>>> > >> > >>>>> > >> > > > the
> >>>>>> > >> > >>>>> GUI
> >>>>>> > >> > >>>>> > >> > > > settings
> >>>>>> > >> > >>>>> > >> > > > and
> >>>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
level
> >>>>>> > >> > >>>>> > >> > > > with
> >>>>>> > >> > >>>>> > >> > > > metviewer has
> >>>>>> > >> > >>>>> > >> > > > significantly
> >>>>>> > >> > >>>>> > >> > > > increased during this time, there are
> >>>>>> > >> > >>>>> > >> > > > some
> >>>>>> > >> > >>>>> > >> > > > remaining
> >>>>>> > >> > >>>>> > >> > > > issues
> >>>>>> > >> > >>>>> > >> > > > that
> >>>>>> > >> > >>>>> > >> > > > I'm
> >>>>>> > >> > >>>>> > >> > > > hoping
> >>>>>> > >> > >>>>> > >> > > > to clear up.
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
> >>>>>> > >> > >>>>> > >> > > > running with
> >>>>>> > >> > >>>>> > >> > > > different
> >>>>>> > >> > >>>>> > >> > > > statistic
> >>>>>> > >> > >>>>> > >> > > > and
> >>>>>> > >> > >>>>> > >> > > > plot
> >>>>>> > >> > >>>>> > >> > > > type:
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
> >>>>>> > >> > >>>>> > >> > > > provided me
> >>>>>> > >> > >>>>> > >> > > > the
> >>>>>> > >> > >>>>> > >> > > > working
> >>>>>> > >> > >>>>> > >> > > > directory
> >>>>>> > >> > >>>>> > >> > > > to get
> >>>>>> > >> > >>>>> > >> > > > started which I copied over:
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I
adjusted
> >>>>>> > >> > >>>>> > >> > > > and
> >>>>>> used
> >>>>>> > >> > >>>>> > >> > > > to
> >>>>>> > >> > >>>>> > >> > > > run his
> >>>>>> > >> > >>>>> > >> > > > example.
> >>>>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
> >>>>>> > >> > >>>>> > >> > > > produced
> >>>>>> a
> >>>>>> > >> > >>>>> > >> > > > CSI
> >>>>>> > >> > >>>>> > >> > > > plot
> >>>>>> > >> > >>>>> that
> >>>>>> > >> > >>>>> > >> > > > was
> >>>>>> > >> > >>>>> > >> > > > stored
> >>>>>> > >> > >>>>> > >> > > > here:
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>>
> >>>>>> > >>
> >>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > Running the script produced the
expected
> >>>>>> result.
> >>>>>> > >> > >>>>> > >> > > > The
> >>>>>> > >> > >>>>> > >> > > > next
> >>>>>> > >> > >>>>> > >> > > > natural
> >>>>>> > >> > >>>>> > >> > > > step was
> >>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and
create a
> >>>>>> > >> > >>>>> > >> > > > new
> >>>>>> xml
> >>>>>> > >> > >>>>> > >> > > > file
> >>>>>> > >> > >>>>> > >> > > > called
> >>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
create
> >>>>>> > >> > >>>>> > >> > > > a
> >>>>>> time
> >>>>>> > >> > >>>>> > >> > > > series
> >>>>>> > >> > >>>>> > >> > > > where
> >>>>>> > >> > >>>>> > >> > > > the
> >>>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
> >>>>>> > >> > >>>>> > >> > > > instead of
> >>>>>> > >> > >>>>> > >> > > > obs_thresh.
> >>>>>> > >> > >>>>> > >> > > > Accordingly,
> >>>>>> > >> > >>>>> > >> > > > I
> >>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
> >>>>>> > >> > >>>>> > >> > > > (removing
> >>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
> >>>>>> > >> > >>>>> and
> >>>>>> > >> > >>>>> > >> > > > changed
> >>>>>> > >> > >>>>> > >> > > > the
> >>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was
> >>>>>> > >> > >>>>> > >> > > > also
> >>>>>> > >> > >>>>> > >> > > > tried).
> >>>>>> > >> > >>>>> > >> > > > The
> >>>>>> > >> > >>>>> error
> >>>>>> > >> > >>>>> > >> > > > that
> >>>>>> > >> > >>>>> > >> > > > resulted
> >>>>>> > >> > >>>>> > >> > > > is as follows:
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
> >>>>>> > >> > >>>>> > >> > > > ('2020-05-01
> >>>>>> > >> > >>>>> > >> > > > 13:00:00',
> >>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>> > >> > >>>>> > >> > > > 01
> >>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
'2020-
> >>>>>> > >> > >>>>> > >> > > > 05-01
> >>>>>> > >> > >>>>> > >> > > > 16:00:00',
> >>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>> > >> > >>>>> > >> > > > 01
> >>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
'2020-
> >>>>>> > >> > >>>>> > >> > > > 05-01
> >>>>>> > >> > >>>>> > >> > > > 19:00:00',
> >>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>> > >> > >>>>> > >> > > > 01
> >>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
'2020-
> >>>>>> > >> > >>>>> > >> > > > 05-01
> >>>>>> > >> > >>>>> > >> > > > 22:00:00',
> >>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>> > >> > >>>>> > >> > > > 01
> >>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
'2020-
> >>>>>> > >> > >>>>> > >> > > > 05-02
> >>>>>> > >> > >>>>> > >> > > > 01:00:00',
> >>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>> > >> > >>>>> > >> > > > 02
> >>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
'2020-
> >>>>>> > >> > >>>>> > >> > > > 05-02
> >>>>>> > >> > >>>>> > >> > > > 04:00:00',
> >>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>> > >> > >>>>> > >> > > > 02
> >>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
'2020-
> >>>>>> > >> > >>>>> > >> > > > 05-02
> >>>>>> > >> > >>>>> > >> > > > 07:00:00',
> >>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>> > >> > >>>>> > >> > > > 02
> >>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
'2020-
> >>>>>> > >> > >>>>> > >> > > > 05-02
> >>>>>> > >> > >>>>> > >> > > > 10:00:00',
> >>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>> > >> > >>>>> > >> > > > 02
> >>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> >>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> >>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
h.stat_header_id
> >>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
> >>>>>> > >> > >>>>> > >> > > > No data was returned from database
> >>>>>> > >> > >>>>> > >> > > > 2020-06-09
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>>
> >>>>>> > >>
> >>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
> >>>>>> > >> > >>>>> > >> > > > ERROR:
> >>>>>> > >> > >>>>> > >> > > > Caught
> >>>>>> > >> > >>>>> > >> > > > class
> >>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> >>>>>> > >> > >>>>> > >> > > > in
> >>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
> >>>>>> > >> > >>>>> > >> > > > res): result set contained no data
> >>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> >>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> >>>>>> > >> > >>>>> > >> > > > in
> >>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
> >>>>>> > >> > >>>>> > >> > > > result set
> >>>>>> > >> > >>>>> > >> > > > contained no
> >>>>>> > >> > >>>>> > >> > > > data
> >>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> >>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> >>>>>> > >> > >>>>> > >> > > > in
> >>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
> >>>>>> > >> > >>>>> > >> > > > result set
> >>>>>> > >> > >>>>> > >> > > > contained no
> >>>>>> > >> > >>>>> > >> > > > data
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > Apparently there was no data during
this
> >>>>>> time. I
> >>>>>> > >> > >>>>> > >> > > > tried
> >>>>>> > >> > >>>>> > >> > > > changing
> >>>>>> > >> > >>>>> > >> > > > the
> >>>>>> > >> > >>>>> > >> > > > range
> >>>>>> > >> > >>>>> > >> > > > of valid times and changed the valid
> >>>>>> > >> > >>>>> > >> > > > time to
> >>>>>> > >> > >>>>> > >> > > > different
> >>>>>> > >> > >>>>> > >> > > > days
> >>>>>> > >> > >>>>> > >> > > > just in
> >>>>>> > >> > >>>>> > >> > > > case
> >>>>>> > >> > >>>>> > >> > > > data was missing. The issue was the
> >>>>>> > >> > >>>>> > >> > > > same. To
> >>>>>> > >> > >>>>> > >> > > > confirm
> >>>>>> > >> > >>>>> > >> > > > my
> >>>>>> > >> > >>>>> > >> > > > logic, I
> >>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
> >>>>>> > >> > >>>>> > >> > > > metviewer,
> >>>>>> > >> > >>>>> > >> > > > which I
> >>>>>> > >> > >>>>> > >> > > > found
> >>>>>> > >> > >>>>> > >> > > > to
> >>>>>> > >> > >>>>> > >> > > > work.
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > I think I lack an understanding of
how
> >>>>>> > >> > >>>>> > >> > > > to set
> >>>>>> up
> >>>>>> > >> > >>>>> > >> > > > batch
> >>>>>> > >> > >>>>> > >> > > > mode
> >>>>>> > >> > >>>>> > >> > > > because
> >>>>>> > >> > >>>>> > >> > > > I
> >>>>>> > >> > >>>>> > >> > > > don't
> >>>>>> > >> > >>>>> > >> > > > see where these definitions are being
> >>>>>> explicitly
> >>>>>> > >> > >>>>> > >> > > > defined:
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > <plot_spec>
> >>>>>> > >> > >>>>> > >> > > > <connection>
> >>>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
> >>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
> >>>>>> > >> > >>>>> > >> > > > </connection>
> >>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
> >>>>>> > >> > >>>>> > >> > > > <folders>
> >>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> >>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
> >>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
> >>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> >>>>>> > >> > >>>>> > >> > > > </folders>
> >>>>>> > >> > >>>>> > >> > > > <plot>
> >>>>>> > >> > >>>>> > >> > > >
<template>series_plot.R_tmpl</template>
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
> >>>>>> > >> > >>>>> > >> > > > grepping all
> >>>>>> > >> > >>>>> > >> > > > available
> >>>>>> > >> > >>>>> files
> >>>>>> > >> > >>>>> > >> > > > does
> >>>>>> > >> > >>>>> > >> > > > not
> >>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being
defined,
> >>>>>> > >> > >>>>> > >> > > > as
> >>>>>> well
> >>>>>> > >> > >>>>> > >> > > > as
> >>>>>> > >> > >>>>> > >> > > > other
> >>>>>> > >> > >>>>> > >> > > > similarly
> >>>>>> > >> > >>>>> > >> > > > defined
> >>>>>> > >> > >>>>> > >> > > > variables. I think if I know how
these
> >>>>>> > >> > >>>>> > >> > > > are
> >>>>>> being
> >>>>>> > >> > >>>>> > >> > > > defined,
> >>>>>> > >> > >>>>> > >> > > > then
> >>>>>> > >> > >>>>> > >> > > > I'll
> >>>>>> > >> > >>>>> > >> > > > be
> >>>>>> > >> > >>>>> > >> > > > able to pipe in my information and
> >>>>>> > >> > >>>>> > >> > > > process the
> >>>>>> > >> > >>>>> > >> > > > data
> >>>>>> > >> > >>>>> > >> > > > using
> >>>>>> > >> > >>>>> met
> >>>>>> > >> > >>>>> > >> > > > and
> >>>>>> > >> > >>>>> > >> > > > produce
> >>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
> >>>>>> > >> > >>>>> > >> > > > that?
> >>>>>> > >> > >>>>> > >> > > > Also, if
> >>>>>> > >> > >>>>> > >> > > > I want
> >>>>>> > >> > >>>>> > >> > > > to use
> >>>>>> > >> > >>>>> > >> > > > obs for
> >>>>>> > >> > >>>>> > >> > > > verification, do I add another line
in
> >>>>>> > >> > >>>>> > >> > > > the XML
> >>>>>> > >> > >>>>> > >> > > > file
> >>>>>> > >> > >>>>> > >> > > > that
> >>>>>> > >> > >>>>> > >> > > > designates
> >>>>>> > >> > >>>>> > >> > > > file
> >>>>>> > >> > >>>>> > >> > > > location of obs? I assume that other
> >>>>>> > >> > >>>>> > >> > > > steps
> >>>>>> have
> >>>>>> > >> > >>>>> > >> > > > to be
> >>>>>> > >> > >>>>> > >> > > > run
> >>>>>> > >> > >>>>> > >> > > > before
> >>>>>> > >> > >>>>> > >> > > > all
> >>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form,
for
> >>>>>> example,
> >>>>>> > >> > >>>>> > >> > > > I
> >>>>>> > >> > >>>>> > >> > > > would
> >>>>>> > >> > >>>>> have
> >>>>>> > >> > >>>>> > >> > > > to
> >>>>>> > >> > >>>>> > >> > > > separately
> >>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then
I
> >>>>>> > >> > >>>>> > >> > > > would
> >>>>>> want
> >>>>>> > >> > >>>>> > >> > > > to
> >>>>>> > >> > >>>>> > >> > > > do that
> >>>>>> > >> > >>>>> > >> > > > as
> >>>>>> > >> > >>>>> > >> > > > well.
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
> >>>>>> > >> > >>>>> > >> > > > online
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of
people/projects
> >>>>>> > >> > >>>>> > >> > > > listed
> >>>>>> > >> > >>>>> > >> > > > here -
> >>>>>> > >> > >>>>> > >> > > > see
> >>>>>> > >> > >>>>> > >> > > > attachment
> >>>>>> > >> > >>>>> > >> > > >
> >>>>>> > >> > >>>>> > >> > > > How do I get the results I generate
to
> >>>>>> > >> > >>>>> > >> > > > be
> >>>>>> > >> > >>>>> > >> > > > uploaded to
> >>>>>> > >> > >>>>> > >> > > > this
> >>>>>> > >> > >>>>> > >> > > > location?
> >>>>>> > >> > >>>>> > >> > > > It's
> >>>>>> > >> > >>>>> > >> > > > not clear how others are setting this
> >>>>>> > >> > >>>>> > >> > > > up.
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >> > >
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>> > >>
> >>>>>> > >> > >>>>> > >> rds_host:3306
> >>>>>> > >> > >>>>> > >> mv_cmaqv5para8_aod_grid2grid_metplus
> >>>>>> > >> > >>>>> > >> rds_user
> >>>>>> > >> > >>>>> rds_pwd
> >>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
rds_plots
> >>>>>> rds_data
> >>>>>> > >> > >>>>> rds_scripts
> >>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
> >>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
> >>>>>> > >> > >>>>> 2020-05-12
> >>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
> >>>>>> > >> > >>>>> > >> 19:00:00
> >>>>>> 2020-
> >>>>>> > >> > >>>>> > >> 05-12
> >>>>>> > >> > >>>>> 20:00:00
> >>>>>> > >> > >>>>> > >> 2020-05-12
> >>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
> >>>>>> > >> > >>>>> > >> 23:00:00
> >>>>>> 2020-
> >>>>>> > >> > >>>>> > >> 05-13
> >>>>>> > >> > >>>>> 10:00:00
> >>>>>> > >> > >>>>> > >> 2020-05-13
> >>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
> >>>>>> > >> > >>>>> > >> 13:00:00
> >>>>>> 2020-
> >>>>>> > >> > >>>>> > >> 05-13
> >>>>>> > >> > >>>>> 14:00:00
> >>>>>> > >> > >>>>> > >> 2020-05-13
> >>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
> >>>>>> > >> > >>>>> > >> 17:00:00
> >>>>>> 2020-
> >>>>>> > >> > >>>>> > >> 05-13
> >>>>>> > >> > >>>>> 18:00:00
> >>>>>> > >> > >>>>> > >> 2020-05-13
> >>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
> >>>>>> > >> > >>>>> > >> 21:00:00
> >>>>>> 2020-
> >>>>>> > >> > >>>>> > >> 05-13
> >>>>>> > >> > >>>>> 22:00:00
> >>>>>> > >> > >>>>> > >> 2020-05-13
> >>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
> >>>>>> > >> > >>>>> > >> 11:00:00
> >>>>>> 2020-
> >>>>>> > >> > >>>>> > >> 05-14
> >>>>>> > >> > >>>>> 12:00:00
> >>>>>> > >> > >>>>> > >> true
> >>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
> >>>>>> > >> > >>>>> > >> plot_20200612_173626.png
> >>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
> >>>>>> > >> > >>>>> > >> y_label
> >>>>>> > >> > >>>>> > >> false list() list() false false false false
> >>>>>> > >> > >>>>> > >> false
> >>>>>> false
> >>>>>> > >> > >>>>> > >> true
> >>>>>> > >> > >>>>> > >> false
> >>>>>> > >> > >>>>> > >> false
> >>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
> >>>>>> c(8,4,5,4)
> >>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
> >>>>>> > >> > >>>>> > >> 1.4 -2
> >>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1
1
> >>>>>> > >> > >>>>> > >> -2
> >>>>>> 0.5 3
> >>>>>> > >> > >>>>> > >> #cccccc 1
> >>>>>> > >> > >>>>> > >> listX 1
> >>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5
o
> >>>>>> > >> > >>>>> > >> c(0,
> >>>>>> -.25)
> >>>>>> > >> > >>>>> > >> 3
> >>>>>> > >> > >>>>> > >> 0.8 1
> >>>>>> > >> > >>>>> > >> #333333
> >>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none")
c(FALSE,FALSE)
> >>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
> >>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
c("b","b")
> >>>>>> c(1,1)
> >>>>>> > >> > >>>>> > >> c(1,1)
> >>>>>> > >> > >>>>> c(1,1)
> >>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>> > > --
> >>>>>> > >> > >>>>> > > Edward Strobach
> >>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
> >>>>>> > >> > >>>>> > > IMSG Contractor
> >>>>>> > >> > >>>>> > > Cubicle#: 2029
> >>>>>> > >> > >>>>> > > 301-683-3717
> >>>>>> > >> > >>>>> > >
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>>
> >>>>>> > >> > >>>>
> >>>>>> > >> > >>>> --
> >>>>>> > >> > >>>> Edward Strobach
> >>>>>> > >> > >>>> EMC/NCEP/NWS/
> >>>>>> > >> > >>>> IMSG Contractor
> >>>>>> > >> > >>>> Cubicle#: 2029
> >>>>>> > >> > >>>> 301-683-3717
> >>>>>> > >> > >>>>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>>
> >>>>>> > >> > >>> --
> >>>>>> > >> > >>> Edward Strobach
> >>>>>> > >> > >>> EMC/NCEP/NWS/
> >>>>>> > >> > >>> IMSG Contractor
> >>>>>> > >> > >>> Cubicle#: 2029
> >>>>>> > >> > >>> 301-683-3717
> >>>>>> > >> > >>>
> >>>>>> > >> > >>
> >>>>>> > >> > >>
> >>>>>> > >> > >> --
> >>>>>> > >> > >> Edward Strobach
> >>>>>> > >> > >> EMC/NCEP/NWS/
> >>>>>> > >> > >> IMSG Contractor
> >>>>>> > >> > >> Cubicle#: 2029
> >>>>>> > >> > >> 301-683-3717
> >>>>>> > >> > >>
> >>>>>> > >> > >
> >>>>>> > >> > >
> >>>>>> > >> > > --
> >>>>>> > >> > > Edward Strobach
> >>>>>> > >> > > EMC/NCEP/NWS/
> >>>>>> > >> > > IMSG Contractor
> >>>>>> > >> > > Cubicle#: 2029
> >>>>>> > >> > > 301-683-3717
> >>>>>> > >> > >
> >>>>>> > >>
> >>>>>> > >>
> >>>>>> > >>
> >>>>>> > >>
> >>>>>> > >
> >>>>>> > > --
> >>>>>> > > Edward Strobach
> >>>>>> > > EMC/NCEP/NWS/
> >>>>>> > > IMSG Contractor
> >>>>>> > > Cubicle#: 2029
> >>>>>> > > 301-683-3717
> >>>>>> > >
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> --
> >>>>> Edward Strobach
> >>>>> EMC/NCEP/NWS/
> >>>>> IMSG Contractor
> >>>>> Cubicle#: 2029
> >>>>> 301-683-3717
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Edward Strobach
> >>>> EMC/NCEP/NWS/
> >>>> IMSG Contractor
> >>>> Cubicle#: 2029
> >>>> 301-683-3717
> >>>>
> >>> --
> >>> Jeffery T. McQueen
> >>> Meteorologist
> >>> National Center for Weather and Climate Prediction
> >>> Room 2095
> >>> 5830 University Research Court
> >>> College Park, MD 20741
> >>> 301-683-3736
> >>> Fax: 301-683-3703
> >>>
> >>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Ho-Chun Huang - NOAA Affiliate
Time: Wed Jun 17 13:34:20 2020
Correction,
Replacing the entire section as stated in previous email, but use your
own
database name [between <database> </database>]
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Wed, Jun 17, 2020 at 3:27 PM Ho-Chun Huang - NOAA Affiliate <
ho-chun.huang at noaa.gov> wrote:
> Ed:
>
> You can try or you can keep waiting, it is your choice.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Wed, Jun 17, 2020 at 3:21 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> Hi Ho-Chun,
>>
>> I've made all the changes that I would need to inside my xml file.
I was
>> told not to change the folders info. I instead changed my database
name,
>> fcst_var, stat options, series1 input, the fix fields, the indep
field, and
>> x_lim.
>>
>> Thanks
>>
>> On Wed, Jun 17, 2020 at 3:16 PM Ho-Chun Huang - NOAA Affiliate <
>> ho-chun.huang at noaa.gov> wrote:
>>
>>> Hi, Ed:
>>>
>>> You should go back to read 1README again from the directory I sent
about
>>> replacing the top portion of your metviewer xml to produce a batch
xml.
>>>
>>> I check your
>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
>>> Please update the section from <connection> to </folder> as those
in
>>> hochun_xml.template. I also do not know whether AWS can handle
comments
>>> <!-- to !--> in batch xml , better make your batch xml clean.
>>>
>>> Ho-Chun Huang
>>>
>>> IMSG at NOAA/NWS/NCEP/EMC
>>>
>>> 5830 University Research Ct., Rm. 2792
>>>
>>> College Park, MD 20740
>>>
>>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>>
>>> 301-683-3958
>>>
>>>
>>> On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA Affiliate <
>>> edward.strobach at noaa.gov> wrote:
>>>
>>>> Just adding more detail to elaborate my concerns...
>>>>
>>>> Not too long ago I was given a test set-up by Ho-Chun to generate
a
>>>> plot for aerosol optical depth. The result was successful and
produced a
>>>> CSI plot for AOD, comparing the production run and experimental
run.
>>>> Although I created a database, I do not seem to have a group
where I can
>>>> access specific databases. I recently created a load_xml file,
which
>>>> should have loaded the data based on what I specified into the
group. It
>>>> seemed successful based on the log output that was generated.
>>>> Additionally, all information that I specified in this step was
create in
>>>> the load_*xml.
>>>>
>>>> The next step is to generate plots since the data should be
loaded.
>>>> Here's what happens when I run Ho-Chun's - the successful case
>>>>
>>>> HO-CHUN:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
>>>> hochun_xml.template+ bash
>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>>>> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # #
# # # # #
>>>> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT
model,
>>>> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
>>>>
>>>> This is early on in the script.
>>>>
>>>> ED_STROBACH:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed
-e
>>>> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>>>> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20
PM
>>>> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
create
>>>> initial connections of pool.java.sql.SQLException: Communications
link
>>>> failure with primary. No active connection found for master.
at
>>>>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
>>>> at
>>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
>>>> at
>>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
>>>> at
>>>>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
>>>> at
>>>>
org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
>>>> at
>>>>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
>>>> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
>>>>
>>>> My interpretation of this is that it fails to bridge a connection
to
>>>> the group or database; therefore, it's unable to proceed with
running the
>>>> XML file to generate the plots based on the specified content.
If I move
>>>> past this step, then I'm golden.
>>>>
>>>> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>
>>>>> Would it be ok to keep perry in the loop
>>>>>
>>>>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA
Affiliate <
>>>>> edward.strobach at noaa.gov> wrote:
>>>>>
>>>>>> you're right. I didn't have the storage problem this time.
I'm not
>>>>>> sure what changed. All I know is that I have now created my
load xml which
>>>>>> now should allow me to move on to the next step of plotting
>>>>>>
>>>>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA
Affiliate <
>>>>>> edward.strobach at noaa.gov> wrote:
>>>>>>
>>>>>>> I'm not following you. How do I now have enough space? What
>>>>>>> changed?
>>>>>>>
>>>>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>
>>>>>>>> Ed, now you should have enough space for the data loading.
>>>>>>>>
>>>>>>>> Tatiana
>>>>>>>>
>>>>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>>>>>>> > I was basically able to build my xml file but it says I
have no
>>>>>>>> > available
>>>>>>>> > space. I'm not sure how that can be since I just started
adding
>>>>>>>> data
>>>>>>>> > to my
>>>>>>>> > account
>>>>>>>> >
>>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > ERROR: Command returned with non-zero status (1): scp -r
./*
>>>>>>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>>>>>>> > 4) Check METviewer AWS database list using
mv_db_size_on_aws.sh
>>>>>>>> > TABLE_SCHEMA USER
SIZE_MB
>>>>>>>> > mv_met_g2o_gefs binbin.zhou
>>>>>>>> 140027.31
>>>>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
>>>>>>>> 71986.11
>>>>>>>> >
>>>>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate <
>>>>>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>>> >
>>>>>>>> > > Thanks Tatiana. Yes, I've created a database which seems
>>>>>>>> different
>>>>>>>> > > from
>>>>>>>> > > mv_group. How can one create their own group so that I
can
>>>>>>>> store and
>>>>>>>> > > access
>>>>>>>> > > the database. The database exists but cannot be viewed
on the
>>>>>>>> > > metviewer
>>>>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
vhagerty. I
>>>>>>>> > > named my
>>>>>>>> > > database Ed_Strobach, but that was really intended for
the group
>>>>>>>> > > name.
>>>>>>>> > >
>>>>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>>>>>>> > > <met_help at ucar.edu>
>>>>>>>> > > wrote:
>>>>>>>> > >
>>>>>>>> > >> We don't have database and METviewer running on wcoss.
>>>>>>>> > >> Instead, users use METviewer-wcoss scripts that help to
>>>>>>>> connect to
>>>>>>>> > >> and
>>>>>>>> > >> use METviewer that is running on AWS. That is why XML
>>>>>>>> connection and
>>>>>>>> > >> folder
>>>>>>>> > >> sections look somewhat different than once from the Docs
page.
>>>>>>>> > >> To create a database on AWS you need to run
>>>>>>>> mv_create_db_on_aws.sh
>>>>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>>>>>>> > >> For data loading :
>>>>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>>>>>>> > >> And for batch:
>>>>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>>>>>>> > >>
>>>>>>>> > >> These scripts copy XML and data files if needed to AWS,
adjust
>>>>>>>> data
>>>>>>>> > >> path
>>>>>>>> > >> and users credentials, run METviewer and copy results
back to
>>>>>>>> wcoss.
>>>>>>>> > >>
>>>>>>>> > >> Tatiana
>>>>>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
wrote:
>>>>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
>>>>>>>> -p[db_password]
>>>>>>>> > >> > -e'create
>>>>>>>> > >> > database [db_name]
>>>>>>>> > >> >
>>>>>>>> > >> > would need to be used to create a database. I assume
that
>>>>>>>> an sql
>>>>>>>> > >> > module is
>>>>>>>> > >> > available on wcoss; I know you don't have access to
those
>>>>>>>> machines
>>>>>>>> > >> > Tatiana. If I create my own database with my own
name, then
>>>>>>>> I
>>>>>>>> > >> > should
>>>>>>>> > >> > be
>>>>>>>> > >> > able to proceed with project specific tasks under that
tab.
>>>>>>>> > >> > Something
>>>>>>>> > >> > like
>>>>>>>> > >> > mv_EdwardStrobach.
>>>>>>>> > >> >
>>>>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add data
to
>>>>>>>> using
>>>>>>>> > >> > load_spec.
>>>>>>>> > >> > Once that step is done, then I can run plot_spec.
That's my
>>>>>>>> > >> > takeaway
>>>>>>>> > >> > currently
>>>>>>>> > >> >
>>>>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
>>>>>>>> Affiliate <
>>>>>>>> > >> > edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> >
>>>>>>>> > >> > > Actually, it appears that I need to run the database
>>>>>>>> loading
>>>>>>>> > >> > > module
>>>>>>>> > >> > > first
>>>>>>>> > >> > > if I want to change/add a data source. From there I
can
>>>>>>>> run the
>>>>>>>> > >> > > plotting
>>>>>>>> > >> > > batch module. Is that right? So if I run the
database
>>>>>>>> loading
>>>>>>>> > >> > > module,
>>>>>>>> > >> > > then I can specify my dates and other related info
>>>>>>>> specific to
>>>>>>>> > >> > > the
>>>>>>>> > >> > > dataset. If I create my database with a unique
name, then
>>>>>>>> it
>>>>>>>> > >> > > should
>>>>>>>> > >> > > be
>>>>>>>> > >> > > recognized on the metviewer gui, presumably.
>>>>>>>> > >> > >
>>>>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach -
NOAA
>>>>>>>> Affiliate
>>>>>>>> > >> > > <
>>>>>>>> > >> > > edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> > >
>>>>>>>> > >> > >> It seems that those options must be treated
separately
>>>>>>>> outside
>>>>>>>> > >> > >> of
>>>>>>>> > >> > >> plot_spec. Is it acceptable to create both a
load_spec
>>>>>>>> and
>>>>>>>> > >> > >> plot_spec in
>>>>>>>> > >> > >> the same xml file? Would the definitions in
load_spec be
>>>>>>>> able
>>>>>>>> > >> > >> to
>>>>>>>> > >> > >> carry
>>>>>>>> > >> > >> over into plot_spec? It appears that data loading
and
>>>>>>>> plotting
>>>>>>>> > >> > >> are
>>>>>>>> > >> > >> treated
>>>>>>>> > >> > >> separately.
>>>>>>>> > >> > >>
>>>>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
NOAA
>>>>>>>> > >> > >> Affiliate <
>>>>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> > >>
>>>>>>>> > >> > >>> I do plan to use date_range; however, in an
example
>>>>>>>> provided
>>>>>>>> > >> > >>> by the
>>>>>>>> > >> > >>> link
>>>>>>>> > >> > >>> I see this:
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <load_spec>
>>>>>>>> > >> > >>> <connection>
>>>>>>>> > >> > >>> <host>kemosabe:3306</host>
>>>>>>>> > >> > >>> <database>metvdb_hwt</database>
>>>>>>>> > >> > >>> <user>pgoldenb</user>
>>>>>>>> > >> > >>> <password>pgoldenb</password>
>>>>>>>> > >> > >>> </connection>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>>> > >> > >>> <start>2010051914V</start>
>>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
>>>>>>>> > >> > >>> </hour></date_offset></end>
>>>>>>>> > >> > >>> <inc>3600</inc>
>>>>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
>>>>>>>> > >> > >>> </date_list>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <met_version>V3.0</met_version>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <verbose>false</verbose>
>>>>>>>> > >> > >>> <insert_size>1</insert_size>
>>>>>>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>>>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>>>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>>>>>>> > >> > >>> <group>Group name</group>
>>>>>>>> > >> > >>> <load_stat>true</load_stat>
>>>>>>>> > >> > >>> <load_mode>true</load_mode>
>>>>>>>> > >> > >>> <load_mtd>true</load_mtd>
>>>>>>>> > >> > >>> <load_mpr>false</load_mpr>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>
>>>>>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>>>>>> > >> > >>> <load_val>
>>>>>>>> > >> > >>> <field name="model">
>>>>>>>> > >> > >>> <val>arw</val>
>>>>>>>> > >> > >>> <val>nmm</val>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <field name="vx_mask">
>>>>>>>> > >> > >>> <val>FULL</val>
>>>>>>>> > >> > >>> <val>SWC</val>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>> </load_val>
>>>>>>>> > >> > >>> </load_spec>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> Are similar settings allowed with <plot_spec>? I
see
>>>>>>>> > >> > >>> <connection>,
>>>>>>>> > >> > >>> which seems carried over. I'm looking to specify
my
>>>>>>>> > >> > >>> folder_tmpl in
>>>>>>>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>>>>>>>> > >> > >>> Since my
>>>>>>>> > >> > >>> directory structure looks like this:
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>
>>>>>>>> > >>
>>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> I would do something like this I would think:
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> .......
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>>> > >> > >>> <start>20200612/start>
>>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset></end>
>>>>>>>> > >> > >>> <inc>86400</inc>
>>>>>>>> > >> > >>> <format>yyyyMMdd</format>
>>>>>>>> > >> > >>> </date_list>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <folder_tmpl>
>>>>>>>> > >> > >>>
>>>>>>>> > >>
>>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>>>>>>> > >> > >>> </folder_tmpl>
>>>>>>>> > >> > >>> <load_val>
>>>>>>>> > >> > >>> <field name="model">
>>>>>>>> > >> > >>> <val>prod</val>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> ......
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach -
NOAA
>>>>>>>> > >> > >>> Affiliate <
>>>>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>> Thank you. This is a very helpful example
>>>>>>>> > >> > >>>>
>>>>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via
RT
>>>>>>>> > >> > >>>> <met_help at ucar.edu>
>>>>>>>> > >> > >>>> wrote:
>>>>>>>> > >> > >>>>
>>>>>>>> > >> > >>>>> Hi,
>>>>>>>> > >> > >>>>> Here is some examples how to use <date_range>
and
>>>>>>>> > >> > >>>>> <date_range_list> in
>>>>>>>> > >> > >>>>> batch XML.
>>>>>>>> > >> > >>>>> First, you need to defined your range. You can
have
>>>>>>>> more
>>>>>>>> > >> > >>>>> then one
>>>>>>>> > >> > >>>>> <date_range>:
>>>>>>>> > >> > >>>>> <date_range name="may">
>>>>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>>>>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>>>>>>>> > >> > >>>>> </date_range>
>>>>>>>> > >> > >>>>> <date_range name="march">
>>>>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>>>>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>>>>>>>> > >> > >>>>> </date_range>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> and then use one or all date_ranges:
>>>>>>>> > >> > >>>>> <plot_fix>
>>>>>>>> > >> > >>>>> <field name="fcst_init_beg">
>>>>>>>> > >> > >>>>> <date_range name="march"/>
>>>>>>>> > >> > >>>>> <date_range name="may"/>
>>>>>>>> > >> > >>>>> </field>...
>>>>>>>> > >> > >>>>> This configuration would include all dates for
March
>>>>>>>> and May
>>>>>>>> > >> > >>>>> of
>>>>>>>> > >> > >>>>> 2019.
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> If you need more flexible configuration you can
use
>>>>>>>> > >> > >>>>> date_range_list.
>>>>>>>> > >> > >>>>> For example, you want to select first 6 hours
every day
>>>>>>>> > >> > >>>>> beetween
>>>>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00.
Your XML
>>>>>>>> would
>>>>>>>> > >> > >>>>> look
>>>>>>>> > >> > >>>>> like this:
>>>>>>>> > >> > >>>>> <date_range_list name = 'all'>
>>>>>>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
>>>>>>>> > >> > >>>>> <range_end>2020-01-05
00:00:00</range_end>
>>>>>>>> > >> > >>>>> <range_length>360</range_length>
>>>>>>>> > >> > >>>>> <inc>1440</inc>
>>>>>>>> > >> > >>>>> </date_range_list>
>>>>>>>> > >> > >>>>> where the 1st and 2nd values are the start and
end,
>>>>>>>> > >> > >>>>> range_length - the length of a range in
minutes(6
>>>>>>>> houres)
>>>>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
houres)
>>>>>>>> > >> > >>>>> This example would create following ranges:
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> Also you can add more parameters to <plot_fix>
to
>>>>>>>> filter
>>>>>>>> > >> > >>>>> your
>>>>>>>> > >> > >>>>> data.
>>>>>>>> > >> > >>>>> For example, if you want to select only dates
with
>>>>>>>> 00:00 you
>>>>>>>> > >> > >>>>> can
>>>>>>>> > >> > >>>>> specify
>>>>>>>> > >> > >>>>> <field name="valid_hour">
>>>>>>>> > >> > >>>>> <val>00</val>
>>>>>>>> > >> > >>>>> </field>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> Tatiana
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
>>>>>>>> wrote:
>>>>>>>> > >> > >>>>> > and if there isn't a better way, I wonder the
>>>>>>>> whereabouts
>>>>>>>> > >> > >>>>> > of a
>>>>>>>> > >> > >>>>> > separate
>>>>>>>> > >> > >>>>> > script that pipes dates (and other relevant
>>>>>>>> information)
>>>>>>>> > >> > >>>>> > into
>>>>>>>> > >> > >>>>> > the XML
>>>>>>>> > >> > >>>>> > file
>>>>>>>> > >> > >>>>> > used to generate the results.
>>>>>>>> > >> > >>>>> >
>>>>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward
Strobach -
>>>>>>>> NOAA
>>>>>>>> > >> > >>>>> > Affiliate <
>>>>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> > >>>>> >
>>>>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are
helpful.
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > One last question. Simply typing lines like
this
>>>>>>>> in can
>>>>>>>> > >> > >>>>> > > take
>>>>>>>> > >> > >>>>> > > forever:
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > <plot_fix>
>>>>>>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
>>>>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>>>>>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>>>>>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>>>>>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>>>>>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>>>>>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > Is there a better way? I see date_range,
>>>>>>>> > >> > >>>>> > > date_range_list,
>>>>>>>> > >> > >>>>> > > or
>>>>>>>> > >> > >>>>> > > date_list.
>>>>>>>> > >> > >>>>> > > Can this be used instead of typing
individual
>>>>>>>> values?
>>>>>>>> > >> > >>>>> > > This
>>>>>>>> > >> > >>>>> > > could
>>>>>>>> > >> > >>>>> > > prove
>>>>>>>> > >> > >>>>> > > pretty daunting if one would have to type in
700
>>>>>>>> entries
>>>>>>>> > >> > >>>>> > > each
>>>>>>>> > >> > >>>>> > > time.
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana
Burek via
>>>>>>>> RT
>>>>>>>> > >> > >>>>> > > <met_help at ucar.edu>
>>>>>>>> > >> > >>>>> > > wrote:
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > >> I found a few problems in your XML.
>>>>>>>> > >> > >>>>> > >> The database you are using doesn't have
data with
>>>>>>>> > >> > >>>>> > >> vx_mask
>>>>>>>> > >> > >>>>> > >> 'abcREGabc'.
>>>>>>>> > >> > >>>>> > >> That is why METviewer returns an empty
dataset and
>>>>>>>> > >> > >>>>> > >> can't
>>>>>>>> > >> > >>>>> > >> create a
>>>>>>>> > >> > >>>>> > >> plot.
>>>>>>>> > >> > >>>>> > >> The other problem is how you define
fcst_lead.
>>>>>>>> This
>>>>>>>> > >> > >>>>> > >> database
>>>>>>>> > >> > >>>>> > >> has
>>>>>>>> > >> > >>>>> > >> values
>>>>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
Since
>>>>>>>> you
>>>>>>>> > >> > >>>>> > >> include
>>>>>>>> > >> > >>>>> > >> all
>>>>>>>> > >> > >>>>> > >> fcst_leads
>>>>>>>> > >> > >>>>> > >> this variable can be omitted.
>>>>>>>> > >> > >>>>> > >> To minimize problems with variables values,
I
>>>>>>>> would
>>>>>>>> > >> > >>>>> > >> suggest
>>>>>>>> > >> > >>>>> > >> to
>>>>>>>> > >> > >>>>> > >> create XML
>>>>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>>>>>>>> > >> > >>>>> > >> I created a plot using you configurations
with GUI
>>>>>>>> > >> > >>>>> > >> (without
>>>>>>>> > >> > >>>>> vx_mack
>>>>>>>> > >> > >>>>> > >> and
>>>>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You
can
>>>>>>>> load it
>>>>>>>> > >> > >>>>> > >> back
>>>>>>>> > >> > >>>>> > >> to
>>>>>>>> > >> > >>>>> the
>>>>>>>> > >> > >>>>> > >> page by
>>>>>>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI).
Find
>>>>>>>> the
>>>>>>>> > >> > >>>>> > >> plot,
>>>>>>>> > >> > >>>>> > >> open XML
>>>>>>>> > >> > >>>>> > >> and
>>>>>>>> > >> > >>>>> > >> click on "Load to page". After that you can
>>>>>>>> recreate a
>>>>>>>> > >> > >>>>> > >> plot
>>>>>>>> > >> > >>>>> > >> and
>>>>>>>> > >> > >>>>> edit
>>>>>>>> > >> > >>>>> > >> the
>>>>>>>> > >> > >>>>> > >> configurations. You can also download a new
XML
>>>>>>>> and
>>>>>>>> > >> > >>>>> > >> edit it
>>>>>>>> > >> > >>>>> > >> for
>>>>>>>> > >> > >>>>> > >> batch.
>>>>>>>> > >> > >>>>> > >> I attached the example of load XML to this
>>>>>>>> message.
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> Description of rds parameters:
>>>>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
METviewer
>>>>>>>> R
>>>>>>>> > >> > >>>>> > >> scripts
>>>>>>>> > >> > >>>>> > >> rds_plots - temporary directory where the
plots
>>>>>>>> will be
>>>>>>>> > >> > >>>>> > >> generated
>>>>>>>> > >> > >>>>> > >> rds_data - temporary directory where the
data
>>>>>>>> retrieved
>>>>>>>> > >> > >>>>> > >> from
>>>>>>>> > >> > >>>>> > >> the
>>>>>>>> > >> > >>>>> > >> database
>>>>>>>> > >> > >>>>> > >> and other intermediate files will be stored
>>>>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
scripts
>>>>>>>> for
>>>>>>>> > >> > >>>>> > >> this
>>>>>>>> > >> > >>>>> > >> run
>>>>>>>> > >> > >>>>> will
>>>>>>>> > >> > >>>>> > >> be
>>>>>>>> > >> > >>>>> > >> stored
>>>>>>>> > >> > >>>>> > >> After the execution is done all the files
from
>>>>>>>> > >> > >>>>> > >> temporary
>>>>>>>> > >> > >>>>> directories
>>>>>>>> > >> > >>>>> > >> would be removed.
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> We don't have the offitial user guide for
>>>>>>>> METviewer.
>>>>>>>> > >> > >>>>> > >> There
>>>>>>>> > >> > >>>>> > >> are
>>>>>>>> > >> > >>>>> some
>>>>>>>> > >> > >>>>> > >> instructions on how to crate XML config
files:
>>>>>>>> > >> > >>>>> > >>
https://metviewer.nws.noaa.gov/doc/index.html
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> Tatiana
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
>>>>>>>> edward.strobach at noaa.gov
>>>>>>>> > >> > >>>>> > >> wrote:
>>>>>>>> > >> > >>>>> > >> > Hi,
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am
having hard
>>>>>>>> time
>>>>>>>> > >> > >>>>> > >> > following
>>>>>>>> > >> > >>>>> how
>>>>>>>> > >> > >>>>> > >> > to
>>>>>>>> > >> > >>>>> > >> > go
>>>>>>>> > >> > >>>>> > >> > about the other steps since I have zero
>>>>>>>> experience on
>>>>>>>> > >> > >>>>> > >> > how
>>>>>>>> > >> > >>>>> > >> > to do
>>>>>>>> > >> > >>>>> > >> > these
>>>>>>>> > >> > >>>>> > >> > things.
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should
>>>>>>>> NOT be
>>>>>>>> > >> > >>>>> > >> > changed
>>>>>>>> > >> > >>>>> and
>>>>>>>> > >> > >>>>> > >> > will
>>>>>>>> > >> > >>>>> > >> > be
>>>>>>>> > >> > >>>>> > >> > replaced by other scripts during the
batch
>>>>>>>> execution.
>>>>>>>> > >> > >>>>> > >> > The
>>>>>>>> > >> > >>>>> > >> > data
>>>>>>>> > >> > >>>>> for
>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>> > >> > >>>>> > >> > batch mode is coming from the database
and not
>>>>>>>> from
>>>>>>>> > >> > >>>>> > >> > files.
>>>>>>>> > >> > >>>>> > >> > The
>>>>>>>> > >> > >>>>> > >> > name of
>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>> > >> > >>>>> > >> > database is in <database> tag.
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > I understand that nothing should be
changed in
>>>>>>>> the
>>>>>>>> > >> > >>>>> > >> > folders
>>>>>>>> > >> > >>>>> > >> > section,
>>>>>>>> > >> > >>>>> > >> > but
>>>>>>>> > >> > >>>>> > >> > there is no indication how and where
these
>>>>>>>> variables
>>>>>>>> > >> > >>>>> > >> > are
>>>>>>>> > >> > >>>>> > >> > being
>>>>>>>> > >> > >>>>> > >> > defined.
>>>>>>>> > >> > >>>>> > >> > Grepping these names of the files
available
>>>>>>>> does not
>>>>>>>> > >> > >>>>> > >> > reveal
>>>>>>>> > >> > >>>>> formal
>>>>>>>> > >> > >>>>> > >> > definitions of these variables. If I
understood
>>>>>>>> > >> > >>>>> > >> > exactly
>>>>>>>> > >> > >>>>> > >> > how
>>>>>>>> > >> > >>>>> these
>>>>>>>> > >> > >>>>> > >> > were
>>>>>>>> > >> > >>>>> > >> > being defined, then I would be able
connect this
>>>>>>>> > >> > >>>>> > >> > better.
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
files.
>>>>>>>> They
>>>>>>>> > >> > >>>>> > >> > use
>>>>>>>> > >> > >>>>> METviewer
>>>>>>>> > >> > >>>>> > >> > load
>>>>>>>> > >> > >>>>> > >> > mode to load these files to the specific
>>>>>>>> database. In
>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>> > >> > >>>>> > >> > config
>>>>>>>> > >> > >>>>> > >> > file
>>>>>>>> > >> > >>>>> > >> > for
>>>>>>>> > >> > >>>>> > >> > this process they also can specify the
database
>>>>>>>> > >> > >>>>> > >> > group. For
>>>>>>>> > >> > >>>>> example
>>>>>>>> > >> > >>>>> > >> > database
>>>>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>>>>>>> > >> > >>>>> > >> > After the data has been loaded to the
database
>>>>>>>> it is
>>>>>>>> > >> > >>>>> > >> > visible and
>>>>>>>> > >> > >>>>> > >> > available
>>>>>>>> > >> > >>>>> > >> > in the GUI and also can be used by the
batch
>>>>>>>> mode.
>>>>>>>> > >> > >>>>> > >> > You can always add more data to the
database
>>>>>>>> using
>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>> > >> > >>>>> > >> > same load
>>>>>>>> > >> > >>>>> > >> > mode.
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > Is there a guide for me to understand the
>>>>>>>> process
>>>>>>>> > >> > >>>>> > >> > that you
>>>>>>>> > >> > >>>>> > >> > are
>>>>>>>> > >> > >>>>> > >> > referring to?
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek
>>>>>>>> via RT
>>>>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>>>>>>> > >> > >>>>> > >> > wrote:
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > > Hi Edward,
>>>>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
servers and
>>>>>>>> > >> > >>>>> > >> > > can't
>>>>>>>> > >> > >>>>> > >> > > view your
>>>>>>>> > >> > >>>>> > >> > > XML
>>>>>>>> > >> > >>>>> > >> > > for the
>>>>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue
>>>>>>>> and I
>>>>>>>> > >> > >>>>> > >> > > will
>>>>>>>> > >> > >>>>> > >> > > take a
>>>>>>>> > >> > >>>>> > >> > > look.
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
>>>>>>>> should NOT
>>>>>>>> > >> > >>>>> > >> > > be
>>>>>>>> > >> > >>>>> > >> > > changed
>>>>>>>> > >> > >>>>> > >> > > and
>>>>>>>> > >> > >>>>> > >> > > will be
>>>>>>>> > >> > >>>>> > >> > > replaced by other scripts during the
batch
>>>>>>>> > >> > >>>>> > >> > > execution.
>>>>>>>> > >> > >>>>> > >> > > The data
>>>>>>>> > >> > >>>>> > >> > > for
>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>> > >> > >>>>> > >> > > batch mode is coming from the database
and
>>>>>>>> not from
>>>>>>>> > >> > >>>>> > >> > > files. The
>>>>>>>> > >> > >>>>> > >> > > name
>>>>>>>> > >> > >>>>> > >> > > of the
>>>>>>>> > >> > >>>>> > >> > > database is in <database> tag.
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET
output
>>>>>>>> files.
>>>>>>>> > >> > >>>>> > >> > > They use
>>>>>>>> > >> > >>>>> > >> > > METviewer
>>>>>>>> > >> > >>>>> > >> > > load
>>>>>>>> > >> > >>>>> > >> > > mode to load these files to the
specific
>>>>>>>> database.
>>>>>>>> > >> > >>>>> > >> > > In
>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>> > >> > >>>>> config
>>>>>>>> > >> > >>>>> > >> > > file
>>>>>>>> > >> > >>>>> > >> > > for
>>>>>>>> > >> > >>>>> > >> > > this process they also can specify the
>>>>>>>> database
>>>>>>>> > >> > >>>>> > >> > > group.
>>>>>>>> > >> > >>>>> > >> > > For
>>>>>>>> > >> > >>>>> > >> > > example
>>>>>>>> > >> > >>>>> > >> > > database
>>>>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>>>>>>> > >> > >>>>> > >> > > After the data has been loaded to the
>>>>>>>> database it
>>>>>>>> > >> > >>>>> > >> > > is
>>>>>>>> > >> > >>>>> > >> > > visible
>>>>>>>> > >> > >>>>> and
>>>>>>>> > >> > >>>>> > >> > > available
>>>>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch
>>>>>>>> mode.
>>>>>>>> > >> > >>>>> > >> > > You can always add more data to the
database
>>>>>>>> using
>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>> > >> > >>>>> > >> > > same
>>>>>>>> > >> > >>>>> load
>>>>>>>> > >> > >>>>> > >> > > mode.
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > > Tatiana
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>>>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>>>>>>> > >> > >>>>> > >> > > wrote:
>>>>>>>> > >> > >>>>> > >> > > > Good morning,
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
effort
>>>>>>>> to
>>>>>>>> > >> > >>>>> > >> > > > understand
>>>>>>>> > >> > >>>>> > >> > > > metviewer,
>>>>>>>> > >> > >>>>> > >> > > > test
>>>>>>>> > >> > >>>>> > >> > > > different cases using the history
tab,
>>>>>>>> changing
>>>>>>>> > >> > >>>>> > >> > > > information
>>>>>>>> > >> > >>>>> to
>>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>>> > >> > >>>>> > >> > > > what
>>>>>>>> > >> > >>>>> > >> > > > works and what does not, and make
>>>>>>>> connections
>>>>>>>> > >> > >>>>> > >> > > > between
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> GUI
>>>>>>>> > >> > >>>>> > >> > > > settings
>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
level
>>>>>>>> with
>>>>>>>> > >> > >>>>> > >> > > > metviewer has
>>>>>>>> > >> > >>>>> > >> > > > significantly
>>>>>>>> > >> > >>>>> > >> > > > increased during this time, there are
some
>>>>>>>> > >> > >>>>> > >> > > > remaining
>>>>>>>> > >> > >>>>> > >> > > > issues
>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>> > >> > >>>>> > >> > > > I'm
>>>>>>>> > >> > >>>>> > >> > > > hoping
>>>>>>>> > >> > >>>>> > >> > > > to clear up.
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
running
>>>>>>>> with
>>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>>> > >> > >>>>> > >> > > > statistic
>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>>> > >> > >>>>> > >> > > > type:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
provided
>>>>>>>> me
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> > >> > > > working
>>>>>>>> > >> > >>>>> > >> > > > directory
>>>>>>>> > >> > >>>>> > >> > > > to get
>>>>>>>> > >> > >>>>> > >> > > > started which I copied over:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I
adjusted and
>>>>>>>> used
>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>> > >> > >>>>> > >> > > > run his
>>>>>>>> > >> > >>>>> > >> > > > example.
>>>>>>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
>>>>>>>> produced a
>>>>>>>> > >> > >>>>> > >> > > > CSI
>>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>>> > >> > >>>>> that
>>>>>>>> > >> > >>>>> > >> > > > was
>>>>>>>> > >> > >>>>> > >> > > > stored
>>>>>>>> > >> > >>>>> > >> > > > here:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >>
>>>>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Running the script produced the
expected
>>>>>>>> result.
>>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>>> > >> > >>>>> > >> > > > next
>>>>>>>> > >> > >>>>> > >> > > > natural
>>>>>>>> > >> > >>>>> > >> > > > step was
>>>>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and
create a
>>>>>>>> new xml
>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>> > >> > >>>>> > >> > > > called
>>>>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
create a
>>>>>>>> time
>>>>>>>> > >> > >>>>> > >> > > > series
>>>>>>>> > >> > >>>>> > >> > > > where
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
instead
>>>>>>>> of
>>>>>>>> > >> > >>>>> > >> > > > obs_thresh.
>>>>>>>> > >> > >>>>> > >> > > > Accordingly,
>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
>>>>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>>>>>>> > >> > >>>>> and
>>>>>>>> > >> > >>>>> > >> > > > changed
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was
also
>>>>>>>> > >> > >>>>> > >> > > > tried).
>>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>>> > >> > >>>>> error
>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>> > >> > >>>>> > >> > > > resulted
>>>>>>>> > >> > >>>>> > >> > > > is as follows:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
>>>>>>>> ('2020-05-01
>>>>>>>> > >> > >>>>> > >> > > > 13:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
>>>>>>>> '2020-05-01
>>>>>>>> > >> > >>>>> > >> > > > 16:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
>>>>>>>> '2020-05-01
>>>>>>>> > >> > >>>>> > >> > > > 19:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
>>>>>>>> '2020-05-01
>>>>>>>> > >> > >>>>> > >> > > > 22:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
>>>>>>>> '2020-05-02
>>>>>>>> > >> > >>>>> > >> > > > 01:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
>>>>>>>> '2020-05-02
>>>>>>>> > >> > >>>>> > >> > > > 04:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
>>>>>>>> '2020-05-02
>>>>>>>> > >> > >>>>> > >> > > > 07:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
>>>>>>>> '2020-05-02
>>>>>>>> > >> > >>>>> > >> > > > 10:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
h.stat_header_id
>>>>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>>>>>>> > >> > >>>>> > >> > > > No data was returned from database
>>>>>>>> > >> > >>>>> > >> > > > 2020-06-09
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >>
>>>>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
>>>>>>>> ERROR:
>>>>>>>> > >> > >>>>> > >> > > > Caught
>>>>>>>> > >> > >>>>> > >> > > > class
>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>> in
>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>>>>>>> > >> > >>>>> > >> > > > res): result set contained no data
>>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
>>>>>>>> set
>>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
>>>>>>>> set
>>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Apparently there was no data during
this
>>>>>>>> time. I
>>>>>>>> > >> > >>>>> > >> > > > tried
>>>>>>>> > >> > >>>>> > >> > > > changing
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> > >> > > > range
>>>>>>>> > >> > >>>>> > >> > > > of valid times and changed the valid
time to
>>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>>> > >> > >>>>> > >> > > > days
>>>>>>>> > >> > >>>>> > >> > > > just in
>>>>>>>> > >> > >>>>> > >> > > > case
>>>>>>>> > >> > >>>>> > >> > > > data was missing. The issue was the
same.
>>>>>>>> To
>>>>>>>> > >> > >>>>> > >> > > > confirm
>>>>>>>> > >> > >>>>> > >> > > > my
>>>>>>>> > >> > >>>>> > >> > > > logic, I
>>>>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
>>>>>>>> metviewer,
>>>>>>>> > >> > >>>>> > >> > > > which I
>>>>>>>> > >> > >>>>> > >> > > > found
>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>> > >> > >>>>> > >> > > > work.
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > I think I lack an understanding of
how to
>>>>>>>> set up
>>>>>>>> > >> > >>>>> > >> > > > batch
>>>>>>>> > >> > >>>>> > >> > > > mode
>>>>>>>> > >> > >>>>> > >> > > > because
>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>> > >> > >>>>> > >> > > > don't
>>>>>>>> > >> > >>>>> > >> > > > see where these definitions are being
>>>>>>>> explicitly
>>>>>>>> > >> > >>>>> > >> > > > defined:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > <plot_spec>
>>>>>>>> > >> > >>>>> > >> > > > <connection>
>>>>>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>>
>>>>>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>>>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>>>>>>> > >> > >>>>> > >> > > > </connection>
>>>>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>>>>>>> > >> > >>>>> > >> > > > <folders>
>>>>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>>>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>>>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>>>>>> > >> > >>>>> > >> > > > </folders>
>>>>>>>> > >> > >>>>> > >> > > > <plot>
>>>>>>>> > >> > >>>>> > >> > > >
<template>series_plot.R_tmpl</template>
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
grepping
>>>>>>>> all
>>>>>>>> > >> > >>>>> > >> > > > available
>>>>>>>> > >> > >>>>> files
>>>>>>>> > >> > >>>>> > >> > > > does
>>>>>>>> > >> > >>>>> > >> > > > not
>>>>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being
defined, as
>>>>>>>> well
>>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>>> > >> > >>>>> > >> > > > other
>>>>>>>> > >> > >>>>> > >> > > > similarly
>>>>>>>> > >> > >>>>> > >> > > > defined
>>>>>>>> > >> > >>>>> > >> > > > variables. I think if I know how
these are
>>>>>>>> being
>>>>>>>> > >> > >>>>> > >> > > > defined,
>>>>>>>> > >> > >>>>> > >> > > > then
>>>>>>>> > >> > >>>>> > >> > > > I'll
>>>>>>>> > >> > >>>>> > >> > > > be
>>>>>>>> > >> > >>>>> > >> > > > able to pipe in my information and
process
>>>>>>>> the
>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>> > >> > >>>>> > >> > > > using
>>>>>>>> > >> > >>>>> met
>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>> > >> > >>>>> > >> > > > produce
>>>>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
that?
>>>>>>>> > >> > >>>>> > >> > > > Also, if
>>>>>>>> > >> > >>>>> > >> > > > I want
>>>>>>>> > >> > >>>>> > >> > > > to use
>>>>>>>> > >> > >>>>> > >> > > > obs for
>>>>>>>> > >> > >>>>> > >> > > > verification, do I add another line
in the
>>>>>>>> XML
>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>> > >> > >>>>> > >> > > > designates
>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>> > >> > >>>>> > >> > > > location of obs? I assume that other
steps
>>>>>>>> have
>>>>>>>> > >> > >>>>> > >> > > > to be
>>>>>>>> > >> > >>>>> > >> > > > run
>>>>>>>> > >> > >>>>> > >> > > > before
>>>>>>>> > >> > >>>>> > >> > > > all
>>>>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form,
for
>>>>>>>> example,
>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>> > >> > >>>>> > >> > > > would
>>>>>>>> > >> > >>>>> have
>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>> > >> > >>>>> > >> > > > separately
>>>>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then
I
>>>>>>>> would want
>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>> > >> > >>>>> > >> > > > do that
>>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>>> > >> > >>>>> > >> > > > well.
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of
people/projects
>>>>>>>> listed
>>>>>>>> > >> > >>>>> > >> > > > here -
>>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>>> > >> > >>>>> > >> > > > attachment
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > How do I get the results I generate
to be
>>>>>>>> > >> > >>>>> > >> > > > uploaded to
>>>>>>>> > >> > >>>>> > >> > > > this
>>>>>>>> > >> > >>>>> > >> > > > location?
>>>>>>>> > >> > >>>>> > >> > > > It's
>>>>>>>> > >> > >>>>> > >> > > > not clear how others are setting this
up.
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>>>>>>>> > >> > >>>>> > >> rds_user
>>>>>>>> > >> > >>>>> rds_pwd
>>>>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
rds_plots
>>>>>>>> rds_data
>>>>>>>> > >> > >>>>> rds_scripts
>>>>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>>>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>>>>>>> > >> > >>>>> 2020-05-12
>>>>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
19:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-12
>>>>>>>> > >> > >>>>> 20:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-12
>>>>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
23:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>> > >> > >>>>> 10:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
13:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>> > >> > >>>>> 14:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
17:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>> > >> > >>>>> 18:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
21:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>> > >> > >>>>> 22:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
11:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-14
>>>>>>>> > >> > >>>>> 12:00:00
>>>>>>>> > >> > >>>>> > >> true
>>>>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>>>>>>> > >> > >>>>> > >> plot_20200612_173626.png
>>>>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
>>>>>>>> > >> > >>>>> > >> false list() list() false false false false
false
>>>>>>>> false
>>>>>>>> > >> > >>>>> > >> true
>>>>>>>> > >> > >>>>> > >> false
>>>>>>>> > >> > >>>>> > >> false
>>>>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
>>>>>>>> c(8,4,5,4)
>>>>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>>>>>>> > >> > >>>>> > >> 1.4 -2
>>>>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1
1 -2
>>>>>>>> 0.5 3
>>>>>>>> > >> > >>>>> > >> #cccccc 1
>>>>>>>> > >> > >>>>> > >> listX 1
>>>>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5
o c(0,
>>>>>>>> -.25)
>>>>>>>> > >> > >>>>> > >> 3
>>>>>>>> > >> > >>>>> > >> 0.8 1
>>>>>>>> > >> > >>>>> > >> #333333
>>>>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none")
c(FALSE,FALSE)
>>>>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>>>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
c("b","b")
>>>>>>>> c(1,1)
>>>>>>>> > >> > >>>>> > >> c(1,1)
>>>>>>>> > >> > >>>>> c(1,1)
>>>>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > --
>>>>>>>> > >> > >>>>> > > Edward Strobach
>>>>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>>>>>>> > >> > >>>>> > > IMSG Contractor
>>>>>>>> > >> > >>>>> > > Cubicle#: 2029
>>>>>>>> > >> > >>>>> > > 301-683-3717
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>
>>>>>>>> > >> > >>>> --
>>>>>>>> > >> > >>>> Edward Strobach
>>>>>>>> > >> > >>>> EMC/NCEP/NWS/
>>>>>>>> > >> > >>>> IMSG Contractor
>>>>>>>> > >> > >>>> Cubicle#: 2029
>>>>>>>> > >> > >>>> 301-683-3717
>>>>>>>> > >> > >>>>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> --
>>>>>>>> > >> > >>> Edward Strobach
>>>>>>>> > >> > >>> EMC/NCEP/NWS/
>>>>>>>> > >> > >>> IMSG Contractor
>>>>>>>> > >> > >>> Cubicle#: 2029
>>>>>>>> > >> > >>> 301-683-3717
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>
>>>>>>>> > >> > >>
>>>>>>>> > >> > >> --
>>>>>>>> > >> > >> Edward Strobach
>>>>>>>> > >> > >> EMC/NCEP/NWS/
>>>>>>>> > >> > >> IMSG Contractor
>>>>>>>> > >> > >> Cubicle#: 2029
>>>>>>>> > >> > >> 301-683-3717
>>>>>>>> > >> > >>
>>>>>>>> > >> > >
>>>>>>>> > >> > >
>>>>>>>> > >> > > --
>>>>>>>> > >> > > Edward Strobach
>>>>>>>> > >> > > EMC/NCEP/NWS/
>>>>>>>> > >> > > IMSG Contractor
>>>>>>>> > >> > > Cubicle#: 2029
>>>>>>>> > >> > > 301-683-3717
>>>>>>>> > >> > >
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> > >
>>>>>>>> > > --
>>>>>>>> > > Edward Strobach
>>>>>>>> > > EMC/NCEP/NWS/
>>>>>>>> > > IMSG Contractor
>>>>>>>> > > Cubicle#: 2029
>>>>>>>> > > 301-683-3717
>>>>>>>> > >
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Edward Strobach
>>>>>>> EMC/NCEP/NWS/
>>>>>>> IMSG Contractor
>>>>>>> Cubicle#: 2029
>>>>>>> 301-683-3717
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Edward Strobach
>>>>>> EMC/NCEP/NWS/
>>>>>> IMSG Contractor
>>>>>> Cubicle#: 2029
>>>>>> 301-683-3717
>>>>>>
>>>>> --
>>>>> Jeffery T. McQueen
>>>>> Meteorologist
>>>>> National Center for Weather and Climate Prediction
>>>>> Room 2095
>>>>> 5830 University Research Court
>>>>> College Park, MD 20741
>>>>> 301-683-3736
>>>>> Fax: 301-683-3703
>>>>>
>>>>
>>>>
>>>> --
>>>> Edward Strobach
>>>> EMC/NCEP/NWS/
>>>> IMSG Contractor
>>>> Cubicle#: 2029
>>>> 301-683-3717
>>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 13:47:45 2020
I'm sorry, I guess I may have been a bit confused by what you were
initially telling me. Are you saying replace what I have with this?
<connection>
*<host>rds_host:3306</host>*
* <database>MY_DATABASEdatabase>*
<user>rds_user</user>
<password>rds_pwd</password>
<management_system>aurora</management_system>
</connection>
* <rscript>Rscript</rscript>*
<folders>
* <r_tmpl>rds_R_tmpl</r_tmpl>
<r_work>rds_R_work</r_work>
<plots>rds_plots</plots> <data>rds_data</data>
<scripts>rds_scripts</scripts>*
On Wed, Jun 17, 2020 at 3:27 PM Ho-Chun Huang - NOAA Affiliate <
ho-chun.huang at noaa.gov> wrote:
> Ed:
>
> You can try or you can keep waiting, it is your choice.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Wed, Jun 17, 2020 at 3:21 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> Hi Ho-Chun,
>>
>> I've made all the changes that I would need to inside my xml file.
I was
>> told not to change the folders info. I instead changed my database
name,
>> fcst_var, stat options, series1 input, the fix fields, the indep
field, and
>> x_lim.
>>
>> Thanks
>>
>> On Wed, Jun 17, 2020 at 3:16 PM Ho-Chun Huang - NOAA Affiliate <
>> ho-chun.huang at noaa.gov> wrote:
>>
>>> Hi, Ed:
>>>
>>> You should go back to read 1README again from the directory I sent
about
>>> replacing the top portion of your metviewer xml to produce a batch
xml.
>>>
>>> I check your
>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
>>> Please update the section from <connection> to </folder> as those
in
>>> hochun_xml.template. I also do not know whether AWS can handle
comments
>>> <!-- to !--> in batch xml , better make your batch xml clean.
>>>
>>> Ho-Chun Huang
>>>
>>> IMSG at NOAA/NWS/NCEP/EMC
>>>
>>> 5830 University Research Ct., Rm. 2792
>>>
>>> College Park, MD 20740
>>>
>>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>>
>>> 301-683-3958
>>>
>>>
>>> On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA Affiliate <
>>> edward.strobach at noaa.gov> wrote:
>>>
>>>> Just adding more detail to elaborate my concerns...
>>>>
>>>> Not too long ago I was given a test set-up by Ho-Chun to generate
a
>>>> plot for aerosol optical depth. The result was successful and
produced a
>>>> CSI plot for AOD, comparing the production run and experimental
run.
>>>> Although I created a database, I do not seem to have a group
where I can
>>>> access specific databases. I recently created a load_xml file,
which
>>>> should have loaded the data based on what I specified into the
group. It
>>>> seemed successful based on the log output that was generated.
>>>> Additionally, all information that I specified in this step was
create in
>>>> the load_*xml.
>>>>
>>>> The next step is to generate plots since the data should be
loaded.
>>>> Here's what happens when I run Ho-Chun's - the successful case
>>>>
>>>> HO-CHUN:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
>>>> hochun_xml.template+ bash
>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>>>> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # #
# # # # #
>>>> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT
model,
>>>> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
>>>>
>>>> This is early on in the script.
>>>>
>>>> ED_STROBACH:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed
-e
>>>> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>>>> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20
PM
>>>> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
create
>>>> initial connections of pool.java.sql.SQLException: Communications
link
>>>> failure with primary. No active connection found for master.
at
>>>>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
>>>> at
>>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
>>>> at
>>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
>>>> at
>>>>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
>>>> at
>>>>
org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
>>>> at
>>>>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
>>>> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
>>>>
>>>> My interpretation of this is that it fails to bridge a connection
to
>>>> the group or database; therefore, it's unable to proceed with
running the
>>>> XML file to generate the plots based on the specified content.
If I move
>>>> past this step, then I'm golden.
>>>>
>>>> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>
>>>>> Would it be ok to keep perry in the loop
>>>>>
>>>>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA
Affiliate <
>>>>> edward.strobach at noaa.gov> wrote:
>>>>>
>>>>>> you're right. I didn't have the storage problem this time.
I'm not
>>>>>> sure what changed. All I know is that I have now created my
load xml which
>>>>>> now should allow me to move on to the next step of plotting
>>>>>>
>>>>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA
Affiliate <
>>>>>> edward.strobach at noaa.gov> wrote:
>>>>>>
>>>>>>> I'm not following you. How do I now have enough space? What
>>>>>>> changed?
>>>>>>>
>>>>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>
>>>>>>>> Ed, now you should have enough space for the data loading.
>>>>>>>>
>>>>>>>> Tatiana
>>>>>>>>
>>>>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>>>>>>> > I was basically able to build my xml file but it says I
have no
>>>>>>>> > available
>>>>>>>> > space. I'm not sure how that can be since I just started
adding
>>>>>>>> data
>>>>>>>> > to my
>>>>>>>> > account
>>>>>>>> >
>>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space
>>>>>>>> left
>>>>>>>> > on
>>>>>>>> > device
>>>>>>>> > ERROR: Command returned with non-zero status (1): scp -r
./*
>>>>>>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>>>>>>> > 4) Check METviewer AWS database list using
mv_db_size_on_aws.sh
>>>>>>>> > TABLE_SCHEMA USER
SIZE_MB
>>>>>>>> > mv_met_g2o_gefs binbin.zhou
>>>>>>>> 140027.31
>>>>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
>>>>>>>> 71986.11
>>>>>>>> >
>>>>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate <
>>>>>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>>> >
>>>>>>>> > > Thanks Tatiana. Yes, I've created a database which seems
>>>>>>>> different
>>>>>>>> > > from
>>>>>>>> > > mv_group. How can one create their own group so that I
can
>>>>>>>> store and
>>>>>>>> > > access
>>>>>>>> > > the database. The database exists but cannot be viewed
on the
>>>>>>>> > > metviewer
>>>>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
vhagerty. I
>>>>>>>> > > named my
>>>>>>>> > > database Ed_Strobach, but that was really intended for
the group
>>>>>>>> > > name.
>>>>>>>> > >
>>>>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>>>>>>> > > <met_help at ucar.edu>
>>>>>>>> > > wrote:
>>>>>>>> > >
>>>>>>>> > >> We don't have database and METviewer running on wcoss.
>>>>>>>> > >> Instead, users use METviewer-wcoss scripts that help to
>>>>>>>> connect to
>>>>>>>> > >> and
>>>>>>>> > >> use METviewer that is running on AWS. That is why XML
>>>>>>>> connection and
>>>>>>>> > >> folder
>>>>>>>> > >> sections look somewhat different than once from the Docs
page.
>>>>>>>> > >> To create a database on AWS you need to run
>>>>>>>> mv_create_db_on_aws.sh
>>>>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>>>>>>> > >> For data loading :
>>>>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>>>>>>> > >> And for batch:
>>>>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>>>>>>> > >>
>>>>>>>> > >> These scripts copy XML and data files if needed to AWS,
adjust
>>>>>>>> data
>>>>>>>> > >> path
>>>>>>>> > >> and users credentials, run METviewer and copy results
back to
>>>>>>>> wcoss.
>>>>>>>> > >>
>>>>>>>> > >> Tatiana
>>>>>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
wrote:
>>>>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
>>>>>>>> -p[db_password]
>>>>>>>> > >> > -e'create
>>>>>>>> > >> > database [db_name]
>>>>>>>> > >> >
>>>>>>>> > >> > would need to be used to create a database. I assume
that
>>>>>>>> an sql
>>>>>>>> > >> > module is
>>>>>>>> > >> > available on wcoss; I know you don't have access to
those
>>>>>>>> machines
>>>>>>>> > >> > Tatiana. If I create my own database with my own
name, then
>>>>>>>> I
>>>>>>>> > >> > should
>>>>>>>> > >> > be
>>>>>>>> > >> > able to proceed with project specific tasks under that
tab.
>>>>>>>> > >> > Something
>>>>>>>> > >> > like
>>>>>>>> > >> > mv_EdwardStrobach.
>>>>>>>> > >> >
>>>>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add data
to
>>>>>>>> using
>>>>>>>> > >> > load_spec.
>>>>>>>> > >> > Once that step is done, then I can run plot_spec.
That's my
>>>>>>>> > >> > takeaway
>>>>>>>> > >> > currently
>>>>>>>> > >> >
>>>>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
>>>>>>>> Affiliate <
>>>>>>>> > >> > edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> >
>>>>>>>> > >> > > Actually, it appears that I need to run the database
>>>>>>>> loading
>>>>>>>> > >> > > module
>>>>>>>> > >> > > first
>>>>>>>> > >> > > if I want to change/add a data source. From there I
can
>>>>>>>> run the
>>>>>>>> > >> > > plotting
>>>>>>>> > >> > > batch module. Is that right? So if I run the
database
>>>>>>>> loading
>>>>>>>> > >> > > module,
>>>>>>>> > >> > > then I can specify my dates and other related info
>>>>>>>> specific to
>>>>>>>> > >> > > the
>>>>>>>> > >> > > dataset. If I create my database with a unique
name, then
>>>>>>>> it
>>>>>>>> > >> > > should
>>>>>>>> > >> > > be
>>>>>>>> > >> > > recognized on the metviewer gui, presumably.
>>>>>>>> > >> > >
>>>>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach -
NOAA
>>>>>>>> Affiliate
>>>>>>>> > >> > > <
>>>>>>>> > >> > > edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> > >
>>>>>>>> > >> > >> It seems that those options must be treated
separately
>>>>>>>> outside
>>>>>>>> > >> > >> of
>>>>>>>> > >> > >> plot_spec. Is it acceptable to create both a
load_spec
>>>>>>>> and
>>>>>>>> > >> > >> plot_spec in
>>>>>>>> > >> > >> the same xml file? Would the definitions in
load_spec be
>>>>>>>> able
>>>>>>>> > >> > >> to
>>>>>>>> > >> > >> carry
>>>>>>>> > >> > >> over into plot_spec? It appears that data loading
and
>>>>>>>> plotting
>>>>>>>> > >> > >> are
>>>>>>>> > >> > >> treated
>>>>>>>> > >> > >> separately.
>>>>>>>> > >> > >>
>>>>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
NOAA
>>>>>>>> > >> > >> Affiliate <
>>>>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> > >>
>>>>>>>> > >> > >>> I do plan to use date_range; however, in an
example
>>>>>>>> provided
>>>>>>>> > >> > >>> by the
>>>>>>>> > >> > >>> link
>>>>>>>> > >> > >>> I see this:
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <load_spec>
>>>>>>>> > >> > >>> <connection>
>>>>>>>> > >> > >>> <host>kemosabe:3306</host>
>>>>>>>> > >> > >>> <database>metvdb_hwt</database>
>>>>>>>> > >> > >>> <user>pgoldenb</user>
>>>>>>>> > >> > >>> <password>pgoldenb</password>
>>>>>>>> > >> > >>> </connection>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>>> > >> > >>> <start>2010051914V</start>
>>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
>>>>>>>> > >> > >>> </hour></date_offset></end>
>>>>>>>> > >> > >>> <inc>3600</inc>
>>>>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
>>>>>>>> > >> > >>> </date_list>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <met_version>V3.0</met_version>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <verbose>false</verbose>
>>>>>>>> > >> > >>> <insert_size>1</insert_size>
>>>>>>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>>>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>>>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>>>>>>> > >> > >>> <group>Group name</group>
>>>>>>>> > >> > >>> <load_stat>true</load_stat>
>>>>>>>> > >> > >>> <load_mode>true</load_mode>
>>>>>>>> > >> > >>> <load_mtd>true</load_mtd>
>>>>>>>> > >> > >>> <load_mpr>false</load_mpr>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>
>>>>>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>>>>>> > >> > >>> <load_val>
>>>>>>>> > >> > >>> <field name="model">
>>>>>>>> > >> > >>> <val>arw</val>
>>>>>>>> > >> > >>> <val>nmm</val>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <field name="vx_mask">
>>>>>>>> > >> > >>> <val>FULL</val>
>>>>>>>> > >> > >>> <val>SWC</val>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>> </load_val>
>>>>>>>> > >> > >>> </load_spec>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> Are similar settings allowed with <plot_spec>? I
see
>>>>>>>> > >> > >>> <connection>,
>>>>>>>> > >> > >>> which seems carried over. I'm looking to specify
my
>>>>>>>> > >> > >>> folder_tmpl in
>>>>>>>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>>>>>>>> > >> > >>> Since my
>>>>>>>> > >> > >>> directory structure looks like this:
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>
>>>>>>>> > >>
>>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> I would do something like this I would think:
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> .......
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>>> > >> > >>> <start>20200612/start>
>>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset></end>
>>>>>>>> > >> > >>> <inc>86400</inc>
>>>>>>>> > >> > >>> <format>yyyyMMdd</format>
>>>>>>>> > >> > >>> </date_list>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <folder_tmpl>
>>>>>>>> > >> > >>>
>>>>>>>> > >>
>>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>>>>>>> > >> > >>> </folder_tmpl>
>>>>>>>> > >> > >>> <load_val>
>>>>>>>> > >> > >>> <field name="model">
>>>>>>>> > >> > >>> <val>prod</val>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>>> > >> > >>> </field>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> ......
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach -
NOAA
>>>>>>>> > >> > >>> Affiliate <
>>>>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>> Thank you. This is a very helpful example
>>>>>>>> > >> > >>>>
>>>>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via
RT
>>>>>>>> > >> > >>>> <met_help at ucar.edu>
>>>>>>>> > >> > >>>> wrote:
>>>>>>>> > >> > >>>>
>>>>>>>> > >> > >>>>> Hi,
>>>>>>>> > >> > >>>>> Here is some examples how to use <date_range>
and
>>>>>>>> > >> > >>>>> <date_range_list> in
>>>>>>>> > >> > >>>>> batch XML.
>>>>>>>> > >> > >>>>> First, you need to defined your range. You can
have
>>>>>>>> more
>>>>>>>> > >> > >>>>> then one
>>>>>>>> > >> > >>>>> <date_range>:
>>>>>>>> > >> > >>>>> <date_range name="may">
>>>>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>>>>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>>>>>>>> > >> > >>>>> </date_range>
>>>>>>>> > >> > >>>>> <date_range name="march">
>>>>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>>>>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>>>>>>>> > >> > >>>>> </date_range>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> and then use one or all date_ranges:
>>>>>>>> > >> > >>>>> <plot_fix>
>>>>>>>> > >> > >>>>> <field name="fcst_init_beg">
>>>>>>>> > >> > >>>>> <date_range name="march"/>
>>>>>>>> > >> > >>>>> <date_range name="may"/>
>>>>>>>> > >> > >>>>> </field>...
>>>>>>>> > >> > >>>>> This configuration would include all dates for
March
>>>>>>>> and May
>>>>>>>> > >> > >>>>> of
>>>>>>>> > >> > >>>>> 2019.
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> If you need more flexible configuration you can
use
>>>>>>>> > >> > >>>>> date_range_list.
>>>>>>>> > >> > >>>>> For example, you want to select first 6 hours
every day
>>>>>>>> > >> > >>>>> beetween
>>>>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00.
Your XML
>>>>>>>> would
>>>>>>>> > >> > >>>>> look
>>>>>>>> > >> > >>>>> like this:
>>>>>>>> > >> > >>>>> <date_range_list name = 'all'>
>>>>>>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
>>>>>>>> > >> > >>>>> <range_end>2020-01-05
00:00:00</range_end>
>>>>>>>> > >> > >>>>> <range_length>360</range_length>
>>>>>>>> > >> > >>>>> <inc>1440</inc>
>>>>>>>> > >> > >>>>> </date_range_list>
>>>>>>>> > >> > >>>>> where the 1st and 2nd values are the start and
end,
>>>>>>>> > >> > >>>>> range_length - the length of a range in
minutes(6
>>>>>>>> houres)
>>>>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
houres)
>>>>>>>> > >> > >>>>> This example would create following ranges:
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>>>>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> Also you can add more parameters to <plot_fix>
to
>>>>>>>> filter
>>>>>>>> > >> > >>>>> your
>>>>>>>> > >> > >>>>> data.
>>>>>>>> > >> > >>>>> For example, if you want to select only dates
with
>>>>>>>> 00:00 you
>>>>>>>> > >> > >>>>> can
>>>>>>>> > >> > >>>>> specify
>>>>>>>> > >> > >>>>> <field name="valid_hour">
>>>>>>>> > >> > >>>>> <val>00</val>
>>>>>>>> > >> > >>>>> </field>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> Tatiana
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
>>>>>>>> wrote:
>>>>>>>> > >> > >>>>> > and if there isn't a better way, I wonder the
>>>>>>>> whereabouts
>>>>>>>> > >> > >>>>> > of a
>>>>>>>> > >> > >>>>> > separate
>>>>>>>> > >> > >>>>> > script that pipes dates (and other relevant
>>>>>>>> information)
>>>>>>>> > >> > >>>>> > into
>>>>>>>> > >> > >>>>> > the XML
>>>>>>>> > >> > >>>>> > file
>>>>>>>> > >> > >>>>> > used to generate the results.
>>>>>>>> > >> > >>>>> >
>>>>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward
Strobach -
>>>>>>>> NOAA
>>>>>>>> > >> > >>>>> > Affiliate <
>>>>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>>> > >> > >>>>> >
>>>>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are
helpful.
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > One last question. Simply typing lines like
this
>>>>>>>> in can
>>>>>>>> > >> > >>>>> > > take
>>>>>>>> > >> > >>>>> > > forever:
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > <plot_fix>
>>>>>>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
>>>>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>>>>>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>>>>>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>>>>>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>>>>>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>>>>>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > Is there a better way? I see date_range,
>>>>>>>> > >> > >>>>> > > date_range_list,
>>>>>>>> > >> > >>>>> > > or
>>>>>>>> > >> > >>>>> > > date_list.
>>>>>>>> > >> > >>>>> > > Can this be used instead of typing
individual
>>>>>>>> values?
>>>>>>>> > >> > >>>>> > > This
>>>>>>>> > >> > >>>>> > > could
>>>>>>>> > >> > >>>>> > > prove
>>>>>>>> > >> > >>>>> > > pretty daunting if one would have to type in
700
>>>>>>>> entries
>>>>>>>> > >> > >>>>> > > each
>>>>>>>> > >> > >>>>> > > time.
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana
Burek via
>>>>>>>> RT
>>>>>>>> > >> > >>>>> > > <met_help at ucar.edu>
>>>>>>>> > >> > >>>>> > > wrote:
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > >> I found a few problems in your XML.
>>>>>>>> > >> > >>>>> > >> The database you are using doesn't have
data with
>>>>>>>> > >> > >>>>> > >> vx_mask
>>>>>>>> > >> > >>>>> > >> 'abcREGabc'.
>>>>>>>> > >> > >>>>> > >> That is why METviewer returns an empty
dataset and
>>>>>>>> > >> > >>>>> > >> can't
>>>>>>>> > >> > >>>>> > >> create a
>>>>>>>> > >> > >>>>> > >> plot.
>>>>>>>> > >> > >>>>> > >> The other problem is how you define
fcst_lead.
>>>>>>>> This
>>>>>>>> > >> > >>>>> > >> database
>>>>>>>> > >> > >>>>> > >> has
>>>>>>>> > >> > >>>>> > >> values
>>>>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
Since
>>>>>>>> you
>>>>>>>> > >> > >>>>> > >> include
>>>>>>>> > >> > >>>>> > >> all
>>>>>>>> > >> > >>>>> > >> fcst_leads
>>>>>>>> > >> > >>>>> > >> this variable can be omitted.
>>>>>>>> > >> > >>>>> > >> To minimize problems with variables values,
I
>>>>>>>> would
>>>>>>>> > >> > >>>>> > >> suggest
>>>>>>>> > >> > >>>>> > >> to
>>>>>>>> > >> > >>>>> > >> create XML
>>>>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>>>>>>>> > >> > >>>>> > >> I created a plot using you configurations
with GUI
>>>>>>>> > >> > >>>>> > >> (without
>>>>>>>> > >> > >>>>> vx_mack
>>>>>>>> > >> > >>>>> > >> and
>>>>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You
can
>>>>>>>> load it
>>>>>>>> > >> > >>>>> > >> back
>>>>>>>> > >> > >>>>> > >> to
>>>>>>>> > >> > >>>>> the
>>>>>>>> > >> > >>>>> > >> page by
>>>>>>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI).
Find
>>>>>>>> the
>>>>>>>> > >> > >>>>> > >> plot,
>>>>>>>> > >> > >>>>> > >> open XML
>>>>>>>> > >> > >>>>> > >> and
>>>>>>>> > >> > >>>>> > >> click on "Load to page". After that you can
>>>>>>>> recreate a
>>>>>>>> > >> > >>>>> > >> plot
>>>>>>>> > >> > >>>>> > >> and
>>>>>>>> > >> > >>>>> edit
>>>>>>>> > >> > >>>>> > >> the
>>>>>>>> > >> > >>>>> > >> configurations. You can also download a new
XML
>>>>>>>> and
>>>>>>>> > >> > >>>>> > >> edit it
>>>>>>>> > >> > >>>>> > >> for
>>>>>>>> > >> > >>>>> > >> batch.
>>>>>>>> > >> > >>>>> > >> I attached the example of load XML to this
>>>>>>>> message.
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> Description of rds parameters:
>>>>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
METviewer
>>>>>>>> R
>>>>>>>> > >> > >>>>> > >> scripts
>>>>>>>> > >> > >>>>> > >> rds_plots - temporary directory where the
plots
>>>>>>>> will be
>>>>>>>> > >> > >>>>> > >> generated
>>>>>>>> > >> > >>>>> > >> rds_data - temporary directory where the
data
>>>>>>>> retrieved
>>>>>>>> > >> > >>>>> > >> from
>>>>>>>> > >> > >>>>> > >> the
>>>>>>>> > >> > >>>>> > >> database
>>>>>>>> > >> > >>>>> > >> and other intermediate files will be stored
>>>>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
scripts
>>>>>>>> for
>>>>>>>> > >> > >>>>> > >> this
>>>>>>>> > >> > >>>>> > >> run
>>>>>>>> > >> > >>>>> will
>>>>>>>> > >> > >>>>> > >> be
>>>>>>>> > >> > >>>>> > >> stored
>>>>>>>> > >> > >>>>> > >> After the execution is done all the files
from
>>>>>>>> > >> > >>>>> > >> temporary
>>>>>>>> > >> > >>>>> directories
>>>>>>>> > >> > >>>>> > >> would be removed.
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> We don't have the offitial user guide for
>>>>>>>> METviewer.
>>>>>>>> > >> > >>>>> > >> There
>>>>>>>> > >> > >>>>> > >> are
>>>>>>>> > >> > >>>>> some
>>>>>>>> > >> > >>>>> > >> instructions on how to crate XML config
files:
>>>>>>>> > >> > >>>>> > >>
https://metviewer.nws.noaa.gov/doc/index.html
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> Tatiana
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
>>>>>>>> edward.strobach at noaa.gov
>>>>>>>> > >> > >>>>> > >> wrote:
>>>>>>>> > >> > >>>>> > >> > Hi,
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am
having hard
>>>>>>>> time
>>>>>>>> > >> > >>>>> > >> > following
>>>>>>>> > >> > >>>>> how
>>>>>>>> > >> > >>>>> > >> > to
>>>>>>>> > >> > >>>>> > >> > go
>>>>>>>> > >> > >>>>> > >> > about the other steps since I have zero
>>>>>>>> experience on
>>>>>>>> > >> > >>>>> > >> > how
>>>>>>>> > >> > >>>>> > >> > to do
>>>>>>>> > >> > >>>>> > >> > these
>>>>>>>> > >> > >>>>> > >> > things.
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should
>>>>>>>> NOT be
>>>>>>>> > >> > >>>>> > >> > changed
>>>>>>>> > >> > >>>>> and
>>>>>>>> > >> > >>>>> > >> > will
>>>>>>>> > >> > >>>>> > >> > be
>>>>>>>> > >> > >>>>> > >> > replaced by other scripts during the
batch
>>>>>>>> execution.
>>>>>>>> > >> > >>>>> > >> > The
>>>>>>>> > >> > >>>>> > >> > data
>>>>>>>> > >> > >>>>> for
>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>> > >> > >>>>> > >> > batch mode is coming from the database
and not
>>>>>>>> from
>>>>>>>> > >> > >>>>> > >> > files.
>>>>>>>> > >> > >>>>> > >> > The
>>>>>>>> > >> > >>>>> > >> > name of
>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>> > >> > >>>>> > >> > database is in <database> tag.
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > I understand that nothing should be
changed in
>>>>>>>> the
>>>>>>>> > >> > >>>>> > >> > folders
>>>>>>>> > >> > >>>>> > >> > section,
>>>>>>>> > >> > >>>>> > >> > but
>>>>>>>> > >> > >>>>> > >> > there is no indication how and where
these
>>>>>>>> variables
>>>>>>>> > >> > >>>>> > >> > are
>>>>>>>> > >> > >>>>> > >> > being
>>>>>>>> > >> > >>>>> > >> > defined.
>>>>>>>> > >> > >>>>> > >> > Grepping these names of the files
available
>>>>>>>> does not
>>>>>>>> > >> > >>>>> > >> > reveal
>>>>>>>> > >> > >>>>> formal
>>>>>>>> > >> > >>>>> > >> > definitions of these variables. If I
understood
>>>>>>>> > >> > >>>>> > >> > exactly
>>>>>>>> > >> > >>>>> > >> > how
>>>>>>>> > >> > >>>>> these
>>>>>>>> > >> > >>>>> > >> > were
>>>>>>>> > >> > >>>>> > >> > being defined, then I would be able
connect this
>>>>>>>> > >> > >>>>> > >> > better.
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
files.
>>>>>>>> They
>>>>>>>> > >> > >>>>> > >> > use
>>>>>>>> > >> > >>>>> METviewer
>>>>>>>> > >> > >>>>> > >> > load
>>>>>>>> > >> > >>>>> > >> > mode to load these files to the specific
>>>>>>>> database. In
>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>> > >> > >>>>> > >> > config
>>>>>>>> > >> > >>>>> > >> > file
>>>>>>>> > >> > >>>>> > >> > for
>>>>>>>> > >> > >>>>> > >> > this process they also can specify the
database
>>>>>>>> > >> > >>>>> > >> > group. For
>>>>>>>> > >> > >>>>> example
>>>>>>>> > >> > >>>>> > >> > database
>>>>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>>>>>>> > >> > >>>>> > >> > After the data has been loaded to the
database
>>>>>>>> it is
>>>>>>>> > >> > >>>>> > >> > visible and
>>>>>>>> > >> > >>>>> > >> > available
>>>>>>>> > >> > >>>>> > >> > in the GUI and also can be used by the
batch
>>>>>>>> mode.
>>>>>>>> > >> > >>>>> > >> > You can always add more data to the
database
>>>>>>>> using
>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>> > >> > >>>>> > >> > same load
>>>>>>>> > >> > >>>>> > >> > mode.
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > Is there a guide for me to understand the
>>>>>>>> process
>>>>>>>> > >> > >>>>> > >> > that you
>>>>>>>> > >> > >>>>> > >> > are
>>>>>>>> > >> > >>>>> > >> > referring to?
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek
>>>>>>>> via RT
>>>>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>>>>>>> > >> > >>>>> > >> > wrote:
>>>>>>>> > >> > >>>>> > >> >
>>>>>>>> > >> > >>>>> > >> > > Hi Edward,
>>>>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
servers and
>>>>>>>> > >> > >>>>> > >> > > can't
>>>>>>>> > >> > >>>>> > >> > > view your
>>>>>>>> > >> > >>>>> > >> > > XML
>>>>>>>> > >> > >>>>> > >> > > for the
>>>>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue
>>>>>>>> and I
>>>>>>>> > >> > >>>>> > >> > > will
>>>>>>>> > >> > >>>>> > >> > > take a
>>>>>>>> > >> > >>>>> > >> > > look.
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
>>>>>>>> should NOT
>>>>>>>> > >> > >>>>> > >> > > be
>>>>>>>> > >> > >>>>> > >> > > changed
>>>>>>>> > >> > >>>>> > >> > > and
>>>>>>>> > >> > >>>>> > >> > > will be
>>>>>>>> > >> > >>>>> > >> > > replaced by other scripts during the
batch
>>>>>>>> > >> > >>>>> > >> > > execution.
>>>>>>>> > >> > >>>>> > >> > > The data
>>>>>>>> > >> > >>>>> > >> > > for
>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>> > >> > >>>>> > >> > > batch mode is coming from the database
and
>>>>>>>> not from
>>>>>>>> > >> > >>>>> > >> > > files. The
>>>>>>>> > >> > >>>>> > >> > > name
>>>>>>>> > >> > >>>>> > >> > > of the
>>>>>>>> > >> > >>>>> > >> > > database is in <database> tag.
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET
output
>>>>>>>> files.
>>>>>>>> > >> > >>>>> > >> > > They use
>>>>>>>> > >> > >>>>> > >> > > METviewer
>>>>>>>> > >> > >>>>> > >> > > load
>>>>>>>> > >> > >>>>> > >> > > mode to load these files to the
specific
>>>>>>>> database.
>>>>>>>> > >> > >>>>> > >> > > In
>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>> > >> > >>>>> config
>>>>>>>> > >> > >>>>> > >> > > file
>>>>>>>> > >> > >>>>> > >> > > for
>>>>>>>> > >> > >>>>> > >> > > this process they also can specify the
>>>>>>>> database
>>>>>>>> > >> > >>>>> > >> > > group.
>>>>>>>> > >> > >>>>> > >> > > For
>>>>>>>> > >> > >>>>> > >> > > example
>>>>>>>> > >> > >>>>> > >> > > database
>>>>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>>>>>>> > >> > >>>>> > >> > > After the data has been loaded to the
>>>>>>>> database it
>>>>>>>> > >> > >>>>> > >> > > is
>>>>>>>> > >> > >>>>> > >> > > visible
>>>>>>>> > >> > >>>>> and
>>>>>>>> > >> > >>>>> > >> > > available
>>>>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch
>>>>>>>> mode.
>>>>>>>> > >> > >>>>> > >> > > You can always add more data to the
database
>>>>>>>> using
>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>> > >> > >>>>> > >> > > same
>>>>>>>> > >> > >>>>> load
>>>>>>>> > >> > >>>>> > >> > > mode.
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > > Tatiana
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>>>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>>>>>>> > >> > >>>>> > >> > > wrote:
>>>>>>>> > >> > >>>>> > >> > > > Good morning,
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
effort
>>>>>>>> to
>>>>>>>> > >> > >>>>> > >> > > > understand
>>>>>>>> > >> > >>>>> > >> > > > metviewer,
>>>>>>>> > >> > >>>>> > >> > > > test
>>>>>>>> > >> > >>>>> > >> > > > different cases using the history
tab,
>>>>>>>> changing
>>>>>>>> > >> > >>>>> > >> > > > information
>>>>>>>> > >> > >>>>> to
>>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>>> > >> > >>>>> > >> > > > what
>>>>>>>> > >> > >>>>> > >> > > > works and what does not, and make
>>>>>>>> connections
>>>>>>>> > >> > >>>>> > >> > > > between
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> GUI
>>>>>>>> > >> > >>>>> > >> > > > settings
>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
level
>>>>>>>> with
>>>>>>>> > >> > >>>>> > >> > > > metviewer has
>>>>>>>> > >> > >>>>> > >> > > > significantly
>>>>>>>> > >> > >>>>> > >> > > > increased during this time, there are
some
>>>>>>>> > >> > >>>>> > >> > > > remaining
>>>>>>>> > >> > >>>>> > >> > > > issues
>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>> > >> > >>>>> > >> > > > I'm
>>>>>>>> > >> > >>>>> > >> > > > hoping
>>>>>>>> > >> > >>>>> > >> > > > to clear up.
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
running
>>>>>>>> with
>>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>>> > >> > >>>>> > >> > > > statistic
>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>>> > >> > >>>>> > >> > > > type:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
provided
>>>>>>>> me
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> > >> > > > working
>>>>>>>> > >> > >>>>> > >> > > > directory
>>>>>>>> > >> > >>>>> > >> > > > to get
>>>>>>>> > >> > >>>>> > >> > > > started which I copied over:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I
adjusted and
>>>>>>>> used
>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>> > >> > >>>>> > >> > > > run his
>>>>>>>> > >> > >>>>> > >> > > > example.
>>>>>>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
>>>>>>>> produced a
>>>>>>>> > >> > >>>>> > >> > > > CSI
>>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>>> > >> > >>>>> that
>>>>>>>> > >> > >>>>> > >> > > > was
>>>>>>>> > >> > >>>>> > >> > > > stored
>>>>>>>> > >> > >>>>> > >> > > > here:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >>
>>>>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Running the script produced the
expected
>>>>>>>> result.
>>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>>> > >> > >>>>> > >> > > > next
>>>>>>>> > >> > >>>>> > >> > > > natural
>>>>>>>> > >> > >>>>> > >> > > > step was
>>>>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and
create a
>>>>>>>> new xml
>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>> > >> > >>>>> > >> > > > called
>>>>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
create a
>>>>>>>> time
>>>>>>>> > >> > >>>>> > >> > > > series
>>>>>>>> > >> > >>>>> > >> > > > where
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
instead
>>>>>>>> of
>>>>>>>> > >> > >>>>> > >> > > > obs_thresh.
>>>>>>>> > >> > >>>>> > >> > > > Accordingly,
>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
>>>>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>>>>>>> > >> > >>>>> and
>>>>>>>> > >> > >>>>> > >> > > > changed
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was
also
>>>>>>>> > >> > >>>>> > >> > > > tried).
>>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>>> > >> > >>>>> error
>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>> > >> > >>>>> > >> > > > resulted
>>>>>>>> > >> > >>>>> > >> > > > is as follows:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
>>>>>>>> ('2020-05-01
>>>>>>>> > >> > >>>>> > >> > > > 13:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
>>>>>>>> '2020-05-01
>>>>>>>> > >> > >>>>> > >> > > > 16:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
>>>>>>>> '2020-05-01
>>>>>>>> > >> > >>>>> > >> > > > 19:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
>>>>>>>> '2020-05-01
>>>>>>>> > >> > >>>>> > >> > > > 22:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
>>>>>>>> '2020-05-02
>>>>>>>> > >> > >>>>> > >> > > > 01:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
>>>>>>>> '2020-05-02
>>>>>>>> > >> > >>>>> > >> > > > 04:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
>>>>>>>> '2020-05-02
>>>>>>>> > >> > >>>>> > >> > > > 07:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
>>>>>>>> '2020-05-02
>>>>>>>> > >> > >>>>> > >> > > > 10:00:00',
>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
h.stat_header_id
>>>>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>>>>>>> > >> > >>>>> > >> > > > No data was returned from database
>>>>>>>> > >> > >>>>> > >> > > > 2020-06-09
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >>
>>>>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable - **
>>>>>>>> ERROR:
>>>>>>>> > >> > >>>>> > >> > > > Caught
>>>>>>>> > >> > >>>>> > >> > > > class
>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>> in
>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>>>>>>> > >> > >>>>> > >> > > > res): result set contained no data
>>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
>>>>>>>> set
>>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
>>>>>>>> set
>>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Apparently there was no data during
this
>>>>>>>> time. I
>>>>>>>> > >> > >>>>> > >> > > > tried
>>>>>>>> > >> > >>>>> > >> > > > changing
>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>> > >> > >>>>> > >> > > > range
>>>>>>>> > >> > >>>>> > >> > > > of valid times and changed the valid
time to
>>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>>> > >> > >>>>> > >> > > > days
>>>>>>>> > >> > >>>>> > >> > > > just in
>>>>>>>> > >> > >>>>> > >> > > > case
>>>>>>>> > >> > >>>>> > >> > > > data was missing. The issue was the
same.
>>>>>>>> To
>>>>>>>> > >> > >>>>> > >> > > > confirm
>>>>>>>> > >> > >>>>> > >> > > > my
>>>>>>>> > >> > >>>>> > >> > > > logic, I
>>>>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
>>>>>>>> metviewer,
>>>>>>>> > >> > >>>>> > >> > > > which I
>>>>>>>> > >> > >>>>> > >> > > > found
>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>> > >> > >>>>> > >> > > > work.
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > I think I lack an understanding of
how to
>>>>>>>> set up
>>>>>>>> > >> > >>>>> > >> > > > batch
>>>>>>>> > >> > >>>>> > >> > > > mode
>>>>>>>> > >> > >>>>> > >> > > > because
>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>> > >> > >>>>> > >> > > > don't
>>>>>>>> > >> > >>>>> > >> > > > see where these definitions are being
>>>>>>>> explicitly
>>>>>>>> > >> > >>>>> > >> > > > defined:
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > <plot_spec>
>>>>>>>> > >> > >>>>> > >> > > > <connection>
>>>>>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>>
>>>>>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>>>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>>>>>>> > >> > >>>>> > >> > > > </connection>
>>>>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>>>>>>> > >> > >>>>> > >> > > > <folders>
>>>>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>>>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>>>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>>>>>> > >> > >>>>> > >> > > > </folders>
>>>>>>>> > >> > >>>>> > >> > > > <plot>
>>>>>>>> > >> > >>>>> > >> > > >
<template>series_plot.R_tmpl</template>
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
grepping
>>>>>>>> all
>>>>>>>> > >> > >>>>> > >> > > > available
>>>>>>>> > >> > >>>>> files
>>>>>>>> > >> > >>>>> > >> > > > does
>>>>>>>> > >> > >>>>> > >> > > > not
>>>>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being
defined, as
>>>>>>>> well
>>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>>> > >> > >>>>> > >> > > > other
>>>>>>>> > >> > >>>>> > >> > > > similarly
>>>>>>>> > >> > >>>>> > >> > > > defined
>>>>>>>> > >> > >>>>> > >> > > > variables. I think if I know how
these are
>>>>>>>> being
>>>>>>>> > >> > >>>>> > >> > > > defined,
>>>>>>>> > >> > >>>>> > >> > > > then
>>>>>>>> > >> > >>>>> > >> > > > I'll
>>>>>>>> > >> > >>>>> > >> > > > be
>>>>>>>> > >> > >>>>> > >> > > > able to pipe in my information and
process
>>>>>>>> the
>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>> > >> > >>>>> > >> > > > using
>>>>>>>> > >> > >>>>> met
>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>> > >> > >>>>> > >> > > > produce
>>>>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right on
that?
>>>>>>>> > >> > >>>>> > >> > > > Also, if
>>>>>>>> > >> > >>>>> > >> > > > I want
>>>>>>>> > >> > >>>>> > >> > > > to use
>>>>>>>> > >> > >>>>> > >> > > > obs for
>>>>>>>> > >> > >>>>> > >> > > > verification, do I add another line
in the
>>>>>>>> XML
>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>> > >> > >>>>> > >> > > > designates
>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>> > >> > >>>>> > >> > > > location of obs? I assume that other
steps
>>>>>>>> have
>>>>>>>> > >> > >>>>> > >> > > > to be
>>>>>>>> > >> > >>>>> > >> > > > run
>>>>>>>> > >> > >>>>> > >> > > > before
>>>>>>>> > >> > >>>>> > >> > > > all
>>>>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form,
for
>>>>>>>> example,
>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>> > >> > >>>>> > >> > > > would
>>>>>>>> > >> > >>>>> have
>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>> > >> > >>>>> > >> > > > separately
>>>>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then
I
>>>>>>>> would want
>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>> > >> > >>>>> > >> > > > do that
>>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>>> > >> > >>>>> > >> > > > well.
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of
people/projects
>>>>>>>> listed
>>>>>>>> > >> > >>>>> > >> > > > here -
>>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>>> > >> > >>>>> > >> > > > attachment
>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>> > >> > >>>>> > >> > > > How do I get the results I generate
to be
>>>>>>>> > >> > >>>>> > >> > > > uploaded to
>>>>>>>> > >> > >>>>> > >> > > > this
>>>>>>>> > >> > >>>>> > >> > > > location?
>>>>>>>> > >> > >>>>> > >> > > > It's
>>>>>>>> > >> > >>>>> > >> > > > not clear how others are setting this
up.
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >>
>>>>>>>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>>>>>>>> > >> > >>>>> > >> rds_user
>>>>>>>> > >> > >>>>> rds_pwd
>>>>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
rds_plots
>>>>>>>> rds_data
>>>>>>>> > >> > >>>>> rds_scripts
>>>>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>>>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>>>>>>> > >> > >>>>> 2020-05-12
>>>>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
19:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-12
>>>>>>>> > >> > >>>>> 20:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-12
>>>>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
23:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>> > >> > >>>>> 10:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
13:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>> > >> > >>>>> 14:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
17:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>> > >> > >>>>> 18:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
21:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>> > >> > >>>>> 22:00:00
>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
11:00:00
>>>>>>>> 2020-
>>>>>>>> > >> > >>>>> > >> 05-14
>>>>>>>> > >> > >>>>> 12:00:00
>>>>>>>> > >> > >>>>> > >> true
>>>>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>>>>>>> > >> > >>>>> > >> plot_20200612_173626.png
>>>>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
>>>>>>>> > >> > >>>>> > >> false list() list() false false false false
false
>>>>>>>> false
>>>>>>>> > >> > >>>>> > >> true
>>>>>>>> > >> > >>>>> > >> false
>>>>>>>> > >> > >>>>> > >> false
>>>>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
>>>>>>>> c(8,4,5,4)
>>>>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>>>>>>> > >> > >>>>> > >> 1.4 -2
>>>>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1
1 -2
>>>>>>>> 0.5 3
>>>>>>>> > >> > >>>>> > >> #cccccc 1
>>>>>>>> > >> > >>>>> > >> listX 1
>>>>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5
o c(0,
>>>>>>>> -.25)
>>>>>>>> > >> > >>>>> > >> 3
>>>>>>>> > >> > >>>>> > >> 0.8 1
>>>>>>>> > >> > >>>>> > >> #333333
>>>>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none")
c(FALSE,FALSE)
>>>>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>>>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
c("b","b")
>>>>>>>> c(1,1)
>>>>>>>> > >> > >>>>> > >> c(1,1)
>>>>>>>> > >> > >>>>> c(1,1)
>>>>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>> > > --
>>>>>>>> > >> > >>>>> > > Edward Strobach
>>>>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>>>>>>> > >> > >>>>> > > IMSG Contractor
>>>>>>>> > >> > >>>>> > > Cubicle#: 2029
>>>>>>>> > >> > >>>>> > > 301-683-3717
>>>>>>>> > >> > >>>>> > >
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>>
>>>>>>>> > >> > >>>>
>>>>>>>> > >> > >>>> --
>>>>>>>> > >> > >>>> Edward Strobach
>>>>>>>> > >> > >>>> EMC/NCEP/NWS/
>>>>>>>> > >> > >>>> IMSG Contractor
>>>>>>>> > >> > >>>> Cubicle#: 2029
>>>>>>>> > >> > >>>> 301-683-3717
>>>>>>>> > >> > >>>>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>> --
>>>>>>>> > >> > >>> Edward Strobach
>>>>>>>> > >> > >>> EMC/NCEP/NWS/
>>>>>>>> > >> > >>> IMSG Contractor
>>>>>>>> > >> > >>> Cubicle#: 2029
>>>>>>>> > >> > >>> 301-683-3717
>>>>>>>> > >> > >>>
>>>>>>>> > >> > >>
>>>>>>>> > >> > >>
>>>>>>>> > >> > >> --
>>>>>>>> > >> > >> Edward Strobach
>>>>>>>> > >> > >> EMC/NCEP/NWS/
>>>>>>>> > >> > >> IMSG Contractor
>>>>>>>> > >> > >> Cubicle#: 2029
>>>>>>>> > >> > >> 301-683-3717
>>>>>>>> > >> > >>
>>>>>>>> > >> > >
>>>>>>>> > >> > >
>>>>>>>> > >> > > --
>>>>>>>> > >> > > Edward Strobach
>>>>>>>> > >> > > EMC/NCEP/NWS/
>>>>>>>> > >> > > IMSG Contractor
>>>>>>>> > >> > > Cubicle#: 2029
>>>>>>>> > >> > > 301-683-3717
>>>>>>>> > >> > >
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> > >>
>>>>>>>> > >
>>>>>>>> > > --
>>>>>>>> > > Edward Strobach
>>>>>>>> > > EMC/NCEP/NWS/
>>>>>>>> > > IMSG Contractor
>>>>>>>> > > Cubicle#: 2029
>>>>>>>> > > 301-683-3717
>>>>>>>> > >
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Edward Strobach
>>>>>>> EMC/NCEP/NWS/
>>>>>>> IMSG Contractor
>>>>>>> Cubicle#: 2029
>>>>>>> 301-683-3717
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Edward Strobach
>>>>>> EMC/NCEP/NWS/
>>>>>> IMSG Contractor
>>>>>> Cubicle#: 2029
>>>>>> 301-683-3717
>>>>>>
>>>>> --
>>>>> Jeffery T. McQueen
>>>>> Meteorologist
>>>>> National Center for Weather and Climate Prediction
>>>>> Room 2095
>>>>> 5830 University Research Court
>>>>> College Park, MD 20741
>>>>> 301-683-3736
>>>>> Fax: 301-683-3703
>>>>>
>>>>
>>>>
>>>> --
>>>> Edward Strobach
>>>> EMC/NCEP/NWS/
>>>> IMSG Contractor
>>>> Cubicle#: 2029
>>>> 301-683-3717
>>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Ho-Chun Huang - NOAA Affiliate
Time: Wed Jun 17 13:55:25 2020
Ed:
That is correct.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Wed, Jun 17, 2020 at 3:47 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> I'm sorry, I guess I may have been a bit confused by what you were
> initially telling me. Are you saying replace what I have with this?
>
> <connection>
> *<host>rds_host:3306</host>*
> * <database>MY_DATABASEdatabase>*
> <user>rds_user</user>
> <password>rds_pwd</password>
> <management_system>aurora</management_system>
> </connection>
> * <rscript>Rscript</rscript>*
> <folders>
>
>
>
>
> * <r_tmpl>rds_R_tmpl</r_tmpl>
<r_work>rds_R_work</r_work>
> <plots>rds_plots</plots> <data>rds_data</data>
> <scripts>rds_scripts</scripts>*
>
> On Wed, Jun 17, 2020 at 3:27 PM Ho-Chun Huang - NOAA Affiliate <
> ho-chun.huang at noaa.gov> wrote:
>
>> Ed:
>>
>> You can try or you can keep waiting, it is your choice.
>>
>> Ho-Chun Huang
>>
>> IMSG at NOAA/NWS/NCEP/EMC
>>
>> 5830 University Research Ct., Rm. 2792
>>
>> College Park, MD 20740
>>
>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>
>> 301-683-3958
>>
>>
>> On Wed, Jun 17, 2020 at 3:21 PM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> Hi Ho-Chun,
>>>
>>> I've made all the changes that I would need to inside my xml file.
I
>>> was told not to change the folders info. I instead changed my
database
>>> name, fcst_var, stat options, series1 input, the fix fields, the
indep
>>> field, and x_lim.
>>>
>>> Thanks
>>>
>>> On Wed, Jun 17, 2020 at 3:16 PM Ho-Chun Huang - NOAA Affiliate <
>>> ho-chun.huang at noaa.gov> wrote:
>>>
>>>> Hi, Ed:
>>>>
>>>> You should go back to read 1README again from the directory I
sent
>>>> about replacing the top portion of your metviewer xml to produce
a batch
>>>> xml.
>>>>
>>>> I check your
>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
>>>> Please update the section from <connection> to </folder> as those
in
>>>> hochun_xml.template. I also do not know whether AWS can handle
comments
>>>> <!-- to !--> in batch xml , better make your batch xml clean.
>>>>
>>>> Ho-Chun Huang
>>>>
>>>> IMSG at NOAA/NWS/NCEP/EMC
>>>>
>>>> 5830 University Research Ct., Rm. 2792
>>>>
>>>> College Park, MD 20740
>>>>
>>>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>>>
>>>> 301-683-3958
>>>>
>>>>
>>>> On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA Affiliate
<
>>>> edward.strobach at noaa.gov> wrote:
>>>>
>>>>> Just adding more detail to elaborate my concerns...
>>>>>
>>>>> Not too long ago I was given a test set-up by Ho-Chun to
generate a
>>>>> plot for aerosol optical depth. The result was successful and
produced a
>>>>> CSI plot for AOD, comparing the production run and experimental
run.
>>>>> Although I created a database, I do not seem to have a group
where I can
>>>>> access specific databases. I recently created a load_xml file,
which
>>>>> should have loaded the data based on what I specified into the
group. It
>>>>> seemed successful based on the log output that was generated.
>>>>> Additionally, all information that I specified in this step was
create in
>>>>> the load_*xml.
>>>>>
>>>>> The next step is to generate plots since the data should be
loaded.
>>>>> Here's what happens when I run Ho-Chun's - the successful case
>>>>>
>>>>> HO-CHUN:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
>>>>> hochun_xml.template+ bash
>>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>>>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>>>>> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # #
# # # # # #
>>>>> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT
model,
>>>>> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
>>>>>
>>>>> This is early on in the script.
>>>>>
>>>>> ED_STROBACH:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed
-e
>>>>> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
>>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>>>> plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
>>>>> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20
PM
>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
create
>>>>> initial connections of pool.java.sql.SQLException:
Communications link
>>>>> failure with primary. No active connection found for master.
at
>>>>>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
>>>>> at
>>>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
>>>>> at
>>>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
>>>>> at
>>>>>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
>>>>> at
>>>>>
org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
>>>>> at
>>>>>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
>>>>> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
>>>>>
>>>>> My interpretation of this is that it fails to bridge a
connection to
>>>>> the group or database; therefore, it's unable to proceed with
running the
>>>>> XML file to generate the plots based on the specified content.
If I move
>>>>> past this step, then I'm golden.
>>>>>
>>>>> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
>>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>>
>>>>>> Would it be ok to keep perry in the loop
>>>>>>
>>>>>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA
Affiliate <
>>>>>> edward.strobach at noaa.gov> wrote:
>>>>>>
>>>>>>> you're right. I didn't have the storage problem this time.
I'm not
>>>>>>> sure what changed. All I know is that I have now created my
load xml which
>>>>>>> now should allow me to move on to the next step of plotting
>>>>>>>
>>>>>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA
Affiliate <
>>>>>>> edward.strobach at noaa.gov> wrote:
>>>>>>>
>>>>>>>> I'm not following you. How do I now have enough space? What
>>>>>>>> changed?
>>>>>>>>
>>>>>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>
>>>>>>>>> Ed, now you should have enough space for the data loading.
>>>>>>>>>
>>>>>>>>> Tatiana
>>>>>>>>>
>>>>>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>>>>>>>> > I was basically able to build my xml file but it says I
have no
>>>>>>>>> > available
>>>>>>>>> > space. I'm not sure how that can be since I just started
adding
>>>>>>>>> data
>>>>>>>>> > to my
>>>>>>>>> > account
>>>>>>>>> >
>>>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space
>>>>>>>>> left
>>>>>>>>> > on
>>>>>>>>> > device
>>>>>>>>> > ERROR: Command returned with non-zero status (1): scp -r
./*
>>>>>>>>> >
edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>>>>>>>> > 4) Check METviewer AWS database list using
mv_db_size_on_aws.sh
>>>>>>>>> > TABLE_SCHEMA USER
>>>>>>>>> SIZE_MB
>>>>>>>>> > mv_met_g2o_gefs binbin.zhou
>>>>>>>>> 140027.31
>>>>>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
>>>>>>>>> 71986.11
>>>>>>>>> >
>>>>>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate
>>>>>>>>> <
>>>>>>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>>>> >
>>>>>>>>> > > Thanks Tatiana. Yes, I've created a database which
seems
>>>>>>>>> different
>>>>>>>>> > > from
>>>>>>>>> > > mv_group. How can one create their own group so that I
can
>>>>>>>>> store and
>>>>>>>>> > > access
>>>>>>>>> > > the database. The database exists but cannot be viewed
on the
>>>>>>>>> > > metviewer
>>>>>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
vhagerty.
>>>>>>>>> I
>>>>>>>>> > > named my
>>>>>>>>> > > database Ed_Strobach, but that was really intended for
the
>>>>>>>>> group
>>>>>>>>> > > name.
>>>>>>>>> > >
>>>>>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>>>>>>>> > > <met_help at ucar.edu>
>>>>>>>>> > > wrote:
>>>>>>>>> > >
>>>>>>>>> > >> We don't have database and METviewer running on wcoss.
>>>>>>>>> > >> Instead, users use METviewer-wcoss scripts that help to
>>>>>>>>> connect to
>>>>>>>>> > >> and
>>>>>>>>> > >> use METviewer that is running on AWS. That is why XML
>>>>>>>>> connection and
>>>>>>>>> > >> folder
>>>>>>>>> > >> sections look somewhat different than once from the
Docs page.
>>>>>>>>> > >> To create a database on AWS you need to run
>>>>>>>>> mv_create_db_on_aws.sh
>>>>>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>>>>>>>> > >> For data loading :
>>>>>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>>>>>>>> > >> And for batch:
>>>>>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>>>>>>>> > >>
>>>>>>>>> > >> These scripts copy XML and data files if needed to AWS,
>>>>>>>>> adjust data
>>>>>>>>> > >> path
>>>>>>>>> > >> and users credentials, run METviewer and copy results
back to
>>>>>>>>> wcoss.
>>>>>>>>> > >>
>>>>>>>>> > >> Tatiana
>>>>>>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
wrote:
>>>>>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
>>>>>>>>> -p[db_password]
>>>>>>>>> > >> > -e'create
>>>>>>>>> > >> > database [db_name]
>>>>>>>>> > >> >
>>>>>>>>> > >> > would need to be used to create a database. I assume
that
>>>>>>>>> an sql
>>>>>>>>> > >> > module is
>>>>>>>>> > >> > available on wcoss; I know you don't have access to
those
>>>>>>>>> machines
>>>>>>>>> > >> > Tatiana. If I create my own database with my own
name,
>>>>>>>>> then I
>>>>>>>>> > >> > should
>>>>>>>>> > >> > be
>>>>>>>>> > >> > able to proceed with project specific tasks under
that tab.
>>>>>>>>> > >> > Something
>>>>>>>>> > >> > like
>>>>>>>>> > >> > mv_EdwardStrobach.
>>>>>>>>> > >> >
>>>>>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add
data to
>>>>>>>>> using
>>>>>>>>> > >> > load_spec.
>>>>>>>>> > >> > Once that step is done, then I can run plot_spec.
That's my
>>>>>>>>> > >> > takeaway
>>>>>>>>> > >> > currently
>>>>>>>>> > >> >
>>>>>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach -
NOAA
>>>>>>>>> Affiliate <
>>>>>>>>> > >> > edward.strobach at noaa.gov> wrote:
>>>>>>>>> > >> >
>>>>>>>>> > >> > > Actually, it appears that I need to run the
database
>>>>>>>>> loading
>>>>>>>>> > >> > > module
>>>>>>>>> > >> > > first
>>>>>>>>> > >> > > if I want to change/add a data source. From there
I can
>>>>>>>>> run the
>>>>>>>>> > >> > > plotting
>>>>>>>>> > >> > > batch module. Is that right? So if I run the
database
>>>>>>>>> loading
>>>>>>>>> > >> > > module,
>>>>>>>>> > >> > > then I can specify my dates and other related info
>>>>>>>>> specific to
>>>>>>>>> > >> > > the
>>>>>>>>> > >> > > dataset. If I create my database with a unique
name,
>>>>>>>>> then it
>>>>>>>>> > >> > > should
>>>>>>>>> > >> > > be
>>>>>>>>> > >> > > recognized on the metviewer gui, presumably.
>>>>>>>>> > >> > >
>>>>>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach -
NOAA
>>>>>>>>> Affiliate
>>>>>>>>> > >> > > <
>>>>>>>>> > >> > > edward.strobach at noaa.gov> wrote:
>>>>>>>>> > >> > >
>>>>>>>>> > >> > >> It seems that those options must be treated
separately
>>>>>>>>> outside
>>>>>>>>> > >> > >> of
>>>>>>>>> > >> > >> plot_spec. Is it acceptable to create both a
load_spec
>>>>>>>>> and
>>>>>>>>> > >> > >> plot_spec in
>>>>>>>>> > >> > >> the same xml file? Would the definitions in
load_spec
>>>>>>>>> be able
>>>>>>>>> > >> > >> to
>>>>>>>>> > >> > >> carry
>>>>>>>>> > >> > >> over into plot_spec? It appears that data loading
and
>>>>>>>>> plotting
>>>>>>>>> > >> > >> are
>>>>>>>>> > >> > >> treated
>>>>>>>>> > >> > >> separately.
>>>>>>>>> > >> > >>
>>>>>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
NOAA
>>>>>>>>> > >> > >> Affiliate <
>>>>>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>>>>>>>> > >> > >>
>>>>>>>>> > >> > >>> I do plan to use date_range; however, in an
example
>>>>>>>>> provided
>>>>>>>>> > >> > >>> by the
>>>>>>>>> > >> > >>> link
>>>>>>>>> > >> > >>> I see this:
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> <load_spec>
>>>>>>>>> > >> > >>> <connection>
>>>>>>>>> > >> > >>> <host>kemosabe:3306</host>
>>>>>>>>> > >> > >>> <database>metvdb_hwt</database>
>>>>>>>>> > >> > >>> <user>pgoldenb</user>
>>>>>>>>> > >> > >>> <password>pgoldenb</password>
>>>>>>>>> > >> > >>> </connection>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>>>> > >> > >>> <start>2010051914V</start>
>>>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
>>>>>>>>> > >> > >>> </hour></date_offset></end>
>>>>>>>>> > >> > >>> <inc>3600</inc>
>>>>>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
>>>>>>>>> > >> > >>> </date_list>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> <met_version>V3.0</met_version>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> <verbose>false</verbose>
>>>>>>>>> > >> > >>> <insert_size>1</insert_size>
>>>>>>>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>>>>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>>>>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>>>>>>>> > >> > >>> <group>Group name</group>
>>>>>>>>> > >> > >>> <load_stat>true</load_stat>
>>>>>>>>> > >> > >>> <load_mode>true</load_mode>
>>>>>>>>> > >> > >>> <load_mtd>true</load_mtd>
>>>>>>>>> > >> > >>> <load_mpr>false</load_mpr>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>>
>>>>>>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>>>>>>> > >> > >>> <load_val>
>>>>>>>>> > >> > >>> <field name="model">
>>>>>>>>> > >> > >>> <val>arw</val>
>>>>>>>>> > >> > >>> <val>nmm</val>
>>>>>>>>> > >> > >>> </field>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>>>> > >> > >>> </field>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> <field name="vx_mask">
>>>>>>>>> > >> > >>> <val>FULL</val>
>>>>>>>>> > >> > >>> <val>SWC</val>
>>>>>>>>> > >> > >>> </field>
>>>>>>>>> > >> > >>> </load_val>
>>>>>>>>> > >> > >>> </load_spec>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> Are similar settings allowed with <plot_spec>? I
see
>>>>>>>>> > >> > >>> <connection>,
>>>>>>>>> > >> > >>> which seems carried over. I'm looking to specify
my
>>>>>>>>> > >> > >>> folder_tmpl in
>>>>>>>>> > >> > >>> <plot_spec>, which according to Tara, should be
>>>>>>>>> possible.
>>>>>>>>> > >> > >>> Since my
>>>>>>>>> > >> > >>> directory structure looks like this:
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >>
>>>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> I would do something like this I would think:
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> .......
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>>>> > >> > >>> <start>20200612/start>
>>>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset></end>
>>>>>>>>> > >> > >>> <inc>86400</inc>
>>>>>>>>> > >> > >>> <format>yyyyMMdd</format>
>>>>>>>>> > >> > >>> </date_list>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> <folder_tmpl>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >>
>>>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>>>>>>>> > >> > >>> </folder_tmpl>
>>>>>>>>> > >> > >>> <load_val>
>>>>>>>>> > >> > >>> <field name="model">
>>>>>>>>> > >> > >>> <val>prod</val>
>>>>>>>>> > >> > >>> </field>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>>>> > >> > >>> </field>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> ......
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach -
NOAA
>>>>>>>>> > >> > >>> Affiliate <
>>>>>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>>> Thank you. This is a very helpful example
>>>>>>>>> > >> > >>>>
>>>>>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek
via RT
>>>>>>>>> > >> > >>>> <met_help at ucar.edu>
>>>>>>>>> > >> > >>>> wrote:
>>>>>>>>> > >> > >>>>
>>>>>>>>> > >> > >>>>> Hi,
>>>>>>>>> > >> > >>>>> Here is some examples how to use <date_range>
and
>>>>>>>>> > >> > >>>>> <date_range_list> in
>>>>>>>>> > >> > >>>>> batch XML.
>>>>>>>>> > >> > >>>>> First, you need to defined your range. You can
have
>>>>>>>>> more
>>>>>>>>> > >> > >>>>> then one
>>>>>>>>> > >> > >>>>> <date_range>:
>>>>>>>>> > >> > >>>>> <date_range name="may">
>>>>>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>>>>>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>>>>>>>>> > >> > >>>>> </date_range>
>>>>>>>>> > >> > >>>>> <date_range name="march">
>>>>>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>>>>>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>>>>>>>>> > >> > >>>>> </date_range>
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>> and then use one or all date_ranges:
>>>>>>>>> > >> > >>>>> <plot_fix>
>>>>>>>>> > >> > >>>>> <field name="fcst_init_beg">
>>>>>>>>> > >> > >>>>> <date_range name="march"/>
>>>>>>>>> > >> > >>>>> <date_range name="may"/>
>>>>>>>>> > >> > >>>>> </field>...
>>>>>>>>> > >> > >>>>> This configuration would include all dates for
March
>>>>>>>>> and May
>>>>>>>>> > >> > >>>>> of
>>>>>>>>> > >> > >>>>> 2019.
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>> If you need more flexible configuration you can
use
>>>>>>>>> > >> > >>>>> date_range_list.
>>>>>>>>> > >> > >>>>> For example, you want to select first 6 hours
every
>>>>>>>>> day
>>>>>>>>> > >> > >>>>> beetween
>>>>>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00.
Your XML
>>>>>>>>> would
>>>>>>>>> > >> > >>>>> look
>>>>>>>>> > >> > >>>>> like this:
>>>>>>>>> > >> > >>>>> <date_range_list name = 'all'>
>>>>>>>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
>>>>>>>>> > >> > >>>>> <range_end>2020-01-05
00:00:00</range_end>
>>>>>>>>> > >> > >>>>> <range_length>360</range_length>
>>>>>>>>> > >> > >>>>> <inc>1440</inc>
>>>>>>>>> > >> > >>>>> </date_range_list>
>>>>>>>>> > >> > >>>>> where the 1st and 2nd values are the start and
end,
>>>>>>>>> > >> > >>>>> range_length - the length of a range in
minutes(6
>>>>>>>>> houres)
>>>>>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
houres)
>>>>>>>>> > >> > >>>>> This example would create following ranges:
>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
>>>>>>>>> 06:00:00'
>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
>>>>>>>>> 06:00:00'
>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
>>>>>>>>> 06:00:00'
>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
>>>>>>>>> 06:00:00'
>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
>>>>>>>>> 06:00:00'
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>> Also you can add more parameters to <plot_fix>
to
>>>>>>>>> filter
>>>>>>>>> > >> > >>>>> your
>>>>>>>>> > >> > >>>>> data.
>>>>>>>>> > >> > >>>>> For example, if you want to select only dates
with
>>>>>>>>> 00:00 you
>>>>>>>>> > >> > >>>>> can
>>>>>>>>> > >> > >>>>> specify
>>>>>>>>> > >> > >>>>> <field name="valid_hour">
>>>>>>>>> > >> > >>>>> <val>00</val>
>>>>>>>>> > >> > >>>>> </field>
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>> Tatiana
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
>>>>>>>>> wrote:
>>>>>>>>> > >> > >>>>> > and if there isn't a better way, I wonder the
>>>>>>>>> whereabouts
>>>>>>>>> > >> > >>>>> > of a
>>>>>>>>> > >> > >>>>> > separate
>>>>>>>>> > >> > >>>>> > script that pipes dates (and other relevant
>>>>>>>>> information)
>>>>>>>>> > >> > >>>>> > into
>>>>>>>>> > >> > >>>>> > the XML
>>>>>>>>> > >> > >>>>> > file
>>>>>>>>> > >> > >>>>> > used to generate the results.
>>>>>>>>> > >> > >>>>> >
>>>>>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward
Strobach -
>>>>>>>>> NOAA
>>>>>>>>> > >> > >>>>> > Affiliate <
>>>>>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>>>> > >> > >>>>> >
>>>>>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are
helpful.
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>> > > One last question. Simply typing lines
like this
>>>>>>>>> in can
>>>>>>>>> > >> > >>>>> > > take
>>>>>>>>> > >> > >>>>> > > forever:
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>> > > <plot_fix>
>>>>>>>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
>>>>>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>>>>>>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>>>>>>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>>>>>>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>>>>>>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>>>>>>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>> > > Is there a better way? I see date_range,
>>>>>>>>> > >> > >>>>> > > date_range_list,
>>>>>>>>> > >> > >>>>> > > or
>>>>>>>>> > >> > >>>>> > > date_list.
>>>>>>>>> > >> > >>>>> > > Can this be used instead of typing
individual
>>>>>>>>> values?
>>>>>>>>> > >> > >>>>> > > This
>>>>>>>>> > >> > >>>>> > > could
>>>>>>>>> > >> > >>>>> > > prove
>>>>>>>>> > >> > >>>>> > > pretty daunting if one would have to type
in 700
>>>>>>>>> entries
>>>>>>>>> > >> > >>>>> > > each
>>>>>>>>> > >> > >>>>> > > time.
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana
Burek via
>>>>>>>>> RT
>>>>>>>>> > >> > >>>>> > > <met_help at ucar.edu>
>>>>>>>>> > >> > >>>>> > > wrote:
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>> > >> I found a few problems in your XML.
>>>>>>>>> > >> > >>>>> > >> The database you are using doesn't have
data with
>>>>>>>>> > >> > >>>>> > >> vx_mask
>>>>>>>>> > >> > >>>>> > >> 'abcREGabc'.
>>>>>>>>> > >> > >>>>> > >> That is why METviewer returns an empty
dataset
>>>>>>>>> and
>>>>>>>>> > >> > >>>>> > >> can't
>>>>>>>>> > >> > >>>>> > >> create a
>>>>>>>>> > >> > >>>>> > >> plot.
>>>>>>>>> > >> > >>>>> > >> The other problem is how you define
fcst_lead.
>>>>>>>>> This
>>>>>>>>> > >> > >>>>> > >> database
>>>>>>>>> > >> > >>>>> > >> has
>>>>>>>>> > >> > >>>>> > >> values
>>>>>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
Since
>>>>>>>>> you
>>>>>>>>> > >> > >>>>> > >> include
>>>>>>>>> > >> > >>>>> > >> all
>>>>>>>>> > >> > >>>>> > >> fcst_leads
>>>>>>>>> > >> > >>>>> > >> this variable can be omitted.
>>>>>>>>> > >> > >>>>> > >> To minimize problems with variables
values, I
>>>>>>>>> would
>>>>>>>>> > >> > >>>>> > >> suggest
>>>>>>>>> > >> > >>>>> > >> to
>>>>>>>>> > >> > >>>>> > >> create XML
>>>>>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
>>>>>>>>> standards.
>>>>>>>>> > >> > >>>>> > >> I created a plot using you configurations
with
>>>>>>>>> GUI
>>>>>>>>> > >> > >>>>> > >> (without
>>>>>>>>> > >> > >>>>> vx_mack
>>>>>>>>> > >> > >>>>> > >> and
>>>>>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626.
You can
>>>>>>>>> load it
>>>>>>>>> > >> > >>>>> > >> back
>>>>>>>>> > >> > >>>>> > >> to
>>>>>>>>> > >> > >>>>> the
>>>>>>>>> > >> > >>>>> > >> page by
>>>>>>>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI).
Find
>>>>>>>>> the
>>>>>>>>> > >> > >>>>> > >> plot,
>>>>>>>>> > >> > >>>>> > >> open XML
>>>>>>>>> > >> > >>>>> > >> and
>>>>>>>>> > >> > >>>>> > >> click on "Load to page". After that you
can
>>>>>>>>> recreate a
>>>>>>>>> > >> > >>>>> > >> plot
>>>>>>>>> > >> > >>>>> > >> and
>>>>>>>>> > >> > >>>>> edit
>>>>>>>>> > >> > >>>>> > >> the
>>>>>>>>> > >> > >>>>> > >> configurations. You can also download a
new XML
>>>>>>>>> and
>>>>>>>>> > >> > >>>>> > >> edit it
>>>>>>>>> > >> > >>>>> > >> for
>>>>>>>>> > >> > >>>>> > >> batch.
>>>>>>>>> > >> > >>>>> > >> I attached the example of load XML to this
>>>>>>>>> message.
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>> > >> Description of rds parameters:
>>>>>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
>>>>>>>>> METviewer R
>>>>>>>>> > >> > >>>>> > >> scripts
>>>>>>>>> > >> > >>>>> > >> rds_plots - temporary directory where the
plots
>>>>>>>>> will be
>>>>>>>>> > >> > >>>>> > >> generated
>>>>>>>>> > >> > >>>>> > >> rds_data - temporary directory where the
data
>>>>>>>>> retrieved
>>>>>>>>> > >> > >>>>> > >> from
>>>>>>>>> > >> > >>>>> > >> the
>>>>>>>>> > >> > >>>>> > >> database
>>>>>>>>> > >> > >>>>> > >> and other intermediate files will be
stored
>>>>>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
>>>>>>>>> scripts for
>>>>>>>>> > >> > >>>>> > >> this
>>>>>>>>> > >> > >>>>> > >> run
>>>>>>>>> > >> > >>>>> will
>>>>>>>>> > >> > >>>>> > >> be
>>>>>>>>> > >> > >>>>> > >> stored
>>>>>>>>> > >> > >>>>> > >> After the execution is done all the files
from
>>>>>>>>> > >> > >>>>> > >> temporary
>>>>>>>>> > >> > >>>>> directories
>>>>>>>>> > >> > >>>>> > >> would be removed.
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>> > >> We don't have the offitial user guide for
>>>>>>>>> METviewer.
>>>>>>>>> > >> > >>>>> > >> There
>>>>>>>>> > >> > >>>>> > >> are
>>>>>>>>> > >> > >>>>> some
>>>>>>>>> > >> > >>>>> > >> instructions on how to crate XML config
files:
>>>>>>>>> > >> > >>>>> > >>
https://metviewer.nws.noaa.gov/doc/index.html
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>> > >> Tatiana
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
>>>>>>>>> edward.strobach at noaa.gov
>>>>>>>>> > >> > >>>>> > >> wrote:
>>>>>>>>> > >> > >>>>> > >> > Hi,
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am
having
>>>>>>>>> hard time
>>>>>>>>> > >> > >>>>> > >> > following
>>>>>>>>> > >> > >>>>> how
>>>>>>>>> > >> > >>>>> > >> > to
>>>>>>>>> > >> > >>>>> > >> > go
>>>>>>>>> > >> > >>>>> > >> > about the other steps since I have zero
>>>>>>>>> experience on
>>>>>>>>> > >> > >>>>> > >> > how
>>>>>>>>> > >> > >>>>> > >> > to do
>>>>>>>>> > >> > >>>>> > >> > these
>>>>>>>>> > >> > >>>>> > >> > things.
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should
>>>>>>>>> NOT be
>>>>>>>>> > >> > >>>>> > >> > changed
>>>>>>>>> > >> > >>>>> and
>>>>>>>>> > >> > >>>>> > >> > will
>>>>>>>>> > >> > >>>>> > >> > be
>>>>>>>>> > >> > >>>>> > >> > replaced by other scripts during the
batch
>>>>>>>>> execution.
>>>>>>>>> > >> > >>>>> > >> > The
>>>>>>>>> > >> > >>>>> > >> > data
>>>>>>>>> > >> > >>>>> for
>>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>>> > >> > >>>>> > >> > batch mode is coming from the database
and not
>>>>>>>>> from
>>>>>>>>> > >> > >>>>> > >> > files.
>>>>>>>>> > >> > >>>>> > >> > The
>>>>>>>>> > >> > >>>>> > >> > name of
>>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>>> > >> > >>>>> > >> > database is in <database> tag.
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> > I understand that nothing should be
changed in
>>>>>>>>> the
>>>>>>>>> > >> > >>>>> > >> > folders
>>>>>>>>> > >> > >>>>> > >> > section,
>>>>>>>>> > >> > >>>>> > >> > but
>>>>>>>>> > >> > >>>>> > >> > there is no indication how and where
these
>>>>>>>>> variables
>>>>>>>>> > >> > >>>>> > >> > are
>>>>>>>>> > >> > >>>>> > >> > being
>>>>>>>>> > >> > >>>>> > >> > defined.
>>>>>>>>> > >> > >>>>> > >> > Grepping these names of the files
available
>>>>>>>>> does not
>>>>>>>>> > >> > >>>>> > >> > reveal
>>>>>>>>> > >> > >>>>> formal
>>>>>>>>> > >> > >>>>> > >> > definitions of these variables. If I
>>>>>>>>> understood
>>>>>>>>> > >> > >>>>> > >> > exactly
>>>>>>>>> > >> > >>>>> > >> > how
>>>>>>>>> > >> > >>>>> these
>>>>>>>>> > >> > >>>>> > >> > were
>>>>>>>>> > >> > >>>>> > >> > being defined, then I would be able
connect
>>>>>>>>> this
>>>>>>>>> > >> > >>>>> > >> > better.
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
>>>>>>>>> files. They
>>>>>>>>> > >> > >>>>> > >> > use
>>>>>>>>> > >> > >>>>> METviewer
>>>>>>>>> > >> > >>>>> > >> > load
>>>>>>>>> > >> > >>>>> > >> > mode to load these files to the specific
>>>>>>>>> database. In
>>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>>> > >> > >>>>> > >> > config
>>>>>>>>> > >> > >>>>> > >> > file
>>>>>>>>> > >> > >>>>> > >> > for
>>>>>>>>> > >> > >>>>> > >> > this process they also can specify the
database
>>>>>>>>> > >> > >>>>> > >> > group. For
>>>>>>>>> > >> > >>>>> example
>>>>>>>>> > >> > >>>>> > >> > database
>>>>>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>>>>>>>> > >> > >>>>> > >> > After the data has been loaded to the
database
>>>>>>>>> it is
>>>>>>>>> > >> > >>>>> > >> > visible and
>>>>>>>>> > >> > >>>>> > >> > available
>>>>>>>>> > >> > >>>>> > >> > in the GUI and also can be used by the
batch
>>>>>>>>> mode.
>>>>>>>>> > >> > >>>>> > >> > You can always add more data to the
database
>>>>>>>>> using
>>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>>> > >> > >>>>> > >> > same load
>>>>>>>>> > >> > >>>>> > >> > mode.
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> > Is there a guide for me to understand
the
>>>>>>>>> process
>>>>>>>>> > >> > >>>>> > >> > that you
>>>>>>>>> > >> > >>>>> > >> > are
>>>>>>>>> > >> > >>>>> > >> > referring to?
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek
>>>>>>>>> via RT
>>>>>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>>>>>>>> > >> > >>>>> > >> > wrote:
>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>> > >> > >>>>> > >> > > Hi Edward,
>>>>>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
servers
>>>>>>>>> and
>>>>>>>>> > >> > >>>>> > >> > > can't
>>>>>>>>> > >> > >>>>> > >> > > view your
>>>>>>>>> > >> > >>>>> > >> > > XML
>>>>>>>>> > >> > >>>>> > >> > > for the
>>>>>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue
>>>>>>>>> and I
>>>>>>>>> > >> > >>>>> > >> > > will
>>>>>>>>> > >> > >>>>> > >> > > take a
>>>>>>>>> > >> > >>>>> > >> > > look.
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders>
section
>>>>>>>>> should NOT
>>>>>>>>> > >> > >>>>> > >> > > be
>>>>>>>>> > >> > >>>>> > >> > > changed
>>>>>>>>> > >> > >>>>> > >> > > and
>>>>>>>>> > >> > >>>>> > >> > > will be
>>>>>>>>> > >> > >>>>> > >> > > replaced by other scripts during the
batch
>>>>>>>>> > >> > >>>>> > >> > > execution.
>>>>>>>>> > >> > >>>>> > >> > > The data
>>>>>>>>> > >> > >>>>> > >> > > for
>>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>>> > >> > >>>>> > >> > > batch mode is coming from the database
and
>>>>>>>>> not from
>>>>>>>>> > >> > >>>>> > >> > > files. The
>>>>>>>>> > >> > >>>>> > >> > > name
>>>>>>>>> > >> > >>>>> > >> > > of the
>>>>>>>>> > >> > >>>>> > >> > > database is in <database> tag.
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET
output
>>>>>>>>> files.
>>>>>>>>> > >> > >>>>> > >> > > They use
>>>>>>>>> > >> > >>>>> > >> > > METviewer
>>>>>>>>> > >> > >>>>> > >> > > load
>>>>>>>>> > >> > >>>>> > >> > > mode to load these files to the
specific
>>>>>>>>> database.
>>>>>>>>> > >> > >>>>> > >> > > In
>>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>>> > >> > >>>>> config
>>>>>>>>> > >> > >>>>> > >> > > file
>>>>>>>>> > >> > >>>>> > >> > > for
>>>>>>>>> > >> > >>>>> > >> > > this process they also can specify the
>>>>>>>>> database
>>>>>>>>> > >> > >>>>> > >> > > group.
>>>>>>>>> > >> > >>>>> > >> > > For
>>>>>>>>> > >> > >>>>> > >> > > example
>>>>>>>>> > >> > >>>>> > >> > > database
>>>>>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>>>>>>>> > >> > >>>>> > >> > > After the data has been loaded to the
>>>>>>>>> database it
>>>>>>>>> > >> > >>>>> > >> > > is
>>>>>>>>> > >> > >>>>> > >> > > visible
>>>>>>>>> > >> > >>>>> and
>>>>>>>>> > >> > >>>>> > >> > > available
>>>>>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch
>>>>>>>>> mode.
>>>>>>>>> > >> > >>>>> > >> > > You can always add more data to the
database
>>>>>>>>> using
>>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>>> > >> > >>>>> > >> > > same
>>>>>>>>> > >> > >>>>> load
>>>>>>>>> > >> > >>>>> > >> > > mode.
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >> > > Tatiana
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>>>>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>>>>>>>> > >> > >>>>> > >> > > wrote:
>>>>>>>>> > >> > >>>>> > >> > > > Good morning,
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
>>>>>>>>> effort to
>>>>>>>>> > >> > >>>>> > >> > > > understand
>>>>>>>>> > >> > >>>>> > >> > > > metviewer,
>>>>>>>>> > >> > >>>>> > >> > > > test
>>>>>>>>> > >> > >>>>> > >> > > > different cases using the history
tab,
>>>>>>>>> changing
>>>>>>>>> > >> > >>>>> > >> > > > information
>>>>>>>>> > >> > >>>>> to
>>>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>>>> > >> > >>>>> > >> > > > what
>>>>>>>>> > >> > >>>>> > >> > > > works and what does not, and make
>>>>>>>>> connections
>>>>>>>>> > >> > >>>>> > >> > > > between
>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>> > >> > >>>>> GUI
>>>>>>>>> > >> > >>>>> > >> > > > settings
>>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
level
>>>>>>>>> with
>>>>>>>>> > >> > >>>>> > >> > > > metviewer has
>>>>>>>>> > >> > >>>>> > >> > > > significantly
>>>>>>>>> > >> > >>>>> > >> > > > increased during this time, there
are some
>>>>>>>>> > >> > >>>>> > >> > > > remaining
>>>>>>>>> > >> > >>>>> > >> > > > issues
>>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>>> > >> > >>>>> > >> > > > I'm
>>>>>>>>> > >> > >>>>> > >> > > > hoping
>>>>>>>>> > >> > >>>>> > >> > > > to clear up.
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
running
>>>>>>>>> with
>>>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>>>> > >> > >>>>> > >> > > > statistic
>>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>>>> > >> > >>>>> > >> > > > type:
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
>>>>>>>>> provided me
>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>> > >> > >>>>> > >> > > > working
>>>>>>>>> > >> > >>>>> > >> > > > directory
>>>>>>>>> > >> > >>>>> > >> > > > to get
>>>>>>>>> > >> > >>>>> > >> > > > started which I copied over:
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I
adjusted
>>>>>>>>> and used
>>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>>> > >> > >>>>> > >> > > > run his
>>>>>>>>> > >> > >>>>> > >> > > > example.
>>>>>>>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
>>>>>>>>> produced a
>>>>>>>>> > >> > >>>>> > >> > > > CSI
>>>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>>>> > >> > >>>>> that
>>>>>>>>> > >> > >>>>> > >> > > > was
>>>>>>>>> > >> > >>>>> > >> > > > stored
>>>>>>>>> > >> > >>>>> > >> > > > here:
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >>
>>>>>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > Running the script produced the
expected
>>>>>>>>> result.
>>>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>>>> > >> > >>>>> > >> > > > next
>>>>>>>>> > >> > >>>>> > >> > > > natural
>>>>>>>>> > >> > >>>>> > >> > > > step was
>>>>>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and
create a
>>>>>>>>> new xml
>>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>>> > >> > >>>>> > >> > > > called
>>>>>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
create a
>>>>>>>>> time
>>>>>>>>> > >> > >>>>> > >> > > > series
>>>>>>>>> > >> > >>>>> > >> > > > where
>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
instead
>>>>>>>>> of
>>>>>>>>> > >> > >>>>> > >> > > > obs_thresh.
>>>>>>>>> > >> > >>>>> > >> > > > Accordingly,
>>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
>>>>>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>>>>>>>> > >> > >>>>> and
>>>>>>>>> > >> > >>>>> > >> > > > changed
>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR
was also
>>>>>>>>> > >> > >>>>> > >> > > > tried).
>>>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>>>> > >> > >>>>> error
>>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>>> > >> > >>>>> > >> > > > resulted
>>>>>>>>> > >> > >>>>> > >> > > > is as follows:
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
>>>>>>>>> ('2020-05-01
>>>>>>>>> > >> > >>>>> > >> > > > 13:00:00',
>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
>>>>>>>>> '2020-05-01
>>>>>>>>> > >> > >>>>> > >> > > > 16:00:00',
>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
>>>>>>>>> '2020-05-01
>>>>>>>>> > >> > >>>>> > >> > > > 19:00:00',
>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
>>>>>>>>> '2020-05-01
>>>>>>>>> > >> > >>>>> > >> > > > 22:00:00',
>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
>>>>>>>>> '2020-05-02
>>>>>>>>> > >> > >>>>> > >> > > > 01:00:00',
>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
>>>>>>>>> '2020-05-02
>>>>>>>>> > >> > >>>>> > >> > > > 04:00:00',
>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
>>>>>>>>> '2020-05-02
>>>>>>>>> > >> > >>>>> > >> > > > 07:00:00',
>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
>>>>>>>>> '2020-05-02
>>>>>>>>> > >> > >>>>> > >> > > > 10:00:00',
>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>>>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>>>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
h.stat_header_id
>>>>>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>>>>>>>> > >> > >>>>> > >> > > > No data was returned from database
>>>>>>>>> > >> > >>>>> > >> > > > 2020-06-09
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >>
>>>>>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>>>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable -
**
>>>>>>>>> ERROR:
>>>>>>>>> > >> > >>>>> > >> > > > Caught
>>>>>>>>> > >> > >>>>> > >> > > > class
>>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>>> in
>>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>>>>>>>> > >> > >>>>> > >> > > > res): result set contained no data
>>>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
>>>>>>>>> set
>>>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
result
>>>>>>>>> set
>>>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > Apparently there was no data during
this
>>>>>>>>> time. I
>>>>>>>>> > >> > >>>>> > >> > > > tried
>>>>>>>>> > >> > >>>>> > >> > > > changing
>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>> > >> > >>>>> > >> > > > range
>>>>>>>>> > >> > >>>>> > >> > > > of valid times and changed the valid
time
>>>>>>>>> to
>>>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>>>> > >> > >>>>> > >> > > > days
>>>>>>>>> > >> > >>>>> > >> > > > just in
>>>>>>>>> > >> > >>>>> > >> > > > case
>>>>>>>>> > >> > >>>>> > >> > > > data was missing. The issue was the
>>>>>>>>> same. To
>>>>>>>>> > >> > >>>>> > >> > > > confirm
>>>>>>>>> > >> > >>>>> > >> > > > my
>>>>>>>>> > >> > >>>>> > >> > > > logic, I
>>>>>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
>>>>>>>>> metviewer,
>>>>>>>>> > >> > >>>>> > >> > > > which I
>>>>>>>>> > >> > >>>>> > >> > > > found
>>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>>> > >> > >>>>> > >> > > > work.
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > I think I lack an understanding of
how to
>>>>>>>>> set up
>>>>>>>>> > >> > >>>>> > >> > > > batch
>>>>>>>>> > >> > >>>>> > >> > > > mode
>>>>>>>>> > >> > >>>>> > >> > > > because
>>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>>> > >> > >>>>> > >> > > > don't
>>>>>>>>> > >> > >>>>> > >> > > > see where these definitions are
being
>>>>>>>>> explicitly
>>>>>>>>> > >> > >>>>> > >> > > > defined:
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > <plot_spec>
>>>>>>>>> > >> > >>>>> > >> > > > <connection>
>>>>>>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>>
>>>>>>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>>>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>>>>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>>>>>>>> > >> > >>>>> > >> > > > </connection>
>>>>>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>>>>>>>> > >> > >>>>> > >> > > > <folders>
>>>>>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>>>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>>>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>>>>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>>>>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>>>>>>> > >> > >>>>> > >> > > > </folders>
>>>>>>>>> > >> > >>>>> > >> > > > <plot>
>>>>>>>>> > >> > >>>>> > >> > > >
<template>series_plot.R_tmpl</template>
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
grepping
>>>>>>>>> all
>>>>>>>>> > >> > >>>>> > >> > > > available
>>>>>>>>> > >> > >>>>> files
>>>>>>>>> > >> > >>>>> > >> > > > does
>>>>>>>>> > >> > >>>>> > >> > > > not
>>>>>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being
defined, as
>>>>>>>>> well
>>>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>>>> > >> > >>>>> > >> > > > other
>>>>>>>>> > >> > >>>>> > >> > > > similarly
>>>>>>>>> > >> > >>>>> > >> > > > defined
>>>>>>>>> > >> > >>>>> > >> > > > variables. I think if I know how
these
>>>>>>>>> are being
>>>>>>>>> > >> > >>>>> > >> > > > defined,
>>>>>>>>> > >> > >>>>> > >> > > > then
>>>>>>>>> > >> > >>>>> > >> > > > I'll
>>>>>>>>> > >> > >>>>> > >> > > > be
>>>>>>>>> > >> > >>>>> > >> > > > able to pipe in my information and
process
>>>>>>>>> the
>>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>>> > >> > >>>>> > >> > > > using
>>>>>>>>> > >> > >>>>> met
>>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>>> > >> > >>>>> > >> > > > produce
>>>>>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right
on that?
>>>>>>>>> > >> > >>>>> > >> > > > Also, if
>>>>>>>>> > >> > >>>>> > >> > > > I want
>>>>>>>>> > >> > >>>>> > >> > > > to use
>>>>>>>>> > >> > >>>>> > >> > > > obs for
>>>>>>>>> > >> > >>>>> > >> > > > verification, do I add another line
in the
>>>>>>>>> XML
>>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>>> > >> > >>>>> > >> > > > designates
>>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>>> > >> > >>>>> > >> > > > location of obs? I assume that
other
>>>>>>>>> steps have
>>>>>>>>> > >> > >>>>> > >> > > > to be
>>>>>>>>> > >> > >>>>> > >> > > > run
>>>>>>>>> > >> > >>>>> > >> > > > before
>>>>>>>>> > >> > >>>>> > >> > > > all
>>>>>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form,
for
>>>>>>>>> example,
>>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>>> > >> > >>>>> > >> > > > would
>>>>>>>>> > >> > >>>>> have
>>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>>> > >> > >>>>> > >> > > > separately
>>>>>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs, then
I
>>>>>>>>> would want
>>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>>> > >> > >>>>> > >> > > > do that
>>>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>>>> > >> > >>>>> > >> > > > well.
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > 3. Communicating results to
metviewer
>>>>>>>>> online
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of
people/projects
>>>>>>>>> listed
>>>>>>>>> > >> > >>>>> > >> > > > here -
>>>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>>>> > >> > >>>>> > >> > > > attachment
>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>> > >> > >>>>> > >> > > > How do I get the results I generate
to be
>>>>>>>>> > >> > >>>>> > >> > > > uploaded to
>>>>>>>>> > >> > >>>>> > >> > > > this
>>>>>>>>> > >> > >>>>> > >> > > > location?
>>>>>>>>> > >> > >>>>> > >> > > > It's
>>>>>>>>> > >> > >>>>> > >> > > > not clear how others are setting
this up.
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>> > >> > >>>>> > >> rds_host:3306
>>>>>>>>> mv_cmaqv5para8_aod_grid2grid_metplus
>>>>>>>>> > >> > >>>>> > >> rds_user
>>>>>>>>> > >> > >>>>> rds_pwd
>>>>>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
rds_plots
>>>>>>>>> rds_data
>>>>>>>>> > >> > >>>>> rds_scripts
>>>>>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>>>>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>>>>>>>> > >> > >>>>> 2020-05-12
>>>>>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
19:00:00
>>>>>>>>> 2020-
>>>>>>>>> > >> > >>>>> > >> 05-12
>>>>>>>>> > >> > >>>>> 20:00:00
>>>>>>>>> > >> > >>>>> > >> 2020-05-12
>>>>>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
23:00:00
>>>>>>>>> 2020-
>>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>>> > >> > >>>>> 10:00:00
>>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
13:00:00
>>>>>>>>> 2020-
>>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>>> > >> > >>>>> 14:00:00
>>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
17:00:00
>>>>>>>>> 2020-
>>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>>> > >> > >>>>> 18:00:00
>>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
21:00:00
>>>>>>>>> 2020-
>>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>>> > >> > >>>>> 22:00:00
>>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
11:00:00
>>>>>>>>> 2020-
>>>>>>>>> > >> > >>>>> > >> 05-14
>>>>>>>>> > >> > >>>>> 12:00:00
>>>>>>>>> > >> > >>>>> > >> true
>>>>>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>>>>>>>> > >> > >>>>> > >> plot_20200612_173626.png
>>>>>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
>>>>>>>>> > >> > >>>>> > >> false list() list() false false false
false
>>>>>>>>> false false
>>>>>>>>> > >> > >>>>> > >> true
>>>>>>>>> > >> > >>>>> > >> false
>>>>>>>>> > >> > >>>>> > >> false
>>>>>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
>>>>>>>>> c(8,4,5,4)
>>>>>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>>>>>>>> > >> > >>>>> > >> 1.4 -2
>>>>>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1
1 1 -2
>>>>>>>>> 0.5 3
>>>>>>>>> > >> > >>>>> > >> #cccccc 1
>>>>>>>>> > >> > >>>>> > >> listX 1
>>>>>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5
o
>>>>>>>>> c(0, -.25)
>>>>>>>>> > >> > >>>>> > >> 3
>>>>>>>>> > >> > >>>>> > >> 0.8 1
>>>>>>>>> > >> > >>>>> > >> #333333
>>>>>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none")
c(FALSE,FALSE)
>>>>>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>>>>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
c("b","b")
>>>>>>>>> c(1,1)
>>>>>>>>> > >> > >>>>> > >> c(1,1)
>>>>>>>>> > >> > >>>>> c(1,1)
>>>>>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>> > > --
>>>>>>>>> > >> > >>>>> > > Edward Strobach
>>>>>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>>>>>>>> > >> > >>>>> > > IMSG Contractor
>>>>>>>>> > >> > >>>>> > > Cubicle#: 2029
>>>>>>>>> > >> > >>>>> > > 301-683-3717
>>>>>>>>> > >> > >>>>> > >
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>>
>>>>>>>>> > >> > >>>>
>>>>>>>>> > >> > >>>> --
>>>>>>>>> > >> > >>>> Edward Strobach
>>>>>>>>> > >> > >>>> EMC/NCEP/NWS/
>>>>>>>>> > >> > >>>> IMSG Contractor
>>>>>>>>> > >> > >>>> Cubicle#: 2029
>>>>>>>>> > >> > >>>> 301-683-3717
>>>>>>>>> > >> > >>>>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>> --
>>>>>>>>> > >> > >>> Edward Strobach
>>>>>>>>> > >> > >>> EMC/NCEP/NWS/
>>>>>>>>> > >> > >>> IMSG Contractor
>>>>>>>>> > >> > >>> Cubicle#: 2029
>>>>>>>>> > >> > >>> 301-683-3717
>>>>>>>>> > >> > >>>
>>>>>>>>> > >> > >>
>>>>>>>>> > >> > >>
>>>>>>>>> > >> > >> --
>>>>>>>>> > >> > >> Edward Strobach
>>>>>>>>> > >> > >> EMC/NCEP/NWS/
>>>>>>>>> > >> > >> IMSG Contractor
>>>>>>>>> > >> > >> Cubicle#: 2029
>>>>>>>>> > >> > >> 301-683-3717
>>>>>>>>> > >> > >>
>>>>>>>>> > >> > >
>>>>>>>>> > >> > >
>>>>>>>>> > >> > > --
>>>>>>>>> > >> > > Edward Strobach
>>>>>>>>> > >> > > EMC/NCEP/NWS/
>>>>>>>>> > >> > > IMSG Contractor
>>>>>>>>> > >> > > Cubicle#: 2029
>>>>>>>>> > >> > > 301-683-3717
>>>>>>>>> > >> > >
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >>
>>>>>>>>> > >
>>>>>>>>> > > --
>>>>>>>>> > > Edward Strobach
>>>>>>>>> > > EMC/NCEP/NWS/
>>>>>>>>> > > IMSG Contractor
>>>>>>>>> > > Cubicle#: 2029
>>>>>>>>> > > 301-683-3717
>>>>>>>>> > >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Edward Strobach
>>>>>>>> EMC/NCEP/NWS/
>>>>>>>> IMSG Contractor
>>>>>>>> Cubicle#: 2029
>>>>>>>> 301-683-3717
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Edward Strobach
>>>>>>> EMC/NCEP/NWS/
>>>>>>> IMSG Contractor
>>>>>>> Cubicle#: 2029
>>>>>>> 301-683-3717
>>>>>>>
>>>>>> --
>>>>>> Jeffery T. McQueen
>>>>>> Meteorologist
>>>>>> National Center for Weather and Climate Prediction
>>>>>> Room 2095
>>>>>> 5830 University Research Court
>>>>>> College Park, MD 20741
>>>>>> 301-683-3736
>>>>>> Fax: 301-683-3703
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Edward Strobach
>>>>> EMC/NCEP/NWS/
>>>>> IMSG Contractor
>>>>> Cubicle#: 2029
>>>>> 301-683-3717
>>>>>
>>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Wed Jun 17 13:56:29 2020
Yes, you need to have these values in your batch XML.
Tatiana
On Wed Jun 17 13:47:45 2020, edward.strobach at noaa.gov wrote:
> I'm sorry, I guess I may have been a bit confused by what you were
> initially telling me. Are you saying replace what I have with this?
>
> <connection>
> *<host>rds_host:3306</host>*
> * <database>MY_DATABASEdatabase>*
> <user>rds_user</user>
> <password>rds_pwd</password>
> <management_system>aurora</management_system>
> </connection>
> * <rscript>Rscript</rscript>*
> <folders>
>
>
>
>
> * <r_tmpl>rds_R_tmpl</r_tmpl>
> <r_work>rds_R_work</r_work>
> <plots>rds_plots</plots> <data>rds_data</data>
> <scripts>rds_scripts</scripts>*
>
> On Wed, Jun 17, 2020 at 3:27 PM Ho-Chun Huang - NOAA Affiliate <
> ho-chun.huang at noaa.gov> wrote:
>
> > Ed:
> >
> > You can try or you can keep waiting, it is your choice.
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Wed, Jun 17, 2020 at 3:21 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> >> Hi Ho-Chun,
> >>
> >> I've made all the changes that I would need to inside my xml
file.
> >> I was
> >> told not to change the folders info. I instead changed my
database
> >> name,
> >> fcst_var, stat options, series1 input, the fix fields, the indep
> >> field, and
> >> x_lim.
> >>
> >> Thanks
> >>
> >> On Wed, Jun 17, 2020 at 3:16 PM Ho-Chun Huang - NOAA Affiliate <
> >> ho-chun.huang at noaa.gov> wrote:
> >>
> >>> Hi, Ed:
> >>>
> >>> You should go back to read 1README again from the directory I
sent
> >>> about
> >>> replacing the top portion of your metviewer xml to produce a
batch
> >>> xml.
> >>>
> >>> I check your
> >>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
> >>> Please update the section from <connection> to </folder> as
those
> >>> in
> >>> hochun_xml.template. I also do not know whether AWS can handle
> >>> comments
> >>> <!-- to !--> in batch xml , better make your batch xml clean.
> >>>
> >>> Ho-Chun Huang
> >>>
> >>> IMSG at NOAA/NWS/NCEP/EMC
> >>>
> >>> 5830 University Research Ct., Rm. 2792
> >>>
> >>> College Park, MD 20740
> >>>
> >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >>>
> >>> 301-683-3958
> >>>
> >>>
> >>> On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA Affiliate
<
> >>> edward.strobach at noaa.gov> wrote:
> >>>
> >>>> Just adding more detail to elaborate my concerns...
> >>>>
> >>>> Not too long ago I was given a test set-up by Ho-Chun to
generate
> >>>> a
> >>>> plot for aerosol optical depth. The result was successful and
> >>>> produced a
> >>>> CSI plot for AOD, comparing the production run and experimental
> >>>> run.
> >>>> Although I created a database, I do not seem to have a group
where
> >>>> I can
> >>>> access specific databases. I recently created a load_xml file,
> >>>> which
> >>>> should have loaded the data based on what I specified into the
> >>>> group. It
> >>>> seemed successful based on the log output that was generated.
> >>>> Additionally, all information that I specified in this step was
> >>>> create in
> >>>> the load_*xml.
> >>>>
> >>>> The next step is to generate plots since the data should be
> >>>> loaded.
> >>>> Here's what happens when I run Ho-Chun's - the successful case
> >>>>
> >>>> HO-CHUN:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
> >>>> hochun_xml.template+ bash
> >>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> >>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> >>>> plot.xmlCALLING: scp plot.xml
edward.strobach at 205.156.8.85:~----
> >>>> MVBatch
> >>>> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # #
#
> >>>> # # # # #
> >>>> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT
> >>>> model,
> >>>> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
> >>>>
> >>>> This is early on in the script.
> >>>>
> >>>> ED_STROBACH:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+
sed
> >>>> -e
> >>>> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
> >>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> >>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> >>>> plot.xmlCALLING: scp plot.xml
edward.strobach at 205.156.8.85:~----
> >>>> MVBatch
> >>>> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20
PM
> >>>> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable
to
> >>>> create
> >>>> initial connections of pool.java.sql.SQLException:
Communications
> >>>> link
> >>>> failure with primary. No active connection found for master.
> >>>> at
> >>>>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
> >>>> at
> >>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
> >>>> at
> >>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
> >>>> at
> >>>>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
> >>>> at
> >>>>
org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
> >>>> at
> >>>>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
> >>>> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
> >>>>
> >>>> My interpretation of this is that it fails to bridge a
connection
> >>>> to
> >>>> the group or database; therefore, it's unable to proceed with
> >>>> running the
> >>>> XML file to generate the plots based on the specified content.
If
> >>>> I move
> >>>> past this step, then I'm golden.
> >>>>
> >>>> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
> >>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>
> >>>>> Would it be ok to keep perry in the loop
> >>>>>
> >>>>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA
Affiliate
> >>>>> <
> >>>>> edward.strobach at noaa.gov> wrote:
> >>>>>
> >>>>>> you're right. I didn't have the storage problem this time.
I'm
> >>>>>> not
> >>>>>> sure what changed. All I know is that I have now created my
> >>>>>> load xml which
> >>>>>> now should allow me to move on to the next step of plotting
> >>>>>>
> >>>>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA
> >>>>>> Affiliate <
> >>>>>> edward.strobach at noaa.gov> wrote:
> >>>>>>
> >>>>>>> I'm not following you. How do I now have enough space?
What
> >>>>>>> changed?
> >>>>>>>
> >>>>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
> >>>>>>> met_help at ucar.edu> wrote:
> >>>>>>>
> >>>>>>>> Ed, now you should have enough space for the data loading.
> >>>>>>>>
> >>>>>>>> Tatiana
> >>>>>>>>
> >>>>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov
wrote:
> >>>>>>>> > I was basically able to build my xml file but it says I
have
> >>>>>>>> > no
> >>>>>>>> > available
> >>>>>>>> > space. I'm not sure how that can be since I just started
> >>>>>>>> > adding
> >>>>>>>> data
> >>>>>>>> > to my
> >>>>>>>> > account
> >>>>>>>> >
> >>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat:
No
> >>>>>>>> > space
> >>>>>>>> left
> >>>>>>>> > on
> >>>>>>>> > device
> >>>>>>>> > ERROR: Command returned with non-zero status (1): scp -r
./*
> >>>>>>>> >
edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
> >>>>>>>> > 4) Check METviewer AWS database list using
> >>>>>>>> > mv_db_size_on_aws.sh
> >>>>>>>> > TABLE_SCHEMA USER
> >>>>>>>> > SIZE_MB
> >>>>>>>> > mv_met_g2o_gefs binbin.zhou
> >>>>>>>> 140027.31
> >>>>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
> >>>>>>>> 71986.11
> >>>>>>>> >
> >>>>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
> >>>>>>>> > Affiliate <
> >>>>>>>> > edward.strobach at noaa.gov> wrote:
> >>>>>>>> >
> >>>>>>>> > > Thanks Tatiana. Yes, I've created a database which
seems
> >>>>>>>> different
> >>>>>>>> > > from
> >>>>>>>> > > mv_group. How can one create their own group so that I
can
> >>>>>>>> store and
> >>>>>>>> > > access
> >>>>>>>> > > the database. The database exists but cannot be viewed
on
> >>>>>>>> > > the
> >>>>>>>> > > metviewer
> >>>>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
> >>>>>>>> > > vhagerty. I
> >>>>>>>> > > named my
> >>>>>>>> > > database Ed_Strobach, but that was really intended for
the
> >>>>>>>> > > group
> >>>>>>>> > > name.
> >>>>>>>> > >
> >>>>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> >>>>>>>> > > <met_help at ucar.edu>
> >>>>>>>> > > wrote:
> >>>>>>>> > >
> >>>>>>>> > >> We don't have database and METviewer running on wcoss.
> >>>>>>>> > >> Instead, users use METviewer-wcoss scripts that help
to
> >>>>>>>> connect to
> >>>>>>>> > >> and
> >>>>>>>> > >> use METviewer that is running on AWS. That is why XML
> >>>>>>>> connection and
> >>>>>>>> > >> folder
> >>>>>>>> > >> sections look somewhat different than once from the
Docs
> >>>>>>>> > >> page.
> >>>>>>>> > >> To create a database on AWS you need to run
> >>>>>>>> mv_create_db_on_aws.sh
> >>>>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
> >>>>>>>> > >> For data loading :
> >>>>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> >>>>>>>> > >> And for batch:
> >>>>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> >>>>>>>> > >>
> >>>>>>>> > >> These scripts copy XML and data files if needed to
AWS,
> >>>>>>>> > >> adjust
> >>>>>>>> data
> >>>>>>>> > >> path
> >>>>>>>> > >> and users credentials, run METviewer and copy results
> >>>>>>>> > >> back to
> >>>>>>>> wcoss.
> >>>>>>>> > >>
> >>>>>>>> > >> Tatiana
> >>>>>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
> >>>>>>>> > >> wrote:
> >>>>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
> >>>>>>>> -p[db_password]
> >>>>>>>> > >> > -e'create
> >>>>>>>> > >> > database [db_name]
> >>>>>>>> > >> >
> >>>>>>>> > >> > would need to be used to create a database. I
assume
> >>>>>>>> > >> > that
> >>>>>>>> an sql
> >>>>>>>> > >> > module is
> >>>>>>>> > >> > available on wcoss; I know you don't have access to
> >>>>>>>> > >> > those
> >>>>>>>> machines
> >>>>>>>> > >> > Tatiana. If I create my own database with my own
name,
> >>>>>>>> > >> > then
> >>>>>>>> I
> >>>>>>>> > >> > should
> >>>>>>>> > >> > be
> >>>>>>>> > >> > able to proceed with project specific tasks under
that
> >>>>>>>> > >> > tab.
> >>>>>>>> > >> > Something
> >>>>>>>> > >> > like
> >>>>>>>> > >> > mv_EdwardStrobach.
> >>>>>>>> > >> >
> >>>>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add
data
> >>>>>>>> > >> > to
> >>>>>>>> using
> >>>>>>>> > >> > load_spec.
> >>>>>>>> > >> > Once that step is done, then I can run plot_spec.
> >>>>>>>> > >> > That's my
> >>>>>>>> > >> > takeaway
> >>>>>>>> > >> > currently
> >>>>>>>> > >> >
> >>>>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach -
NOAA
> >>>>>>>> Affiliate <
> >>>>>>>> > >> > edward.strobach at noaa.gov> wrote:
> >>>>>>>> > >> >
> >>>>>>>> > >> > > Actually, it appears that I need to run the
database
> >>>>>>>> loading
> >>>>>>>> > >> > > module
> >>>>>>>> > >> > > first
> >>>>>>>> > >> > > if I want to change/add a data source. From there
I
> >>>>>>>> > >> > > can
> >>>>>>>> run the
> >>>>>>>> > >> > > plotting
> >>>>>>>> > >> > > batch module. Is that right? So if I run the
> >>>>>>>> > >> > > database
> >>>>>>>> loading
> >>>>>>>> > >> > > module,
> >>>>>>>> > >> > > then I can specify my dates and other related info
> >>>>>>>> specific to
> >>>>>>>> > >> > > the
> >>>>>>>> > >> > > dataset. If I create my database with a unique
name,
> >>>>>>>> > >> > > then
> >>>>>>>> it
> >>>>>>>> > >> > > should
> >>>>>>>> > >> > > be
> >>>>>>>> > >> > > recognized on the metviewer gui, presumably.
> >>>>>>>> > >> > >
> >>>>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach -
> >>>>>>>> > >> > > NOAA
> >>>>>>>> Affiliate
> >>>>>>>> > >> > > <
> >>>>>>>> > >> > > edward.strobach at noaa.gov> wrote:
> >>>>>>>> > >> > >
> >>>>>>>> > >> > >> It seems that those options must be treated
> >>>>>>>> > >> > >> separately
> >>>>>>>> outside
> >>>>>>>> > >> > >> of
> >>>>>>>> > >> > >> plot_spec. Is it acceptable to create both a
> >>>>>>>> > >> > >> load_spec
> >>>>>>>> and
> >>>>>>>> > >> > >> plot_spec in
> >>>>>>>> > >> > >> the same xml file? Would the definitions in
> >>>>>>>> > >> > >> load_spec be
> >>>>>>>> able
> >>>>>>>> > >> > >> to
> >>>>>>>> > >> > >> carry
> >>>>>>>> > >> > >> over into plot_spec? It appears that data
loading
> >>>>>>>> > >> > >> and
> >>>>>>>> plotting
> >>>>>>>> > >> > >> are
> >>>>>>>> > >> > >> treated
> >>>>>>>> > >> > >> separately.
> >>>>>>>> > >> > >>
> >>>>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
> >>>>>>>> > >> > >> NOAA
> >>>>>>>> > >> > >> Affiliate <
> >>>>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
> >>>>>>>> > >> > >>
> >>>>>>>> > >> > >>> I do plan to use date_range; however, in an
example
> >>>>>>>> provided
> >>>>>>>> > >> > >>> by the
> >>>>>>>> > >> > >>> link
> >>>>>>>> > >> > >>> I see this:
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> <load_spec>
> >>>>>>>> > >> > >>> <connection>
> >>>>>>>> > >> > >>> <host>kemosabe:3306</host>
> >>>>>>>> > >> > >>> <database>metvdb_hwt</database>
> >>>>>>>> > >> > >>> <user>pgoldenb</user>
> >>>>>>>> > >> > >>> <password>pgoldenb</password>
> >>>>>>>> > >> > >>> </connection>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> <date_list name="folder_dates">
> >>>>>>>> > >> > >>> <start>2010051914V</start>
> >>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
> >>>>>>>> > >> > >>> </hour></date_offset></end>
> >>>>>>>> > >> > >>> <inc>3600</inc>
> >>>>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
> >>>>>>>> > >> > >>> </date_list>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> <met_version>V3.0</met_version>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> <verbose>false</verbose>
> >>>>>>>> > >> > >>> <insert_size>1</insert_size>
> >>>>>>>> > >> > >>>
<mode_header_db_check>true</mode_header_db_check>
> >>>>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
> >>>>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
> >>>>>>>> > >> > >>> <group>Group name</group>
> >>>>>>>> > >> > >>> <load_stat>true</load_stat>
> >>>>>>>> > >> > >>> <load_mode>true</load_mode>
> >>>>>>>> > >> > >>> <load_mtd>true</load_mtd>
> >>>>>>>> > >> > >>> <load_mpr>false</load_mpr>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>>
> >>>>>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> >>>>>>>> > >> > >>> <load_val>
> >>>>>>>> > >> > >>> <field name="model">
> >>>>>>>> > >> > >>> <val>arw</val>
> >>>>>>>> > >> > >>> <val>nmm</val>
> >>>>>>>> > >> > >>> </field>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> <field name="valid_time">
> >>>>>>>> > >> > >>> <date_list name="folder_dates"/>
> >>>>>>>> > >> > >>> </field>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> <field name="vx_mask">
> >>>>>>>> > >> > >>> <val>FULL</val>
> >>>>>>>> > >> > >>> <val>SWC</val>
> >>>>>>>> > >> > >>> </field>
> >>>>>>>> > >> > >>> </load_val>
> >>>>>>>> > >> > >>> </load_spec>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> Are similar settings allowed with <plot_spec>?
I
> >>>>>>>> > >> > >>> see
> >>>>>>>> > >> > >>> <connection>,
> >>>>>>>> > >> > >>> which seems carried over. I'm looking to
specify
> >>>>>>>> > >> > >>> my
> >>>>>>>> > >> > >>> folder_tmpl in
> >>>>>>>> > >> > >>> <plot_spec>, which according to Tara, should be
> >>>>>>>> > >> > >>> possible.
> >>>>>>>> > >> > >>> Since my
> >>>>>>>> > >> > >>> directory structure looks like this:
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >>
> >>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> I would do something like this I would think:
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> .......
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> <date_list name="folder_dates">
> >>>>>>>> > >> > >>> <start>20200612/start>
> >>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset></end>
> >>>>>>>> > >> > >>> <inc>86400</inc>
> >>>>>>>> > >> > >>> <format>yyyyMMdd</format>
> >>>>>>>> > >> > >>> </date_list>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> <folder_tmpl>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >>
> >>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> >>>>>>>> > >> > >>> </folder_tmpl>
> >>>>>>>> > >> > >>> <load_val>
> >>>>>>>> > >> > >>> <field name="model">
> >>>>>>>> > >> > >>> <val>prod</val>
> >>>>>>>> > >> > >>> </field>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> <field name="valid_time">
> >>>>>>>> > >> > >>> <date_list name="folder_dates"/>
> >>>>>>>> > >> > >>> </field>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> ......
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach
-
> >>>>>>>> > >> > >>> NOAA
> >>>>>>>> > >> > >>> Affiliate <
> >>>>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>>> Thank you. This is a very helpful example
> >>>>>>>> > >> > >>>>
> >>>>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek
via
> >>>>>>>> > >> > >>>> RT
> >>>>>>>> > >> > >>>> <met_help at ucar.edu>
> >>>>>>>> > >> > >>>> wrote:
> >>>>>>>> > >> > >>>>
> >>>>>>>> > >> > >>>>> Hi,
> >>>>>>>> > >> > >>>>> Here is some examples how to use <date_range>
and
> >>>>>>>> > >> > >>>>> <date_range_list> in
> >>>>>>>> > >> > >>>>> batch XML.
> >>>>>>>> > >> > >>>>> First, you need to defined your range. You can
> >>>>>>>> > >> > >>>>> have
> >>>>>>>> more
> >>>>>>>> > >> > >>>>> then one
> >>>>>>>> > >> > >>>>> <date_range>:
> >>>>>>>> > >> > >>>>> <date_range name="may">
> >>>>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
> >>>>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
> >>>>>>>> > >> > >>>>> </date_range>
> >>>>>>>> > >> > >>>>> <date_range name="march">
> >>>>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
> >>>>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
> >>>>>>>> > >> > >>>>> </date_range>
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>> and then use one or all date_ranges:
> >>>>>>>> > >> > >>>>> <plot_fix>
> >>>>>>>> > >> > >>>>> <field name="fcst_init_beg">
> >>>>>>>> > >> > >>>>> <date_range name="march"/>
> >>>>>>>> > >> > >>>>> <date_range name="may"/>
> >>>>>>>> > >> > >>>>> </field>...
> >>>>>>>> > >> > >>>>> This configuration would include all dates for
> >>>>>>>> > >> > >>>>> March
> >>>>>>>> and May
> >>>>>>>> > >> > >>>>> of
> >>>>>>>> > >> > >>>>> 2019.
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>> If you need more flexible configuration you
can
> >>>>>>>> > >> > >>>>> use
> >>>>>>>> > >> > >>>>> date_range_list.
> >>>>>>>> > >> > >>>>> For example, you want to select first 6 hours
> >>>>>>>> > >> > >>>>> every day
> >>>>>>>> > >> > >>>>> beetween
> >>>>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00.
Your
> >>>>>>>> > >> > >>>>> XML
> >>>>>>>> would
> >>>>>>>> > >> > >>>>> look
> >>>>>>>> > >> > >>>>> like this:
> >>>>>>>> > >> > >>>>> <date_range_list name = 'all'>
> >>>>>>>> > >> > >>>>> <range_start>2020-01-01
> >>>>>>>> > >> > >>>>> 00:00:00</range_start>
> >>>>>>>> > >> > >>>>> <range_end>2020-01-05
> >>>>>>>> > >> > >>>>> 00:00:00</range_end>
> >>>>>>>> > >> > >>>>> <range_length>360</range_length>
> >>>>>>>> > >> > >>>>> <inc>1440</inc>
> >>>>>>>> > >> > >>>>> </date_range_list>
> >>>>>>>> > >> > >>>>> where the 1st and 2nd values are the start and
> >>>>>>>> > >> > >>>>> end,
> >>>>>>>> > >> > >>>>> range_length - the length of a range in
minutes(6
> >>>>>>>> houres)
> >>>>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
> >>>>>>>> > >> > >>>>> houres)
> >>>>>>>> > >> > >>>>> This example would create following ranges:
> >>>>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
> >>>>>>>> > >> > >>>>> 06:00:00'
> >>>>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
> >>>>>>>> > >> > >>>>> 06:00:00'
> >>>>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
> >>>>>>>> > >> > >>>>> 06:00:00'
> >>>>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
> >>>>>>>> > >> > >>>>> 06:00:00'
> >>>>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
> >>>>>>>> > >> > >>>>> 06:00:00'
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>> Also you can add more parameters to <plot_fix>
to
> >>>>>>>> filter
> >>>>>>>> > >> > >>>>> your
> >>>>>>>> > >> > >>>>> data.
> >>>>>>>> > >> > >>>>> For example, if you want to select only dates
> >>>>>>>> > >> > >>>>> with
> >>>>>>>> 00:00 you
> >>>>>>>> > >> > >>>>> can
> >>>>>>>> > >> > >>>>> specify
> >>>>>>>> > >> > >>>>> <field name="valid_hour">
> >>>>>>>> > >> > >>>>> <val>00</val>
> >>>>>>>> > >> > >>>>> </field>
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>> Tatiana
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
> >>>>>>>> > >> > >>>>> edward.strobach at noaa.gov
> >>>>>>>> wrote:
> >>>>>>>> > >> > >>>>> > and if there isn't a better way, I wonder
the
> >>>>>>>> whereabouts
> >>>>>>>> > >> > >>>>> > of a
> >>>>>>>> > >> > >>>>> > separate
> >>>>>>>> > >> > >>>>> > script that pipes dates (and other relevant
> >>>>>>>> information)
> >>>>>>>> > >> > >>>>> > into
> >>>>>>>> > >> > >>>>> > the XML
> >>>>>>>> > >> > >>>>> > file
> >>>>>>>> > >> > >>>>> > used to generate the results.
> >>>>>>>> > >> > >>>>> >
> >>>>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward
> >>>>>>>> > >> > >>>>> > Strobach -
> >>>>>>>> NOAA
> >>>>>>>> > >> > >>>>> > Affiliate <
> >>>>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
> >>>>>>>> > >> > >>>>> >
> >>>>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are
helpful.
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>> > > One last question. Simply typing lines
like
> >>>>>>>> > >> > >>>>> > > this
> >>>>>>>> in can
> >>>>>>>> > >> > >>>>> > > take
> >>>>>>>> > >> > >>>>> > > forever:
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>> > > <plot_fix>
> >>>>>>>> > >> > >>>>> > > <field equalize="false"
> >>>>>>>> > >> > >>>>> > > name="fcst_init_beg">
> >>>>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
> >>>>>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
> >>>>>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
> >>>>>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
> >>>>>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
> >>>>>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>> > > Is there a better way? I see date_range,
> >>>>>>>> > >> > >>>>> > > date_range_list,
> >>>>>>>> > >> > >>>>> > > or
> >>>>>>>> > >> > >>>>> > > date_list.
> >>>>>>>> > >> > >>>>> > > Can this be used instead of typing
individual
> >>>>>>>> values?
> >>>>>>>> > >> > >>>>> > > This
> >>>>>>>> > >> > >>>>> > > could
> >>>>>>>> > >> > >>>>> > > prove
> >>>>>>>> > >> > >>>>> > > pretty daunting if one would have to type
in
> >>>>>>>> > >> > >>>>> > > 700
> >>>>>>>> entries
> >>>>>>>> > >> > >>>>> > > each
> >>>>>>>> > >> > >>>>> > > time.
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana
Burek
> >>>>>>>> > >> > >>>>> > > via
> >>>>>>>> RT
> >>>>>>>> > >> > >>>>> > > <met_help at ucar.edu>
> >>>>>>>> > >> > >>>>> > > wrote:
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>> > >> I found a few problems in your XML.
> >>>>>>>> > >> > >>>>> > >> The database you are using doesn't have
data
> >>>>>>>> > >> > >>>>> > >> with
> >>>>>>>> > >> > >>>>> > >> vx_mask
> >>>>>>>> > >> > >>>>> > >> 'abcREGabc'.
> >>>>>>>> > >> > >>>>> > >> That is why METviewer returns an empty
> >>>>>>>> > >> > >>>>> > >> dataset and
> >>>>>>>> > >> > >>>>> > >> can't
> >>>>>>>> > >> > >>>>> > >> create a
> >>>>>>>> > >> > >>>>> > >> plot.
> >>>>>>>> > >> > >>>>> > >> The other problem is how you define
> >>>>>>>> > >> > >>>>> > >> fcst_lead.
> >>>>>>>> This
> >>>>>>>> > >> > >>>>> > >> database
> >>>>>>>> > >> > >>>>> > >> has
> >>>>>>>> > >> > >>>>> > >> values
> >>>>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
> >>>>>>>> > >> > >>>>> > >> Since
> >>>>>>>> you
> >>>>>>>> > >> > >>>>> > >> include
> >>>>>>>> > >> > >>>>> > >> all
> >>>>>>>> > >> > >>>>> > >> fcst_leads
> >>>>>>>> > >> > >>>>> > >> this variable can be omitted.
> >>>>>>>> > >> > >>>>> > >> To minimize problems with variables
values,
> >>>>>>>> > >> > >>>>> > >> I
> >>>>>>>> would
> >>>>>>>> > >> > >>>>> > >> suggest
> >>>>>>>> > >> > >>>>> > >> to
> >>>>>>>> > >> > >>>>> > >> create XML
> >>>>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
> >>>>>>>> > >> > >>>>> > >> standards.
> >>>>>>>> > >> > >>>>> > >> I created a plot using you configurations
> >>>>>>>> > >> > >>>>> > >> with GUI
> >>>>>>>> > >> > >>>>> > >> (without
> >>>>>>>> > >> > >>>>> vx_mack
> >>>>>>>> > >> > >>>>> > >> and
> >>>>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626.
You
> >>>>>>>> > >> > >>>>> > >> can
> >>>>>>>> load it
> >>>>>>>> > >> > >>>>> > >> back
> >>>>>>>> > >> > >>>>> > >> to
> >>>>>>>> > >> > >>>>> the
> >>>>>>>> > >> > >>>>> > >> page by
> >>>>>>>> > >> > >>>>> > >> using the "History" tab (left bar on
GUI).
> >>>>>>>> > >> > >>>>> > >> Find
> >>>>>>>> the
> >>>>>>>> > >> > >>>>> > >> plot,
> >>>>>>>> > >> > >>>>> > >> open XML
> >>>>>>>> > >> > >>>>> > >> and
> >>>>>>>> > >> > >>>>> > >> click on "Load to page". After that you
can
> >>>>>>>> recreate a
> >>>>>>>> > >> > >>>>> > >> plot
> >>>>>>>> > >> > >>>>> > >> and
> >>>>>>>> > >> > >>>>> edit
> >>>>>>>> > >> > >>>>> > >> the
> >>>>>>>> > >> > >>>>> > >> configurations. You can also download a
new
> >>>>>>>> > >> > >>>>> > >> XML
> >>>>>>>> and
> >>>>>>>> > >> > >>>>> > >> edit it
> >>>>>>>> > >> > >>>>> > >> for
> >>>>>>>> > >> > >>>>> > >> batch.
> >>>>>>>> > >> > >>>>> > >> I attached the example of load XML to
this
> >>>>>>>> message.
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>> > >> Description of rds parameters:
> >>>>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
> >>>>>>>> > >> > >>>>> > >> METviewer
> >>>>>>>> R
> >>>>>>>> > >> > >>>>> > >> scripts
> >>>>>>>> > >> > >>>>> > >> rds_plots - temporary directory where the
> >>>>>>>> > >> > >>>>> > >> plots
> >>>>>>>> will be
> >>>>>>>> > >> > >>>>> > >> generated
> >>>>>>>> > >> > >>>>> > >> rds_data - temporary directory where the
> >>>>>>>> > >> > >>>>> > >> data
> >>>>>>>> retrieved
> >>>>>>>> > >> > >>>>> > >> from
> >>>>>>>> > >> > >>>>> > >> the
> >>>>>>>> > >> > >>>>> > >> database
> >>>>>>>> > >> > >>>>> > >> and other intermediate files will be
stored
> >>>>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
> >>>>>>>> > >> > >>>>> > >> scripts
> >>>>>>>> for
> >>>>>>>> > >> > >>>>> > >> this
> >>>>>>>> > >> > >>>>> > >> run
> >>>>>>>> > >> > >>>>> will
> >>>>>>>> > >> > >>>>> > >> be
> >>>>>>>> > >> > >>>>> > >> stored
> >>>>>>>> > >> > >>>>> > >> After the execution is done all the files
> >>>>>>>> > >> > >>>>> > >> from
> >>>>>>>> > >> > >>>>> > >> temporary
> >>>>>>>> > >> > >>>>> directories
> >>>>>>>> > >> > >>>>> > >> would be removed.
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>> > >> We don't have the offitial user guide for
> >>>>>>>> METviewer.
> >>>>>>>> > >> > >>>>> > >> There
> >>>>>>>> > >> > >>>>> > >> are
> >>>>>>>> > >> > >>>>> some
> >>>>>>>> > >> > >>>>> > >> instructions on how to crate XML config
> >>>>>>>> > >> > >>>>> > >> files:
> >>>>>>>> > >> > >>>>> > >>
https://metviewer.nws.noaa.gov/doc/index.html
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>> > >> Tatiana
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
> >>>>>>>> edward.strobach at noaa.gov
> >>>>>>>> > >> > >>>>> > >> wrote:
> >>>>>>>> > >> > >>>>> > >> > Hi,
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am
having
> >>>>>>>> > >> > >>>>> > >> > hard
> >>>>>>>> time
> >>>>>>>> > >> > >>>>> > >> > following
> >>>>>>>> > >> > >>>>> how
> >>>>>>>> > >> > >>>>> > >> > to
> >>>>>>>> > >> > >>>>> > >> > go
> >>>>>>>> > >> > >>>>> > >> > about the other steps since I have zero
> >>>>>>>> experience on
> >>>>>>>> > >> > >>>>> > >> > how
> >>>>>>>> > >> > >>>>> > >> > to do
> >>>>>>>> > >> > >>>>> > >> > these
> >>>>>>>> > >> > >>>>> > >> > things.
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
> >>>>>>>> > >> > >>>>> > >> > should
> >>>>>>>> NOT be
> >>>>>>>> > >> > >>>>> > >> > changed
> >>>>>>>> > >> > >>>>> and
> >>>>>>>> > >> > >>>>> > >> > will
> >>>>>>>> > >> > >>>>> > >> > be
> >>>>>>>> > >> > >>>>> > >> > replaced by other scripts during the
batch
> >>>>>>>> execution.
> >>>>>>>> > >> > >>>>> > >> > The
> >>>>>>>> > >> > >>>>> > >> > data
> >>>>>>>> > >> > >>>>> for
> >>>>>>>> > >> > >>>>> > >> > the
> >>>>>>>> > >> > >>>>> > >> > batch mode is coming from the database
and
> >>>>>>>> > >> > >>>>> > >> > not
> >>>>>>>> from
> >>>>>>>> > >> > >>>>> > >> > files.
> >>>>>>>> > >> > >>>>> > >> > The
> >>>>>>>> > >> > >>>>> > >> > name of
> >>>>>>>> > >> > >>>>> > >> > the
> >>>>>>>> > >> > >>>>> > >> > database is in <database> tag.
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> > I understand that nothing should be
> >>>>>>>> > >> > >>>>> > >> > changed in
> >>>>>>>> the
> >>>>>>>> > >> > >>>>> > >> > folders
> >>>>>>>> > >> > >>>>> > >> > section,
> >>>>>>>> > >> > >>>>> > >> > but
> >>>>>>>> > >> > >>>>> > >> > there is no indication how and where
these
> >>>>>>>> variables
> >>>>>>>> > >> > >>>>> > >> > are
> >>>>>>>> > >> > >>>>> > >> > being
> >>>>>>>> > >> > >>>>> > >> > defined.
> >>>>>>>> > >> > >>>>> > >> > Grepping these names of the files
> >>>>>>>> > >> > >>>>> > >> > available
> >>>>>>>> does not
> >>>>>>>> > >> > >>>>> > >> > reveal
> >>>>>>>> > >> > >>>>> formal
> >>>>>>>> > >> > >>>>> > >> > definitions of these variables. If I
> >>>>>>>> > >> > >>>>> > >> > understood
> >>>>>>>> > >> > >>>>> > >> > exactly
> >>>>>>>> > >> > >>>>> > >> > how
> >>>>>>>> > >> > >>>>> these
> >>>>>>>> > >> > >>>>> > >> > were
> >>>>>>>> > >> > >>>>> > >> > being defined, then I would be able
> >>>>>>>> > >> > >>>>> > >> > connect this
> >>>>>>>> > >> > >>>>> > >> > better.
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET
output
> >>>>>>>> > >> > >>>>> > >> > files.
> >>>>>>>> They
> >>>>>>>> > >> > >>>>> > >> > use
> >>>>>>>> > >> > >>>>> METviewer
> >>>>>>>> > >> > >>>>> > >> > load
> >>>>>>>> > >> > >>>>> > >> > mode to load these files to the
specific
> >>>>>>>> database. In
> >>>>>>>> > >> > >>>>> > >> > the
> >>>>>>>> > >> > >>>>> > >> > config
> >>>>>>>> > >> > >>>>> > >> > file
> >>>>>>>> > >> > >>>>> > >> > for
> >>>>>>>> > >> > >>>>> > >> > this process they also can specify the
> >>>>>>>> > >> > >>>>> > >> > database
> >>>>>>>> > >> > >>>>> > >> > group. For
> >>>>>>>> > >> > >>>>> example
> >>>>>>>> > >> > >>>>> > >> > database
> >>>>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
> >>>>>>>> > >> > >>>>> > >> > After the data has been loaded to the
> >>>>>>>> > >> > >>>>> > >> > database
> >>>>>>>> it is
> >>>>>>>> > >> > >>>>> > >> > visible and
> >>>>>>>> > >> > >>>>> > >> > available
> >>>>>>>> > >> > >>>>> > >> > in the GUI and also can be used by the
> >>>>>>>> > >> > >>>>> > >> > batch
> >>>>>>>> mode.
> >>>>>>>> > >> > >>>>> > >> > You can always add more data to the
> >>>>>>>> > >> > >>>>> > >> > database
> >>>>>>>> using
> >>>>>>>> > >> > >>>>> > >> > the
> >>>>>>>> > >> > >>>>> > >> > same load
> >>>>>>>> > >> > >>>>> > >> > mode.
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> > Is there a guide for me to understand
the
> >>>>>>>> process
> >>>>>>>> > >> > >>>>> > >> > that you
> >>>>>>>> > >> > >>>>> > >> > are
> >>>>>>>> > >> > >>>>> > >> > referring to?
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM
Tatiana
> >>>>>>>> > >> > >>>>> > >> > Burek
> >>>>>>>> via RT
> >>>>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
> >>>>>>>> > >> > >>>>> > >> > wrote:
> >>>>>>>> > >> > >>>>> > >> >
> >>>>>>>> > >> > >>>>> > >> > > Hi Edward,
> >>>>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
> >>>>>>>> > >> > >>>>> > >> > > servers and
> >>>>>>>> > >> > >>>>> > >> > > can't
> >>>>>>>> > >> > >>>>> > >> > > view your
> >>>>>>>> > >> > >>>>> > >> > > XML
> >>>>>>>> > >> > >>>>> > >> > > for the
> >>>>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to
this
> >>>>>>>> > >> > >>>>> > >> > > issue
> >>>>>>>> and I
> >>>>>>>> > >> > >>>>> > >> > > will
> >>>>>>>> > >> > >>>>> > >> > > take a
> >>>>>>>> > >> > >>>>> > >> > > look.
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders>
section
> >>>>>>>> should NOT
> >>>>>>>> > >> > >>>>> > >> > > be
> >>>>>>>> > >> > >>>>> > >> > > changed
> >>>>>>>> > >> > >>>>> > >> > > and
> >>>>>>>> > >> > >>>>> > >> > > will be
> >>>>>>>> > >> > >>>>> > >> > > replaced by other scripts during the
> >>>>>>>> > >> > >>>>> > >> > > batch
> >>>>>>>> > >> > >>>>> > >> > > execution.
> >>>>>>>> > >> > >>>>> > >> > > The data
> >>>>>>>> > >> > >>>>> > >> > > for
> >>>>>>>> > >> > >>>>> > >> > > the
> >>>>>>>> > >> > >>>>> > >> > > batch mode is coming from the
database
> >>>>>>>> > >> > >>>>> > >> > > and
> >>>>>>>> not from
> >>>>>>>> > >> > >>>>> > >> > > files. The
> >>>>>>>> > >> > >>>>> > >> > > name
> >>>>>>>> > >> > >>>>> > >> > > of the
> >>>>>>>> > >> > >>>>> > >> > > database is in <database> tag.
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET
output
> >>>>>>>> files.
> >>>>>>>> > >> > >>>>> > >> > > They use
> >>>>>>>> > >> > >>>>> > >> > > METviewer
> >>>>>>>> > >> > >>>>> > >> > > load
> >>>>>>>> > >> > >>>>> > >> > > mode to load these files to the
specific
> >>>>>>>> database.
> >>>>>>>> > >> > >>>>> > >> > > In
> >>>>>>>> > >> > >>>>> > >> > > the
> >>>>>>>> > >> > >>>>> config
> >>>>>>>> > >> > >>>>> > >> > > file
> >>>>>>>> > >> > >>>>> > >> > > for
> >>>>>>>> > >> > >>>>> > >> > > this process they also can specify
the
> >>>>>>>> database
> >>>>>>>> > >> > >>>>> > >> > > group.
> >>>>>>>> > >> > >>>>> > >> > > For
> >>>>>>>> > >> > >>>>> > >> > > example
> >>>>>>>> > >> > >>>>> > >> > > database
> >>>>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
> >>>>>>>> > >> > >>>>> > >> > > After the data has been loaded to the
> >>>>>>>> database it
> >>>>>>>> > >> > >>>>> > >> > > is
> >>>>>>>> > >> > >>>>> > >> > > visible
> >>>>>>>> > >> > >>>>> and
> >>>>>>>> > >> > >>>>> > >> > > available
> >>>>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by
the
> >>>>>>>> > >> > >>>>> > >> > > batch
> >>>>>>>> mode.
> >>>>>>>> > >> > >>>>> > >> > > You can always add more data to the
> >>>>>>>> > >> > >>>>> > >> > > database
> >>>>>>>> using
> >>>>>>>> > >> > >>>>> > >> > > the
> >>>>>>>> > >> > >>>>> > >> > > same
> >>>>>>>> > >> > >>>>> load
> >>>>>>>> > >> > >>>>> > >> > > mode.
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >> > > Tatiana
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> >>>>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
> >>>>>>>> > >> > >>>>> > >> > > wrote:
> >>>>>>>> > >> > >>>>> > >> > > > Good morning,
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made
an
> >>>>>>>> > >> > >>>>> > >> > > > effort
> >>>>>>>> to
> >>>>>>>> > >> > >>>>> > >> > > > understand
> >>>>>>>> > >> > >>>>> > >> > > > metviewer,
> >>>>>>>> > >> > >>>>> > >> > > > test
> >>>>>>>> > >> > >>>>> > >> > > > different cases using the history
tab,
> >>>>>>>> changing
> >>>>>>>> > >> > >>>>> > >> > > > information
> >>>>>>>> > >> > >>>>> to
> >>>>>>>> > >> > >>>>> > >> > > > see
> >>>>>>>> > >> > >>>>> > >> > > > what
> >>>>>>>> > >> > >>>>> > >> > > > works and what does not, and make
> >>>>>>>> connections
> >>>>>>>> > >> > >>>>> > >> > > > between
> >>>>>>>> > >> > >>>>> > >> > > > the
> >>>>>>>> > >> > >>>>> GUI
> >>>>>>>> > >> > >>>>> > >> > > > settings
> >>>>>>>> > >> > >>>>> > >> > > > and
> >>>>>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
> >>>>>>>> > >> > >>>>> > >> > > > level
> >>>>>>>> with
> >>>>>>>> > >> > >>>>> > >> > > > metviewer has
> >>>>>>>> > >> > >>>>> > >> > > > significantly
> >>>>>>>> > >> > >>>>> > >> > > > increased during this time, there
are
> >>>>>>>> > >> > >>>>> > >> > > > some
> >>>>>>>> > >> > >>>>> > >> > > > remaining
> >>>>>>>> > >> > >>>>> > >> > > > issues
> >>>>>>>> > >> > >>>>> > >> > > > that
> >>>>>>>> > >> > >>>>> > >> > > > I'm
> >>>>>>>> > >> > >>>>> > >> > > > hoping
> >>>>>>>> > >> > >>>>> > >> > > > to clear up.
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
> >>>>>>>> > >> > >>>>> > >> > > > running
> >>>>>>>> with
> >>>>>>>> > >> > >>>>> > >> > > > different
> >>>>>>>> > >> > >>>>> > >> > > > statistic
> >>>>>>>> > >> > >>>>> > >> > > > and
> >>>>>>>> > >> > >>>>> > >> > > > plot
> >>>>>>>> > >> > >>>>> > >> > > > type:
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
> >>>>>>>> > >> > >>>>> > >> > > > provided
> >>>>>>>> me
> >>>>>>>> > >> > >>>>> > >> > > > the
> >>>>>>>> > >> > >>>>> > >> > > > working
> >>>>>>>> > >> > >>>>> > >> > > > directory
> >>>>>>>> > >> > >>>>> > >> > > > to get
> >>>>>>>> > >> > >>>>> > >> > > > started which I copied over:
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I
> >>>>>>>> > >> > >>>>> > >> > > > adjusted and
> >>>>>>>> used
> >>>>>>>> > >> > >>>>> > >> > > > to
> >>>>>>>> > >> > >>>>> > >> > > > run his
> >>>>>>>> > >> > >>>>> > >> > > > example.
> >>>>>>>> > >> > >>>>> > >> > > > running of the script "bash
test.sh"
> >>>>>>>> produced a
> >>>>>>>> > >> > >>>>> > >> > > > CSI
> >>>>>>>> > >> > >>>>> > >> > > > plot
> >>>>>>>> > >> > >>>>> that
> >>>>>>>> > >> > >>>>> > >> > > > was
> >>>>>>>> > >> > >>>>> > >> > > > stored
> >>>>>>>> > >> > >>>>> > >> > > > here:
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >>
> >>>>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > Running the script produced the
> >>>>>>>> > >> > >>>>> > >> > > > expected
> >>>>>>>> result.
> >>>>>>>> > >> > >>>>> > >> > > > The
> >>>>>>>> > >> > >>>>> > >> > > > next
> >>>>>>>> > >> > >>>>> > >> > > > natural
> >>>>>>>> > >> > >>>>> > >> > > > step was
> >>>>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and
create
> >>>>>>>> > >> > >>>>> > >> > > > a
> >>>>>>>> new xml
> >>>>>>>> > >> > >>>>> > >> > > > file
> >>>>>>>> > >> > >>>>> > >> > > > called
> >>>>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
> >>>>>>>> > >> > >>>>> > >> > > > create a
> >>>>>>>> time
> >>>>>>>> > >> > >>>>> > >> > > > series
> >>>>>>>> > >> > >>>>> > >> > > > where
> >>>>>>>> > >> > >>>>> > >> > > > the
> >>>>>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
> >>>>>>>> > >> > >>>>> > >> > > > instead
> >>>>>>>> of
> >>>>>>>> > >> > >>>>> > >> > > > obs_thresh.
> >>>>>>>> > >> > >>>>> > >> > > > Accordingly,
> >>>>>>>> > >> > >>>>> > >> > > > I
> >>>>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
> >>>>>>>> > >> > >>>>> > >> > > > (removing
> >>>>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
> >>>>>>>> > >> > >>>>> and
> >>>>>>>> > >> > >>>>> > >> > > > changed
> >>>>>>>> > >> > >>>>> > >> > > > the
> >>>>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR
was
> >>>>>>>> > >> > >>>>> > >> > > > also
> >>>>>>>> > >> > >>>>> > >> > > > tried).
> >>>>>>>> > >> > >>>>> > >> > > > The
> >>>>>>>> > >> > >>>>> error
> >>>>>>>> > >> > >>>>> > >> > > > that
> >>>>>>>> > >> > >>>>> > >> > > > resulted
> >>>>>>>> > >> > >>>>> > >> > > > is as follows:
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
> >>>>>>>> ('2020-05-01
> >>>>>>>> > >> > >>>>> > >> > > > 13:00:00',
> >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>>>> > >> > >>>>> > >> > > > 01
> >>>>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
> >>>>>>>> '2020-05-01
> >>>>>>>> > >> > >>>>> > >> > > > 16:00:00',
> >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>>>> > >> > >>>>> > >> > > > 01
> >>>>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
> >>>>>>>> '2020-05-01
> >>>>>>>> > >> > >>>>> > >> > > > 19:00:00',
> >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>>>> > >> > >>>>> > >> > > > 01
> >>>>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
> >>>>>>>> '2020-05-01
> >>>>>>>> > >> > >>>>> > >> > > > 22:00:00',
> >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>>>> > >> > >>>>> > >> > > > 01
> >>>>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
> >>>>>>>> '2020-05-02
> >>>>>>>> > >> > >>>>> > >> > > > 01:00:00',
> >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>>>> > >> > >>>>> > >> > > > 02
> >>>>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
> >>>>>>>> '2020-05-02
> >>>>>>>> > >> > >>>>> > >> > > > 04:00:00',
> >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>>>> > >> > >>>>> > >> > > > 02
> >>>>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
> >>>>>>>> '2020-05-02
> >>>>>>>> > >> > >>>>> > >> > > > 07:00:00',
> >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>>>> > >> > >>>>> > >> > > > 02
> >>>>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
> >>>>>>>> '2020-05-02
> >>>>>>>> > >> > >>>>> > >> > > > 10:00:00',
> >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> >>>>>>>> > >> > >>>>> > >> > > > 02
> >>>>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> >>>>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> >>>>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
> >>>>>>>> > >> > >>>>> > >> > > > h.stat_header_id
> >>>>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
> >>>>>>>> > >> > >>>>> > >> > > > No data was returned from database
> >>>>>>>> > >> > >>>>> > >> > > > 2020-06-09
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >>
> >>>>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >>>>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable -
**
> >>>>>>>> ERROR:
> >>>>>>>> > >> > >>>>> > >> > > > Caught
> >>>>>>>> > >> > >>>>> > >> > > > class
> >>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> >>>>>>>> in
> >>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
> >>>>>>>> > >> > >>>>> > >> > > > res): result set contained no data
> >>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> >>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> >>>>>>>> > >> > >>>>> > >> > > > in
> >>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
> >>>>>>>> > >> > >>>>> > >> > > > result
> >>>>>>>> set
> >>>>>>>> > >> > >>>>> > >> > > > contained no
> >>>>>>>> > >> > >>>>> > >> > > > data
> >>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> >>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
> >>>>>>>> > >> > >>>>> > >> > > > in
> >>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
> >>>>>>>> > >> > >>>>> > >> > > > result
> >>>>>>>> set
> >>>>>>>> > >> > >>>>> > >> > > > contained no
> >>>>>>>> > >> > >>>>> > >> > > > data
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > Apparently there was no data during
> >>>>>>>> > >> > >>>>> > >> > > > this
> >>>>>>>> time. I
> >>>>>>>> > >> > >>>>> > >> > > > tried
> >>>>>>>> > >> > >>>>> > >> > > > changing
> >>>>>>>> > >> > >>>>> > >> > > > the
> >>>>>>>> > >> > >>>>> > >> > > > range
> >>>>>>>> > >> > >>>>> > >> > > > of valid times and changed the
valid
> >>>>>>>> > >> > >>>>> > >> > > > time to
> >>>>>>>> > >> > >>>>> > >> > > > different
> >>>>>>>> > >> > >>>>> > >> > > > days
> >>>>>>>> > >> > >>>>> > >> > > > just in
> >>>>>>>> > >> > >>>>> > >> > > > case
> >>>>>>>> > >> > >>>>> > >> > > > data was missing. The issue was
the
> >>>>>>>> > >> > >>>>> > >> > > > same.
> >>>>>>>> To
> >>>>>>>> > >> > >>>>> > >> > > > confirm
> >>>>>>>> > >> > >>>>> > >> > > > my
> >>>>>>>> > >> > >>>>> > >> > > > logic, I
> >>>>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
> >>>>>>>> metviewer,
> >>>>>>>> > >> > >>>>> > >> > > > which I
> >>>>>>>> > >> > >>>>> > >> > > > found
> >>>>>>>> > >> > >>>>> > >> > > > to
> >>>>>>>> > >> > >>>>> > >> > > > work.
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > I think I lack an understanding of
how
> >>>>>>>> > >> > >>>>> > >> > > > to
> >>>>>>>> set up
> >>>>>>>> > >> > >>>>> > >> > > > batch
> >>>>>>>> > >> > >>>>> > >> > > > mode
> >>>>>>>> > >> > >>>>> > >> > > > because
> >>>>>>>> > >> > >>>>> > >> > > > I
> >>>>>>>> > >> > >>>>> > >> > > > don't
> >>>>>>>> > >> > >>>>> > >> > > > see where these definitions are
being
> >>>>>>>> explicitly
> >>>>>>>> > >> > >>>>> > >> > > > defined:
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > <plot_spec>
> >>>>>>>> > >> > >>>>> > >> > > > <connection>
> >>>>>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>>
> >>>>>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >>>>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
> >>>>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
> >>>>>>>> > >> > >>>>> > >> > > > </connection>
> >>>>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
> >>>>>>>> > >> > >>>>> > >> > > > <folders>
> >>>>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >>>>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> >>>>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
> >>>>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
> >>>>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> >>>>>>>> > >> > >>>>> > >> > > > </folders>
> >>>>>>>> > >> > >>>>> > >> > > > <plot>
> >>>>>>>> > >> > >>>>> > >> > > >
<template>series_plot.R_tmpl</template>
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
> >>>>>>>> > >> > >>>>> > >> > > > grepping
> >>>>>>>> all
> >>>>>>>> > >> > >>>>> > >> > > > available
> >>>>>>>> > >> > >>>>> files
> >>>>>>>> > >> > >>>>> > >> > > > does
> >>>>>>>> > >> > >>>>> > >> > > > not
> >>>>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being
> >>>>>>>> > >> > >>>>> > >> > > > defined, as
> >>>>>>>> well
> >>>>>>>> > >> > >>>>> > >> > > > as
> >>>>>>>> > >> > >>>>> > >> > > > other
> >>>>>>>> > >> > >>>>> > >> > > > similarly
> >>>>>>>> > >> > >>>>> > >> > > > defined
> >>>>>>>> > >> > >>>>> > >> > > > variables. I think if I know how
> >>>>>>>> > >> > >>>>> > >> > > > these are
> >>>>>>>> being
> >>>>>>>> > >> > >>>>> > >> > > > defined,
> >>>>>>>> > >> > >>>>> > >> > > > then
> >>>>>>>> > >> > >>>>> > >> > > > I'll
> >>>>>>>> > >> > >>>>> > >> > > > be
> >>>>>>>> > >> > >>>>> > >> > > > able to pipe in my information and
> >>>>>>>> > >> > >>>>> > >> > > > process
> >>>>>>>> the
> >>>>>>>> > >> > >>>>> > >> > > > data
> >>>>>>>> > >> > >>>>> > >> > > > using
> >>>>>>>> > >> > >>>>> met
> >>>>>>>> > >> > >>>>> > >> > > > and
> >>>>>>>> > >> > >>>>> > >> > > > produce
> >>>>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right
on
> >>>>>>>> > >> > >>>>> > >> > > > that?
> >>>>>>>> > >> > >>>>> > >> > > > Also, if
> >>>>>>>> > >> > >>>>> > >> > > > I want
> >>>>>>>> > >> > >>>>> > >> > > > to use
> >>>>>>>> > >> > >>>>> > >> > > > obs for
> >>>>>>>> > >> > >>>>> > >> > > > verification, do I add another line
in
> >>>>>>>> > >> > >>>>> > >> > > > the
> >>>>>>>> XML
> >>>>>>>> > >> > >>>>> > >> > > > file
> >>>>>>>> > >> > >>>>> > >> > > > that
> >>>>>>>> > >> > >>>>> > >> > > > designates
> >>>>>>>> > >> > >>>>> > >> > > > file
> >>>>>>>> > >> > >>>>> > >> > > > location of obs? I assume that
other
> >>>>>>>> > >> > >>>>> > >> > > > steps
> >>>>>>>> have
> >>>>>>>> > >> > >>>>> > >> > > > to be
> >>>>>>>> > >> > >>>>> > >> > > > run
> >>>>>>>> > >> > >>>>> > >> > > > before
> >>>>>>>> > >> > >>>>> > >> > > > all
> >>>>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form,
> >>>>>>>> > >> > >>>>> > >> > > > for
> >>>>>>>> example,
> >>>>>>>> > >> > >>>>> > >> > > > I
> >>>>>>>> > >> > >>>>> > >> > > > would
> >>>>>>>> > >> > >>>>> have
> >>>>>>>> > >> > >>>>> > >> > > > to
> >>>>>>>> > >> > >>>>> > >> > > > separately
> >>>>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs,
then I
> >>>>>>>> would want
> >>>>>>>> > >> > >>>>> > >> > > > to
> >>>>>>>> > >> > >>>>> > >> > > > do that
> >>>>>>>> > >> > >>>>> > >> > > > as
> >>>>>>>> > >> > >>>>> > >> > > > well.
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > 3. Communicating results to
metviewer
> >>>>>>>> > >> > >>>>> > >> > > > online
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of
people/projects
> >>>>>>>> listed
> >>>>>>>> > >> > >>>>> > >> > > > here -
> >>>>>>>> > >> > >>>>> > >> > > > see
> >>>>>>>> > >> > >>>>> > >> > > > attachment
> >>>>>>>> > >> > >>>>> > >> > > >
> >>>>>>>> > >> > >>>>> > >> > > > How do I get the results I generate
to
> >>>>>>>> > >> > >>>>> > >> > > > be
> >>>>>>>> > >> > >>>>> > >> > > > uploaded to
> >>>>>>>> > >> > >>>>> > >> > > > this
> >>>>>>>> > >> > >>>>> > >> > > > location?
> >>>>>>>> > >> > >>>>> > >> > > > It's
> >>>>>>>> > >> > >>>>> > >> > > > not clear how others are setting
this
> >>>>>>>> > >> > >>>>> > >> > > > up.
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >> > >
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>> > >>
> >>>>>>>> > >> > >>>>> > >> rds_host:3306
> >>>>>>>> > >> > >>>>> > >> mv_cmaqv5para8_aod_grid2grid_metplus
> >>>>>>>> > >> > >>>>> > >> rds_user
> >>>>>>>> > >> > >>>>> rds_pwd
> >>>>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
> >>>>>>>> > >> > >>>>> > >> rds_plots
> >>>>>>>> rds_data
> >>>>>>>> > >> > >>>>> rds_scripts
> >>>>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
> >>>>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
> >>>>>>>> > >> > >>>>> 2020-05-12
> >>>>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
> >>>>>>>> > >> > >>>>> > >> 19:00:00
> >>>>>>>> 2020-
> >>>>>>>> > >> > >>>>> > >> 05-12
> >>>>>>>> > >> > >>>>> 20:00:00
> >>>>>>>> > >> > >>>>> > >> 2020-05-12
> >>>>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
> >>>>>>>> > >> > >>>>> > >> 23:00:00
> >>>>>>>> 2020-
> >>>>>>>> > >> > >>>>> > >> 05-13
> >>>>>>>> > >> > >>>>> 10:00:00
> >>>>>>>> > >> > >>>>> > >> 2020-05-13
> >>>>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
> >>>>>>>> > >> > >>>>> > >> 13:00:00
> >>>>>>>> 2020-
> >>>>>>>> > >> > >>>>> > >> 05-13
> >>>>>>>> > >> > >>>>> 14:00:00
> >>>>>>>> > >> > >>>>> > >> 2020-05-13
> >>>>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
> >>>>>>>> > >> > >>>>> > >> 17:00:00
> >>>>>>>> 2020-
> >>>>>>>> > >> > >>>>> > >> 05-13
> >>>>>>>> > >> > >>>>> 18:00:00
> >>>>>>>> > >> > >>>>> > >> 2020-05-13
> >>>>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
> >>>>>>>> > >> > >>>>> > >> 21:00:00
> >>>>>>>> 2020-
> >>>>>>>> > >> > >>>>> > >> 05-13
> >>>>>>>> > >> > >>>>> 22:00:00
> >>>>>>>> > >> > >>>>> > >> 2020-05-13
> >>>>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
> >>>>>>>> > >> > >>>>> > >> 11:00:00
> >>>>>>>> 2020-
> >>>>>>>> > >> > >>>>> > >> 05-14
> >>>>>>>> > >> > >>>>> 12:00:00
> >>>>>>>> > >> > >>>>> > >> true
> >>>>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
> >>>>>>>> > >> > >>>>> > >> plot_20200612_173626.png
> >>>>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
> >>>>>>>> > >> > >>>>> > >> y_label
> >>>>>>>> > >> > >>>>> > >> false list() list() false false false
false
> >>>>>>>> > >> > >>>>> > >> false
> >>>>>>>> false
> >>>>>>>> > >> > >>>>> > >> true
> >>>>>>>> > >> > >>>>> > >> false
> >>>>>>>> > >> > >>>>> > >> false
> >>>>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72
in
> >>>>>>>> c(8,4,5,4)
> >>>>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
> >>>>>>>> > >> > >>>>> > >> 1.4 -2
> >>>>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1
1
> >>>>>>>> > >> > >>>>> > >> 1 -2
> >>>>>>>> 0.5 3
> >>>>>>>> > >> > >>>>> > >> #cccccc 1
> >>>>>>>> > >> > >>>>> > >> listX 1
> >>>>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1
0.5 o
> >>>>>>>> > >> > >>>>> > >> c(0,
> >>>>>>>> -.25)
> >>>>>>>> > >> > >>>>> > >> 3
> >>>>>>>> > >> > >>>>> > >> 0.8 1
> >>>>>>>> > >> > >>>>> > >> #333333
> >>>>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none")
c(FALSE,FALSE)
> >>>>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
> >>>>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
> >>>>>>>> > >> > >>>>> > >> c("b","b")
> >>>>>>>> c(1,1)
> >>>>>>>> > >> > >>>>> > >> c(1,1)
> >>>>>>>> > >> > >>>>> c(1,1)
> >>>>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>> > > --
> >>>>>>>> > >> > >>>>> > > Edward Strobach
> >>>>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
> >>>>>>>> > >> > >>>>> > > IMSG Contractor
> >>>>>>>> > >> > >>>>> > > Cubicle#: 2029
> >>>>>>>> > >> > >>>>> > > 301-683-3717
> >>>>>>>> > >> > >>>>> > >
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>>
> >>>>>>>> > >> > >>>>
> >>>>>>>> > >> > >>>> --
> >>>>>>>> > >> > >>>> Edward Strobach
> >>>>>>>> > >> > >>>> EMC/NCEP/NWS/
> >>>>>>>> > >> > >>>> IMSG Contractor
> >>>>>>>> > >> > >>>> Cubicle#: 2029
> >>>>>>>> > >> > >>>> 301-683-3717
> >>>>>>>> > >> > >>>>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>> --
> >>>>>>>> > >> > >>> Edward Strobach
> >>>>>>>> > >> > >>> EMC/NCEP/NWS/
> >>>>>>>> > >> > >>> IMSG Contractor
> >>>>>>>> > >> > >>> Cubicle#: 2029
> >>>>>>>> > >> > >>> 301-683-3717
> >>>>>>>> > >> > >>>
> >>>>>>>> > >> > >>
> >>>>>>>> > >> > >>
> >>>>>>>> > >> > >> --
> >>>>>>>> > >> > >> Edward Strobach
> >>>>>>>> > >> > >> EMC/NCEP/NWS/
> >>>>>>>> > >> > >> IMSG Contractor
> >>>>>>>> > >> > >> Cubicle#: 2029
> >>>>>>>> > >> > >> 301-683-3717
> >>>>>>>> > >> > >>
> >>>>>>>> > >> > >
> >>>>>>>> > >> > >
> >>>>>>>> > >> > > --
> >>>>>>>> > >> > > Edward Strobach
> >>>>>>>> > >> > > EMC/NCEP/NWS/
> >>>>>>>> > >> > > IMSG Contractor
> >>>>>>>> > >> > > Cubicle#: 2029
> >>>>>>>> > >> > > 301-683-3717
> >>>>>>>> > >> > >
> >>>>>>>> > >>
> >>>>>>>> > >>
> >>>>>>>> > >>
> >>>>>>>> > >>
> >>>>>>>> > >
> >>>>>>>> > > --
> >>>>>>>> > > Edward Strobach
> >>>>>>>> > > EMC/NCEP/NWS/
> >>>>>>>> > > IMSG Contractor
> >>>>>>>> > > Cubicle#: 2029
> >>>>>>>> > > 301-683-3717
> >>>>>>>> > >
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Edward Strobach
> >>>>>>> EMC/NCEP/NWS/
> >>>>>>> IMSG Contractor
> >>>>>>> Cubicle#: 2029
> >>>>>>> 301-683-3717
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Edward Strobach
> >>>>>> EMC/NCEP/NWS/
> >>>>>> IMSG Contractor
> >>>>>> Cubicle#: 2029
> >>>>>> 301-683-3717
> >>>>>>
> >>>>> --
> >>>>> Jeffery T. McQueen
> >>>>> Meteorologist
> >>>>> National Center for Weather and Climate Prediction
> >>>>> Room 2095
> >>>>> 5830 University Research Court
> >>>>> College Park, MD 20741
> >>>>> 301-683-3736
> >>>>> Fax: 301-683-3703
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Edward Strobach
> >>>> EMC/NCEP/NWS/
> >>>> IMSG Contractor
> >>>> Cubicle#: 2029
> >>>> 301-683-3717
> >>>>
> >>>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 13:57:47 2020
Great, thanks Ho-Chun. Sorry if I originally misunderstood.
On Wed, Jun 17, 2020 at 3:55 PM Ho-Chun Huang - NOAA Affiliate <
ho-chun.huang at noaa.gov> wrote:
> Ed:
>
> That is correct.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Wed, Jun 17, 2020 at 3:47 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> I'm sorry, I guess I may have been a bit confused by what you were
>> initially telling me. Are you saying replace what I have with
this?
>>
>> <connection>
>> *<host>rds_host:3306</host>*
>> * <database>MY_DATABASEdatabase>*
>> <user>rds_user</user>
>> <password>rds_pwd</password>
>> <management_system>aurora</management_system>
>> </connection>
>> * <rscript>Rscript</rscript>*
>> <folders>
>>
>>
>>
>>
>> * <r_tmpl>rds_R_tmpl</r_tmpl>
<r_work>rds_R_work</r_work>
>> <plots>rds_plots</plots> <data>rds_data</data>
>> <scripts>rds_scripts</scripts>*
>>
>> On Wed, Jun 17, 2020 at 3:27 PM Ho-Chun Huang - NOAA Affiliate <
>> ho-chun.huang at noaa.gov> wrote:
>>
>>> Ed:
>>>
>>> You can try or you can keep waiting, it is your choice.
>>>
>>> Ho-Chun Huang
>>>
>>> IMSG at NOAA/NWS/NCEP/EMC
>>>
>>> 5830 University Research Ct., Rm. 2792
>>>
>>> College Park, MD 20740
>>>
>>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>>
>>> 301-683-3958
>>>
>>>
>>> On Wed, Jun 17, 2020 at 3:21 PM Edward Strobach - NOAA Affiliate <
>>> edward.strobach at noaa.gov> wrote:
>>>
>>>> Hi Ho-Chun,
>>>>
>>>> I've made all the changes that I would need to inside my xml
file. I
>>>> was told not to change the folders info. I instead changed my
database
>>>> name, fcst_var, stat options, series1 input, the fix fields, the
indep
>>>> field, and x_lim.
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Jun 17, 2020 at 3:16 PM Ho-Chun Huang - NOAA Affiliate <
>>>> ho-chun.huang at noaa.gov> wrote:
>>>>
>>>>> Hi, Ed:
>>>>>
>>>>> You should go back to read 1README again from the directory I
sent
>>>>> about replacing the top portion of your metviewer xml to produce
a batch
>>>>> xml.
>>>>>
>>>>> I check your
>>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
>>>>> Please update the section from <connection> to </folder> as
those in
>>>>> hochun_xml.template. I also do not know whether AWS can handle
comments
>>>>> <!-- to !--> in batch xml , better make your batch xml clean.
>>>>>
>>>>> Ho-Chun Huang
>>>>>
>>>>> IMSG at NOAA/NWS/NCEP/EMC
>>>>>
>>>>> 5830 University Research Ct., Rm. 2792
>>>>>
>>>>> College Park, MD 20740
>>>>>
>>>>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>>>>
>>>>> 301-683-3958
>>>>>
>>>>>
>>>>> On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA Affiliate
<
>>>>> edward.strobach at noaa.gov> wrote:
>>>>>
>>>>>> Just adding more detail to elaborate my concerns...
>>>>>>
>>>>>> Not too long ago I was given a test set-up by Ho-Chun to
generate a
>>>>>> plot for aerosol optical depth. The result was successful and
produced a
>>>>>> CSI plot for AOD, comparing the production run and experimental
run.
>>>>>> Although I created a database, I do not seem to have a group
where I can
>>>>>> access specific databases. I recently created a load_xml file,
which
>>>>>> should have loaded the data based on what I specified into the
group. It
>>>>>> seemed successful based on the log output that was generated.
>>>>>> Additionally, all information that I specified in this step was
create in
>>>>>> the load_*xml.
>>>>>>
>>>>>> The next step is to generate plots since the data should be
loaded.
>>>>>> Here's what happens when I run Ho-Chun's - the successful case
>>>>>>
>>>>>> HO-CHUN:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
>>>>>> hochun_xml.template+ bash
>>>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>>>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>>>>> plot.xmlCALLING: scp plot.xml
edward.strobach at 205.156.8.85:~---- MVBatch
>>>>>> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # #
# # # # # #
>>>>>> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT
model,
>>>>>> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
>>>>>>
>>>>>> This is early on in the script.
>>>>>>
>>>>>> ED_STROBACH:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+
sed -e
>>>>>> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
>>>>>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
>>>>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
>>>>>> plot.xmlCALLING: scp plot.xml
edward.strobach at 205.156.8.85:~---- MVBatch
>>>>>> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20
PM
>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable
to create
>>>>>> initial connections of pool.java.sql.SQLException:
Communications link
>>>>>> failure with primary. No active connection found for master.
at
>>>>>>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
>>>>>> at
>>>>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
>>>>>> at
>>>>>>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
>>>>>> at
>>>>>>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
>>>>>> at
>>>>>>
org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
>>>>>> at
>>>>>>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
>>>>>> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
>>>>>>
>>>>>> My interpretation of this is that it fails to bridge a
connection to
>>>>>> the group or database; therefore, it's unable to proceed with
running the
>>>>>> XML file to generate the plots based on the specified content.
If I move
>>>>>> past this step, then I'm golden.
>>>>>>
>>>>>> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
>>>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>>>
>>>>>>> Would it be ok to keep perry in the loop
>>>>>>>
>>>>>>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA
Affiliate <
>>>>>>> edward.strobach at noaa.gov> wrote:
>>>>>>>
>>>>>>>> you're right. I didn't have the storage problem this time.
I'm
>>>>>>>> not sure what changed. All I know is that I have now created
my load xml
>>>>>>>> which now should allow me to move on to the next step of
plotting
>>>>>>>>
>>>>>>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA
Affiliate <
>>>>>>>> edward.strobach at noaa.gov> wrote:
>>>>>>>>
>>>>>>>>> I'm not following you. How do I now have enough space?
What
>>>>>>>>> changed?
>>>>>>>>>
>>>>>>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
>>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>>
>>>>>>>>>> Ed, now you should have enough space for the data loading.
>>>>>>>>>>
>>>>>>>>>> Tatiana
>>>>>>>>>>
>>>>>>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov
wrote:
>>>>>>>>>> > I was basically able to build my xml file but it says I
have no
>>>>>>>>>> > available
>>>>>>>>>> > space. I'm not sure how that can be since I just started
>>>>>>>>>> adding data
>>>>>>>>>> > to my
>>>>>>>>>> > account
>>>>>>>>>> >
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat:
No
>>>>>>>>>> space left
>>>>>>>>>> > on
>>>>>>>>>> > device
>>>>>>>>>> > ERROR: Command returned with non-zero status (1): scp -r
./*
>>>>>>>>>> >
edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>>>>>>>>> > 4) Check METviewer AWS database list using
mv_db_size_on_aws.sh
>>>>>>>>>> > TABLE_SCHEMA USER
>>>>>>>>>> SIZE_MB
>>>>>>>>>> > mv_met_g2o_gefs binbin.zhou
>>>>>>>>>> 140027.31
>>>>>>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
>>>>>>>>>> 71986.11
>>>>>>>>>> >
>>>>>>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
>>>>>>>>>> Affiliate <
>>>>>>>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>>>>> >
>>>>>>>>>> > > Thanks Tatiana. Yes, I've created a database which
seems
>>>>>>>>>> different
>>>>>>>>>> > > from
>>>>>>>>>> > > mv_group. How can one create their own group so that I
can
>>>>>>>>>> store and
>>>>>>>>>> > > access
>>>>>>>>>> > > the database. The database exists but cannot be viewed
on the
>>>>>>>>>> > > metviewer
>>>>>>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
>>>>>>>>>> vhagerty. I
>>>>>>>>>> > > named my
>>>>>>>>>> > > database Ed_Strobach, but that was really intended for
the
>>>>>>>>>> group
>>>>>>>>>> > > name.
>>>>>>>>>> > >
>>>>>>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>>>>>>>>> > > <met_help at ucar.edu>
>>>>>>>>>> > > wrote:
>>>>>>>>>> > >
>>>>>>>>>> > >> We don't have database and METviewer running on wcoss.
>>>>>>>>>> > >> Instead, users use METviewer-wcoss scripts that help
to
>>>>>>>>>> connect to
>>>>>>>>>> > >> and
>>>>>>>>>> > >> use METviewer that is running on AWS. That is why XML
>>>>>>>>>> connection and
>>>>>>>>>> > >> folder
>>>>>>>>>> > >> sections look somewhat different than once from the
Docs
>>>>>>>>>> page.
>>>>>>>>>> > >> To create a database on AWS you need to run
>>>>>>>>>> mv_create_db_on_aws.sh
>>>>>>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>>>>>>>>> > >> For data loading :
>>>>>>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>>>>>>>>> > >> And for batch:
>>>>>>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>>>>>>>>> > >>
>>>>>>>>>> > >> These scripts copy XML and data files if needed to
AWS,
>>>>>>>>>> adjust data
>>>>>>>>>> > >> path
>>>>>>>>>> > >> and users credentials, run METviewer and copy results
back
>>>>>>>>>> to wcoss.
>>>>>>>>>> > >>
>>>>>>>>>> > >> Tatiana
>>>>>>>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov
wrote:
>>>>>>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
>>>>>>>>>> -p[db_password]
>>>>>>>>>> > >> > -e'create
>>>>>>>>>> > >> > database [db_name]
>>>>>>>>>> > >> >
>>>>>>>>>> > >> > would need to be used to create a database. I
assume that
>>>>>>>>>> an sql
>>>>>>>>>> > >> > module is
>>>>>>>>>> > >> > available on wcoss; I know you don't have access to
those
>>>>>>>>>> machines
>>>>>>>>>> > >> > Tatiana. If I create my own database with my own
name,
>>>>>>>>>> then I
>>>>>>>>>> > >> > should
>>>>>>>>>> > >> > be
>>>>>>>>>> > >> > able to proceed with project specific tasks under
that tab.
>>>>>>>>>> > >> > Something
>>>>>>>>>> > >> > like
>>>>>>>>>> > >> > mv_EdwardStrobach.
>>>>>>>>>> > >> >
>>>>>>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add
data to
>>>>>>>>>> using
>>>>>>>>>> > >> > load_spec.
>>>>>>>>>> > >> > Once that step is done, then I can run plot_spec.
That's
>>>>>>>>>> my
>>>>>>>>>> > >> > takeaway
>>>>>>>>>> > >> > currently
>>>>>>>>>> > >> >
>>>>>>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach -
NOAA
>>>>>>>>>> Affiliate <
>>>>>>>>>> > >> > edward.strobach at noaa.gov> wrote:
>>>>>>>>>> > >> >
>>>>>>>>>> > >> > > Actually, it appears that I need to run the
database
>>>>>>>>>> loading
>>>>>>>>>> > >> > > module
>>>>>>>>>> > >> > > first
>>>>>>>>>> > >> > > if I want to change/add a data source. From there
I can
>>>>>>>>>> run the
>>>>>>>>>> > >> > > plotting
>>>>>>>>>> > >> > > batch module. Is that right? So if I run the
database
>>>>>>>>>> loading
>>>>>>>>>> > >> > > module,
>>>>>>>>>> > >> > > then I can specify my dates and other related info
>>>>>>>>>> specific to
>>>>>>>>>> > >> > > the
>>>>>>>>>> > >> > > dataset. If I create my database with a unique
name,
>>>>>>>>>> then it
>>>>>>>>>> > >> > > should
>>>>>>>>>> > >> > > be
>>>>>>>>>> > >> > > recognized on the metviewer gui, presumably.
>>>>>>>>>> > >> > >
>>>>>>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach -
NOAA
>>>>>>>>>> Affiliate
>>>>>>>>>> > >> > > <
>>>>>>>>>> > >> > > edward.strobach at noaa.gov> wrote:
>>>>>>>>>> > >> > >
>>>>>>>>>> > >> > >> It seems that those options must be treated
separately
>>>>>>>>>> outside
>>>>>>>>>> > >> > >> of
>>>>>>>>>> > >> > >> plot_spec. Is it acceptable to create both a
load_spec
>>>>>>>>>> and
>>>>>>>>>> > >> > >> plot_spec in
>>>>>>>>>> > >> > >> the same xml file? Would the definitions in
load_spec
>>>>>>>>>> be able
>>>>>>>>>> > >> > >> to
>>>>>>>>>> > >> > >> carry
>>>>>>>>>> > >> > >> over into plot_spec? It appears that data
loading and
>>>>>>>>>> plotting
>>>>>>>>>> > >> > >> are
>>>>>>>>>> > >> > >> treated
>>>>>>>>>> > >> > >> separately.
>>>>>>>>>> > >> > >>
>>>>>>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach -
NOAA
>>>>>>>>>> > >> > >> Affiliate <
>>>>>>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>>>>>>>>> > >> > >>
>>>>>>>>>> > >> > >>> I do plan to use date_range; however, in an
example
>>>>>>>>>> provided
>>>>>>>>>> > >> > >>> by the
>>>>>>>>>> > >> > >>> link
>>>>>>>>>> > >> > >>> I see this:
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> <load_spec>
>>>>>>>>>> > >> > >>> <connection>
>>>>>>>>>> > >> > >>> <host>kemosabe:3306</host>
>>>>>>>>>> > >> > >>> <database>metvdb_hwt</database>
>>>>>>>>>> > >> > >>> <user>pgoldenb</user>
>>>>>>>>>> > >> > >>> <password>pgoldenb</password>
>>>>>>>>>> > >> > >>> </connection>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>>>>> > >> > >>> <start>2010051914V</start>
>>>>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
>>>>>>>>>> > >> > >>> </hour></date_offset></end>
>>>>>>>>>> > >> > >>> <inc>3600</inc>
>>>>>>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
>>>>>>>>>> > >> > >>> </date_list>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> <met_version>V3.0</met_version>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> <verbose>false</verbose>
>>>>>>>>>> > >> > >>> <insert_size>1</insert_size>
>>>>>>>>>> > >> > >>>
<mode_header_db_check>true</mode_header_db_check>
>>>>>>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>>>>>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>>>>>>>>> > >> > >>> <group>Group name</group>
>>>>>>>>>> > >> > >>> <load_stat>true</load_stat>
>>>>>>>>>> > >> > >>> <load_mode>true</load_mode>
>>>>>>>>>> > >> > >>> <load_mtd>true</load_mtd>
>>>>>>>>>> > >> > >>> <load_mpr>false</load_mpr>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>>
>>>>>>>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>>>>>>>> > >> > >>> <load_val>
>>>>>>>>>> > >> > >>> <field name="model">
>>>>>>>>>> > >> > >>> <val>arw</val>
>>>>>>>>>> > >> > >>> <val>nmm</val>
>>>>>>>>>> > >> > >>> </field>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>>>>> > >> > >>> </field>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> <field name="vx_mask">
>>>>>>>>>> > >> > >>> <val>FULL</val>
>>>>>>>>>> > >> > >>> <val>SWC</val>
>>>>>>>>>> > >> > >>> </field>
>>>>>>>>>> > >> > >>> </load_val>
>>>>>>>>>> > >> > >>> </load_spec>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> Are similar settings allowed with <plot_spec>?
I see
>>>>>>>>>> > >> > >>> <connection>,
>>>>>>>>>> > >> > >>> which seems carried over. I'm looking to
specify my
>>>>>>>>>> > >> > >>> folder_tmpl in
>>>>>>>>>> > >> > >>> <plot_spec>, which according to Tara, should be
>>>>>>>>>> possible.
>>>>>>>>>> > >> > >>> Since my
>>>>>>>>>> > >> > >>> directory structure looks like this:
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >>
>>>>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> I would do something like this I would think:
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> .......
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> <date_list name="folder_dates">
>>>>>>>>>> > >> > >>> <start>20200612/start>
>>>>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset></end>
>>>>>>>>>> > >> > >>> <inc>86400</inc>
>>>>>>>>>> > >> > >>> <format>yyyyMMdd</format>
>>>>>>>>>> > >> > >>> </date_list>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> <folder_tmpl>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >>
>>>>>>>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>>>>>>>>> > >> > >>> </folder_tmpl>
>>>>>>>>>> > >> > >>> <load_val>
>>>>>>>>>> > >> > >>> <field name="model">
>>>>>>>>>> > >> > >>> <val>prod</val>
>>>>>>>>>> > >> > >>> </field>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> <field name="valid_time">
>>>>>>>>>> > >> > >>> <date_list name="folder_dates"/>
>>>>>>>>>> > >> > >>> </field>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> ......
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach
- NOAA
>>>>>>>>>> > >> > >>> Affiliate <
>>>>>>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>>> Thank you. This is a very helpful example
>>>>>>>>>> > >> > >>>>
>>>>>>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek
via RT
>>>>>>>>>> > >> > >>>> <met_help at ucar.edu>
>>>>>>>>>> > >> > >>>> wrote:
>>>>>>>>>> > >> > >>>>
>>>>>>>>>> > >> > >>>>> Hi,
>>>>>>>>>> > >> > >>>>> Here is some examples how to use <date_range>
and
>>>>>>>>>> > >> > >>>>> <date_range_list> in
>>>>>>>>>> > >> > >>>>> batch XML.
>>>>>>>>>> > >> > >>>>> First, you need to defined your range. You can
have
>>>>>>>>>> more
>>>>>>>>>> > >> > >>>>> then one
>>>>>>>>>> > >> > >>>>> <date_range>:
>>>>>>>>>> > >> > >>>>> <date_range name="may">
>>>>>>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
>>>>>>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
>>>>>>>>>> > >> > >>>>> </date_range>
>>>>>>>>>> > >> > >>>>> <date_range name="march">
>>>>>>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
>>>>>>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
>>>>>>>>>> > >> > >>>>> </date_range>
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>> and then use one or all date_ranges:
>>>>>>>>>> > >> > >>>>> <plot_fix>
>>>>>>>>>> > >> > >>>>> <field name="fcst_init_beg">
>>>>>>>>>> > >> > >>>>> <date_range name="march"/>
>>>>>>>>>> > >> > >>>>> <date_range name="may"/>
>>>>>>>>>> > >> > >>>>> </field>...
>>>>>>>>>> > >> > >>>>> This configuration would include all dates for
March
>>>>>>>>>> and May
>>>>>>>>>> > >> > >>>>> of
>>>>>>>>>> > >> > >>>>> 2019.
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>> If you need more flexible configuration you
can use
>>>>>>>>>> > >> > >>>>> date_range_list.
>>>>>>>>>> > >> > >>>>> For example, you want to select first 6 hours
every
>>>>>>>>>> day
>>>>>>>>>> > >> > >>>>> beetween
>>>>>>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00.
Your
>>>>>>>>>> XML would
>>>>>>>>>> > >> > >>>>> look
>>>>>>>>>> > >> > >>>>> like this:
>>>>>>>>>> > >> > >>>>> <date_range_list name = 'all'>
>>>>>>>>>> > >> > >>>>> <range_start>2020-01-01
00:00:00</range_start>
>>>>>>>>>> > >> > >>>>> <range_end>2020-01-05
00:00:00</range_end>
>>>>>>>>>> > >> > >>>>> <range_length>360</range_length>
>>>>>>>>>> > >> > >>>>> <inc>1440</inc>
>>>>>>>>>> > >> > >>>>> </date_range_list>
>>>>>>>>>> > >> > >>>>> where the 1st and 2nd values are the start and
end,
>>>>>>>>>> > >> > >>>>> range_length - the length of a range in
minutes(6
>>>>>>>>>> houres)
>>>>>>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
houres)
>>>>>>>>>> > >> > >>>>> This example would create following ranges:
>>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
>>>>>>>>>> 06:00:00'
>>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
>>>>>>>>>> 06:00:00'
>>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
>>>>>>>>>> 06:00:00'
>>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
>>>>>>>>>> 06:00:00'
>>>>>>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
>>>>>>>>>> 06:00:00'
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>> Also you can add more parameters to <plot_fix>
to
>>>>>>>>>> filter
>>>>>>>>>> > >> > >>>>> your
>>>>>>>>>> > >> > >>>>> data.
>>>>>>>>>> > >> > >>>>> For example, if you want to select only dates
with
>>>>>>>>>> 00:00 you
>>>>>>>>>> > >> > >>>>> can
>>>>>>>>>> > >> > >>>>> specify
>>>>>>>>>> > >> > >>>>> <field name="valid_hour">
>>>>>>>>>> > >> > >>>>> <val>00</val>
>>>>>>>>>> > >> > >>>>> </field>
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>> Tatiana
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
>>>>>>>>>> edward.strobach at noaa.gov wrote:
>>>>>>>>>> > >> > >>>>> > and if there isn't a better way, I wonder
the
>>>>>>>>>> whereabouts
>>>>>>>>>> > >> > >>>>> > of a
>>>>>>>>>> > >> > >>>>> > separate
>>>>>>>>>> > >> > >>>>> > script that pipes dates (and other relevant
>>>>>>>>>> information)
>>>>>>>>>> > >> > >>>>> > into
>>>>>>>>>> > >> > >>>>> > the XML
>>>>>>>>>> > >> > >>>>> > file
>>>>>>>>>> > >> > >>>>> > used to generate the results.
>>>>>>>>>> > >> > >>>>> >
>>>>>>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward
Strobach -
>>>>>>>>>> NOAA
>>>>>>>>>> > >> > >>>>> > Affiliate <
>>>>>>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>>>>>>>>> > >> > >>>>> >
>>>>>>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are
helpful.
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>> > > One last question. Simply typing lines
like
>>>>>>>>>> this in can
>>>>>>>>>> > >> > >>>>> > > take
>>>>>>>>>> > >> > >>>>> > > forever:
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>> > > <plot_fix>
>>>>>>>>>> > >> > >>>>> > > <field equalize="false"
name="fcst_init_beg">
>>>>>>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
>>>>>>>>>> > >> > >>>>> > > <val>2019-05-01 12:00:00</val>
>>>>>>>>>> > >> > >>>>> > > <val>2019-05-02 06:00:00</val>
>>>>>>>>>> > >> > >>>>> > > <val>2019-05-02 12:00:00</val>
>>>>>>>>>> > >> > >>>>> > > <val>2019-05-03 06:00:00</val>
>>>>>>>>>> > >> > >>>>> > > <val>2019-05-03 12:00:00</val>
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>> > > Is there a better way? I see date_range,
>>>>>>>>>> > >> > >>>>> > > date_range_list,
>>>>>>>>>> > >> > >>>>> > > or
>>>>>>>>>> > >> > >>>>> > > date_list.
>>>>>>>>>> > >> > >>>>> > > Can this be used instead of typing
individual
>>>>>>>>>> values?
>>>>>>>>>> > >> > >>>>> > > This
>>>>>>>>>> > >> > >>>>> > > could
>>>>>>>>>> > >> > >>>>> > > prove
>>>>>>>>>> > >> > >>>>> > > pretty daunting if one would have to type
in 700
>>>>>>>>>> entries
>>>>>>>>>> > >> > >>>>> > > each
>>>>>>>>>> > >> > >>>>> > > time.
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana
Burek
>>>>>>>>>> via RT
>>>>>>>>>> > >> > >>>>> > > <met_help at ucar.edu>
>>>>>>>>>> > >> > >>>>> > > wrote:
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>> > >> I found a few problems in your XML.
>>>>>>>>>> > >> > >>>>> > >> The database you are using doesn't have
data
>>>>>>>>>> with
>>>>>>>>>> > >> > >>>>> > >> vx_mask
>>>>>>>>>> > >> > >>>>> > >> 'abcREGabc'.
>>>>>>>>>> > >> > >>>>> > >> That is why METviewer returns an empty
dataset
>>>>>>>>>> and
>>>>>>>>>> > >> > >>>>> > >> can't
>>>>>>>>>> > >> > >>>>> > >> create a
>>>>>>>>>> > >> > >>>>> > >> plot.
>>>>>>>>>> > >> > >>>>> > >> The other problem is how you define
fcst_lead.
>>>>>>>>>> This
>>>>>>>>>> > >> > >>>>> > >> database
>>>>>>>>>> > >> > >>>>> > >> has
>>>>>>>>>> > >> > >>>>> > >> values
>>>>>>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000.
Since
>>>>>>>>>> you
>>>>>>>>>> > >> > >>>>> > >> include
>>>>>>>>>> > >> > >>>>> > >> all
>>>>>>>>>> > >> > >>>>> > >> fcst_leads
>>>>>>>>>> > >> > >>>>> > >> this variable can be omitted.
>>>>>>>>>> > >> > >>>>> > >> To minimize problems with variables
values, I
>>>>>>>>>> would
>>>>>>>>>> > >> > >>>>> > >> suggest
>>>>>>>>>> > >> > >>>>> > >> to
>>>>>>>>>> > >> > >>>>> > >> create XML
>>>>>>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
>>>>>>>>>> standards.
>>>>>>>>>> > >> > >>>>> > >> I created a plot using you configurations
with
>>>>>>>>>> GUI
>>>>>>>>>> > >> > >>>>> > >> (without
>>>>>>>>>> > >> > >>>>> vx_mack
>>>>>>>>>> > >> > >>>>> > >> and
>>>>>>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626.
You can
>>>>>>>>>> load it
>>>>>>>>>> > >> > >>>>> > >> back
>>>>>>>>>> > >> > >>>>> > >> to
>>>>>>>>>> > >> > >>>>> the
>>>>>>>>>> > >> > >>>>> > >> page by
>>>>>>>>>> > >> > >>>>> > >> using the "History" tab (left bar on
GUI). Find
>>>>>>>>>> the
>>>>>>>>>> > >> > >>>>> > >> plot,
>>>>>>>>>> > >> > >>>>> > >> open XML
>>>>>>>>>> > >> > >>>>> > >> and
>>>>>>>>>> > >> > >>>>> > >> click on "Load to page". After that you
can
>>>>>>>>>> recreate a
>>>>>>>>>> > >> > >>>>> > >> plot
>>>>>>>>>> > >> > >>>>> > >> and
>>>>>>>>>> > >> > >>>>> edit
>>>>>>>>>> > >> > >>>>> > >> the
>>>>>>>>>> > >> > >>>>> > >> configurations. You can also download a
new XML
>>>>>>>>>> and
>>>>>>>>>> > >> > >>>>> > >> edit it
>>>>>>>>>> > >> > >>>>> > >> for
>>>>>>>>>> > >> > >>>>> > >> batch.
>>>>>>>>>> > >> > >>>>> > >> I attached the example of load XML to
this
>>>>>>>>>> message.
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>> > >> Description of rds parameters:
>>>>>>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
>>>>>>>>>> METviewer R
>>>>>>>>>> > >> > >>>>> > >> scripts
>>>>>>>>>> > >> > >>>>> > >> rds_plots - temporary directory where the
plots
>>>>>>>>>> will be
>>>>>>>>>> > >> > >>>>> > >> generated
>>>>>>>>>> > >> > >>>>> > >> rds_data - temporary directory where the
data
>>>>>>>>>> retrieved
>>>>>>>>>> > >> > >>>>> > >> from
>>>>>>>>>> > >> > >>>>> > >> the
>>>>>>>>>> > >> > >>>>> > >> database
>>>>>>>>>> > >> > >>>>> > >> and other intermediate files will be
stored
>>>>>>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
>>>>>>>>>> scripts for
>>>>>>>>>> > >> > >>>>> > >> this
>>>>>>>>>> > >> > >>>>> > >> run
>>>>>>>>>> > >> > >>>>> will
>>>>>>>>>> > >> > >>>>> > >> be
>>>>>>>>>> > >> > >>>>> > >> stored
>>>>>>>>>> > >> > >>>>> > >> After the execution is done all the files
from
>>>>>>>>>> > >> > >>>>> > >> temporary
>>>>>>>>>> > >> > >>>>> directories
>>>>>>>>>> > >> > >>>>> > >> would be removed.
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>> > >> We don't have the offitial user guide for
>>>>>>>>>> METviewer.
>>>>>>>>>> > >> > >>>>> > >> There
>>>>>>>>>> > >> > >>>>> > >> are
>>>>>>>>>> > >> > >>>>> some
>>>>>>>>>> > >> > >>>>> > >> instructions on how to crate XML config
files:
>>>>>>>>>> > >> > >>>>> > >>
https://metviewer.nws.noaa.gov/doc/index.html
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>> > >> Tatiana
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
>>>>>>>>>> edward.strobach at noaa.gov
>>>>>>>>>> > >> > >>>>> > >> wrote:
>>>>>>>>>> > >> > >>>>> > >> > Hi,
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am
having
>>>>>>>>>> hard time
>>>>>>>>>> > >> > >>>>> > >> > following
>>>>>>>>>> > >> > >>>>> how
>>>>>>>>>> > >> > >>>>> > >> > to
>>>>>>>>>> > >> > >>>>> > >> > go
>>>>>>>>>> > >> > >>>>> > >> > about the other steps since I have zero
>>>>>>>>>> experience on
>>>>>>>>>> > >> > >>>>> > >> > how
>>>>>>>>>> > >> > >>>>> > >> > to do
>>>>>>>>>> > >> > >>>>> > >> > these
>>>>>>>>>> > >> > >>>>> > >> > things.
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
>>>>>>>>>> should NOT be
>>>>>>>>>> > >> > >>>>> > >> > changed
>>>>>>>>>> > >> > >>>>> and
>>>>>>>>>> > >> > >>>>> > >> > will
>>>>>>>>>> > >> > >>>>> > >> > be
>>>>>>>>>> > >> > >>>>> > >> > replaced by other scripts during the
batch
>>>>>>>>>> execution.
>>>>>>>>>> > >> > >>>>> > >> > The
>>>>>>>>>> > >> > >>>>> > >> > data
>>>>>>>>>> > >> > >>>>> for
>>>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>>>> > >> > >>>>> > >> > batch mode is coming from the database
and
>>>>>>>>>> not from
>>>>>>>>>> > >> > >>>>> > >> > files.
>>>>>>>>>> > >> > >>>>> > >> > The
>>>>>>>>>> > >> > >>>>> > >> > name of
>>>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>>>> > >> > >>>>> > >> > database is in <database> tag.
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> > I understand that nothing should be
changed
>>>>>>>>>> in the
>>>>>>>>>> > >> > >>>>> > >> > folders
>>>>>>>>>> > >> > >>>>> > >> > section,
>>>>>>>>>> > >> > >>>>> > >> > but
>>>>>>>>>> > >> > >>>>> > >> > there is no indication how and where
these
>>>>>>>>>> variables
>>>>>>>>>> > >> > >>>>> > >> > are
>>>>>>>>>> > >> > >>>>> > >> > being
>>>>>>>>>> > >> > >>>>> > >> > defined.
>>>>>>>>>> > >> > >>>>> > >> > Grepping these names of the files
available
>>>>>>>>>> does not
>>>>>>>>>> > >> > >>>>> > >> > reveal
>>>>>>>>>> > >> > >>>>> formal
>>>>>>>>>> > >> > >>>>> > >> > definitions of these variables. If I
>>>>>>>>>> understood
>>>>>>>>>> > >> > >>>>> > >> > exactly
>>>>>>>>>> > >> > >>>>> > >> > how
>>>>>>>>>> > >> > >>>>> these
>>>>>>>>>> > >> > >>>>> > >> > were
>>>>>>>>>> > >> > >>>>> > >> > being defined, then I would be able
connect
>>>>>>>>>> this
>>>>>>>>>> > >> > >>>>> > >> > better.
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET
output
>>>>>>>>>> files. They
>>>>>>>>>> > >> > >>>>> > >> > use
>>>>>>>>>> > >> > >>>>> METviewer
>>>>>>>>>> > >> > >>>>> > >> > load
>>>>>>>>>> > >> > >>>>> > >> > mode to load these files to the
specific
>>>>>>>>>> database. In
>>>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>>>> > >> > >>>>> > >> > config
>>>>>>>>>> > >> > >>>>> > >> > file
>>>>>>>>>> > >> > >>>>> > >> > for
>>>>>>>>>> > >> > >>>>> > >> > this process they also can specify the
>>>>>>>>>> database
>>>>>>>>>> > >> > >>>>> > >> > group. For
>>>>>>>>>> > >> > >>>>> example
>>>>>>>>>> > >> > >>>>> > >> > database
>>>>>>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>>>>>>>>> > >> > >>>>> > >> > After the data has been loaded to the
>>>>>>>>>> database it is
>>>>>>>>>> > >> > >>>>> > >> > visible and
>>>>>>>>>> > >> > >>>>> > >> > available
>>>>>>>>>> > >> > >>>>> > >> > in the GUI and also can be used by the
batch
>>>>>>>>>> mode.
>>>>>>>>>> > >> > >>>>> > >> > You can always add more data to the
database
>>>>>>>>>> using
>>>>>>>>>> > >> > >>>>> > >> > the
>>>>>>>>>> > >> > >>>>> > >> > same load
>>>>>>>>>> > >> > >>>>> > >> > mode.
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> > Is there a guide for me to understand
the
>>>>>>>>>> process
>>>>>>>>>> > >> > >>>>> > >> > that you
>>>>>>>>>> > >> > >>>>> > >> > are
>>>>>>>>>> > >> > >>>>> > >> > referring to?
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM
Tatiana
>>>>>>>>>> Burek via RT
>>>>>>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>>>>>>>>> > >> > >>>>> > >> > wrote:
>>>>>>>>>> > >> > >>>>> > >> >
>>>>>>>>>> > >> > >>>>> > >> > > Hi Edward,
>>>>>>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
servers
>>>>>>>>>> and
>>>>>>>>>> > >> > >>>>> > >> > > can't
>>>>>>>>>> > >> > >>>>> > >> > > view your
>>>>>>>>>> > >> > >>>>> > >> > > XML
>>>>>>>>>> > >> > >>>>> > >> > > for the
>>>>>>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to
this
>>>>>>>>>> issue and I
>>>>>>>>>> > >> > >>>>> > >> > > will
>>>>>>>>>> > >> > >>>>> > >> > > take a
>>>>>>>>>> > >> > >>>>> > >> > > look.
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders>
section
>>>>>>>>>> should NOT
>>>>>>>>>> > >> > >>>>> > >> > > be
>>>>>>>>>> > >> > >>>>> > >> > > changed
>>>>>>>>>> > >> > >>>>> > >> > > and
>>>>>>>>>> > >> > >>>>> > >> > > will be
>>>>>>>>>> > >> > >>>>> > >> > > replaced by other scripts during the
batch
>>>>>>>>>> > >> > >>>>> > >> > > execution.
>>>>>>>>>> > >> > >>>>> > >> > > The data
>>>>>>>>>> > >> > >>>>> > >> > > for
>>>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>>>> > >> > >>>>> > >> > > batch mode is coming from the
database and
>>>>>>>>>> not from
>>>>>>>>>> > >> > >>>>> > >> > > files. The
>>>>>>>>>> > >> > >>>>> > >> > > name
>>>>>>>>>> > >> > >>>>> > >> > > of the
>>>>>>>>>> > >> > >>>>> > >> > > database is in <database> tag.
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET
output
>>>>>>>>>> files.
>>>>>>>>>> > >> > >>>>> > >> > > They use
>>>>>>>>>> > >> > >>>>> > >> > > METviewer
>>>>>>>>>> > >> > >>>>> > >> > > load
>>>>>>>>>> > >> > >>>>> > >> > > mode to load these files to the
specific
>>>>>>>>>> database.
>>>>>>>>>> > >> > >>>>> > >> > > In
>>>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>>>> > >> > >>>>> config
>>>>>>>>>> > >> > >>>>> > >> > > file
>>>>>>>>>> > >> > >>>>> > >> > > for
>>>>>>>>>> > >> > >>>>> > >> > > this process they also can specify
the
>>>>>>>>>> database
>>>>>>>>>> > >> > >>>>> > >> > > group.
>>>>>>>>>> > >> > >>>>> > >> > > For
>>>>>>>>>> > >> > >>>>> > >> > > example
>>>>>>>>>> > >> > >>>>> > >> > > database
>>>>>>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>>>>>>>>> > >> > >>>>> > >> > > After the data has been loaded to the
>>>>>>>>>> database it
>>>>>>>>>> > >> > >>>>> > >> > > is
>>>>>>>>>> > >> > >>>>> > >> > > visible
>>>>>>>>>> > >> > >>>>> and
>>>>>>>>>> > >> > >>>>> > >> > > available
>>>>>>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by
the
>>>>>>>>>> batch mode.
>>>>>>>>>> > >> > >>>>> > >> > > You can always add more data to the
>>>>>>>>>> database using
>>>>>>>>>> > >> > >>>>> > >> > > the
>>>>>>>>>> > >> > >>>>> > >> > > same
>>>>>>>>>> > >> > >>>>> load
>>>>>>>>>> > >> > >>>>> > >> > > mode.
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >> > > Tatiana
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>>>>>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>>>>>>>>> > >> > >>>>> > >> > > wrote:
>>>>>>>>>> > >> > >>>>> > >> > > > Good morning,
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made
an
>>>>>>>>>> effort to
>>>>>>>>>> > >> > >>>>> > >> > > > understand
>>>>>>>>>> > >> > >>>>> > >> > > > metviewer,
>>>>>>>>>> > >> > >>>>> > >> > > > test
>>>>>>>>>> > >> > >>>>> > >> > > > different cases using the history
tab,
>>>>>>>>>> changing
>>>>>>>>>> > >> > >>>>> > >> > > > information
>>>>>>>>>> > >> > >>>>> to
>>>>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>>>>> > >> > >>>>> > >> > > > what
>>>>>>>>>> > >> > >>>>> > >> > > > works and what does not, and make
>>>>>>>>>> connections
>>>>>>>>>> > >> > >>>>> > >> > > > between
>>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>>> > >> > >>>>> GUI
>>>>>>>>>> > >> > >>>>> > >> > > > settings
>>>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>>>> > >> > >>>>> > >> > > > the XML file. Although my comfort
level
>>>>>>>>>> with
>>>>>>>>>> > >> > >>>>> > >> > > > metviewer has
>>>>>>>>>> > >> > >>>>> > >> > > > significantly
>>>>>>>>>> > >> > >>>>> > >> > > > increased during this time, there
are some
>>>>>>>>>> > >> > >>>>> > >> > > > remaining
>>>>>>>>>> > >> > >>>>> > >> > > > issues
>>>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>>>> > >> > >>>>> > >> > > > I'm
>>>>>>>>>> > >> > >>>>> > >> > > > hoping
>>>>>>>>>> > >> > >>>>> > >> > > > to clear up.
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
running
>>>>>>>>>> with
>>>>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>>>>> > >> > >>>>> > >> > > > statistic
>>>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>>>>> > >> > >>>>> > >> > > > type:
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
>>>>>>>>>> provided me
>>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>>> > >> > >>>>> > >> > > > working
>>>>>>>>>> > >> > >>>>> > >> > > > directory
>>>>>>>>>> > >> > >>>>> > >> > > > to get
>>>>>>>>>> > >> > >>>>> > >> > > > started which I copied over:
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I
adjusted
>>>>>>>>>> and used
>>>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>>>> > >> > >>>>> > >> > > > run his
>>>>>>>>>> > >> > >>>>> > >> > > > example.
>>>>>>>>>> > >> > >>>>> > >> > > > running of the script "bash
test.sh"
>>>>>>>>>> produced a
>>>>>>>>>> > >> > >>>>> > >> > > > CSI
>>>>>>>>>> > >> > >>>>> > >> > > > plot
>>>>>>>>>> > >> > >>>>> that
>>>>>>>>>> > >> > >>>>> > >> > > > was
>>>>>>>>>> > >> > >>>>> > >> > > > stored
>>>>>>>>>> > >> > >>>>> > >> > > > here:
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >>
>>>>>>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > Running the script produced the
expected
>>>>>>>>>> result.
>>>>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>>>>> > >> > >>>>> > >> > > > next
>>>>>>>>>> > >> > >>>>> > >> > > > natural
>>>>>>>>>> > >> > >>>>> > >> > > > step was
>>>>>>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and
create a
>>>>>>>>>> new xml
>>>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>>>> > >> > >>>>> > >> > > > called
>>>>>>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
create
>>>>>>>>>> a time
>>>>>>>>>> > >> > >>>>> > >> > > > series
>>>>>>>>>> > >> > >>>>> > >> > > > where
>>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg
>>>>>>>>>> instead of
>>>>>>>>>> > >> > >>>>> > >> > > > obs_thresh.
>>>>>>>>>> > >> > >>>>> > >> > > > Accordingly,
>>>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
(removing
>>>>>>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>>>>>>>>> > >> > >>>>> and
>>>>>>>>>> > >> > >>>>> > >> > > > changed
>>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR
was also
>>>>>>>>>> > >> > >>>>> > >> > > > tried).
>>>>>>>>>> > >> > >>>>> > >> > > > The
>>>>>>>>>> > >> > >>>>> error
>>>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>>>> > >> > >>>>> > >> > > > resulted
>>>>>>>>>> > >> > >>>>> > >> > > > is as follows:
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
>>>>>>>>>> ('2020-05-01
>>>>>>>>>> > >> > >>>>> > >> > > > 13:00:00',
>>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
>>>>>>>>>> '2020-05-01
>>>>>>>>>> > >> > >>>>> > >> > > > 16:00:00',
>>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
>>>>>>>>>> '2020-05-01
>>>>>>>>>> > >> > >>>>> > >> > > > 19:00:00',
>>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
>>>>>>>>>> '2020-05-01
>>>>>>>>>> > >> > >>>>> > >> > > > 22:00:00',
>>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>>> > >> > >>>>> > >> > > > 01
>>>>>>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
>>>>>>>>>> '2020-05-02
>>>>>>>>>> > >> > >>>>> > >> > > > 01:00:00',
>>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
>>>>>>>>>> '2020-05-02
>>>>>>>>>> > >> > >>>>> > >> > > > 04:00:00',
>>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
>>>>>>>>>> '2020-05-02
>>>>>>>>>> > >> > >>>>> > >> > > > 07:00:00',
>>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
>>>>>>>>>> '2020-05-02
>>>>>>>>>> > >> > >>>>> > >> > > > 10:00:00',
>>>>>>>>>> > >> > >>>>> > >> > > > '2020-05-
>>>>>>>>>> > >> > >>>>> > >> > > > 02
>>>>>>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>>>>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
>>>>>>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
h.stat_header_id
>>>>>>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>>>>>>>>> > >> > >>>>> > >> > > > No data was returned from database
>>>>>>>>>> > >> > >>>>> > >> > > > 2020-06-09
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >>
>>>>>>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>>>>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable -
**
>>>>>>>>>> ERROR:
>>>>>>>>>> > >> > >>>>> > >> > > > Caught
>>>>>>>>>> > >> > >>>>> > >> > > > class
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> edu.ucar.metviewer.EmptyResultSetException in
>>>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>>>>>>>>> > >> > >>>>> > >> > > > res): result set contained no data
>>>>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
>>>>>>>>>> result set
>>>>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
>>>>>>>>>> > >> > >>>>> > >> > > >
edu.ucar.metviewer.EmptyResultSetException
>>>>>>>>>> > >> > >>>>> > >> > > > in
>>>>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res):
>>>>>>>>>> result set
>>>>>>>>>> > >> > >>>>> > >> > > > contained no
>>>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > Apparently there was no data during
this
>>>>>>>>>> time. I
>>>>>>>>>> > >> > >>>>> > >> > > > tried
>>>>>>>>>> > >> > >>>>> > >> > > > changing
>>>>>>>>>> > >> > >>>>> > >> > > > the
>>>>>>>>>> > >> > >>>>> > >> > > > range
>>>>>>>>>> > >> > >>>>> > >> > > > of valid times and changed the
valid time
>>>>>>>>>> to
>>>>>>>>>> > >> > >>>>> > >> > > > different
>>>>>>>>>> > >> > >>>>> > >> > > > days
>>>>>>>>>> > >> > >>>>> > >> > > > just in
>>>>>>>>>> > >> > >>>>> > >> > > > case
>>>>>>>>>> > >> > >>>>> > >> > > > data was missing. The issue was
the
>>>>>>>>>> same. To
>>>>>>>>>> > >> > >>>>> > >> > > > confirm
>>>>>>>>>> > >> > >>>>> > >> > > > my
>>>>>>>>>> > >> > >>>>> > >> > > > logic, I
>>>>>>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
>>>>>>>>>> metviewer,
>>>>>>>>>> > >> > >>>>> > >> > > > which I
>>>>>>>>>> > >> > >>>>> > >> > > > found
>>>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>>>> > >> > >>>>> > >> > > > work.
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > I think I lack an understanding of
how to
>>>>>>>>>> set up
>>>>>>>>>> > >> > >>>>> > >> > > > batch
>>>>>>>>>> > >> > >>>>> > >> > > > mode
>>>>>>>>>> > >> > >>>>> > >> > > > because
>>>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>>>> > >> > >>>>> > >> > > > don't
>>>>>>>>>> > >> > >>>>> > >> > > > see where these definitions are
being
>>>>>>>>>> explicitly
>>>>>>>>>> > >> > >>>>> > >> > > > defined:
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > <plot_spec>
>>>>>>>>>> > >> > >>>>> > >> > > > <connection>
>>>>>>>>>> > >> > >>>>> > >> > > > <host>rds_host:3306</host>
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>>>>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>>>>>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>>>>>>>>> > >> > >>>>> > >> > > > </connection>
>>>>>>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>>>>>>>>> > >> > >>>>> > >> > > > <folders>
>>>>>>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>>>>>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>>>>>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>>>>>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>>>>>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>>>>>>>>> > >> > >>>>> > >> > > > </folders>
>>>>>>>>>> > >> > >>>>> > >> > > > <plot>
>>>>>>>>>> > >> > >>>>> > >> > > >
<template>series_plot.R_tmpl</template>
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
grepping
>>>>>>>>>> all
>>>>>>>>>> > >> > >>>>> > >> > > > available
>>>>>>>>>> > >> > >>>>> files
>>>>>>>>>> > >> > >>>>> > >> > > > does
>>>>>>>>>> > >> > >>>>> > >> > > > not
>>>>>>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being
defined,
>>>>>>>>>> as well
>>>>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>>>>> > >> > >>>>> > >> > > > other
>>>>>>>>>> > >> > >>>>> > >> > > > similarly
>>>>>>>>>> > >> > >>>>> > >> > > > defined
>>>>>>>>>> > >> > >>>>> > >> > > > variables. I think if I know how
these
>>>>>>>>>> are being
>>>>>>>>>> > >> > >>>>> > >> > > > defined,
>>>>>>>>>> > >> > >>>>> > >> > > > then
>>>>>>>>>> > >> > >>>>> > >> > > > I'll
>>>>>>>>>> > >> > >>>>> > >> > > > be
>>>>>>>>>> > >> > >>>>> > >> > > > able to pipe in my information and
>>>>>>>>>> process the
>>>>>>>>>> > >> > >>>>> > >> > > > data
>>>>>>>>>> > >> > >>>>> > >> > > > using
>>>>>>>>>> > >> > >>>>> met
>>>>>>>>>> > >> > >>>>> > >> > > > and
>>>>>>>>>> > >> > >>>>> > >> > > > produce
>>>>>>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I right
on
>>>>>>>>>> that?
>>>>>>>>>> > >> > >>>>> > >> > > > Also, if
>>>>>>>>>> > >> > >>>>> > >> > > > I want
>>>>>>>>>> > >> > >>>>> > >> > > > to use
>>>>>>>>>> > >> > >>>>> > >> > > > obs for
>>>>>>>>>> > >> > >>>>> > >> > > > verification, do I add another line
in
>>>>>>>>>> the XML
>>>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>>>> > >> > >>>>> > >> > > > that
>>>>>>>>>> > >> > >>>>> > >> > > > designates
>>>>>>>>>> > >> > >>>>> > >> > > > file
>>>>>>>>>> > >> > >>>>> > >> > > > location of obs? I assume that
other
>>>>>>>>>> steps have
>>>>>>>>>> > >> > >>>>> > >> > > > to be
>>>>>>>>>> > >> > >>>>> > >> > > > run
>>>>>>>>>> > >> > >>>>> > >> > > > before
>>>>>>>>>> > >> > >>>>> > >> > > > all
>>>>>>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr form,
for
>>>>>>>>>> example,
>>>>>>>>>> > >> > >>>>> > >> > > > I
>>>>>>>>>> > >> > >>>>> > >> > > > would
>>>>>>>>>> > >> > >>>>> have
>>>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>>>> > >> > >>>>> > >> > > > separately
>>>>>>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs,
then I
>>>>>>>>>> would want
>>>>>>>>>> > >> > >>>>> > >> > > > to
>>>>>>>>>> > >> > >>>>> > >> > > > do that
>>>>>>>>>> > >> > >>>>> > >> > > > as
>>>>>>>>>> > >> > >>>>> > >> > > > well.
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > 3. Communicating results to
metviewer
>>>>>>>>>> online
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of
people/projects
>>>>>>>>>> listed
>>>>>>>>>> > >> > >>>>> > >> > > > here -
>>>>>>>>>> > >> > >>>>> > >> > > > see
>>>>>>>>>> > >> > >>>>> > >> > > > attachment
>>>>>>>>>> > >> > >>>>> > >> > > >
>>>>>>>>>> > >> > >>>>> > >> > > > How do I get the results I generate
to be
>>>>>>>>>> > >> > >>>>> > >> > > > uploaded to
>>>>>>>>>> > >> > >>>>> > >> > > > this
>>>>>>>>>> > >> > >>>>> > >> > > > location?
>>>>>>>>>> > >> > >>>>> > >> > > > It's
>>>>>>>>>> > >> > >>>>> > >> > > > not clear how others are setting
this up.
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >> > >
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>> > >>
>>>>>>>>>> > >> > >>>>> > >> rds_host:3306
>>>>>>>>>> mv_cmaqv5para8_aod_grid2grid_metplus
>>>>>>>>>> > >> > >>>>> > >> rds_user
>>>>>>>>>> > >> > >>>>> rds_pwd
>>>>>>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
rds_plots
>>>>>>>>>> rds_data
>>>>>>>>>> > >> > >>>>> rds_scripts
>>>>>>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>>>>>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>>>>>>>>> > >> > >>>>> 2020-05-12
>>>>>>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
>>>>>>>>>> 19:00:00 2020-
>>>>>>>>>> > >> > >>>>> > >> 05-12
>>>>>>>>>> > >> > >>>>> 20:00:00
>>>>>>>>>> > >> > >>>>> > >> 2020-05-12
>>>>>>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
>>>>>>>>>> 23:00:00 2020-
>>>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>>>> > >> > >>>>> 10:00:00
>>>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
>>>>>>>>>> 13:00:00 2020-
>>>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>>>> > >> > >>>>> 14:00:00
>>>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
>>>>>>>>>> 17:00:00 2020-
>>>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>>>> > >> > >>>>> 18:00:00
>>>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
>>>>>>>>>> 21:00:00 2020-
>>>>>>>>>> > >> > >>>>> > >> 05-13
>>>>>>>>>> > >> > >>>>> 22:00:00
>>>>>>>>>> > >> > >>>>> > >> 2020-05-13
>>>>>>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
>>>>>>>>>> 11:00:00 2020-
>>>>>>>>>> > >> > >>>>> > >> 05-14
>>>>>>>>>> > >> > >>>>> 12:00:00
>>>>>>>>>> > >> > >>>>> > >> true
>>>>>>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>>>>>>>>> > >> > >>>>> > >> plot_20200612_173626.png
>>>>>>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test
y_label
>>>>>>>>>> > >> > >>>>> > >> false list() list() false false false
false
>>>>>>>>>> false false
>>>>>>>>>> > >> > >>>>> > >> true
>>>>>>>>>> > >> > >>>>> > >> false
>>>>>>>>>> > >> > >>>>> > >> false
>>>>>>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72
in
>>>>>>>>>> c(8,4,5,4)
>>>>>>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>>>>>>>>> > >> > >>>>> > >> 1.4 -2
>>>>>>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1
1 1
>>>>>>>>>> -2 0.5 3
>>>>>>>>>> > >> > >>>>> > >> #cccccc 1
>>>>>>>>>> > >> > >>>>> > >> listX 1
>>>>>>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1
0.5 o
>>>>>>>>>> c(0, -.25)
>>>>>>>>>> > >> > >>>>> > >> 3
>>>>>>>>>> > >> > >>>>> > >> 0.8 1
>>>>>>>>>> > >> > >>>>> > >> #333333
>>>>>>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none")
c(FALSE,FALSE)
>>>>>>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>>>>>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
c("b","b")
>>>>>>>>>> c(1,1)
>>>>>>>>>> > >> > >>>>> > >> c(1,1)
>>>>>>>>>> > >> > >>>>> c(1,1)
>>>>>>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>> > > --
>>>>>>>>>> > >> > >>>>> > > Edward Strobach
>>>>>>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>>>>>>>>> > >> > >>>>> > > IMSG Contractor
>>>>>>>>>> > >> > >>>>> > > Cubicle#: 2029
>>>>>>>>>> > >> > >>>>> > > 301-683-3717
>>>>>>>>>> > >> > >>>>> > >
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>>
>>>>>>>>>> > >> > >>>>
>>>>>>>>>> > >> > >>>> --
>>>>>>>>>> > >> > >>>> Edward Strobach
>>>>>>>>>> > >> > >>>> EMC/NCEP/NWS/
>>>>>>>>>> > >> > >>>> IMSG Contractor
>>>>>>>>>> > >> > >>>> Cubicle#: 2029
>>>>>>>>>> > >> > >>>> 301-683-3717
>>>>>>>>>> > >> > >>>>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>> --
>>>>>>>>>> > >> > >>> Edward Strobach
>>>>>>>>>> > >> > >>> EMC/NCEP/NWS/
>>>>>>>>>> > >> > >>> IMSG Contractor
>>>>>>>>>> > >> > >>> Cubicle#: 2029
>>>>>>>>>> > >> > >>> 301-683-3717
>>>>>>>>>> > >> > >>>
>>>>>>>>>> > >> > >>
>>>>>>>>>> > >> > >>
>>>>>>>>>> > >> > >> --
>>>>>>>>>> > >> > >> Edward Strobach
>>>>>>>>>> > >> > >> EMC/NCEP/NWS/
>>>>>>>>>> > >> > >> IMSG Contractor
>>>>>>>>>> > >> > >> Cubicle#: 2029
>>>>>>>>>> > >> > >> 301-683-3717
>>>>>>>>>> > >> > >>
>>>>>>>>>> > >> > >
>>>>>>>>>> > >> > >
>>>>>>>>>> > >> > > --
>>>>>>>>>> > >> > > Edward Strobach
>>>>>>>>>> > >> > > EMC/NCEP/NWS/
>>>>>>>>>> > >> > > IMSG Contractor
>>>>>>>>>> > >> > > Cubicle#: 2029
>>>>>>>>>> > >> > > 301-683-3717
>>>>>>>>>> > >> > >
>>>>>>>>>> > >>
>>>>>>>>>> > >>
>>>>>>>>>> > >>
>>>>>>>>>> > >>
>>>>>>>>>> > >
>>>>>>>>>> > > --
>>>>>>>>>> > > Edward Strobach
>>>>>>>>>> > > EMC/NCEP/NWS/
>>>>>>>>>> > > IMSG Contractor
>>>>>>>>>> > > Cubicle#: 2029
>>>>>>>>>> > > 301-683-3717
>>>>>>>>>> > >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Edward Strobach
>>>>>>>>> EMC/NCEP/NWS/
>>>>>>>>> IMSG Contractor
>>>>>>>>> Cubicle#: 2029
>>>>>>>>> 301-683-3717
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Edward Strobach
>>>>>>>> EMC/NCEP/NWS/
>>>>>>>> IMSG Contractor
>>>>>>>> Cubicle#: 2029
>>>>>>>> 301-683-3717
>>>>>>>>
>>>>>>> --
>>>>>>> Jeffery T. McQueen
>>>>>>> Meteorologist
>>>>>>> National Center for Weather and Climate Prediction
>>>>>>> Room 2095
>>>>>>> 5830 University Research Court
>>>>>>> College Park, MD 20741
>>>>>>> 301-683-3736
>>>>>>> Fax: 301-683-3703
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Edward Strobach
>>>>>> EMC/NCEP/NWS/
>>>>>> IMSG Contractor
>>>>>> Cubicle#: 2029
>>>>>> 301-683-3717
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Edward Strobach
>>>> EMC/NCEP/NWS/
>>>> IMSG Contractor
>>>> Cubicle#: 2029
>>>> 301-683-3717
>>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 13:58:59 2020
Thank you for confirming
On Wed, Jun 17, 2020 at 3:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Yes, you need to have these values in your batch XML.
>
> Tatiana
> On Wed Jun 17 13:47:45 2020, edward.strobach at noaa.gov wrote:
> > I'm sorry, I guess I may have been a bit confused by what you were
> > initially telling me. Are you saying replace what I have with
this?
> >
> > <connection>
> > *<host>rds_host:3306</host>*
> > * <database>MY_DATABASEdatabase>*
> > <user>rds_user</user>
> > <password>rds_pwd</password>
> > <management_system>aurora</management_system>
> > </connection>
> > * <rscript>Rscript</rscript>*
> > <folders>
> >
> >
> >
> >
> > * <r_tmpl>rds_R_tmpl</r_tmpl>
> > <r_work>rds_R_work</r_work>
> > <plots>rds_plots</plots> <data>rds_data</data>
> > <scripts>rds_scripts</scripts>*
> >
> > On Wed, Jun 17, 2020 at 3:27 PM Ho-Chun Huang - NOAA Affiliate <
> > ho-chun.huang at noaa.gov> wrote:
> >
> > > Ed:
> > >
> > > You can try or you can keep waiting, it is your choice.
> > >
> > > Ho-Chun Huang
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct., Rm. 2792
> > >
> > > College Park, MD 20740
> > >
> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >
> > > 301-683-3958
> > >
> > >
> > > On Wed, Jun 17, 2020 at 3:21 PM Edward Strobach - NOAA Affiliate
<
> > > edward.strobach at noaa.gov> wrote:
> > >
> > >> Hi Ho-Chun,
> > >>
> > >> I've made all the changes that I would need to inside my xml
file.
> > >> I was
> > >> told not to change the folders info. I instead changed my
database
> > >> name,
> > >> fcst_var, stat options, series1 input, the fix fields, the
indep
> > >> field, and
> > >> x_lim.
> > >>
> > >> Thanks
> > >>
> > >> On Wed, Jun 17, 2020 at 3:16 PM Ho-Chun Huang - NOAA Affiliate
<
> > >> ho-chun.huang at noaa.gov> wrote:
> > >>
> > >>> Hi, Ed:
> > >>>
> > >>> You should go back to read 1README again from the directory I
sent
> > >>> about
> > >>> replacing the top portion of your metviewer xml to produce a
batch
> > >>> xml.
> > >>>
> > >>> I check your
> > >>>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/Strobach_Time_Series_xml.template.
> > >>> Please update the section from <connection> to </folder> as
those
> > >>> in
> > >>> hochun_xml.template. I also do not know whether AWS can handle
> > >>> comments
> > >>> <!-- to !--> in batch xml , better make your batch xml clean.
> > >>>
> > >>> Ho-Chun Huang
> > >>>
> > >>> IMSG at NOAA/NWS/NCEP/EMC
> > >>>
> > >>> 5830 University Research Ct., Rm. 2792
> > >>>
> > >>> College Park, MD 20740
> > >>>
> > >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >>>
> > >>> 301-683-3958
> > >>>
> > >>>
> > >>> On Wed, Jun 17, 2020 at 2:28 PM Edward Strobach - NOAA
Affiliate <
> > >>> edward.strobach at noaa.gov> wrote:
> > >>>
> > >>>> Just adding more detail to elaborate my concerns...
> > >>>>
> > >>>> Not too long ago I was given a test set-up by Ho-Chun to
generate
> > >>>> a
> > >>>> plot for aerosol optical depth. The result was successful
and
> > >>>> produced a
> > >>>> CSI plot for AOD, comparing the production run and
experimental
> > >>>> run.
> > >>>> Although I created a database, I do not seem to have a group
where
> > >>>> I can
> > >>>> access specific databases. I recently created a load_xml
file,
> > >>>> which
> > >>>> should have loaded the data based on what I specified into
the
> > >>>> group. It
> > >>>> seemed successful based on the log output that was generated.
> > >>>> Additionally, all information that I specified in this step
was
> > >>>> create in
> > >>>> the load_*xml.
> > >>>>
> > >>>> The next step is to generate plots since the data should be
> > >>>> loaded.
> > >>>> Here's what happens when I run Ho-Chun's - the successful
case
> > >>>>
> > >>>> HO-CHUN:
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> *+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
> > >>>> hochun_xml.template+ bash
> > >>>>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> > >>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> > >>>> plot.xmlCALLING: scp plot.xml
edward.strobach at 205.156.8.85:~----
> > >>>> MVBatch
> > >>>> ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# #
# #
> > >>>> # # # # #
> > >>>> # # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT
> > >>>> model,
> > >>>> ld.fcst_init_beg, ld.fcst_valid_beg, ld.fcst_lead,*
> > >>>>
> > >>>> This is early on in the script.
> > >>>>
> > >>>> ED_STROBACH:
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> *+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+
sed
> > >>>> -e
> > >>>> 's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
> > >>>>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
> > >>>> edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
> > >>>> plot.xmlCALLING: scp plot.xml
edward.strobach at 205.156.8.85:~----
> > >>>> MVBatch
> > >>>> ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020
6:18:20 PM
> > >>>> org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable
to
> > >>>> create
> > >>>> initial connections of pool.java.sql.SQLException:
Communications
> > >>>> link
> > >>>> failure with primary. No active connection found for master.
> > >>>> at
> > >>>>
>
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
> > >>>> at
> > >>>>
>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
> > >>>> at
> > >>>>
>
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
> > >>>> at
> > >>>>
>
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
> > >>>> at
> > >>>>
org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
> > >>>> at
> > >>>>
>
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
> > >>>> at org.mariadb.jdbc.Driver.connect(Driver.java:92)*
> > >>>>
> > >>>> My interpretation of this is that it fails to bridge a
connection
> > >>>> to
> > >>>> the group or database; therefore, it's unable to proceed with
> > >>>> running the
> > >>>> XML file to generate the plots based on the specified
content. If
> > >>>> I move
> > >>>> past this step, then I'm golden.
> > >>>>
> > >>>> On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
> > >>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>
> > >>>>> Would it be ok to keep perry in the loop
> > >>>>>
> > >>>>> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA
Affiliate
> > >>>>> <
> > >>>>> edward.strobach at noaa.gov> wrote:
> > >>>>>
> > >>>>>> you're right. I didn't have the storage problem this time.
I'm
> > >>>>>> not
> > >>>>>> sure what changed. All I know is that I have now created
my
> > >>>>>> load xml which
> > >>>>>> now should allow me to move on to the next step of plotting
> > >>>>>>
> > >>>>>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA
> > >>>>>> Affiliate <
> > >>>>>> edward.strobach at noaa.gov> wrote:
> > >>>>>>
> > >>>>>>> I'm not following you. How do I now have enough space?
What
> > >>>>>>> changed?
> > >>>>>>>
> > >>>>>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT <
> > >>>>>>> met_help at ucar.edu> wrote:
> > >>>>>>>
> > >>>>>>>> Ed, now you should have enough space for the data
loading.
> > >>>>>>>>
> > >>>>>>>> Tatiana
> > >>>>>>>>
> > >>>>>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov
wrote:
> > >>>>>>>> > I was basically able to build my xml file but it says I
have
> > >>>>>>>> > no
> > >>>>>>>> > available
> > >>>>>>>> > space. I'm not sure how that can be since I just
started
> > >>>>>>>> > adding
> > >>>>>>>> data
> > >>>>>>>> > to my
> > >>>>>>>> > account
> > >>>>>>>> >
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat:
No
> > >>>>>>>> > space
> > >>>>>>>> left
> > >>>>>>>> > on
> > >>>>>>>> > device
> > >>>>>>>> > ERROR: Command returned with non-zero status (1): scp
-r ./*
> > >>>>>>>> >
edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
> > >>>>>>>> > 4) Check METviewer AWS database list using
> > >>>>>>>> > mv_db_size_on_aws.sh
> > >>>>>>>> > TABLE_SCHEMA USER
> > >>>>>>>> > SIZE_MB
> > >>>>>>>> > mv_met_g2o_gefs binbin.zhou
> > >>>>>>>> 140027.31
> > >>>>>>>> > mv_met_system_hrefv2_v3 binbin.zhou
> > >>>>>>>> 71986.11
> > >>>>>>>> >
> > >>>>>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
> > >>>>>>>> > Affiliate <
> > >>>>>>>> > edward.strobach at noaa.gov> wrote:
> > >>>>>>>> >
> > >>>>>>>> > > Thanks Tatiana. Yes, I've created a database which
seems
> > >>>>>>>> different
> > >>>>>>>> > > from
> > >>>>>>>> > > mv_group. How can one create their own group so that
I can
> > >>>>>>>> store and
> > >>>>>>>> > > access
> > >>>>>>>> > > the database. The database exists but cannot be
viewed on
> > >>>>>>>> > > the
> > >>>>>>>> > > metviewer
> > >>>>>>>> > > GUI. The list starts with EMC HREFv3 and ends with
> > >>>>>>>> > > vhagerty. I
> > >>>>>>>> > > named my
> > >>>>>>>> > > database Ed_Strobach, but that was really intended
for the
> > >>>>>>>> > > group
> > >>>>>>>> > > name.
> > >>>>>>>> > >
> > >>>>>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> > >>>>>>>> > > <met_help at ucar.edu>
> > >>>>>>>> > > wrote:
> > >>>>>>>> > >
> > >>>>>>>> > >> We don't have database and METviewer running on
wcoss.
> > >>>>>>>> > >> Instead, users use METviewer-wcoss scripts that help
to
> > >>>>>>>> connect to
> > >>>>>>>> > >> and
> > >>>>>>>> > >> use METviewer that is running on AWS. That is why
XML
> > >>>>>>>> connection and
> > >>>>>>>> > >> folder
> > >>>>>>>> > >> sections look somewhat different than once from the
Docs
> > >>>>>>>> > >> page.
> > >>>>>>>> > >> To create a database on AWS you need to run
> > >>>>>>>> mv_create_db_on_aws.sh
> > >>>>>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
> > >>>>>>>> > >> For data loading :
> > >>>>>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> > >>>>>>>> > >> And for batch:
> > >>>>>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir>
<xml_file>
> > >>>>>>>> > >>
> > >>>>>>>> > >> These scripts copy XML and data files if needed to
AWS,
> > >>>>>>>> > >> adjust
> > >>>>>>>> data
> > >>>>>>>> > >> path
> > >>>>>>>> > >> and users credentials, run METviewer and copy
results
> > >>>>>>>> > >> back to
> > >>>>>>>> wcoss.
> > >>>>>>>> > >>
> > >>>>>>>> > >> Tatiana
> > >>>>>>>> > >> On Tue Jun 16 08:26:32 2020,
edward.strobach at noaa.gov
> > >>>>>>>> > >> wrote:
> > >>>>>>>> > >> > lastly, it does seem thatmysql -u[db_username]
> > >>>>>>>> -p[db_password]
> > >>>>>>>> > >> > -e'create
> > >>>>>>>> > >> > database [db_name]
> > >>>>>>>> > >> >
> > >>>>>>>> > >> > would need to be used to create a database. I
assume
> > >>>>>>>> > >> > that
> > >>>>>>>> an sql
> > >>>>>>>> > >> > module is
> > >>>>>>>> > >> > available on wcoss; I know you don't have access
to
> > >>>>>>>> > >> > those
> > >>>>>>>> machines
> > >>>>>>>> > >> > Tatiana. If I create my own database with my own
name,
> > >>>>>>>> > >> > then
> > >>>>>>>> I
> > >>>>>>>> > >> > should
> > >>>>>>>> > >> > be
> > >>>>>>>> > >> > able to proceed with project specific tasks under
that
> > >>>>>>>> > >> > tab.
> > >>>>>>>> > >> > Something
> > >>>>>>>> > >> > like
> > >>>>>>>> > >> > mv_EdwardStrobach.
> > >>>>>>>> > >> >
> > >>>>>>>> > >> > I can then create mv_cmaq_prod_v_para which I add
data
> > >>>>>>>> > >> > to
> > >>>>>>>> using
> > >>>>>>>> > >> > load_spec.
> > >>>>>>>> > >> > Once that step is done, then I can run plot_spec.
> > >>>>>>>> > >> > That's my
> > >>>>>>>> > >> > takeaway
> > >>>>>>>> > >> > currently
> > >>>>>>>> > >> >
> > >>>>>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach -
NOAA
> > >>>>>>>> Affiliate <
> > >>>>>>>> > >> > edward.strobach at noaa.gov> wrote:
> > >>>>>>>> > >> >
> > >>>>>>>> > >> > > Actually, it appears that I need to run the
database
> > >>>>>>>> loading
> > >>>>>>>> > >> > > module
> > >>>>>>>> > >> > > first
> > >>>>>>>> > >> > > if I want to change/add a data source. From
there I
> > >>>>>>>> > >> > > can
> > >>>>>>>> run the
> > >>>>>>>> > >> > > plotting
> > >>>>>>>> > >> > > batch module. Is that right? So if I run the
> > >>>>>>>> > >> > > database
> > >>>>>>>> loading
> > >>>>>>>> > >> > > module,
> > >>>>>>>> > >> > > then I can specify my dates and other related
info
> > >>>>>>>> specific to
> > >>>>>>>> > >> > > the
> > >>>>>>>> > >> > > dataset. If I create my database with a unique
name,
> > >>>>>>>> > >> > > then
> > >>>>>>>> it
> > >>>>>>>> > >> > > should
> > >>>>>>>> > >> > > be
> > >>>>>>>> > >> > > recognized on the metviewer gui, presumably.
> > >>>>>>>> > >> > >
> > >>>>>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach
-
> > >>>>>>>> > >> > > NOAA
> > >>>>>>>> Affiliate
> > >>>>>>>> > >> > > <
> > >>>>>>>> > >> > > edward.strobach at noaa.gov> wrote:
> > >>>>>>>> > >> > >
> > >>>>>>>> > >> > >> It seems that those options must be treated
> > >>>>>>>> > >> > >> separately
> > >>>>>>>> outside
> > >>>>>>>> > >> > >> of
> > >>>>>>>> > >> > >> plot_spec. Is it acceptable to create both a
> > >>>>>>>> > >> > >> load_spec
> > >>>>>>>> and
> > >>>>>>>> > >> > >> plot_spec in
> > >>>>>>>> > >> > >> the same xml file? Would the definitions in
> > >>>>>>>> > >> > >> load_spec be
> > >>>>>>>> able
> > >>>>>>>> > >> > >> to
> > >>>>>>>> > >> > >> carry
> > >>>>>>>> > >> > >> over into plot_spec? It appears that data
loading
> > >>>>>>>> > >> > >> and
> > >>>>>>>> plotting
> > >>>>>>>> > >> > >> are
> > >>>>>>>> > >> > >> treated
> > >>>>>>>> > >> > >> separately.
> > >>>>>>>> > >> > >>
> > >>>>>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach
-
> > >>>>>>>> > >> > >> NOAA
> > >>>>>>>> > >> > >> Affiliate <
> > >>>>>>>> > >> > >> edward.strobach at noaa.gov> wrote:
> > >>>>>>>> > >> > >>
> > >>>>>>>> > >> > >>> I do plan to use date_range; however, in an
example
> > >>>>>>>> provided
> > >>>>>>>> > >> > >>> by the
> > >>>>>>>> > >> > >>> link
> > >>>>>>>> > >> > >>> I see this:
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> <load_spec>
> > >>>>>>>> > >> > >>> <connection>
> > >>>>>>>> > >> > >>> <host>kemosabe:3306</host>
> > >>>>>>>> > >> > >>> <database>metvdb_hwt</database>
> > >>>>>>>> > >> > >>> <user>pgoldenb</user>
> > >>>>>>>> > >> > >>> <password>pgoldenb</password>
> > >>>>>>>> > >> > >>> </connection>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> <date_list name="folder_dates">
> > >>>>>>>> > >> > >>> <start>2010051914V</start>
> > >>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset><hour>6
> > >>>>>>>> > >> > >>> </hour></date_offset></end>
> > >>>>>>>> > >> > >>> <inc>3600</inc>
> > >>>>>>>> > >> > >>> <format>yyyyMMddHH'V'</format>
> > >>>>>>>> > >> > >>> </date_list>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> <met_version>V3.0</met_version>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> <verbose>false</verbose>
> > >>>>>>>> > >> > >>> <insert_size>1</insert_size>
> > >>>>>>>> > >> > >>>
<mode_header_db_check>true</mode_header_db_check>
> > >>>>>>>> > >> > >>> <drop_indexes>false</drop_indexes>
> > >>>>>>>> > >> > >>> <apply_indexes>true</apply_indexes>
> > >>>>>>>> > >> > >>> <group>Group name</group>
> > >>>>>>>> > >> > >>> <load_stat>true</load_stat>
> > >>>>>>>> > >> > >>> <load_mode>true</load_mode>
> > >>>>>>>> > >> > >>> <load_mtd>true</load_mtd>
> > >>>>>>>> > >> > >>> <load_mpr>false</load_mpr>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>>
> > >>>>>>>>
> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > >>>>>>>> > >> > >>> <load_val>
> > >>>>>>>> > >> > >>> <field name="model">
> > >>>>>>>> > >> > >>> <val>arw</val>
> > >>>>>>>> > >> > >>> <val>nmm</val>
> > >>>>>>>> > >> > >>> </field>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> <field name="valid_time">
> > >>>>>>>> > >> > >>> <date_list name="folder_dates"/>
> > >>>>>>>> > >> > >>> </field>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> <field name="vx_mask">
> > >>>>>>>> > >> > >>> <val>FULL</val>
> > >>>>>>>> > >> > >>> <val>SWC</val>
> > >>>>>>>> > >> > >>> </field>
> > >>>>>>>> > >> > >>> </load_val>
> > >>>>>>>> > >> > >>> </load_spec>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> Are similar settings allowed with <plot_spec>?
I
> > >>>>>>>> > >> > >>> see
> > >>>>>>>> > >> > >>> <connection>,
> > >>>>>>>> > >> > >>> which seems carried over. I'm looking to
specify
> > >>>>>>>> > >> > >>> my
> > >>>>>>>> > >> > >>> folder_tmpl in
> > >>>>>>>> > >> > >>> <plot_spec>, which according to Tara, should
be
> > >>>>>>>> > >> > >>> possible.
> > >>>>>>>> > >> > >>> Since my
> > >>>>>>>> > >> > >>> directory structure looks like this:
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >>
> > >>>>>>>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> I would do something like this I would think:
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> .......
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> <date_list name="folder_dates">
> > >>>>>>>> > >> > >>> <start>20200612/start>
> > >>>>>>>> > >> > >>>
<end><date_offset><day_offset>0</day_offset></end>
> > >>>>>>>> > >> > >>> <inc>86400</inc>
> > >>>>>>>> > >> > >>> <format>yyyyMMdd</format>
> > >>>>>>>> > >> > >>> </date_list>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> <folder_tmpl>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >>
> > >>>>>>>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > >>>>>>>> > >> > >>> </folder_tmpl>
> > >>>>>>>> > >> > >>> <load_val>
> > >>>>>>>> > >> > >>> <field name="model">
> > >>>>>>>> > >> > >>> <val>prod</val>
> > >>>>>>>> > >> > >>> </field>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> <field name="valid_time">
> > >>>>>>>> > >> > >>> <date_list name="folder_dates"/>
> > >>>>>>>> > >> > >>> </field>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> ......
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward
Strobach -
> > >>>>>>>> > >> > >>> NOAA
> > >>>>>>>> > >> > >>> Affiliate <
> > >>>>>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>>> Thank you. This is a very helpful example
> > >>>>>>>> > >> > >>>>
> > >>>>>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek
via
> > >>>>>>>> > >> > >>>> RT
> > >>>>>>>> > >> > >>>> <met_help at ucar.edu>
> > >>>>>>>> > >> > >>>> wrote:
> > >>>>>>>> > >> > >>>>
> > >>>>>>>> > >> > >>>>> Hi,
> > >>>>>>>> > >> > >>>>> Here is some examples how to use
<date_range> and
> > >>>>>>>> > >> > >>>>> <date_range_list> in
> > >>>>>>>> > >> > >>>>> batch XML.
> > >>>>>>>> > >> > >>>>> First, you need to defined your range. You
can
> > >>>>>>>> > >> > >>>>> have
> > >>>>>>>> more
> > >>>>>>>> > >> > >>>>> then one
> > >>>>>>>> > >> > >>>>> <date_range>:
> > >>>>>>>> > >> > >>>>> <date_range name="may">
> > >>>>>>>> > >> > >>>>> <start>2019-05-01 00:00:00</start>
> > >>>>>>>> > >> > >>>>> <end>2019-05-31 23:59:59</end>
> > >>>>>>>> > >> > >>>>> </date_range>
> > >>>>>>>> > >> > >>>>> <date_range name="march">
> > >>>>>>>> > >> > >>>>> <start>2019-03-01 00:00:00</start>
> > >>>>>>>> > >> > >>>>> <end>2019-03-31 23:59:59</end>
> > >>>>>>>> > >> > >>>>> </date_range>
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>> and then use one or all date_ranges:
> > >>>>>>>> > >> > >>>>> <plot_fix>
> > >>>>>>>> > >> > >>>>> <field name="fcst_init_beg">
> > >>>>>>>> > >> > >>>>> <date_range name="march"/>
> > >>>>>>>> > >> > >>>>> <date_range name="may"/>
> > >>>>>>>> > >> > >>>>> </field>...
> > >>>>>>>> > >> > >>>>> This configuration would include all dates
for
> > >>>>>>>> > >> > >>>>> March
> > >>>>>>>> and May
> > >>>>>>>> > >> > >>>>> of
> > >>>>>>>> > >> > >>>>> 2019.
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>> If you need more flexible configuration you
can
> > >>>>>>>> > >> > >>>>> use
> > >>>>>>>> > >> > >>>>> date_range_list.
> > >>>>>>>> > >> > >>>>> For example, you want to select first 6
hours
> > >>>>>>>> > >> > >>>>> every day
> > >>>>>>>> > >> > >>>>> beetween
> > >>>>>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00.
Your
> > >>>>>>>> > >> > >>>>> XML
> > >>>>>>>> would
> > >>>>>>>> > >> > >>>>> look
> > >>>>>>>> > >> > >>>>> like this:
> > >>>>>>>> > >> > >>>>> <date_range_list name = 'all'>
> > >>>>>>>> > >> > >>>>> <range_start>2020-01-01
> > >>>>>>>> > >> > >>>>> 00:00:00</range_start>
> > >>>>>>>> > >> > >>>>> <range_end>2020-01-05
> > >>>>>>>> > >> > >>>>> 00:00:00</range_end>
> > >>>>>>>> > >> > >>>>> <range_length>360</range_length>
> > >>>>>>>> > >> > >>>>> <inc>1440</inc>
> > >>>>>>>> > >> > >>>>> </date_range_list>
> > >>>>>>>> > >> > >>>>> where the 1st and 2nd values are the start
and
> > >>>>>>>> > >> > >>>>> end,
> > >>>>>>>> > >> > >>>>> range_length - the length of a range in
minutes(6
> > >>>>>>>> houres)
> > >>>>>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24
> > >>>>>>>> > >> > >>>>> houres)
> > >>>>>>>> > >> > >>>>> This example would create following ranges:
> > >>>>>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-
01
> > >>>>>>>> > >> > >>>>> 06:00:00'
> > >>>>>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-
02
> > >>>>>>>> > >> > >>>>> 06:00:00'
> > >>>>>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-
03
> > >>>>>>>> > >> > >>>>> 06:00:00'
> > >>>>>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-
04
> > >>>>>>>> > >> > >>>>> 06:00:00'
> > >>>>>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-
05
> > >>>>>>>> > >> > >>>>> 06:00:00'
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>> Also you can add more parameters to
<plot_fix> to
> > >>>>>>>> filter
> > >>>>>>>> > >> > >>>>> your
> > >>>>>>>> > >> > >>>>> data.
> > >>>>>>>> > >> > >>>>> For example, if you want to select only
dates
> > >>>>>>>> > >> > >>>>> with
> > >>>>>>>> 00:00 you
> > >>>>>>>> > >> > >>>>> can
> > >>>>>>>> > >> > >>>>> specify
> > >>>>>>>> > >> > >>>>> <field name="valid_hour">
> > >>>>>>>> > >> > >>>>> <val>00</val>
> > >>>>>>>> > >> > >>>>> </field>
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>> Tatiana
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
> > >>>>>>>> > >> > >>>>> edward.strobach at noaa.gov
> > >>>>>>>> wrote:
> > >>>>>>>> > >> > >>>>> > and if there isn't a better way, I wonder
the
> > >>>>>>>> whereabouts
> > >>>>>>>> > >> > >>>>> > of a
> > >>>>>>>> > >> > >>>>> > separate
> > >>>>>>>> > >> > >>>>> > script that pipes dates (and other
relevant
> > >>>>>>>> information)
> > >>>>>>>> > >> > >>>>> > into
> > >>>>>>>> > >> > >>>>> > the XML
> > >>>>>>>> > >> > >>>>> > file
> > >>>>>>>> > >> > >>>>> > used to generate the results.
> > >>>>>>>> > >> > >>>>> >
> > >>>>>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward
> > >>>>>>>> > >> > >>>>> > Strobach -
> > >>>>>>>> NOAA
> > >>>>>>>> > >> > >>>>> > Affiliate <
> > >>>>>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
> > >>>>>>>> > >> > >>>>> >
> > >>>>>>>> > >> > >>>>> > > Thanks Tatiana. These resources are
helpful.
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>> > > One last question. Simply typing lines
like
> > >>>>>>>> > >> > >>>>> > > this
> > >>>>>>>> in can
> > >>>>>>>> > >> > >>>>> > > take
> > >>>>>>>> > >> > >>>>> > > forever:
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>> > > <plot_fix>
> > >>>>>>>> > >> > >>>>> > > <field equalize="false"
> > >>>>>>>> > >> > >>>>> > > name="fcst_init_beg">
> > >>>>>>>> > >> > >>>>> > > <set name="fcst_init_beg_0">
> > >>>>>>>> > >> > >>>>> > > <val>2019-05-01
12:00:00</val>
> > >>>>>>>> > >> > >>>>> > > <val>2019-05-02
06:00:00</val>
> > >>>>>>>> > >> > >>>>> > > <val>2019-05-02
12:00:00</val>
> > >>>>>>>> > >> > >>>>> > > <val>2019-05-03
06:00:00</val>
> > >>>>>>>> > >> > >>>>> > > <val>2019-05-03
12:00:00</val>
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>> > > Is there a better way? I see
date_range,
> > >>>>>>>> > >> > >>>>> > > date_range_list,
> > >>>>>>>> > >> > >>>>> > > or
> > >>>>>>>> > >> > >>>>> > > date_list.
> > >>>>>>>> > >> > >>>>> > > Can this be used instead of typing
individual
> > >>>>>>>> values?
> > >>>>>>>> > >> > >>>>> > > This
> > >>>>>>>> > >> > >>>>> > > could
> > >>>>>>>> > >> > >>>>> > > prove
> > >>>>>>>> > >> > >>>>> > > pretty daunting if one would have to
type in
> > >>>>>>>> > >> > >>>>> > > 700
> > >>>>>>>> entries
> > >>>>>>>> > >> > >>>>> > > each
> > >>>>>>>> > >> > >>>>> > > time.
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana
Burek
> > >>>>>>>> > >> > >>>>> > > via
> > >>>>>>>> RT
> > >>>>>>>> > >> > >>>>> > > <met_help at ucar.edu>
> > >>>>>>>> > >> > >>>>> > > wrote:
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>> > >> I found a few problems in your XML.
> > >>>>>>>> > >> > >>>>> > >> The database you are using doesn't have
data
> > >>>>>>>> > >> > >>>>> > >> with
> > >>>>>>>> > >> > >>>>> > >> vx_mask
> > >>>>>>>> > >> > >>>>> > >> 'abcREGabc'.
> > >>>>>>>> > >> > >>>>> > >> That is why METviewer returns an empty
> > >>>>>>>> > >> > >>>>> > >> dataset and
> > >>>>>>>> > >> > >>>>> > >> can't
> > >>>>>>>> > >> > >>>>> > >> create a
> > >>>>>>>> > >> > >>>>> > >> plot.
> > >>>>>>>> > >> > >>>>> > >> The other problem is how you define
> > >>>>>>>> > >> > >>>>> > >> fcst_lead.
> > >>>>>>>> This
> > >>>>>>>> > >> > >>>>> > >> database
> > >>>>>>>> > >> > >>>>> > >> has
> > >>>>>>>> > >> > >>>>> > >> values
> > >>>>>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000,
020000.
> > >>>>>>>> > >> > >>>>> > >> Since
> > >>>>>>>> you
> > >>>>>>>> > >> > >>>>> > >> include
> > >>>>>>>> > >> > >>>>> > >> all
> > >>>>>>>> > >> > >>>>> > >> fcst_leads
> > >>>>>>>> > >> > >>>>> > >> this variable can be omitted.
> > >>>>>>>> > >> > >>>>> > >> To minimize problems with variables
values,
> > >>>>>>>> > >> > >>>>> > >> I
> > >>>>>>>> would
> > >>>>>>>> > >> > >>>>> > >> suggest
> > >>>>>>>> > >> > >>>>> > >> to
> > >>>>>>>> > >> > >>>>> > >> create XML
> > >>>>>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
> > >>>>>>>> > >> > >>>>> > >> standards.
> > >>>>>>>> > >> > >>>>> > >> I created a plot using you
configurations
> > >>>>>>>> > >> > >>>>> > >> with GUI
> > >>>>>>>> > >> > >>>>> > >> (without
> > >>>>>>>> > >> > >>>>> vx_mack
> > >>>>>>>> > >> > >>>>> > >> and
> > >>>>>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626.
You
> > >>>>>>>> > >> > >>>>> > >> can
> > >>>>>>>> load it
> > >>>>>>>> > >> > >>>>> > >> back
> > >>>>>>>> > >> > >>>>> > >> to
> > >>>>>>>> > >> > >>>>> the
> > >>>>>>>> > >> > >>>>> > >> page by
> > >>>>>>>> > >> > >>>>> > >> using the "History" tab (left bar on
GUI).
> > >>>>>>>> > >> > >>>>> > >> Find
> > >>>>>>>> the
> > >>>>>>>> > >> > >>>>> > >> plot,
> > >>>>>>>> > >> > >>>>> > >> open XML
> > >>>>>>>> > >> > >>>>> > >> and
> > >>>>>>>> > >> > >>>>> > >> click on "Load to page". After that you
can
> > >>>>>>>> recreate a
> > >>>>>>>> > >> > >>>>> > >> plot
> > >>>>>>>> > >> > >>>>> > >> and
> > >>>>>>>> > >> > >>>>> edit
> > >>>>>>>> > >> > >>>>> > >> the
> > >>>>>>>> > >> > >>>>> > >> configurations. You can also download a
new
> > >>>>>>>> > >> > >>>>> > >> XML
> > >>>>>>>> and
> > >>>>>>>> > >> > >>>>> > >> edit it
> > >>>>>>>> > >> > >>>>> > >> for
> > >>>>>>>> > >> > >>>>> > >> batch.
> > >>>>>>>> > >> > >>>>> > >> I attached the example of load XML to
this
> > >>>>>>>> message.
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>> > >> Description of rds parameters:
> > >>>>>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
> > >>>>>>>> > >> > >>>>> > >> METviewer
> > >>>>>>>> R
> > >>>>>>>> > >> > >>>>> > >> scripts
> > >>>>>>>> > >> > >>>>> > >> rds_plots - temporary directory where
the
> > >>>>>>>> > >> > >>>>> > >> plots
> > >>>>>>>> will be
> > >>>>>>>> > >> > >>>>> > >> generated
> > >>>>>>>> > >> > >>>>> > >> rds_data - temporary directory where
the
> > >>>>>>>> > >> > >>>>> > >> data
> > >>>>>>>> retrieved
> > >>>>>>>> > >> > >>>>> > >> from
> > >>>>>>>> > >> > >>>>> > >> the
> > >>>>>>>> > >> > >>>>> > >> database
> > >>>>>>>> > >> > >>>>> > >> and other intermediate files will be
stored
> > >>>>>>>> > >> > >>>>> > >> rds_scripts - temporary directory where
R
> > >>>>>>>> > >> > >>>>> > >> scripts
> > >>>>>>>> for
> > >>>>>>>> > >> > >>>>> > >> this
> > >>>>>>>> > >> > >>>>> > >> run
> > >>>>>>>> > >> > >>>>> will
> > >>>>>>>> > >> > >>>>> > >> be
> > >>>>>>>> > >> > >>>>> > >> stored
> > >>>>>>>> > >> > >>>>> > >> After the execution is done all the
files
> > >>>>>>>> > >> > >>>>> > >> from
> > >>>>>>>> > >> > >>>>> > >> temporary
> > >>>>>>>> > >> > >>>>> directories
> > >>>>>>>> > >> > >>>>> > >> would be removed.
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>> > >> We don't have the offitial user guide
for
> > >>>>>>>> METviewer.
> > >>>>>>>> > >> > >>>>> > >> There
> > >>>>>>>> > >> > >>>>> > >> are
> > >>>>>>>> > >> > >>>>> some
> > >>>>>>>> > >> > >>>>> > >> instructions on how to crate XML config
> > >>>>>>>> > >> > >>>>> > >> files:
> > >>>>>>>> > >> > >>>>> > >>
https://metviewer.nws.noaa.gov/doc/index.html
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>> > >> Tatiana
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
> > >>>>>>>> edward.strobach at noaa.gov
> > >>>>>>>> > >> > >>>>> > >> wrote:
> > >>>>>>>> > >> > >>>>> > >> > Hi,
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am
having
> > >>>>>>>> > >> > >>>>> > >> > hard
> > >>>>>>>> time
> > >>>>>>>> > >> > >>>>> > >> > following
> > >>>>>>>> > >> > >>>>> how
> > >>>>>>>> > >> > >>>>> > >> > to
> > >>>>>>>> > >> > >>>>> > >> > go
> > >>>>>>>> > >> > >>>>> > >> > about the other steps since I have
zero
> > >>>>>>>> experience on
> > >>>>>>>> > >> > >>>>> > >> > how
> > >>>>>>>> > >> > >>>>> > >> > to do
> > >>>>>>>> > >> > >>>>> > >> > these
> > >>>>>>>> > >> > >>>>> > >> > things.
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> > #2. All parameters in <folders>
section
> > >>>>>>>> > >> > >>>>> > >> > should
> > >>>>>>>> NOT be
> > >>>>>>>> > >> > >>>>> > >> > changed
> > >>>>>>>> > >> > >>>>> and
> > >>>>>>>> > >> > >>>>> > >> > will
> > >>>>>>>> > >> > >>>>> > >> > be
> > >>>>>>>> > >> > >>>>> > >> > replaced by other scripts during the
batch
> > >>>>>>>> execution.
> > >>>>>>>> > >> > >>>>> > >> > The
> > >>>>>>>> > >> > >>>>> > >> > data
> > >>>>>>>> > >> > >>>>> for
> > >>>>>>>> > >> > >>>>> > >> > the
> > >>>>>>>> > >> > >>>>> > >> > batch mode is coming from the
database and
> > >>>>>>>> > >> > >>>>> > >> > not
> > >>>>>>>> from
> > >>>>>>>> > >> > >>>>> > >> > files.
> > >>>>>>>> > >> > >>>>> > >> > The
> > >>>>>>>> > >> > >>>>> > >> > name of
> > >>>>>>>> > >> > >>>>> > >> > the
> > >>>>>>>> > >> > >>>>> > >> > database is in <database> tag.
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> > I understand that nothing should be
> > >>>>>>>> > >> > >>>>> > >> > changed in
> > >>>>>>>> the
> > >>>>>>>> > >> > >>>>> > >> > folders
> > >>>>>>>> > >> > >>>>> > >> > section,
> > >>>>>>>> > >> > >>>>> > >> > but
> > >>>>>>>> > >> > >>>>> > >> > there is no indication how and where
these
> > >>>>>>>> variables
> > >>>>>>>> > >> > >>>>> > >> > are
> > >>>>>>>> > >> > >>>>> > >> > being
> > >>>>>>>> > >> > >>>>> > >> > defined.
> > >>>>>>>> > >> > >>>>> > >> > Grepping these names of the files
> > >>>>>>>> > >> > >>>>> > >> > available
> > >>>>>>>> does not
> > >>>>>>>> > >> > >>>>> > >> > reveal
> > >>>>>>>> > >> > >>>>> formal
> > >>>>>>>> > >> > >>>>> > >> > definitions of these variables. If I
> > >>>>>>>> > >> > >>>>> > >> > understood
> > >>>>>>>> > >> > >>>>> > >> > exactly
> > >>>>>>>> > >> > >>>>> > >> > how
> > >>>>>>>> > >> > >>>>> these
> > >>>>>>>> > >> > >>>>> > >> > were
> > >>>>>>>> > >> > >>>>> > >> > being defined, then I would be able
> > >>>>>>>> > >> > >>>>> > >> > connect this
> > >>>>>>>> > >> > >>>>> > >> > better.
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET
output
> > >>>>>>>> > >> > >>>>> > >> > files.
> > >>>>>>>> They
> > >>>>>>>> > >> > >>>>> > >> > use
> > >>>>>>>> > >> > >>>>> METviewer
> > >>>>>>>> > >> > >>>>> > >> > load
> > >>>>>>>> > >> > >>>>> > >> > mode to load these files to the
specific
> > >>>>>>>> database. In
> > >>>>>>>> > >> > >>>>> > >> > the
> > >>>>>>>> > >> > >>>>> > >> > config
> > >>>>>>>> > >> > >>>>> > >> > file
> > >>>>>>>> > >> > >>>>> > >> > for
> > >>>>>>>> > >> > >>>>> > >> > this process they also can specify
the
> > >>>>>>>> > >> > >>>>> > >> > database
> > >>>>>>>> > >> > >>>>> > >> > group. For
> > >>>>>>>> > >> > >>>>> example
> > >>>>>>>> > >> > >>>>> > >> > database
> > >>>>>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
> > >>>>>>>> > >> > >>>>> > >> > After the data has been loaded to the
> > >>>>>>>> > >> > >>>>> > >> > database
> > >>>>>>>> it is
> > >>>>>>>> > >> > >>>>> > >> > visible and
> > >>>>>>>> > >> > >>>>> > >> > available
> > >>>>>>>> > >> > >>>>> > >> > in the GUI and also can be used by
the
> > >>>>>>>> > >> > >>>>> > >> > batch
> > >>>>>>>> mode.
> > >>>>>>>> > >> > >>>>> > >> > You can always add more data to the
> > >>>>>>>> > >> > >>>>> > >> > database
> > >>>>>>>> using
> > >>>>>>>> > >> > >>>>> > >> > the
> > >>>>>>>> > >> > >>>>> > >> > same load
> > >>>>>>>> > >> > >>>>> > >> > mode.
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> > Is there a guide for me to understand
the
> > >>>>>>>> process
> > >>>>>>>> > >> > >>>>> > >> > that you
> > >>>>>>>> > >> > >>>>> > >> > are
> > >>>>>>>> > >> > >>>>> > >> > referring to?
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM
Tatiana
> > >>>>>>>> > >> > >>>>> > >> > Burek
> > >>>>>>>> via RT
> > >>>>>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
> > >>>>>>>> > >> > >>>>> > >> > wrote:
> > >>>>>>>> > >> > >>>>> > >> >
> > >>>>>>>> > >> > >>>>> > >> > > Hi Edward,
> > >>>>>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA
> > >>>>>>>> > >> > >>>>> > >> > > servers and
> > >>>>>>>> > >> > >>>>> > >> > > can't
> > >>>>>>>> > >> > >>>>> > >> > > view your
> > >>>>>>>> > >> > >>>>> > >> > > XML
> > >>>>>>>> > >> > >>>>> > >> > > for the
> > >>>>>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to
this
> > >>>>>>>> > >> > >>>>> > >> > > issue
> > >>>>>>>> and I
> > >>>>>>>> > >> > >>>>> > >> > > will
> > >>>>>>>> > >> > >>>>> > >> > > take a
> > >>>>>>>> > >> > >>>>> > >> > > look.
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >> > > #2. All parameters in <folders>
section
> > >>>>>>>> should NOT
> > >>>>>>>> > >> > >>>>> > >> > > be
> > >>>>>>>> > >> > >>>>> > >> > > changed
> > >>>>>>>> > >> > >>>>> > >> > > and
> > >>>>>>>> > >> > >>>>> > >> > > will be
> > >>>>>>>> > >> > >>>>> > >> > > replaced by other scripts during
the
> > >>>>>>>> > >> > >>>>> > >> > > batch
> > >>>>>>>> > >> > >>>>> > >> > > execution.
> > >>>>>>>> > >> > >>>>> > >> > > The data
> > >>>>>>>> > >> > >>>>> > >> > > for
> > >>>>>>>> > >> > >>>>> > >> > > the
> > >>>>>>>> > >> > >>>>> > >> > > batch mode is coming from the
database
> > >>>>>>>> > >> > >>>>> > >> > > and
> > >>>>>>>> not from
> > >>>>>>>> > >> > >>>>> > >> > > files. The
> > >>>>>>>> > >> > >>>>> > >> > > name
> > >>>>>>>> > >> > >>>>> > >> > > of the
> > >>>>>>>> > >> > >>>>> > >> > > database is in <database> tag.
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET
output
> > >>>>>>>> files.
> > >>>>>>>> > >> > >>>>> > >> > > They use
> > >>>>>>>> > >> > >>>>> > >> > > METviewer
> > >>>>>>>> > >> > >>>>> > >> > > load
> > >>>>>>>> > >> > >>>>> > >> > > mode to load these files to the
specific
> > >>>>>>>> database.
> > >>>>>>>> > >> > >>>>> > >> > > In
> > >>>>>>>> > >> > >>>>> > >> > > the
> > >>>>>>>> > >> > >>>>> config
> > >>>>>>>> > >> > >>>>> > >> > > file
> > >>>>>>>> > >> > >>>>> > >> > > for
> > >>>>>>>> > >> > >>>>> > >> > > this process they also can specify
the
> > >>>>>>>> database
> > >>>>>>>> > >> > >>>>> > >> > > group.
> > >>>>>>>> > >> > >>>>> > >> > > For
> > >>>>>>>> > >> > >>>>> > >> > > example
> > >>>>>>>> > >> > >>>>> > >> > > database
> > >>>>>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
> > >>>>>>>> > >> > >>>>> > >> > > After the data has been loaded to
the
> > >>>>>>>> database it
> > >>>>>>>> > >> > >>>>> > >> > > is
> > >>>>>>>> > >> > >>>>> > >> > > visible
> > >>>>>>>> > >> > >>>>> and
> > >>>>>>>> > >> > >>>>> > >> > > available
> > >>>>>>>> > >> > >>>>> > >> > > in the GUI and also can be used by
the
> > >>>>>>>> > >> > >>>>> > >> > > batch
> > >>>>>>>> mode.
> > >>>>>>>> > >> > >>>>> > >> > > You can always add more data to the
> > >>>>>>>> > >> > >>>>> > >> > > database
> > >>>>>>>> using
> > >>>>>>>> > >> > >>>>> > >> > > the
> > >>>>>>>> > >> > >>>>> > >> > > same
> > >>>>>>>> > >> > >>>>> load
> > >>>>>>>> > >> > >>>>> > >> > > mode.
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >> > > Tatiana
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> > >>>>>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
> > >>>>>>>> > >> > >>>>> > >> > > wrote:
> > >>>>>>>> > >> > >>>>> > >> > > > Good morning,
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made
an
> > >>>>>>>> > >> > >>>>> > >> > > > effort
> > >>>>>>>> to
> > >>>>>>>> > >> > >>>>> > >> > > > understand
> > >>>>>>>> > >> > >>>>> > >> > > > metviewer,
> > >>>>>>>> > >> > >>>>> > >> > > > test
> > >>>>>>>> > >> > >>>>> > >> > > > different cases using the history
tab,
> > >>>>>>>> changing
> > >>>>>>>> > >> > >>>>> > >> > > > information
> > >>>>>>>> > >> > >>>>> to
> > >>>>>>>> > >> > >>>>> > >> > > > see
> > >>>>>>>> > >> > >>>>> > >> > > > what
> > >>>>>>>> > >> > >>>>> > >> > > > works and what does not, and make
> > >>>>>>>> connections
> > >>>>>>>> > >> > >>>>> > >> > > > between
> > >>>>>>>> > >> > >>>>> > >> > > > the
> > >>>>>>>> > >> > >>>>> GUI
> > >>>>>>>> > >> > >>>>> > >> > > > settings
> > >>>>>>>> > >> > >>>>> > >> > > > and
> > >>>>>>>> > >> > >>>>> > >> > > > the XML file. Although my
comfort
> > >>>>>>>> > >> > >>>>> > >> > > > level
> > >>>>>>>> with
> > >>>>>>>> > >> > >>>>> > >> > > > metviewer has
> > >>>>>>>> > >> > >>>>> > >> > > > significantly
> > >>>>>>>> > >> > >>>>> > >> > > > increased during this time, there
are
> > >>>>>>>> > >> > >>>>> > >> > > > some
> > >>>>>>>> > >> > >>>>> > >> > > > remaining
> > >>>>>>>> > >> > >>>>> > >> > > > issues
> > >>>>>>>> > >> > >>>>> > >> > > > that
> > >>>>>>>> > >> > >>>>> > >> > > > I'm
> > >>>>>>>> > >> > >>>>> > >> > > > hoping
> > >>>>>>>> > >> > >>>>> > >> > > > to clear up.
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > 1. Problem changing XML file and
> > >>>>>>>> > >> > >>>>> > >> > > > running
> > >>>>>>>> with
> > >>>>>>>> > >> > >>>>> > >> > > > different
> > >>>>>>>> > >> > >>>>> > >> > > > statistic
> > >>>>>>>> > >> > >>>>> > >> > > > and
> > >>>>>>>> > >> > >>>>> > >> > > > plot
> > >>>>>>>> > >> > >>>>> > >> > > > type:
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at
EMC,
> > >>>>>>>> > >> > >>>>> > >> > > > provided
> > >>>>>>>> me
> > >>>>>>>> > >> > >>>>> > >> > > > the
> > >>>>>>>> > >> > >>>>> > >> > > > working
> > >>>>>>>> > >> > >>>>> > >> > > > directory
> > >>>>>>>> > >> > >>>>> > >> > > > to get
> > >>>>>>>> > >> > >>>>> > >> > > > started which I copied over:
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > He has a test.sh script that I
> > >>>>>>>> > >> > >>>>> > >> > > > adjusted and
> > >>>>>>>> used
> > >>>>>>>> > >> > >>>>> > >> > > > to
> > >>>>>>>> > >> > >>>>> > >> > > > run his
> > >>>>>>>> > >> > >>>>> > >> > > > example.
> > >>>>>>>> > >> > >>>>> > >> > > > running of the script "bash
test.sh"
> > >>>>>>>> produced a
> > >>>>>>>> > >> > >>>>> > >> > > > CSI
> > >>>>>>>> > >> > >>>>> > >> > > > plot
> > >>>>>>>> > >> > >>>>> that
> > >>>>>>>> > >> > >>>>> > >> > > > was
> > >>>>>>>> > >> > >>>>> > >> > > > stored
> > >>>>>>>> > >> > >>>>> > >> > > > here:
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >>
> > >>>>>>>>
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > Running the script produced the
> > >>>>>>>> > >> > >>>>> > >> > > > expected
> > >>>>>>>> result.
> > >>>>>>>> > >> > >>>>> > >> > > > The
> > >>>>>>>> > >> > >>>>> > >> > > > next
> > >>>>>>>> > >> > >>>>> > >> > > > natural
> > >>>>>>>> > >> > >>>>> > >> > > > step was
> > >>>>>>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and
create
> > >>>>>>>> > >> > >>>>> > >> > > > a
> > >>>>>>>> new xml
> > >>>>>>>> > >> > >>>>> > >> > > > file
> > >>>>>>>> > >> > >>>>> > >> > > > called
> > >>>>>>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to
> > >>>>>>>> > >> > >>>>> > >> > > > create a
> > >>>>>>>> time
> > >>>>>>>> > >> > >>>>> > >> > > > series
> > >>>>>>>> > >> > >>>>> > >> > > > where
> > >>>>>>>> > >> > >>>>> > >> > > > the
> > >>>>>>>> > >> > >>>>> > >> > > > independent time is
fcst_valid_beg
> > >>>>>>>> > >> > >>>>> > >> > > > instead
> > >>>>>>>> of
> > >>>>>>>> > >> > >>>>> > >> > > > obs_thresh.
> > >>>>>>>> > >> > >>>>> > >> > > > Accordingly,
> > >>>>>>>> > >> > >>>>> > >> > > > I
> > >>>>>>>> > >> > >>>>> > >> > > > made changes to the fixed fields
> > >>>>>>>> > >> > >>>>> > >> > > > (removing
> > >>>>>>>> > >> > >>>>> > >> > > > fcst_valid_beg)
> > >>>>>>>> > >> > >>>>> and
> > >>>>>>>> > >> > >>>>> > >> > > > changed
> > >>>>>>>> > >> > >>>>> > >> > > > the
> > >>>>>>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR
was
> > >>>>>>>> > >> > >>>>> > >> > > > also
> > >>>>>>>> > >> > >>>>> > >> > > > tried).
> > >>>>>>>> > >> > >>>>> > >> > > > The
> > >>>>>>>> > >> > >>>>> error
> > >>>>>>>> > >> > >>>>> > >> > > > that
> > >>>>>>>> > >> > >>>>> > >> > > > resulted
> > >>>>>>>> > >> > >>>>> > >> > > > is as follows:
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > AND BINARY ld.fcst_valid_beg IN
> > >>>>>>>> ('2020-05-01
> > >>>>>>>> > >> > >>>>> > >> > > > 13:00:00',
> > >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>>>>>> > >> > >>>>> > >> > > > 01
> > >>>>>>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00',
> > >>>>>>>> '2020-05-01
> > >>>>>>>> > >> > >>>>> > >> > > > 16:00:00',
> > >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>>>>>> > >> > >>>>> > >> > > > 01
> > >>>>>>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00',
> > >>>>>>>> '2020-05-01
> > >>>>>>>> > >> > >>>>> > >> > > > 19:00:00',
> > >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>>>>>> > >> > >>>>> > >> > > > 01
> > >>>>>>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00',
> > >>>>>>>> '2020-05-01
> > >>>>>>>> > >> > >>>>> > >> > > > 22:00:00',
> > >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>>>>>> > >> > >>>>> > >> > > > 01
> > >>>>>>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00',
> > >>>>>>>> '2020-05-02
> > >>>>>>>> > >> > >>>>> > >> > > > 01:00:00',
> > >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>>>>>> > >> > >>>>> > >> > > > 02
> > >>>>>>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00',
> > >>>>>>>> '2020-05-02
> > >>>>>>>> > >> > >>>>> > >> > > > 04:00:00',
> > >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>>>>>> > >> > >>>>> > >> > > > 02
> > >>>>>>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00',
> > >>>>>>>> '2020-05-02
> > >>>>>>>> > >> > >>>>> > >> > > > 07:00:00',
> > >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>>>>>> > >> > >>>>> > >> > > > 02
> > >>>>>>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00',
> > >>>>>>>> '2020-05-02
> > >>>>>>>> > >> > >>>>> > >> > > > 10:00:00',
> > >>>>>>>> > >> > >>>>> > >> > > > '2020-05-
> > >>>>>>>> > >> > >>>>> > >> > > > 02
> > >>>>>>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > >>>>>>>> > >> > >>>>> > >> > > > AND BINARY h.fcst_var = 'AOTK'
> > >>>>>>>> > >> > >>>>> > >> > > > AND ld.stat_header_id =
> > >>>>>>>> > >> > >>>>> > >> > > > h.stat_header_id
> > >>>>>>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
> > >>>>>>>> > >> > >>>>> > >> > > > No data was returned from
database
> > >>>>>>>> > >> > >>>>> > >> > > > 2020-06-09
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >>
> > >>>>>>>>
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > >>>>>>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable -
**
> > >>>>>>>> ERROR:
> > >>>>>>>> > >> > >>>>> > >> > > > Caught
> > >>>>>>>> > >> > >>>>> > >> > > > class
> > >>>>>>>> > >> > >>>>> > >> > > >
> edu.ucar.metviewer.EmptyResultSetException
> > >>>>>>>> in
> > >>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
> > >>>>>>>> > >> > >>>>> > >> > > > res): result set contained no
data
> > >>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> > >>>>>>>> > >> > >>>>> > >> > > >
> edu.ucar.metviewer.EmptyResultSetException
> > >>>>>>>> > >> > >>>>> > >> > > > in
> > >>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
res):
> > >>>>>>>> > >> > >>>>> > >> > > > result
> > >>>>>>>> set
> > >>>>>>>> > >> > >>>>> > >> > > > contained no
> > >>>>>>>> > >> > >>>>> > >> > > > data
> > >>>>>>>> > >> > >>>>> > >> > > > ** ERROR: Caught class
> > >>>>>>>> > >> > >>>>> > >> > > >
> edu.ucar.metviewer.EmptyResultSetException
> > >>>>>>>> > >> > >>>>> > >> > > > in
> > >>>>>>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
res):
> > >>>>>>>> > >> > >>>>> > >> > > > result
> > >>>>>>>> set
> > >>>>>>>> > >> > >>>>> > >> > > > contained no
> > >>>>>>>> > >> > >>>>> > >> > > > data
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > Apparently there was no data
during
> > >>>>>>>> > >> > >>>>> > >> > > > this
> > >>>>>>>> time. I
> > >>>>>>>> > >> > >>>>> > >> > > > tried
> > >>>>>>>> > >> > >>>>> > >> > > > changing
> > >>>>>>>> > >> > >>>>> > >> > > > the
> > >>>>>>>> > >> > >>>>> > >> > > > range
> > >>>>>>>> > >> > >>>>> > >> > > > of valid times and changed the
valid
> > >>>>>>>> > >> > >>>>> > >> > > > time to
> > >>>>>>>> > >> > >>>>> > >> > > > different
> > >>>>>>>> > >> > >>>>> > >> > > > days
> > >>>>>>>> > >> > >>>>> > >> > > > just in
> > >>>>>>>> > >> > >>>>> > >> > > > case
> > >>>>>>>> > >> > >>>>> > >> > > > data was missing. The issue was
the
> > >>>>>>>> > >> > >>>>> > >> > > > same.
> > >>>>>>>> To
> > >>>>>>>> > >> > >>>>> > >> > > > confirm
> > >>>>>>>> > >> > >>>>> > >> > > > my
> > >>>>>>>> > >> > >>>>> > >> > > > logic, I
> > >>>>>>>> > >> > >>>>> > >> > > > reproduced my attempted approach
on
> > >>>>>>>> metviewer,
> > >>>>>>>> > >> > >>>>> > >> > > > which I
> > >>>>>>>> > >> > >>>>> > >> > > > found
> > >>>>>>>> > >> > >>>>> > >> > > > to
> > >>>>>>>> > >> > >>>>> > >> > > > work.
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > 2. Running in batch mode
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > I think I lack an understanding
of how
> > >>>>>>>> > >> > >>>>> > >> > > > to
> > >>>>>>>> set up
> > >>>>>>>> > >> > >>>>> > >> > > > batch
> > >>>>>>>> > >> > >>>>> > >> > > > mode
> > >>>>>>>> > >> > >>>>> > >> > > > because
> > >>>>>>>> > >> > >>>>> > >> > > > I
> > >>>>>>>> > >> > >>>>> > >> > > > don't
> > >>>>>>>> > >> > >>>>> > >> > > > see where these definitions are
being
> > >>>>>>>> explicitly
> > >>>>>>>> > >> > >>>>> > >> > > > defined:
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > <plot_spec>
> > >>>>>>>> > >> > >>>>> > >> > > > <connection>
> > >>>>>>>> > >> > >>>>> > >> > > >
<host>rds_host:3306</host>
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > >>>>>>>> > >> > >>>>> > >> > > > <user>rds_user</user>
> > >>>>>>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
> > >>>>>>>> > >> > >>>>> > >> > > > </connection>
> > >>>>>>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
> > >>>>>>>> > >> > >>>>> > >> > > > <folders>
> > >>>>>>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > >>>>>>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > >>>>>>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
> > >>>>>>>> > >> > >>>>> > >> > > > <data>rds_data</data>
> > >>>>>>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > >>>>>>>> > >> > >>>>> > >> > > > </folders>
> > >>>>>>>> > >> > >>>>> > >> > > > <plot>
> > >>>>>>>> > >> > >>>>> > >> > > >
<template>series_plot.R_tmpl</template>
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > I see the <folders> section, but
> > >>>>>>>> > >> > >>>>> > >> > > > grepping
> > >>>>>>>> all
> > >>>>>>>> > >> > >>>>> > >> > > > available
> > >>>>>>>> > >> > >>>>> files
> > >>>>>>>> > >> > >>>>> > >> > > > does
> > >>>>>>>> > >> > >>>>> > >> > > > not
> > >>>>>>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being
> > >>>>>>>> > >> > >>>>> > >> > > > defined, as
> > >>>>>>>> well
> > >>>>>>>> > >> > >>>>> > >> > > > as
> > >>>>>>>> > >> > >>>>> > >> > > > other
> > >>>>>>>> > >> > >>>>> > >> > > > similarly
> > >>>>>>>> > >> > >>>>> > >> > > > defined
> > >>>>>>>> > >> > >>>>> > >> > > > variables. I think if I know how
> > >>>>>>>> > >> > >>>>> > >> > > > these are
> > >>>>>>>> being
> > >>>>>>>> > >> > >>>>> > >> > > > defined,
> > >>>>>>>> > >> > >>>>> > >> > > > then
> > >>>>>>>> > >> > >>>>> > >> > > > I'll
> > >>>>>>>> > >> > >>>>> > >> > > > be
> > >>>>>>>> > >> > >>>>> > >> > > > able to pipe in my information
and
> > >>>>>>>> > >> > >>>>> > >> > > > process
> > >>>>>>>> the
> > >>>>>>>> > >> > >>>>> > >> > > > data
> > >>>>>>>> > >> > >>>>> > >> > > > using
> > >>>>>>>> > >> > >>>>> met
> > >>>>>>>> > >> > >>>>> > >> > > > and
> > >>>>>>>> > >> > >>>>> > >> > > > produce
> > >>>>>>>> > >> > >>>>> > >> > > > results in batch mode. Am I
right on
> > >>>>>>>> > >> > >>>>> > >> > > > that?
> > >>>>>>>> > >> > >>>>> > >> > > > Also, if
> > >>>>>>>> > >> > >>>>> > >> > > > I want
> > >>>>>>>> > >> > >>>>> > >> > > > to use
> > >>>>>>>> > >> > >>>>> > >> > > > obs for
> > >>>>>>>> > >> > >>>>> > >> > > > verification, do I add another
line in
> > >>>>>>>> > >> > >>>>> > >> > > > the
> > >>>>>>>> XML
> > >>>>>>>> > >> > >>>>> > >> > > > file
> > >>>>>>>> > >> > >>>>> > >> > > > that
> > >>>>>>>> > >> > >>>>> > >> > > > designates
> > >>>>>>>> > >> > >>>>> > >> > > > file
> > >>>>>>>> > >> > >>>>> > >> > > > location of obs? I assume that
other
> > >>>>>>>> > >> > >>>>> > >> > > > steps
> > >>>>>>>> have
> > >>>>>>>> > >> > >>>>> > >> > > > to be
> > >>>>>>>> > >> > >>>>> > >> > > > run
> > >>>>>>>> > >> > >>>>> > >> > > > before
> > >>>>>>>> > >> > >>>>> > >> > > > all
> > >>>>>>>> > >> > >>>>> > >> > > > this. If data is in prepbufr
form,
> > >>>>>>>> > >> > >>>>> > >> > > > for
> > >>>>>>>> example,
> > >>>>>>>> > >> > >>>>> > >> > > > I
> > >>>>>>>> > >> > >>>>> > >> > > > would
> > >>>>>>>> > >> > >>>>> have
> > >>>>>>>> > >> > >>>>> > >> > > > to
> > >>>>>>>> > >> > >>>>> > >> > > > separately
> > >>>>>>>> > >> > >>>>> > >> > > > run PB2NC. If I want grid2obs,
then I
> > >>>>>>>> would want
> > >>>>>>>> > >> > >>>>> > >> > > > to
> > >>>>>>>> > >> > >>>>> > >> > > > do that
> > >>>>>>>> > >> > >>>>> > >> > > > as
> > >>>>>>>> > >> > >>>>> > >> > > > well.
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > 3. Communicating results to
metviewer
> > >>>>>>>> > >> > >>>>> > >> > > > online
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > Lastly, I see a lot of
people/projects
> > >>>>>>>> listed
> > >>>>>>>> > >> > >>>>> > >> > > > here -
> > >>>>>>>> > >> > >>>>> > >> > > > see
> > >>>>>>>> > >> > >>>>> > >> > > > attachment
> > >>>>>>>> > >> > >>>>> > >> > > >
> > >>>>>>>> > >> > >>>>> > >> > > > How do I get the results I
generate to
> > >>>>>>>> > >> > >>>>> > >> > > > be
> > >>>>>>>> > >> > >>>>> > >> > > > uploaded to
> > >>>>>>>> > >> > >>>>> > >> > > > this
> > >>>>>>>> > >> > >>>>> > >> > > > location?
> > >>>>>>>> > >> > >>>>> > >> > > > It's
> > >>>>>>>> > >> > >>>>> > >> > > > not clear how others are setting
this
> > >>>>>>>> > >> > >>>>> > >> > > > up.
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >> > >
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>> > >>
> > >>>>>>>> > >> > >>>>> > >> rds_host:3306
> > >>>>>>>> > >> > >>>>> > >> mv_cmaqv5para8_aod_grid2grid_metplus
> > >>>>>>>> > >> > >>>>> > >> rds_user
> > >>>>>>>> > >> > >>>>> rds_pwd
> > >>>>>>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work
> > >>>>>>>> > >> > >>>>> > >> rds_plots
> > >>>>>>>> rds_data
> > >>>>>>>> > >> > >>>>> rds_scripts
> > >>>>>>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR
CMAQAODPARA8HIGH
> > >>>>>>>> > >> > >>>>> > >> CMAQAODPRODHIGH
> > >>>>>>>> > >> > >>>>> 2020-05-12
> > >>>>>>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12
> > >>>>>>>> > >> > >>>>> > >> 19:00:00
> > >>>>>>>> 2020-
> > >>>>>>>> > >> > >>>>> > >> 05-12
> > >>>>>>>> > >> > >>>>> 20:00:00
> > >>>>>>>> > >> > >>>>> > >> 2020-05-12
> > >>>>>>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12
> > >>>>>>>> > >> > >>>>> > >> 23:00:00
> > >>>>>>>> 2020-
> > >>>>>>>> > >> > >>>>> > >> 05-13
> > >>>>>>>> > >> > >>>>> 10:00:00
> > >>>>>>>> > >> > >>>>> > >> 2020-05-13
> > >>>>>>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13
> > >>>>>>>> > >> > >>>>> > >> 13:00:00
> > >>>>>>>> 2020-
> > >>>>>>>> > >> > >>>>> > >> 05-13
> > >>>>>>>> > >> > >>>>> 14:00:00
> > >>>>>>>> > >> > >>>>> > >> 2020-05-13
> > >>>>>>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13
> > >>>>>>>> > >> > >>>>> > >> 17:00:00
> > >>>>>>>> 2020-
> > >>>>>>>> > >> > >>>>> > >> 05-13
> > >>>>>>>> > >> > >>>>> 18:00:00
> > >>>>>>>> > >> > >>>>> > >> 2020-05-13
> > >>>>>>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13
> > >>>>>>>> > >> > >>>>> > >> 21:00:00
> > >>>>>>>> 2020-
> > >>>>>>>> > >> > >>>>> > >> 05-13
> > >>>>>>>> > >> > >>>>> 22:00:00
> > >>>>>>>> > >> > >>>>> > >> 2020-05-13
> > >>>>>>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14
> > >>>>>>>> > >> > >>>>> > >> 11:00:00
> > >>>>>>>> 2020-
> > >>>>>>>> > >> > >>>>> > >> 05-14
> > >>>>>>>> > >> > >>>>> 12:00:00
> > >>>>>>>> > >> > >>>>> > >> true
> > >>>>>>>> > >> > >>>>> > >> median plot_20200612_173626.data
> > >>>>>>>> > >> > >>>>> > >> plot_20200612_173626.png
> > >>>>>>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label
test
> > >>>>>>>> > >> > >>>>> > >> y_label
> > >>>>>>>> > >> > >>>>> > >> false list() list() false false false
false
> > >>>>>>>> > >> > >>>>> > >> false
> > >>>>>>>> false
> > >>>>>>>> > >> > >>>>> > >> true
> > >>>>>>>> > >> > >>>>> > >> false
> > >>>>>>>> > >> > >>>>> > >> false
> > >>>>>>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72
in
> > >>>>>>>> c(8,4,5,4)
> > >>>>>>>> > >> > >>>>> > >> c(1,1,0) 1 2
> > >>>>>>>> > >> > >>>>> > >> 1.4 -2
> > >>>>>>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5
1 1
> > >>>>>>>> > >> > >>>>> > >> 1 -2
> > >>>>>>>> 0.5 3
> > >>>>>>>> > >> > >>>>> > >> #cccccc 1
> > >>>>>>>> > >> > >>>>> > >> listX 1
> > >>>>>>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1
0.5 o
> > >>>>>>>> > >> > >>>>> > >> c(0,
> > >>>>>>>> -.25)
> > >>>>>>>> > >> > >>>>> > >> 3
> > >>>>>>>> > >> > >>>>> > >> 0.8 1
> > >>>>>>>> > >> > >>>>> > >> #333333
> > >>>>>>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none")
c(FALSE,FALSE)
> > >>>>>>>> > >> > >>>>> > >> c(TRUE,TRUE)
> > >>>>>>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20)
> > >>>>>>>> > >> > >>>>> > >> c("b","b")
> > >>>>>>>> c(1,1)
> > >>>>>>>> > >> > >>>>> > >> c(1,1)
> > >>>>>>>> > >> > >>>>> c(1,1)
> > >>>>>>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>> > > --
> > >>>>>>>> > >> > >>>>> > > Edward Strobach
> > >>>>>>>> > >> > >>>>> > > EMC/NCEP/NWS/
> > >>>>>>>> > >> > >>>>> > > IMSG Contractor
> > >>>>>>>> > >> > >>>>> > > Cubicle#: 2029
> > >>>>>>>> > >> > >>>>> > > 301-683-3717
> > >>>>>>>> > >> > >>>>> > >
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>>
> > >>>>>>>> > >> > >>>>
> > >>>>>>>> > >> > >>>> --
> > >>>>>>>> > >> > >>>> Edward Strobach
> > >>>>>>>> > >> > >>>> EMC/NCEP/NWS/
> > >>>>>>>> > >> > >>>> IMSG Contractor
> > >>>>>>>> > >> > >>>> Cubicle#: 2029
> > >>>>>>>> > >> > >>>> 301-683-3717
> > >>>>>>>> > >> > >>>>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>> --
> > >>>>>>>> > >> > >>> Edward Strobach
> > >>>>>>>> > >> > >>> EMC/NCEP/NWS/
> > >>>>>>>> > >> > >>> IMSG Contractor
> > >>>>>>>> > >> > >>> Cubicle#: 2029
> > >>>>>>>> > >> > >>> 301-683-3717
> > >>>>>>>> > >> > >>>
> > >>>>>>>> > >> > >>
> > >>>>>>>> > >> > >>
> > >>>>>>>> > >> > >> --
> > >>>>>>>> > >> > >> Edward Strobach
> > >>>>>>>> > >> > >> EMC/NCEP/NWS/
> > >>>>>>>> > >> > >> IMSG Contractor
> > >>>>>>>> > >> > >> Cubicle#: 2029
> > >>>>>>>> > >> > >> 301-683-3717
> > >>>>>>>> > >> > >>
> > >>>>>>>> > >> > >
> > >>>>>>>> > >> > >
> > >>>>>>>> > >> > > --
> > >>>>>>>> > >> > > Edward Strobach
> > >>>>>>>> > >> > > EMC/NCEP/NWS/
> > >>>>>>>> > >> > > IMSG Contractor
> > >>>>>>>> > >> > > Cubicle#: 2029
> > >>>>>>>> > >> > > 301-683-3717
> > >>>>>>>> > >> > >
> > >>>>>>>> > >>
> > >>>>>>>> > >>
> > >>>>>>>> > >>
> > >>>>>>>> > >>
> > >>>>>>>> > >
> > >>>>>>>> > > --
> > >>>>>>>> > > Edward Strobach
> > >>>>>>>> > > EMC/NCEP/NWS/
> > >>>>>>>> > > IMSG Contractor
> > >>>>>>>> > > Cubicle#: 2029
> > >>>>>>>> > > 301-683-3717
> > >>>>>>>> > >
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Edward Strobach
> > >>>>>>> EMC/NCEP/NWS/
> > >>>>>>> IMSG Contractor
> > >>>>>>> Cubicle#: 2029
> > >>>>>>> 301-683-3717
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Edward Strobach
> > >>>>>> EMC/NCEP/NWS/
> > >>>>>> IMSG Contractor
> > >>>>>> Cubicle#: 2029
> > >>>>>> 301-683-3717
> > >>>>>>
> > >>>>> --
> > >>>>> Jeffery T. McQueen
> > >>>>> Meteorologist
> > >>>>> National Center for Weather and Climate Prediction
> > >>>>> Room 2095
> > >>>>> 5830 University Research Court
> > >>>>> College Park, MD 20741
> > >>>>> 301-683-3736
> > >>>>> Fax: 301-683-3703
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Edward Strobach
> > >>>> EMC/NCEP/NWS/
> > >>>> IMSG Contractor
> > >>>> Cubicle#: 2029
> > >>>> 301-683-3717
> > >>>>
> > >>>
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
More information about the Met_help
mailing list