[Met_help] [rt.rap.ucar.edu #95596] History for setting up batch mode; communicating results with metviewer online; other remaining issues

Tatiana Burek via RT met_help at ucar.edu
Tue Oct 13 12:13:19 MDT 2020


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Good morning,

Over the last two weeks I've made an effort to understand metviewer, test
different cases using the history tab, changing information to see what
works and what does not, and make connections between the GUI settings and
the XML file.  Although my comfort level with metviewer has significantly
increased during this time, there are some remaining issues that I'm hoping
to clear up.

1.  Problem changing XML file and running with different statistic and plot
type:

Ho-Chun Huang, a colleague at EMC, provided me the working directory to get
started which I copied over:
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer

He has a test.sh script that I adjusted and used to run his example.
running of the script "bash test.sh" produced a CSI plot that was stored
here:
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png

Running the script produced the expected result.  The next natural step was
to copy hochum_xml.template and create a new xml file called
Strobach_xml.template in order to create a time series where the
independent time is fcst_valid_beg instead of obs_thresh. Accordingly, I
made changes to the fixed fields (removing fcst_valid_beg) and changed the
statistic from CSI to FMEAN (FBAR was also tried).  The error that resulted
is as follows:

  AND   BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00', '2020-05-01
14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00', '2020-05-01
17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00', '2020-05-01
20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00', '2020-05-01
23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00', '2020-05-02
02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00', '2020-05-02
05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00', '2020-05-02
08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00', '2020-05-02
11:00:00', '2020-05-02 12:00:00')
  AND   BINARY h.fcst_var = 'AOTK'
  AND ld.stat_header_id = h.stat_header_id
  AND ld.fmean != -9999;
No data was returned from database
2020-06-09
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
Source) -1 printFormattedTable -   **  ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException in printFormattedTable(ResultSet
res): result set contained no data
  **  ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException in
printFormattedTable(ResultSet res): result set contained no data
  **  ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException in
printFormattedTable(ResultSet res): result set contained no data

Apparently there was no data during this time.  I tried changing the range
of valid times and changed the valid time to different days just in case
data was missing.  The issue was the same.  To confirm my logic, I
reproduced my attempted approach on metviewer, which I found to work.

2.  Running in batch mode

I think I lack an understanding of how to set up batch mode because I don't
see where these definitions are being explicitly defined:

<plot_spec>
    <connection>
        <host>rds_host:3306</host>
        <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
        <user>rds_user</user>
        <password>rds_pwd</password>
    </connection>
    <rscript>Rscript</rscript>
    <folders>
        <r_tmpl>rds_R_tmpl</r_tmpl>
        <r_work>rds_R_work</r_work>
        <plots>rds_plots</plots>
        <data>rds_data</data>
        <scripts>rds_scripts</scripts>
    </folders>
    <plot>
        <template>series_plot.R_tmpl</template>

I see the <folders> section, but grepping all available files does not
reveal how rds_R_work is being defined, as well as other similarly defined
variables.  I think if I know how these are being defined, then I'll be
able to pipe in my information and process the data using met and produce
results in batch mode.  Am I right on that?  Also, if I want to use obs for
verification, do I add another line in the XML file that designates file
location of obs?  I assume that other steps have to be run before all
this.  If data is in prepbufr form, for example, I would have to separately
run PB2NC.  If I want grid2obs, then I would want to do that as well.

3. Communicating results to metviewer online

Lastly, I see a lot of people/projects listed here - see attachment

How do I get the results I generate to be uploaded to this location?  It's
not clear how others are setting this up.



-- 
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Fri Jun 12 10:51:21 2020

Hi Edward,
#1. I don't have an access to NOAA servers and can't view your XML for
the batch mode. Please, attache it to this issue and I will take a
look.

#2. All parameters in <folders> section should NOT be changed and will
be replaced by other scripts during the batch execution. The data for
the batch mode is coming from the database and not from files. The
name of the database is in <database> tag.

#3. Users rum MET and produce MET output files. They use METviewer
load mode to load these files to the specific database. In the config
file for this process they also can specify the database group. For
example database mv_dev is in Test12 group.
After the data has been loaded to the database it is visible and
available in the GUI and also can be used by the batch mode.
You can always add more data to the database using the same load mode.

Tatiana

On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> Good morning,
>
> Over the last two weeks I've made an effort to understand metviewer,
> test
> different cases using the history tab, changing information to see
> what
> works and what does not, and make connections between the GUI
settings
> and
> the XML file.  Although my comfort level with metviewer has
> significantly
> increased during this time, there are some remaining issues that I'm
> hoping
> to clear up.
>
> 1.  Problem changing XML file and running with different statistic
and
> plot
> type:
>
> Ho-Chun Huang, a colleague at EMC, provided me the working directory
> to get
> started which I copied over:
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>
> He has a test.sh script that I adjusted and used to run his example.
> running of the script "bash test.sh" produced a CSI plot that was
> stored
> here:
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>
> Running the script produced the expected result.  The next natural
> step was
> to copy hochum_xml.template and create a new xml file called
> Strobach_xml.template in order to create a time series where the
> independent time is fcst_valid_beg instead of obs_thresh.
Accordingly,
> I
> made changes to the fixed fields (removing fcst_valid_beg) and
changed
> the
> statistic from CSI to FMEAN (FBAR was also tried).  The error that
> resulted
> is as follows:
>
> AND   BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00', '2020-05-
01
> 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00', '2020-05-01
> 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00', '2020-05-01
> 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00', '2020-05-01
> 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00', '2020-05-02
> 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00', '2020-05-02
> 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00', '2020-05-02
> 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00', '2020-05-02
> 11:00:00', '2020-05-02 12:00:00')
> AND   BINARY h.fcst_var = 'AOTK'
> AND ld.stat_header_id = h.stat_header_id
> AND ld.fmean != -9999;
> No data was returned from database
> 2020-06-09
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> Source) -1 printFormattedTable -   **  ERROR: Caught class
> edu.ucar.metviewer.EmptyResultSetException in
> printFormattedTable(ResultSet
> res): result set contained no data
> **  ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException
in
> printFormattedTable(ResultSet res): result set contained no data
> **  ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException
in
> printFormattedTable(ResultSet res): result set contained no data
>
> Apparently there was no data during this time.  I tried changing the
> range
> of valid times and changed the valid time to different days just in
> case
> data was missing.  The issue was the same.  To confirm my logic, I
> reproduced my attempted approach on metviewer, which I found to
work.
>
> 2.  Running in batch mode
>
> I think I lack an understanding of how to set up batch mode because
I
> don't
> see where these definitions are being explicitly defined:
>
> <plot_spec>
>     <connection>
>         <host>rds_host:3306</host>
>         <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>         <user>rds_user</user>
>         <password>rds_pwd</password>
>     </connection>
>     <rscript>Rscript</rscript>
>     <folders>
>         <r_tmpl>rds_R_tmpl</r_tmpl>
>         <r_work>rds_R_work</r_work>
>         <plots>rds_plots</plots>
>         <data>rds_data</data>
>         <scripts>rds_scripts</scripts>
>     </folders>
>     <plot>
>         <template>series_plot.R_tmpl</template>
>
> I see the <folders> section, but grepping all available files does
not
> reveal how rds_R_work is being defined, as well as other similarly
> defined
> variables.  I think if I know how these are being defined, then I'll
> be
> able to pipe in my information and process the data using met and
> produce
> results in batch mode.  Am I right on that?  Also, if I want to use
> obs for
> verification, do I add another line in the XML file that designates
> file
> location of obs?  I assume that other steps have to be run before
all
> this.  If data is in prepbufr form, for example, I would have to
> separately
> run PB2NC.  If I want grid2obs, then I would want to do that as
well.
>
> 3. Communicating results to metviewer online
>
> Lastly, I see a lot of people/projects listed here - see attachment
>
> How do I get the results I generate to be uploaded to this location?
> It's
> not clear how others are setting this up.



------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Jun 12 11:08:43 2020

Hi,

I'm attaching the xml below, but am having hard time following how to
go
about the other steps since I have zero experience on how to do these
things.

#2. All parameters in <folders> section should NOT be changed and will
be
replaced by other scripts during the batch execution. The data for the
batch mode is coming from the database and not from files. The name of
the
database is in <database> tag.

I understand that nothing should be changed in the folders section,
but
there is no indication how and where these variables are being
defined.
Grepping these names of the files available does not reveal formal
definitions of these variables.  If I understood exactly how these
were
being defined, then I would be able connect this better.

#3. Users rum MET and produce MET output files. They use METviewer
load
mode to load these files to the specific database. In the config file
for
this process they also can specify the database group. For example
database
mv_dev is in Test12 group.
After the data has been loaded to the database it is visible and
available
in the GUI and also can be used by the batch mode.
You can always add more data to the database using the same load mode.

Is there a guide for me to understand the process that you are
referring to?










On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Hi Edward,
> #1. I don't have an access to NOAA servers and can't view your XML
for the
> batch mode. Please, attache it to this issue and I will take a look.
>
> #2. All parameters in <folders> section should NOT be changed and
will be
> replaced by other scripts during the batch execution. The data for
the
> batch mode is coming from the database and not from files. The name
of the
> database is in <database> tag.
>
> #3. Users rum MET and produce MET output files. They use METviewer
load
> mode to load these files to the specific database. In the config
file for
> this process they also can specify the database group. For example
database
> mv_dev is in Test12 group.
> After the data has been loaded to the database it is visible and
available
> in the GUI and also can be used by the batch mode.
> You can always add more data to the database using the same load
mode.
>
> Tatiana
>
> On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> > Good morning,
> >
> > Over the last two weeks I've made an effort to understand
metviewer,
> > test
> > different cases using the history tab, changing information to see
> > what
> > works and what does not, and make connections between the GUI
settings
> > and
> > the XML file.  Although my comfort level with metviewer has
> > significantly
> > increased during this time, there are some remaining issues that
I'm
> > hoping
> > to clear up.
> >
> > 1.  Problem changing XML file and running with different statistic
and
> > plot
> > type:
> >
> > Ho-Chun Huang, a colleague at EMC, provided me the working
directory
> > to get
> > started which I copied over:
> > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >
> > He has a test.sh script that I adjusted and used to run his
example.
> > running of the script "bash test.sh" produced a CSI plot that was
> > stored
> > here:
> >
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >
> > Running the script produced the expected result.  The next natural
> > step was
> > to copy hochum_xml.template and create a new xml file called
> > Strobach_xml.template in order to create a time series where the
> > independent time is fcst_valid_beg instead of obs_thresh.
Accordingly,
> > I
> > made changes to the fixed fields (removing fcst_valid_beg) and
changed
> > the
> > statistic from CSI to FMEAN (FBAR was also tried).  The error that
> > resulted
> > is as follows:
> >
> > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00', '2020-
05-01
> > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00', '2020-05-
01
> > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00', '2020-05-
01
> > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00', '2020-05-
01
> > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00', '2020-05-
02
> > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00', '2020-05-
02
> > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00', '2020-05-
02
> > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00', '2020-05-
02
> > 11:00:00', '2020-05-02 12:00:00')
> > AND   BINARY h.fcst_var = 'AOTK'
> > AND ld.stat_header_id = h.stat_header_id
> > AND ld.fmean != -9999;
> > No data was returned from database
> > 2020-06-09
> >
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > Source) -1 printFormattedTable -   **  ERROR: Caught class
> > edu.ucar.metviewer.EmptyResultSetException in
> > printFormattedTable(ResultSet
> > res): result set contained no data
> > **  ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException
in
> > printFormattedTable(ResultSet res): result set contained no data
> > **  ERROR: Caught class edu.ucar.metviewer.EmptyResultSetException
in
> > printFormattedTable(ResultSet res): result set contained no data
> >
> > Apparently there was no data during this time.  I tried changing
the
> > range
> > of valid times and changed the valid time to different days just
in
> > case
> > data was missing.  The issue was the same.  To confirm my logic, I
> > reproduced my attempted approach on metviewer, which I found to
work.
> >
> > 2.  Running in batch mode
> >
> > I think I lack an understanding of how to set up batch mode
because I
> > don't
> > see where these definitions are being explicitly defined:
> >
> > <plot_spec>
> >     <connection>
> >         <host>rds_host:3306</host>
> >         <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >         <user>rds_user</user>
> >         <password>rds_pwd</password>
> >     </connection>
> >     <rscript>Rscript</rscript>
> >     <folders>
> >         <r_tmpl>rds_R_tmpl</r_tmpl>
> >         <r_work>rds_R_work</r_work>
> >         <plots>rds_plots</plots>
> >         <data>rds_data</data>
> >         <scripts>rds_scripts</scripts>
> >     </folders>
> >     <plot>
> >         <template>series_plot.R_tmpl</template>
> >
> > I see the <folders> section, but grepping all available files does
not
> > reveal how rds_R_work is being defined, as well as other similarly
> > defined
> > variables.  I think if I know how these are being defined, then
I'll
> > be
> > able to pipe in my information and process the data using met and
> > produce
> > results in batch mode.  Am I right on that?  Also, if I want to
use
> > obs for
> > verification, do I add another line in the XML file that
designates
> > file
> > location of obs?  I assume that other steps have to be run before
all
> > this.  If data is in prepbufr form, for example, I would have to
> > separately
> > run PB2NC.  If I want grid2obs, then I would want to do that as
well.
> >
> > 3. Communicating results to metviewer online
> >
> > Lastly, I see a lot of people/projects listed here - see
attachment
> >
> > How do I get the results I generate to be uploaded to this
location?
> > It's
> > not clear how others are setting this up.
>
>
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Fri Jun 12 12:22:08 2020

I found a few problems in your XML.
The database you are using doesn't have data with vx_mask 'abcREGabc'.
That is why METviewer returns an empty dataset and can't create a
plot.
The other problem is how you define fcst_lead. This database has
values like 10000, 20000 and not 010000, 020000. Since you include all
fcst_leads this variable can be omitted.
To minimize problems with variables values, I would suggest to create
XML using GUI and then edit it to fit batch standards.
I created a plot using you configurations with GUI (without vx_mack
and plot and axis titles) #20200612_173626. You can load it back to
the page by using the "History" tab (left bar on GUI). Find the plot,
open XML and click on "Load to page". After that you can recreate a
plot and edit the configurations. You can also download a new XML and
edit it for batch.
I attached the example of load XML to this message.

Description of rds parameters:
rds_R_tmpl and rds_R_work - location of METviewer R scripts
rds_plots - temporary directory where the plots will be generated
rds_data - temporary directory where the data retrieved from the
database and other intermediate files will be stored
rds_scripts - temporary directory where R scripts for this run will be
stored
After the execution is done all the files from temporary directories
would be removed.


We don't have the offitial user guide for METviewer. There are some
instructions on how to crate XML config files:
https://metviewer.nws.noaa.gov/doc/index.html

Tatiana

On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
> Hi,
>
> I'm attaching the xml below, but am having hard time following how
to
> go
> about the other steps since I have zero experience on how to do
these
> things.
>
> #2. All parameters in <folders> section should NOT be changed and
will
> be
> replaced by other scripts during the batch execution. The data for
the
> batch mode is coming from the database and not from files. The name
of
> the
> database is in <database> tag.
>
> I understand that nothing should be changed in the folders section,
> but
> there is no indication how and where these variables are being
> defined.
> Grepping these names of the files available does not reveal formal
> definitions of these variables.  If I understood exactly how these
> were
> being defined, then I would be able connect this better.
>
> #3. Users rum MET and produce MET output files. They use METviewer
> load
> mode to load these files to the specific database. In the config
file
> for
> this process they also can specify the database group. For example
> database
> mv_dev is in Test12 group.
> After the data has been loaded to the database it is visible and
> available
> in the GUI and also can be used by the batch mode.
> You can always add more data to the database using the same load
mode.
>
> Is there a guide for me to understand the process that you are
> referring to?
>
>
>
>
>
>
>
>
>
>
> On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> > #1. I don't have an access to NOAA servers and can't view your XML
> > for the
> > batch mode. Please, attache it to this issue and I will take a
look.
> >
> > #2. All parameters in <folders> section should NOT be changed and
> > will be
> > replaced by other scripts during the batch execution. The data for
> > the
> > batch mode is coming from the database and not from files. The
name
> > of the
> > database is in <database> tag.
> >
> > #3. Users rum MET and produce MET output files. They use METviewer
> > load
> > mode to load these files to the specific database. In the config
file
> > for
> > this process they also can specify the database group. For example
> > database
> > mv_dev is in Test12 group.
> > After the data has been loaded to the database it is visible and
> > available
> > in the GUI and also can be used by the batch mode.
> > You can always add more data to the database using the same load
> > mode.
> >
> > Tatiana
> >
> > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> > > Good morning,
> > >
> > > Over the last two weeks I've made an effort to understand
> > > metviewer,
> > > test
> > > different cases using the history tab, changing information to
see
> > > what
> > > works and what does not, and make connections between the GUI
> > > settings
> > > and
> > > the XML file.  Although my comfort level with metviewer has
> > > significantly
> > > increased during this time, there are some remaining issues that
> > > I'm
> > > hoping
> > > to clear up.
> > >
> > > 1.  Problem changing XML file and running with different
statistic
> > > and
> > > plot
> > > type:
> > >
> > > Ho-Chun Huang, a colleague at EMC, provided me the working
> > > directory
> > > to get
> > > started which I copied over:
> > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >
> > > He has a test.sh script that I adjusted and used to run his
> > > example.
> > > running of the script "bash test.sh" produced a CSI plot that
was
> > > stored
> > > here:
> > >
> >
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >
> > > Running the script produced the expected result.  The next
natural
> > > step was
> > > to copy hochum_xml.template and create a new xml file called
> > > Strobach_xml.template in order to create a time series where the
> > > independent time is fcst_valid_beg instead of obs_thresh.
> > > Accordingly,
> > > I
> > > made changes to the fixed fields (removing fcst_valid_beg) and
> > > changed
> > > the
> > > statistic from CSI to FMEAN (FBAR was also tried).  The error
that
> > > resulted
> > > is as follows:
> > >
> > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00', '2020-
05-
> > > 01
> > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00', '2020-
05-
> > > 01
> > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00', '2020-
05-
> > > 01
> > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00', '2020-
05-
> > > 01
> > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00', '2020-
05-
> > > 02
> > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00', '2020-
05-
> > > 02
> > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00', '2020-
05-
> > > 02
> > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00', '2020-
05-
> > > 02
> > > 11:00:00', '2020-05-02 12:00:00')
> > > AND   BINARY h.fcst_var = 'AOTK'
> > > AND ld.stat_header_id = h.stat_header_id
> > > AND ld.fmean != -9999;
> > > No data was returned from database
> > > 2020-06-09
> > >
> >
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > > Source) -1 printFormattedTable -   **  ERROR: Caught class
> > > edu.ucar.metviewer.EmptyResultSetException in
> > > printFormattedTable(ResultSet
> > > res): result set contained no data
> > > **  ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
> > > in
> > > printFormattedTable(ResultSet res): result set contained no data
> > > **  ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
> > > in
> > > printFormattedTable(ResultSet res): result set contained no data
> > >
> > > Apparently there was no data during this time.  I tried changing
> > > the
> > > range
> > > of valid times and changed the valid time to different days just
in
> > > case
> > > data was missing.  The issue was the same.  To confirm my logic,
I
> > > reproduced my attempted approach on metviewer, which I found to
> > > work.
> > >
> > > 2.  Running in batch mode
> > >
> > > I think I lack an understanding of how to set up batch mode
because
> > > I
> > > don't
> > > see where these definitions are being explicitly defined:
> > >
> > > <plot_spec>
> > >     <connection>
> > >         <host>rds_host:3306</host>
> > >
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > >         <user>rds_user</user>
> > >         <password>rds_pwd</password>
> > >     </connection>
> > >     <rscript>Rscript</rscript>
> > >     <folders>
> > >         <r_tmpl>rds_R_tmpl</r_tmpl>
> > >         <r_work>rds_R_work</r_work>
> > >         <plots>rds_plots</plots>
> > >         <data>rds_data</data>
> > >         <scripts>rds_scripts</scripts>
> > >     </folders>
> > >     <plot>
> > >         <template>series_plot.R_tmpl</template>
> > >
> > > I see the <folders> section, but grepping all available files
does
> > > not
> > > reveal how rds_R_work is being defined, as well as other
similarly
> > > defined
> > > variables.  I think if I know how these are being defined, then
> > > I'll
> > > be
> > > able to pipe in my information and process the data using met
and
> > > produce
> > > results in batch mode.  Am I right on that?  Also, if I want to
use
> > > obs for
> > > verification, do I add another line in the XML file that
designates
> > > file
> > > location of obs?  I assume that other steps have to be run
before
> > > all
> > > this.  If data is in prepbufr form, for example, I would have to
> > > separately
> > > run PB2NC.  If I want grid2obs, then I would want to do that as
> > > well.
> > >
> > > 3. Communicating results to metviewer online
> > >
> > > Lastly, I see a lot of people/projects listed here - see
attachment
> > >
> > > How do I get the results I generate to be uploaded to this
> > > location?
> > > It's
> > > not clear how others are setting this up.
> >
> >
> >
> >



------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Sun Jun 14 09:42:44 2020

Thanks Tatiana.  These resources are helpful.

One last question.  Simply typing lines like this in can take forever:

        <plot_fix>
            <field equalize="false" name="fcst_init_beg">
                <set name="fcst_init_beg_0">
                    <val>2019-05-01 12:00:00</val>
                    <val>2019-05-02 06:00:00</val>
                    <val>2019-05-02 12:00:00</val>
                    <val>2019-05-03 06:00:00</val>
                    <val>2019-05-03 12:00:00</val>


Is there a better way?   I see date_range, date_range_list, or
date_list.
Can this be used instead of typing individual values?  This could
prove
pretty daunting if one would have to type in 700 entries each time.

On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> I found a few problems in your XML.
> The database you are using doesn't have data with vx_mask
'abcREGabc'.
> That is why METviewer returns an empty dataset and can't create a
plot.
> The other problem is how you define fcst_lead. This database has
values
> like 10000, 20000 and not 010000, 020000. Since you include all
fcst_leads
> this variable can be omitted.
> To minimize problems with variables values, I would suggest to
create XML
> using GUI and then edit it to fit batch standards.
> I created a plot using you configurations with GUI (without vx_mack
and
> plot and axis titles) #20200612_173626. You can load it back to the
page by
> using the "History" tab (left bar on GUI). Find the plot, open XML
and
> click on "Load to page". After that you can recreate a plot and edit
the
> configurations. You can also download a new XML and edit it for
batch.
> I attached the example of load XML to this message.
>
> Description of rds parameters:
> rds_R_tmpl and rds_R_work - location of METviewer R scripts
> rds_plots - temporary directory where the plots will be generated
> rds_data - temporary directory where the data retrieved from the
database
> and other intermediate files will be stored
> rds_scripts - temporary directory where R scripts for this run will
be
> stored
> After the execution is done all the files from temporary directories
would
> be removed.
>
>
> We don't have the offitial user guide for METviewer. There are some
> instructions on how to crate XML config files:
> https://metviewer.nws.noaa.gov/doc/index.html
>
> Tatiana
>
> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
> > Hi,
> >
> > I'm attaching the xml below, but am having hard time following how
to
> > go
> > about the other steps since I have zero experience on how to do
these
> > things.
> >
> > #2. All parameters in <folders> section should NOT be changed and
will
> > be
> > replaced by other scripts during the batch execution. The data for
the
> > batch mode is coming from the database and not from files. The
name of
> > the
> > database is in <database> tag.
> >
> > I understand that nothing should be changed in the folders
section,
> > but
> > there is no indication how and where these variables are being
> > defined.
> > Grepping these names of the files available does not reveal formal
> > definitions of these variables.  If I understood exactly how these
> > were
> > being defined, then I would be able connect this better.
> >
> > #3. Users rum MET and produce MET output files. They use METviewer
> > load
> > mode to load these files to the specific database. In the config
file
> > for
> > this process they also can specify the database group. For example
> > database
> > mv_dev is in Test12 group.
> > After the data has been loaded to the database it is visible and
> > available
> > in the GUI and also can be used by the batch mode.
> > You can always add more data to the database using the same load
mode.
> >
> > Is there a guide for me to understand the process that you are
> > referring to?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward,
> > > #1. I don't have an access to NOAA servers and can't view your
XML
> > > for the
> > > batch mode. Please, attache it to this issue and I will take a
look.
> > >
> > > #2. All parameters in <folders> section should NOT be changed
and
> > > will be
> > > replaced by other scripts during the batch execution. The data
for
> > > the
> > > batch mode is coming from the database and not from files. The
name
> > > of the
> > > database is in <database> tag.
> > >
> > > #3. Users rum MET and produce MET output files. They use
METviewer
> > > load
> > > mode to load these files to the specific database. In the config
file
> > > for
> > > this process they also can specify the database group. For
example
> > > database
> > > mv_dev is in Test12 group.
> > > After the data has been loaded to the database it is visible and
> > > available
> > > in the GUI and also can be used by the batch mode.
> > > You can always add more data to the database using the same load
> > > mode.
> > >
> > > Tatiana
> > >
> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> > > > Good morning,
> > > >
> > > > Over the last two weeks I've made an effort to understand
> > > > metviewer,
> > > > test
> > > > different cases using the history tab, changing information to
see
> > > > what
> > > > works and what does not, and make connections between the GUI
> > > > settings
> > > > and
> > > > the XML file.  Although my comfort level with metviewer has
> > > > significantly
> > > > increased during this time, there are some remaining issues
that
> > > > I'm
> > > > hoping
> > > > to clear up.
> > > >
> > > > 1.  Problem changing XML file and running with different
statistic
> > > > and
> > > > plot
> > > > type:
> > > >
> > > > Ho-Chun Huang, a colleague at EMC, provided me the working
> > > > directory
> > > > to get
> > > > started which I copied over:
> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > >
> > > > He has a test.sh script that I adjusted and used to run his
> > > > example.
> > > > running of the script "bash test.sh" produced a CSI plot that
was
> > > > stored
> > > > here:
> > > >
> > >
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > > >
> > > > Running the script produced the expected result.  The next
natural
> > > > step was
> > > > to copy hochum_xml.template and create a new xml file called
> > > > Strobach_xml.template in order to create a time series where
the
> > > > independent time is fcst_valid_beg instead of obs_thresh.
> > > > Accordingly,
> > > > I
> > > > made changes to the fixed fields (removing fcst_valid_beg) and
> > > > changed
> > > > the
> > > > statistic from CSI to FMEAN (FBAR was also tried).  The error
that
> > > > resulted
> > > > is as follows:
> > > >
> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00',
'2020-05-
> > > > 01
> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
'2020-05-
> > > > 01
> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
'2020-05-
> > > > 01
> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
'2020-05-
> > > > 01
> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
'2020-05-
> > > > 02
> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
'2020-05-
> > > > 02
> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
'2020-05-
> > > > 02
> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
'2020-05-
> > > > 02
> > > > 11:00:00', '2020-05-02 12:00:00')
> > > > AND   BINARY h.fcst_var = 'AOTK'
> > > > AND ld.stat_header_id = h.stat_header_id
> > > > AND ld.fmean != -9999;
> > > > No data was returned from database
> > > > 2020-06-09
> > > >
> > >
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > > > Source) -1 printFormattedTable -   **  ERROR: Caught class
> > > > edu.ucar.metviewer.EmptyResultSetException in
> > > > printFormattedTable(ResultSet
> > > > res): result set contained no data
> > > > **  ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
> > > > in
> > > > printFormattedTable(ResultSet res): result set contained no
data
> > > > **  ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
> > > > in
> > > > printFormattedTable(ResultSet res): result set contained no
data
> > > >
> > > > Apparently there was no data during this time.  I tried
changing
> > > > the
> > > > range
> > > > of valid times and changed the valid time to different days
just in
> > > > case
> > > > data was missing.  The issue was the same.  To confirm my
logic, I
> > > > reproduced my attempted approach on metviewer, which I found
to
> > > > work.
> > > >
> > > > 2.  Running in batch mode
> > > >
> > > > I think I lack an understanding of how to set up batch mode
because
> > > > I
> > > > don't
> > > > see where these definitions are being explicitly defined:
> > > >
> > > > <plot_spec>
> > > >     <connection>
> > > >         <host>rds_host:3306</host>
> > > >
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > > >         <user>rds_user</user>
> > > >         <password>rds_pwd</password>
> > > >     </connection>
> > > >     <rscript>Rscript</rscript>
> > > >     <folders>
> > > >         <r_tmpl>rds_R_tmpl</r_tmpl>
> > > >         <r_work>rds_R_work</r_work>
> > > >         <plots>rds_plots</plots>
> > > >         <data>rds_data</data>
> > > >         <scripts>rds_scripts</scripts>
> > > >     </folders>
> > > >     <plot>
> > > >         <template>series_plot.R_tmpl</template>
> > > >
> > > > I see the <folders> section, but grepping all available files
does
> > > > not
> > > > reveal how rds_R_work is being defined, as well as other
similarly
> > > > defined
> > > > variables.  I think if I know how these are being defined,
then
> > > > I'll
> > > > be
> > > > able to pipe in my information and process the data using met
and
> > > > produce
> > > > results in batch mode.  Am I right on that?  Also, if I want
to use
> > > > obs for
> > > > verification, do I add another line in the XML file that
designates
> > > > file
> > > > location of obs?  I assume that other steps have to be run
before
> > > > all
> > > > this.  If data is in prepbufr form, for example, I would have
to
> > > > separately
> > > > run PB2NC.  If I want grid2obs, then I would want to do that
as
> > > > well.
> > > >
> > > > 3. Communicating results to metviewer online
> > > >
> > > > Lastly, I see a lot of people/projects listed here - see
attachment
> > > >
> > > > How do I get the results I generate to be uploaded to this
> > > > location?
> > > > It's
> > > > not clear how others are setting this up.
> > >
> > >
> > >
> > >
>
>
>
> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user rds_pwd
aurora
> Rscript rds_R_tmpl rds_R_work rds_plots rds_data rds_scripts
> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-05-12
> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12 20:00:00
2020-05-12
> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13 10:00:00
2020-05-13
> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13 14:00:00
2020-05-13
> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13 18:00:00
2020-05-13
> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13 22:00:00
2020-05-13
> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14 12:00:00
true
> median plot_20200612_173626.data plot_20200612_173626.png
> plot_20200612_173626.R test x_label test y_label
> false list() list() false false false false false false true false
false
> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0) 1 2
1.4 -2
> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc 1
listX 1 1
> 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8 1
#333333 0.8
> 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1) c(1,1)
c(1,2)
> c("","") c() c() 0.04 c()



--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Sun Jun 14 09:48:24 2020

and if there isn't a better way, I wonder the whereabouts of a
separate
script that pipes dates (and other relevant information) into the XML
file
used to generate the results.

On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> Thanks Tatiana.  These resources are helpful.
>
> One last question.  Simply typing lines like this in can take
forever:
>
>         <plot_fix>
>             <field equalize="false" name="fcst_init_beg">
>                 <set name="fcst_init_beg_0">
>                     <val>2019-05-01 12:00:00</val>
>                     <val>2019-05-02 06:00:00</val>
>                     <val>2019-05-02 12:00:00</val>
>                     <val>2019-05-03 06:00:00</val>
>                     <val>2019-05-03 12:00:00</val>
>
>
> Is there a better way?   I see date_range, date_range_list, or
date_list.
> Can this be used instead of typing individual values?  This could
prove
> pretty daunting if one would have to type in 700 entries each time.
>
> On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> I found a few problems in your XML.
>> The database you are using doesn't have data with vx_mask
'abcREGabc'.
>> That is why METviewer returns an empty dataset and can't create a
plot.
>> The other problem is how you define fcst_lead. This database has
values
>> like 10000, 20000 and not 010000, 020000. Since you include all
fcst_leads
>> this variable can be omitted.
>> To minimize problems with variables values, I would suggest to
create XML
>> using GUI and then edit it to fit batch standards.
>> I created a plot using you configurations with GUI (without vx_mack
and
>> plot and axis titles) #20200612_173626. You can load it back to the
page by
>> using the "History" tab (left bar on GUI). Find the plot, open XML
and
>> click on "Load to page". After that you can recreate a plot and
edit the
>> configurations. You can also download a new XML and edit it for
batch.
>> I attached the example of load XML to this message.
>>
>> Description of rds parameters:
>> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>> rds_plots - temporary directory where the plots will be generated
>> rds_data - temporary directory where the data retrieved from the
database
>> and other intermediate files will be stored
>> rds_scripts - temporary directory where R scripts for this run will
be
>> stored
>> After the execution is done all the files from temporary
directories
>> would be removed.
>>
>>
>> We don't have the offitial user guide for METviewer. There are some
>> instructions on how to crate XML config files:
>> https://metviewer.nws.noaa.gov/doc/index.html
>>
>> Tatiana
>>
>> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
>> > Hi,
>> >
>> > I'm attaching the xml below, but am having hard time following
how to
>> > go
>> > about the other steps since I have zero experience on how to do
these
>> > things.
>> >
>> > #2. All parameters in <folders> section should NOT be changed and
will
>> > be
>> > replaced by other scripts during the batch execution. The data
for the
>> > batch mode is coming from the database and not from files. The
name of
>> > the
>> > database is in <database> tag.
>> >
>> > I understand that nothing should be changed in the folders
section,
>> > but
>> > there is no indication how and where these variables are being
>> > defined.
>> > Grepping these names of the files available does not reveal
formal
>> > definitions of these variables.  If I understood exactly how
these
>> > were
>> > being defined, then I would be able connect this better.
>> >
>> > #3. Users rum MET and produce MET output files. They use
METviewer
>> > load
>> > mode to load these files to the specific database. In the config
file
>> > for
>> > this process they also can specify the database group. For
example
>> > database
>> > mv_dev is in Test12 group.
>> > After the data has been loaded to the database it is visible and
>> > available
>> > in the GUI and also can be used by the batch mode.
>> > You can always add more data to the database using the same load
mode.
>> >
>> > Is there a guide for me to understand the process that you are
>> > referring to?
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>> > <met_help at ucar.edu>
>> > wrote:
>> >
>> > > Hi Edward,
>> > > #1. I don't have an access to NOAA servers and can't view your
XML
>> > > for the
>> > > batch mode. Please, attache it to this issue and I will take a
look.
>> > >
>> > > #2. All parameters in <folders> section should NOT be changed
and
>> > > will be
>> > > replaced by other scripts during the batch execution. The data
for
>> > > the
>> > > batch mode is coming from the database and not from files. The
name
>> > > of the
>> > > database is in <database> tag.
>> > >
>> > > #3. Users rum MET and produce MET output files. They use
METviewer
>> > > load
>> > > mode to load these files to the specific database. In the
config file
>> > > for
>> > > this process they also can specify the database group. For
example
>> > > database
>> > > mv_dev is in Test12 group.
>> > > After the data has been loaded to the database it is visible
and
>> > > available
>> > > in the GUI and also can be used by the batch mode.
>> > > You can always add more data to the database using the same
load
>> > > mode.
>> > >
>> > > Tatiana
>> > >
>> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
>> > > > Good morning,
>> > > >
>> > > > Over the last two weeks I've made an effort to understand
>> > > > metviewer,
>> > > > test
>> > > > different cases using the history tab, changing information
to see
>> > > > what
>> > > > works and what does not, and make connections between the GUI
>> > > > settings
>> > > > and
>> > > > the XML file.  Although my comfort level with metviewer has
>> > > > significantly
>> > > > increased during this time, there are some remaining issues
that
>> > > > I'm
>> > > > hoping
>> > > > to clear up.
>> > > >
>> > > > 1.  Problem changing XML file and running with different
statistic
>> > > > and
>> > > > plot
>> > > > type:
>> > > >
>> > > > Ho-Chun Huang, a colleague at EMC, provided me the working
>> > > > directory
>> > > > to get
>> > > > started which I copied over:
>> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > > >
>> > > > He has a test.sh script that I adjusted and used to run his
>> > > > example.
>> > > > running of the script "bash test.sh" produced a CSI plot that
was
>> > > > stored
>> > > > here:
>> > > >
>> > >
>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>> > > >
>> > > > Running the script produced the expected result.  The next
natural
>> > > > step was
>> > > > to copy hochum_xml.template and create a new xml file called
>> > > > Strobach_xml.template in order to create a time series where
the
>> > > > independent time is fcst_valid_beg instead of obs_thresh.
>> > > > Accordingly,
>> > > > I
>> > > > made changes to the fixed fields (removing fcst_valid_beg)
and
>> > > > changed
>> > > > the
>> > > > statistic from CSI to FMEAN (FBAR was also tried).  The error
that
>> > > > resulted
>> > > > is as follows:
>> > > >
>> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00',
'2020-05-
>> > > > 01
>> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
'2020-05-
>> > > > 01
>> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
'2020-05-
>> > > > 01
>> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
'2020-05-
>> > > > 01
>> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
'2020-05-
>> > > > 02
>> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
'2020-05-
>> > > > 02
>> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
'2020-05-
>> > > > 02
>> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
'2020-05-
>> > > > 02
>> > > > 11:00:00', '2020-05-02 12:00:00')
>> > > > AND   BINARY h.fcst_var = 'AOTK'
>> > > > AND ld.stat_header_id = h.stat_header_id
>> > > > AND ld.fmean != -9999;
>> > > > No data was returned from database
>> > > > 2020-06-09
>> > > >
>> > >
>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>> > > > Source) -1 printFormattedTable -   **  ERROR: Caught class
>> > > > edu.ucar.metviewer.EmptyResultSetException in
>> > > > printFormattedTable(ResultSet
>> > > > res): result set contained no data
>> > > > **  ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
>> > > > in
>> > > > printFormattedTable(ResultSet res): result set contained no
data
>> > > > **  ERROR: Caught class
edu.ucar.metviewer.EmptyResultSetException
>> > > > in
>> > > > printFormattedTable(ResultSet res): result set contained no
data
>> > > >
>> > > > Apparently there was no data during this time.  I tried
changing
>> > > > the
>> > > > range
>> > > > of valid times and changed the valid time to different days
just in
>> > > > case
>> > > > data was missing.  The issue was the same.  To confirm my
logic, I
>> > > > reproduced my attempted approach on metviewer, which I found
to
>> > > > work.
>> > > >
>> > > > 2.  Running in batch mode
>> > > >
>> > > > I think I lack an understanding of how to set up batch mode
because
>> > > > I
>> > > > don't
>> > > > see where these definitions are being explicitly defined:
>> > > >
>> > > > <plot_spec>
>> > > >     <connection>
>> > > >         <host>rds_host:3306</host>
>> > > >
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>> > > >         <user>rds_user</user>
>> > > >         <password>rds_pwd</password>
>> > > >     </connection>
>> > > >     <rscript>Rscript</rscript>
>> > > >     <folders>
>> > > >         <r_tmpl>rds_R_tmpl</r_tmpl>
>> > > >         <r_work>rds_R_work</r_work>
>> > > >         <plots>rds_plots</plots>
>> > > >         <data>rds_data</data>
>> > > >         <scripts>rds_scripts</scripts>
>> > > >     </folders>
>> > > >     <plot>
>> > > >         <template>series_plot.R_tmpl</template>
>> > > >
>> > > > I see the <folders> section, but grepping all available files
does
>> > > > not
>> > > > reveal how rds_R_work is being defined, as well as other
similarly
>> > > > defined
>> > > > variables.  I think if I know how these are being defined,
then
>> > > > I'll
>> > > > be
>> > > > able to pipe in my information and process the data using met
and
>> > > > produce
>> > > > results in batch mode.  Am I right on that?  Also, if I want
to use
>> > > > obs for
>> > > > verification, do I add another line in the XML file that
designates
>> > > > file
>> > > > location of obs?  I assume that other steps have to be run
before
>> > > > all
>> > > > this.  If data is in prepbufr form, for example, I would have
to
>> > > > separately
>> > > > run PB2NC.  If I want grid2obs, then I would want to do that
as
>> > > > well.
>> > > >
>> > > > 3. Communicating results to metviewer online
>> > > >
>> > > > Lastly, I see a lot of people/projects listed here - see
attachment
>> > > >
>> > > > How do I get the results I generate to be uploaded to this
>> > > > location?
>> > > > It's
>> > > > not clear how others are setting this up.
>> > >
>> > >
>> > >
>> > >
>>
>>
>>
>> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user rds_pwd
>> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data rds_scripts
>> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-05-12
>> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00 2020-05-12
>> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00 2020-05-13
>> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00 2020-05-13
>> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00 2020-05-13
>> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00 2020-05-13
>> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00 true
>> median plot_20200612_173626.data plot_20200612_173626.png
>> plot_20200612_173626.R test x_label test y_label
>> false list() list() false false false false false false true false
false
>> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0) 1 2
1.4 -2
>> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc 1
listX 1
>> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8 1
#333333
>> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1) c(1,1)
>> c(1,2) c("","") c() c() 0.04 c()
>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>


--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Mon Jun 15 11:55:54 2020

Hi,
Here is some examples how to use <date_range> and <date_range_list> in
batch XML.
First, you need to defined your range. You can have more then one
<date_range>:
<date_range name="may">
        <start>2019-05-01 00:00:00</start>
        <end>2019-05-31 23:59:59</end>
</date_range>
<date_range name="march">
        <start>2019-03-01 00:00:00</start>
        <end>2019-03-31 23:59:59</end>
</date_range>

and then use one or all date_ranges:
<plot_fix>
    <field name="fcst_init_beg">
        <date_range name="march"/>
        <date_range name="may"/>
   </field>...
This configuration would include all dates for March and May of 2019.

If you need more flexible configuration you can use date_range_list.
For example, you want to select first 6 hours every day beetween 2020-
01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look like this:
<date_range_list name = 'all'>
       <range_start>2020-01-01 00:00:00</range_start>
        <range_end>2020-01-05 00:00:00</range_end>
        <range_length>360</range_length>
        <inc>1440</inc>
</date_range_list>
where the 1st and 2nd values are the start and end,
range_length - the length of a range in minutes(6 houres)
inc - lenght beetween ranges in minutes (24 houres)
This example would create following ranges:
BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'

Also you can add more parameters to <plot_fix> to filter your data.
For example, if you want to select only dates with 00:00 you can
specify
 <field name="valid_hour">
        <val>00</val>
 </field>

Tatiana



On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> and if there isn't a better way, I wonder the whereabouts of a
> separate
> script that pipes dates (and other relevant information) into the
XML
> file
> used to generate the results.
>
> On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Thanks Tatiana.  These resources are helpful.
> >
> > One last question.  Simply typing lines like this in can take
> > forever:
> >
> > <plot_fix>
> >     <field equalize="false" name="fcst_init_beg">
> >         <set name="fcst_init_beg_0">
> >             <val>2019-05-01 12:00:00</val>
> >             <val>2019-05-02 06:00:00</val>
> >             <val>2019-05-02 12:00:00</val>
> >             <val>2019-05-03 06:00:00</val>
> >             <val>2019-05-03 12:00:00</val>
> >
> >
> > Is there a better way?   I see date_range, date_range_list, or
> > date_list.
> > Can this be used instead of typing individual values?  This could
> > prove
> > pretty daunting if one would have to type in 700 entries each
time.
> >
> > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> >> I found a few problems in your XML.
> >> The database you are using doesn't have data with vx_mask
> >> 'abcREGabc'.
> >> That is why METviewer returns an empty dataset and can't create a
> >> plot.
> >> The other problem is how you define fcst_lead. This database has
> >> values
> >> like 10000, 20000 and not 010000, 020000. Since you include all
> >> fcst_leads
> >> this variable can be omitted.
> >> To minimize problems with variables values, I would suggest to
> >> create XML
> >> using GUI and then edit it to fit batch standards.
> >> I created a plot using you configurations with GUI (without
vx_mack
> >> and
> >> plot and axis titles) #20200612_173626. You can load it back to
the
> >> page by
> >> using the "History" tab (left bar on GUI). Find the plot, open
XML
> >> and
> >> click on "Load to page". After that you can recreate a plot and
edit
> >> the
> >> configurations. You can also download a new XML and edit it for
> >> batch.
> >> I attached the example of load XML to this message.
> >>
> >> Description of rds parameters:
> >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
> >> rds_plots - temporary directory where the plots will be generated
> >> rds_data - temporary directory where the data retrieved from the
> >> database
> >> and other intermediate files will be stored
> >> rds_scripts - temporary directory where R scripts for this run
will
> >> be
> >> stored
> >> After the execution is done all the files from temporary
directories
> >> would be removed.
> >>
> >>
> >> We don't have the offitial user guide for METviewer. There are
some
> >> instructions on how to crate XML config files:
> >> https://metviewer.nws.noaa.gov/doc/index.html
> >>
> >> Tatiana
> >>
> >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
> >> > Hi,
> >> >
> >> > I'm attaching the xml below, but am having hard time following
how
> >> > to
> >> > go
> >> > about the other steps since I have zero experience on how to do
> >> > these
> >> > things.
> >> >
> >> > #2. All parameters in <folders> section should NOT be changed
and
> >> > will
> >> > be
> >> > replaced by other scripts during the batch execution. The data
for
> >> > the
> >> > batch mode is coming from the database and not from files. The
> >> > name of
> >> > the
> >> > database is in <database> tag.
> >> >
> >> > I understand that nothing should be changed in the folders
> >> > section,
> >> > but
> >> > there is no indication how and where these variables are being
> >> > defined.
> >> > Grepping these names of the files available does not reveal
formal
> >> > definitions of these variables.  If I understood exactly how
these
> >> > were
> >> > being defined, then I would be able connect this better.
> >> >
> >> > #3. Users rum MET and produce MET output files. They use
METviewer
> >> > load
> >> > mode to load these files to the specific database. In the
config
> >> > file
> >> > for
> >> > this process they also can specify the database group. For
example
> >> > database
> >> > mv_dev is in Test12 group.
> >> > After the data has been loaded to the database it is visible
and
> >> > available
> >> > in the GUI and also can be used by the batch mode.
> >> > You can always add more data to the database using the same
load
> >> > mode.
> >> >
> >> > Is there a guide for me to understand the process that you are
> >> > referring to?
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> >> > <met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Hi Edward,
> >> > > #1. I don't have an access to NOAA servers and can't view
your
> >> > > XML
> >> > > for the
> >> > > batch mode. Please, attache it to this issue and I will take
a
> >> > > look.
> >> > >
> >> > > #2. All parameters in <folders> section should NOT be changed
> >> > > and
> >> > > will be
> >> > > replaced by other scripts during the batch execution. The
data
> >> > > for
> >> > > the
> >> > > batch mode is coming from the database and not from files.
The
> >> > > name
> >> > > of the
> >> > > database is in <database> tag.
> >> > >
> >> > > #3. Users rum MET and produce MET output files. They use
> >> > > METviewer
> >> > > load
> >> > > mode to load these files to the specific database. In the
config
> >> > > file
> >> > > for
> >> > > this process they also can specify the database group. For
> >> > > example
> >> > > database
> >> > > mv_dev is in Test12 group.
> >> > > After the data has been loaded to the database it is visible
and
> >> > > available
> >> > > in the GUI and also can be used by the batch mode.
> >> > > You can always add more data to the database using the same
load
> >> > > mode.
> >> > >
> >> > > Tatiana
> >> > >
> >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov wrote:
> >> > > > Good morning,
> >> > > >
> >> > > > Over the last two weeks I've made an effort to understand
> >> > > > metviewer,
> >> > > > test
> >> > > > different cases using the history tab, changing information
to
> >> > > > see
> >> > > > what
> >> > > > works and what does not, and make connections between the
GUI
> >> > > > settings
> >> > > > and
> >> > > > the XML file.  Although my comfort level with metviewer has
> >> > > > significantly
> >> > > > increased during this time, there are some remaining issues
> >> > > > that
> >> > > > I'm
> >> > > > hoping
> >> > > > to clear up.
> >> > > >
> >> > > > 1.  Problem changing XML file and running with different
> >> > > > statistic
> >> > > > and
> >> > > > plot
> >> > > > type:
> >> > > >
> >> > > > Ho-Chun Huang, a colleague at EMC, provided me the working
> >> > > > directory
> >> > > > to get
> >> > > > started which I copied over:
> >> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >> > > >
> >> > > > He has a test.sh script that I adjusted and used to run his
> >> > > > example.
> >> > > > running of the script "bash test.sh" produced a CSI plot
that
> >> > > > was
> >> > > > stored
> >> > > > here:
> >> > > >
> >> > >
> >>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >> > > >
> >> > > > Running the script produced the expected result.  The next
> >> > > > natural
> >> > > > step was
> >> > > > to copy hochum_xml.template and create a new xml file
called
> >> > > > Strobach_xml.template in order to create a time series
where
> >> > > > the
> >> > > > independent time is fcst_valid_beg instead of obs_thresh.
> >> > > > Accordingly,
> >> > > > I
> >> > > > made changes to the fixed fields (removing fcst_valid_beg)
and
> >> > > > changed
> >> > > > the
> >> > > > statistic from CSI to FMEAN (FBAR was also tried).  The
error
> >> > > > that
> >> > > > resulted
> >> > > > is as follows:
> >> > > >
> >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00',
> >> > > > '2020-05-
> >> > > > 01
> >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
> >> > > > '2020-05-
> >> > > > 01
> >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
> >> > > > '2020-05-
> >> > > > 01
> >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
> >> > > > '2020-05-
> >> > > > 01
> >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
> >> > > > '2020-05-
> >> > > > 02
> >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
> >> > > > '2020-05-
> >> > > > 02
> >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
> >> > > > '2020-05-
> >> > > > 02
> >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
> >> > > > '2020-05-
> >> > > > 02
> >> > > > 11:00:00', '2020-05-02 12:00:00')
> >> > > > AND   BINARY h.fcst_var = 'AOTK'
> >> > > > AND ld.stat_header_id = h.stat_header_id
> >> > > > AND ld.fmean != -9999;
> >> > > > No data was returned from database
> >> > > > 2020-06-09
> >> > > >
> >> > >
> >>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >> > > > Source) -1 printFormattedTable -   **  ERROR: Caught class
> >> > > > edu.ucar.metviewer.EmptyResultSetException in
> >> > > > printFormattedTable(ResultSet
> >> > > > res): result set contained no data
> >> > > > **  ERROR: Caught class
> >> > > > edu.ucar.metviewer.EmptyResultSetException
> >> > > > in
> >> > > > printFormattedTable(ResultSet res): result set contained no
> >> > > > data
> >> > > > **  ERROR: Caught class
> >> > > > edu.ucar.metviewer.EmptyResultSetException
> >> > > > in
> >> > > > printFormattedTable(ResultSet res): result set contained no
> >> > > > data
> >> > > >
> >> > > > Apparently there was no data during this time.  I tried
> >> > > > changing
> >> > > > the
> >> > > > range
> >> > > > of valid times and changed the valid time to different days
> >> > > > just in
> >> > > > case
> >> > > > data was missing.  The issue was the same.  To confirm my
> >> > > > logic, I
> >> > > > reproduced my attempted approach on metviewer, which I
found
> >> > > > to
> >> > > > work.
> >> > > >
> >> > > > 2.  Running in batch mode
> >> > > >
> >> > > > I think I lack an understanding of how to set up batch mode
> >> > > > because
> >> > > > I
> >> > > > don't
> >> > > > see where these definitions are being explicitly defined:
> >> > > >
> >> > > > <plot_spec>
> >> > > >     <connection>
> >> > > >         <host>rds_host:3306</host>
> >> > > >
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >> > > >         <user>rds_user</user>
> >> > > >         <password>rds_pwd</password>
> >> > > >     </connection>
> >> > > >     <rscript>Rscript</rscript>
> >> > > >     <folders>
> >> > > >         <r_tmpl>rds_R_tmpl</r_tmpl>
> >> > > >         <r_work>rds_R_work</r_work>
> >> > > >         <plots>rds_plots</plots>
> >> > > >         <data>rds_data</data>
> >> > > >         <scripts>rds_scripts</scripts>
> >> > > >     </folders>
> >> > > >     <plot>
> >> > > >         <template>series_plot.R_tmpl</template>
> >> > > >
> >> > > > I see the <folders> section, but grepping all available
files
> >> > > > does
> >> > > > not
> >> > > > reveal how rds_R_work is being defined, as well as other
> >> > > > similarly
> >> > > > defined
> >> > > > variables.  I think if I know how these are being defined,
> >> > > > then
> >> > > > I'll
> >> > > > be
> >> > > > able to pipe in my information and process the data using
met
> >> > > > and
> >> > > > produce
> >> > > > results in batch mode.  Am I right on that?  Also, if I
want
> >> > > > to use
> >> > > > obs for
> >> > > > verification, do I add another line in the XML file that
> >> > > > designates
> >> > > > file
> >> > > > location of obs?  I assume that other steps have to be run
> >> > > > before
> >> > > > all
> >> > > > this.  If data is in prepbufr form, for example, I would
have
> >> > > > to
> >> > > > separately
> >> > > > run PB2NC.  If I want grid2obs, then I would want to do
that
> >> > > > as
> >> > > > well.
> >> > > >
> >> > > > 3. Communicating results to metviewer online
> >> > > >
> >> > > > Lastly, I see a lot of people/projects listed here - see
> >> > > > attachment
> >> > > >
> >> > > > How do I get the results I generate to be uploaded to this
> >> > > > location?
> >> > > > It's
> >> > > > not clear how others are setting this up.
> >> > >
> >> > >
> >> > >
> >> > >
> >>
> >>
> >>
> >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
> >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
> >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-05-
12
> >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00
> >> 2020-05-12
> >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00
> >> 2020-05-13
> >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00
> >> 2020-05-13
> >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00
> >> 2020-05-13
> >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00
> >> 2020-05-13
> >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00
> >> true
> >> median plot_20200612_173626.data plot_20200612_173626.png
> >> plot_20200612_173626.R test x_label test y_label
> >> false list() list() false false false false false false true
false
> >> false
> >> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0) 1
2
> >> 1.4 -2
> >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc 1
> >> listX 1
> >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8 1
> >> #333333
> >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
> >> c(1,2) c("","") c() c() 0.04 c()
> >
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >



------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Mon Jun 15 14:28:45 2020

Thank you.  This is a very helpful example

On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Hi,
> Here is some examples how to use <date_range> and <date_range_list>
in
> batch XML.
> First, you need to defined your range. You can have more then one
> <date_range>:
> <date_range name="may">
>         <start>2019-05-01 00:00:00</start>
>         <end>2019-05-31 23:59:59</end>
> </date_range>
> <date_range name="march">
>         <start>2019-03-01 00:00:00</start>
>         <end>2019-03-31 23:59:59</end>
> </date_range>
>
> and then use one or all date_ranges:
> <plot_fix>
>     <field name="fcst_init_beg">
>         <date_range name="march"/>
>         <date_range name="may"/>
>    </field>...
> This configuration would include all dates for March and May of
2019.
>
> If you need more flexible configuration you can use date_range_list.
> For example, you want to select first 6 hours every day beetween
> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
> <date_range_list name = 'all'>
>        <range_start>2020-01-01 00:00:00</range_start>
>         <range_end>2020-01-05 00:00:00</range_end>
>         <range_length>360</range_length>
>         <inc>1440</inc>
> </date_range_list>
> where the 1st and 2nd values are the start and end,
> range_length - the length of a range in minutes(6 houres)
> inc - lenght beetween ranges in minutes (24 houres)
> This example would create following ranges:
> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>
> Also you can add more parameters to <plot_fix> to filter your data.
For
> example, if you want to select only dates with 00:00 you can specify
>  <field name="valid_hour">
>         <val>00</val>
>  </field>
>
> Tatiana
>
>
>
> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> > and if there isn't a better way, I wonder the whereabouts of a
> > separate
> > script that pipes dates (and other relevant information) into the
XML
> > file
> > used to generate the results.
> >
> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA Affiliate
<
> > edward.strobach at noaa.gov> wrote:
> >
> > > Thanks Tatiana.  These resources are helpful.
> > >
> > > One last question.  Simply typing lines like this in can take
> > > forever:
> > >
> > > <plot_fix>
> > >     <field equalize="false" name="fcst_init_beg">
> > >         <set name="fcst_init_beg_0">
> > >             <val>2019-05-01 12:00:00</val>
> > >             <val>2019-05-02 06:00:00</val>
> > >             <val>2019-05-02 12:00:00</val>
> > >             <val>2019-05-03 06:00:00</val>
> > >             <val>2019-05-03 12:00:00</val>
> > >
> > >
> > > Is there a better way?   I see date_range, date_range_list, or
> > > date_list.
> > > Can this be used instead of typing individual values?  This
could
> > > prove
> > > pretty daunting if one would have to type in 700 entries each
time.
> > >
> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > >> I found a few problems in your XML.
> > >> The database you are using doesn't have data with vx_mask
> > >> 'abcREGabc'.
> > >> That is why METviewer returns an empty dataset and can't create
a
> > >> plot.
> > >> The other problem is how you define fcst_lead. This database
has
> > >> values
> > >> like 10000, 20000 and not 010000, 020000. Since you include all
> > >> fcst_leads
> > >> this variable can be omitted.
> > >> To minimize problems with variables values, I would suggest to
> > >> create XML
> > >> using GUI and then edit it to fit batch standards.
> > >> I created a plot using you configurations with GUI (without
vx_mack
> > >> and
> > >> plot and axis titles) #20200612_173626. You can load it back to
the
> > >> page by
> > >> using the "History" tab (left bar on GUI). Find the plot, open
XML
> > >> and
> > >> click on "Load to page". After that you can recreate a plot and
edit
> > >> the
> > >> configurations. You can also download a new XML and edit it for
> > >> batch.
> > >> I attached the example of load XML to this message.
> > >>
> > >> Description of rds parameters:
> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
> > >> rds_plots - temporary directory where the plots will be
generated
> > >> rds_data - temporary directory where the data retrieved from
the
> > >> database
> > >> and other intermediate files will be stored
> > >> rds_scripts - temporary directory where R scripts for this run
will
> > >> be
> > >> stored
> > >> After the execution is done all the files from temporary
directories
> > >> would be removed.
> > >>
> > >>
> > >> We don't have the offitial user guide for METviewer. There are
some
> > >> instructions on how to crate XML config files:
> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > >>
> > >> Tatiana
> > >>
> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
> > >> > Hi,
> > >> >
> > >> > I'm attaching the xml below, but am having hard time
following how
> > >> > to
> > >> > go
> > >> > about the other steps since I have zero experience on how to
do
> > >> > these
> > >> > things.
> > >> >
> > >> > #2. All parameters in <folders> section should NOT be changed
and
> > >> > will
> > >> > be
> > >> > replaced by other scripts during the batch execution. The
data for
> > >> > the
> > >> > batch mode is coming from the database and not from files.
The
> > >> > name of
> > >> > the
> > >> > database is in <database> tag.
> > >> >
> > >> > I understand that nothing should be changed in the folders
> > >> > section,
> > >> > but
> > >> > there is no indication how and where these variables are
being
> > >> > defined.
> > >> > Grepping these names of the files available does not reveal
formal
> > >> > definitions of these variables.  If I understood exactly how
these
> > >> > were
> > >> > being defined, then I would be able connect this better.
> > >> >
> > >> > #3. Users rum MET and produce MET output files. They use
METviewer
> > >> > load
> > >> > mode to load these files to the specific database. In the
config
> > >> > file
> > >> > for
> > >> > this process they also can specify the database group. For
example
> > >> > database
> > >> > mv_dev is in Test12 group.
> > >> > After the data has been loaded to the database it is visible
and
> > >> > available
> > >> > in the GUI and also can be used by the batch mode.
> > >> > You can always add more data to the database using the same
load
> > >> > mode.
> > >> >
> > >> > Is there a guide for me to understand the process that you
are
> > >> > referring to?
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> > >> > <met_help at ucar.edu>
> > >> > wrote:
> > >> >
> > >> > > Hi Edward,
> > >> > > #1. I don't have an access to NOAA servers and can't view
your
> > >> > > XML
> > >> > > for the
> > >> > > batch mode. Please, attache it to this issue and I will
take a
> > >> > > look.
> > >> > >
> > >> > > #2. All parameters in <folders> section should NOT be
changed
> > >> > > and
> > >> > > will be
> > >> > > replaced by other scripts during the batch execution. The
data
> > >> > > for
> > >> > > the
> > >> > > batch mode is coming from the database and not from files.
The
> > >> > > name
> > >> > > of the
> > >> > > database is in <database> tag.
> > >> > >
> > >> > > #3. Users rum MET and produce MET output files. They use
> > >> > > METviewer
> > >> > > load
> > >> > > mode to load these files to the specific database. In the
config
> > >> > > file
> > >> > > for
> > >> > > this process they also can specify the database group. For
> > >> > > example
> > >> > > database
> > >> > > mv_dev is in Test12 group.
> > >> > > After the data has been loaded to the database it is
visible and
> > >> > > available
> > >> > > in the GUI and also can be used by the batch mode.
> > >> > > You can always add more data to the database using the same
load
> > >> > > mode.
> > >> > >
> > >> > > Tatiana
> > >> > >
> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
> > >> > > > Good morning,
> > >> > > >
> > >> > > > Over the last two weeks I've made an effort to understand
> > >> > > > metviewer,
> > >> > > > test
> > >> > > > different cases using the history tab, changing
information to
> > >> > > > see
> > >> > > > what
> > >> > > > works and what does not, and make connections between the
GUI
> > >> > > > settings
> > >> > > > and
> > >> > > > the XML file.  Although my comfort level with metviewer
has
> > >> > > > significantly
> > >> > > > increased during this time, there are some remaining
issues
> > >> > > > that
> > >> > > > I'm
> > >> > > > hoping
> > >> > > > to clear up.
> > >> > > >
> > >> > > > 1.  Problem changing XML file and running with different
> > >> > > > statistic
> > >> > > > and
> > >> > > > plot
> > >> > > > type:
> > >> > > >
> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
> > >> > > > directory
> > >> > > > to get
> > >> > > > started which I copied over:
> > >> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >> > > >
> > >> > > > He has a test.sh script that I adjusted and used to run
his
> > >> > > > example.
> > >> > > > running of the script "bash test.sh" produced a CSI plot
that
> > >> > > > was
> > >> > > > stored
> > >> > > > here:
> > >> > > >
> > >> > >
> > >>
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >> > > >
> > >> > > > Running the script produced the expected result.  The
next
> > >> > > > natural
> > >> > > > step was
> > >> > > > to copy hochum_xml.template and create a new xml file
called
> > >> > > > Strobach_xml.template in order to create a time series
where
> > >> > > > the
> > >> > > > independent time is fcst_valid_beg instead of obs_thresh.
> > >> > > > Accordingly,
> > >> > > > I
> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg) and
> > >> > > > changed
> > >> > > > the
> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).  The
error
> > >> > > > that
> > >> > > > resulted
> > >> > > > is as follows:
> > >> > > >
> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01 13:00:00',
> > >> > > > '2020-05-
> > >> > > > 01
> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
> > >> > > > '2020-05-
> > >> > > > 01
> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
> > >> > > > '2020-05-
> > >> > > > 01
> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
> > >> > > > '2020-05-
> > >> > > > 01
> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
> > >> > > > '2020-05-
> > >> > > > 02
> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
> > >> > > > '2020-05-
> > >> > > > 02
> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
> > >> > > > '2020-05-
> > >> > > > 02
> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
> > >> > > > '2020-05-
> > >> > > > 02
> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > >> > > > AND ld.fmean != -9999;
> > >> > > > No data was returned from database
> > >> > > > 2020-06-09
> > >> > > >
> > >> > >
> > >>
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > >> > > > Source) -1 printFormattedTable -   **  ERROR: Caught
class
> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> > >> > > > printFormattedTable(ResultSet
> > >> > > > res): result set contained no data
> > >> > > > **  ERROR: Caught class
> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >> > > > in
> > >> > > > printFormattedTable(ResultSet res): result set contained
no
> > >> > > > data
> > >> > > > **  ERROR: Caught class
> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >> > > > in
> > >> > > > printFormattedTable(ResultSet res): result set contained
no
> > >> > > > data
> > >> > > >
> > >> > > > Apparently there was no data during this time.  I tried
> > >> > > > changing
> > >> > > > the
> > >> > > > range
> > >> > > > of valid times and changed the valid time to different
days
> > >> > > > just in
> > >> > > > case
> > >> > > > data was missing.  The issue was the same.  To confirm my
> > >> > > > logic, I
> > >> > > > reproduced my attempted approach on metviewer, which I
found
> > >> > > > to
> > >> > > > work.
> > >> > > >
> > >> > > > 2.  Running in batch mode
> > >> > > >
> > >> > > > I think I lack an understanding of how to set up batch
mode
> > >> > > > because
> > >> > > > I
> > >> > > > don't
> > >> > > > see where these definitions are being explicitly defined:
> > >> > > >
> > >> > > > <plot_spec>
> > >> > > >     <connection>
> > >> > > >         <host>rds_host:3306</host>
> > >> > > >
>  <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > >> > > >         <user>rds_user</user>
> > >> > > >         <password>rds_pwd</password>
> > >> > > >     </connection>
> > >> > > >     <rscript>Rscript</rscript>
> > >> > > >     <folders>
> > >> > > >         <r_tmpl>rds_R_tmpl</r_tmpl>
> > >> > > >         <r_work>rds_R_work</r_work>
> > >> > > >         <plots>rds_plots</plots>
> > >> > > >         <data>rds_data</data>
> > >> > > >         <scripts>rds_scripts</scripts>
> > >> > > >     </folders>
> > >> > > >     <plot>
> > >> > > >         <template>series_plot.R_tmpl</template>
> > >> > > >
> > >> > > > I see the <folders> section, but grepping all available
files
> > >> > > > does
> > >> > > > not
> > >> > > > reveal how rds_R_work is being defined, as well as other
> > >> > > > similarly
> > >> > > > defined
> > >> > > > variables.  I think if I know how these are being
defined,
> > >> > > > then
> > >> > > > I'll
> > >> > > > be
> > >> > > > able to pipe in my information and process the data using
met
> > >> > > > and
> > >> > > > produce
> > >> > > > results in batch mode.  Am I right on that?  Also, if I
want
> > >> > > > to use
> > >> > > > obs for
> > >> > > > verification, do I add another line in the XML file that
> > >> > > > designates
> > >> > > > file
> > >> > > > location of obs?  I assume that other steps have to be
run
> > >> > > > before
> > >> > > > all
> > >> > > > this.  If data is in prepbufr form, for example, I would
have
> > >> > > > to
> > >> > > > separately
> > >> > > > run PB2NC.  If I want grid2obs, then I would want to do
that
> > >> > > > as
> > >> > > > well.
> > >> > > >
> > >> > > > 3. Communicating results to metviewer online
> > >> > > >
> > >> > > > Lastly, I see a lot of people/projects listed here - see
> > >> > > > attachment
> > >> > > >
> > >> > > > How do I get the results I generate to be uploaded to
this
> > >> > > > location?
> > >> > > > It's
> > >> > > > not clear how others are setting this up.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >>
> > >>
> > >>
> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-
05-12
> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00
> > >> 2020-05-12
> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00
> > >> 2020-05-13
> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00
> > >> 2020-05-13
> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00
> > >> 2020-05-13
> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00
> > >> 2020-05-13
> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00
> > >> true
> > >> median plot_20200612_173626.data plot_20200612_173626.png
> > >> plot_20200612_173626.R test x_label test y_label
> > >> false list() list() false false false false false false true
false
> > >> false
> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0)
1 2
> > >> 1.4 -2
> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc
1
> > >> listX 1
> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8 1
> > >> #333333
> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
> > >> c(1,2) c("","") c() c() 0.04 c()
> > >
> > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
>
>
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Mon Jun 15 15:47:17 2020

I do plan to use date_range; however, in an example provided by the
link I
see this:

<load_spec>
  <connection>
    <host>kemosabe:3306</host>
    <database>metvdb_hwt</database>
    <user>pgoldenb</user>
    <password>pgoldenb</password>
  </connection>

  <date_list name="folder_dates">
    <start>2010051914V</start>
    <end><date_offset><day_offset>0</day_offset><hour>6
</hour></date_offset></end>
    <inc>3600</inc>
    <format>yyyyMMddHH'V'</format>
  </date_list>

  <met_version>V3.0</met_version>

  <verbose>false</verbose>
  <insert_size>1</insert_size>
  <mode_header_db_check>true</mode_header_db_check>
  <drop_indexes>false</drop_indexes>
  <apply_indexes>true</apply_indexes>
  <group>Group name</group>
  <load_stat>true</load_stat>
  <load_mode>true</load_mode>
  <load_mtd>true</load_mtd>
  <load_mpr>false</load_mpr>

  <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
  <load_val>
    <field name="model">
      <val>arw</val>
      <val>nmm</val>
    </field>

    <field name="valid_time">
      <date_list name="folder_dates"/>
    </field>

    <field name="vx_mask">
      <val>FULL</val>
      <val>SWC</val>
    </field>
  </load_val>
</load_spec>

Are similar settings allowed with <plot_spec>?  I see <connection>,
which
seems carried over.  I'm looking to specify my folder_tmpl in
<plot_spec>,
which according to Tara, should be possible.  Since my directory
structure
looks like this:

/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612

I would do something like this I would think:

.......

  <date_list name="folder_dates">
    <start>20200612/start>
    <end><date_offset><day_offset>0</day_offset></end>
    <inc>86400</inc>
    <format>yyyyMMdd</format>
  </date_list>

  <folder_tmpl>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
</folder_tmpl>
  <load_val>
    <field name="model">
      <val>prod</val>
    </field>

    <field name="valid_time">
      <date_list name="folder_dates"/>
    </field>

......

On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> Thank you.  This is a very helpful example
>
> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> Hi,
>> Here is some examples how to use <date_range> and <date_range_list>
in
>> batch XML.
>> First, you need to defined your range. You can have more then one
>> <date_range>:
>> <date_range name="may">
>>         <start>2019-05-01 00:00:00</start>
>>         <end>2019-05-31 23:59:59</end>
>> </date_range>
>> <date_range name="march">
>>         <start>2019-03-01 00:00:00</start>
>>         <end>2019-03-31 23:59:59</end>
>> </date_range>
>>
>> and then use one or all date_ranges:
>> <plot_fix>
>>     <field name="fcst_init_beg">
>>         <date_range name="march"/>
>>         <date_range name="may"/>
>>    </field>...
>> This configuration would include all dates for March and May of
2019.
>>
>> If you need more flexible configuration you can use
date_range_list.
>> For example, you want to select first 6 hours every day beetween
>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
>> <date_range_list name = 'all'>
>>        <range_start>2020-01-01 00:00:00</range_start>
>>         <range_end>2020-01-05 00:00:00</range_end>
>>         <range_length>360</range_length>
>>         <inc>1440</inc>
>> </date_range_list>
>> where the 1st and 2nd values are the start and end,
>> range_length - the length of a range in minutes(6 houres)
>> inc - lenght beetween ranges in minutes (24 houres)
>> This example would create following ranges:
>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>>
>> Also you can add more parameters to <plot_fix> to filter your data.
For
>> example, if you want to select only dates with 00:00 you can
specify
>>  <field name="valid_hour">
>>         <val>00</val>
>>  </field>
>>
>> Tatiana
>>
>>
>>
>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
>> > and if there isn't a better way, I wonder the whereabouts of a
>> > separate
>> > script that pipes dates (and other relevant information) into the
XML
>> > file
>> > used to generate the results.
>> >
>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA Affiliate
<
>> > edward.strobach at noaa.gov> wrote:
>> >
>> > > Thanks Tatiana.  These resources are helpful.
>> > >
>> > > One last question.  Simply typing lines like this in can take
>> > > forever:
>> > >
>> > > <plot_fix>
>> > >     <field equalize="false" name="fcst_init_beg">
>> > >         <set name="fcst_init_beg_0">
>> > >             <val>2019-05-01 12:00:00</val>
>> > >             <val>2019-05-02 06:00:00</val>
>> > >             <val>2019-05-02 12:00:00</val>
>> > >             <val>2019-05-03 06:00:00</val>
>> > >             <val>2019-05-03 12:00:00</val>
>> > >
>> > >
>> > > Is there a better way?   I see date_range, date_range_list, or
>> > > date_list.
>> > > Can this be used instead of typing individual values?  This
could
>> > > prove
>> > > pretty daunting if one would have to type in 700 entries each
time.
>> > >
>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>> > > <met_help at ucar.edu>
>> > > wrote:
>> > >
>> > >> I found a few problems in your XML.
>> > >> The database you are using doesn't have data with vx_mask
>> > >> 'abcREGabc'.
>> > >> That is why METviewer returns an empty dataset and can't
create a
>> > >> plot.
>> > >> The other problem is how you define fcst_lead. This database
has
>> > >> values
>> > >> like 10000, 20000 and not 010000, 020000. Since you include
all
>> > >> fcst_leads
>> > >> this variable can be omitted.
>> > >> To minimize problems with variables values, I would suggest to
>> > >> create XML
>> > >> using GUI and then edit it to fit batch standards.
>> > >> I created a plot using you configurations with GUI (without
vx_mack
>> > >> and
>> > >> plot and axis titles) #20200612_173626. You can load it back
to the
>> > >> page by
>> > >> using the "History" tab (left bar on GUI). Find the plot, open
XML
>> > >> and
>> > >> click on "Load to page". After that you can recreate a plot
and edit
>> > >> the
>> > >> configurations. You can also download a new XML and edit it
for
>> > >> batch.
>> > >> I attached the example of load XML to this message.
>> > >>
>> > >> Description of rds parameters:
>> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>> > >> rds_plots - temporary directory where the plots will be
generated
>> > >> rds_data - temporary directory where the data retrieved from
the
>> > >> database
>> > >> and other intermediate files will be stored
>> > >> rds_scripts - temporary directory where R scripts for this run
will
>> > >> be
>> > >> stored
>> > >> After the execution is done all the files from temporary
directories
>> > >> would be removed.
>> > >>
>> > >>
>> > >> We don't have the offitial user guide for METviewer. There are
some
>> > >> instructions on how to crate XML config files:
>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>> > >>
>> > >> Tatiana
>> > >>
>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
>> > >> > Hi,
>> > >> >
>> > >> > I'm attaching the xml below, but am having hard time
following how
>> > >> > to
>> > >> > go
>> > >> > about the other steps since I have zero experience on how to
do
>> > >> > these
>> > >> > things.
>> > >> >
>> > >> > #2. All parameters in <folders> section should NOT be
changed and
>> > >> > will
>> > >> > be
>> > >> > replaced by other scripts during the batch execution. The
data for
>> > >> > the
>> > >> > batch mode is coming from the database and not from files.
The
>> > >> > name of
>> > >> > the
>> > >> > database is in <database> tag.
>> > >> >
>> > >> > I understand that nothing should be changed in the folders
>> > >> > section,
>> > >> > but
>> > >> > there is no indication how and where these variables are
being
>> > >> > defined.
>> > >> > Grepping these names of the files available does not reveal
formal
>> > >> > definitions of these variables.  If I understood exactly how
these
>> > >> > were
>> > >> > being defined, then I would be able connect this better.
>> > >> >
>> > >> > #3. Users rum MET and produce MET output files. They use
METviewer
>> > >> > load
>> > >> > mode to load these files to the specific database. In the
config
>> > >> > file
>> > >> > for
>> > >> > this process they also can specify the database group. For
example
>> > >> > database
>> > >> > mv_dev is in Test12 group.
>> > >> > After the data has been loaded to the database it is visible
and
>> > >> > available
>> > >> > in the GUI and also can be used by the batch mode.
>> > >> > You can always add more data to the database using the same
load
>> > >> > mode.
>> > >> >
>> > >> > Is there a guide for me to understand the process that you
are
>> > >> > referring to?
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>> > >> > <met_help at ucar.edu>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Edward,
>> > >> > > #1. I don't have an access to NOAA servers and can't view
your
>> > >> > > XML
>> > >> > > for the
>> > >> > > batch mode. Please, attache it to this issue and I will
take a
>> > >> > > look.
>> > >> > >
>> > >> > > #2. All parameters in <folders> section should NOT be
changed
>> > >> > > and
>> > >> > > will be
>> > >> > > replaced by other scripts during the batch execution. The
data
>> > >> > > for
>> > >> > > the
>> > >> > > batch mode is coming from the database and not from files.
The
>> > >> > > name
>> > >> > > of the
>> > >> > > database is in <database> tag.
>> > >> > >
>> > >> > > #3. Users rum MET and produce MET output files. They use
>> > >> > > METviewer
>> > >> > > load
>> > >> > > mode to load these files to the specific database. In the
config
>> > >> > > file
>> > >> > > for
>> > >> > > this process they also can specify the database group. For
>> > >> > > example
>> > >> > > database
>> > >> > > mv_dev is in Test12 group.
>> > >> > > After the data has been loaded to the database it is
visible and
>> > >> > > available
>> > >> > > in the GUI and also can be used by the batch mode.
>> > >> > > You can always add more data to the database using the
same load
>> > >> > > mode.
>> > >> > >
>> > >> > > Tatiana
>> > >> > >
>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
>> > >> > > > Good morning,
>> > >> > > >
>> > >> > > > Over the last two weeks I've made an effort to
understand
>> > >> > > > metviewer,
>> > >> > > > test
>> > >> > > > different cases using the history tab, changing
information to
>> > >> > > > see
>> > >> > > > what
>> > >> > > > works and what does not, and make connections between
the GUI
>> > >> > > > settings
>> > >> > > > and
>> > >> > > > the XML file.  Although my comfort level with metviewer
has
>> > >> > > > significantly
>> > >> > > > increased during this time, there are some remaining
issues
>> > >> > > > that
>> > >> > > > I'm
>> > >> > > > hoping
>> > >> > > > to clear up.
>> > >> > > >
>> > >> > > > 1.  Problem changing XML file and running with different
>> > >> > > > statistic
>> > >> > > > and
>> > >> > > > plot
>> > >> > > > type:
>> > >> > > >
>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
>> > >> > > > directory
>> > >> > > > to get
>> > >> > > > started which I copied over:
>> > >> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > >> > > >
>> > >> > > > He has a test.sh script that I adjusted and used to run
his
>> > >> > > > example.
>> > >> > > > running of the script "bash test.sh" produced a CSI plot
that
>> > >> > > > was
>> > >> > > > stored
>> > >> > > > here:
>> > >> > > >
>> > >> > >
>> > >>
>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>> > >> > > >
>> > >> > > > Running the script produced the expected result.  The
next
>> > >> > > > natural
>> > >> > > > step was
>> > >> > > > to copy hochum_xml.template and create a new xml file
called
>> > >> > > > Strobach_xml.template in order to create a time series
where
>> > >> > > > the
>> > >> > > > independent time is fcst_valid_beg instead of
obs_thresh.
>> > >> > > > Accordingly,
>> > >> > > > I
>> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg) and
>> > >> > > > changed
>> > >> > > > the
>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).  The
error
>> > >> > > > that
>> > >> > > > resulted
>> > >> > > > is as follows:
>> > >> > > >
>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
13:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 01
>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01 16:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 01
>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01 19:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 01
>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01 22:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 01
>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02 01:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 02
>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02 04:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 02
>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02 07:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 02
>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02 10:00:00',
>> > >> > > > '2020-05-
>> > >> > > > 02
>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>> > >> > > > AND ld.fmean != -9999;
>> > >> > > > No data was returned from database
>> > >> > > > 2020-06-09
>> > >> > > >
>> > >> > >
>> > >>
>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>> > >> > > > Source) -1 printFormattedTable -   **  ERROR: Caught
class
>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>> > >> > > > printFormattedTable(ResultSet
>> > >> > > > res): result set contained no data
>> > >> > > > **  ERROR: Caught class
>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >> > > > in
>> > >> > > > printFormattedTable(ResultSet res): result set contained
no
>> > >> > > > data
>> > >> > > > **  ERROR: Caught class
>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >> > > > in
>> > >> > > > printFormattedTable(ResultSet res): result set contained
no
>> > >> > > > data
>> > >> > > >
>> > >> > > > Apparently there was no data during this time.  I tried
>> > >> > > > changing
>> > >> > > > the
>> > >> > > > range
>> > >> > > > of valid times and changed the valid time to different
days
>> > >> > > > just in
>> > >> > > > case
>> > >> > > > data was missing.  The issue was the same.  To confirm
my
>> > >> > > > logic, I
>> > >> > > > reproduced my attempted approach on metviewer, which I
found
>> > >> > > > to
>> > >> > > > work.
>> > >> > > >
>> > >> > > > 2.  Running in batch mode
>> > >> > > >
>> > >> > > > I think I lack an understanding of how to set up batch
mode
>> > >> > > > because
>> > >> > > > I
>> > >> > > > don't
>> > >> > > > see where these definitions are being explicitly
defined:
>> > >> > > >
>> > >> > > > <plot_spec>
>> > >> > > >     <connection>
>> > >> > > >         <host>rds_host:3306</host>
>> > >> > > >
>>  <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>> > >> > > >         <user>rds_user</user>
>> > >> > > >         <password>rds_pwd</password>
>> > >> > > >     </connection>
>> > >> > > >     <rscript>Rscript</rscript>
>> > >> > > >     <folders>
>> > >> > > >         <r_tmpl>rds_R_tmpl</r_tmpl>
>> > >> > > >         <r_work>rds_R_work</r_work>
>> > >> > > >         <plots>rds_plots</plots>
>> > >> > > >         <data>rds_data</data>
>> > >> > > >         <scripts>rds_scripts</scripts>
>> > >> > > >     </folders>
>> > >> > > >     <plot>
>> > >> > > >         <template>series_plot.R_tmpl</template>
>> > >> > > >
>> > >> > > > I see the <folders> section, but grepping all available
files
>> > >> > > > does
>> > >> > > > not
>> > >> > > > reveal how rds_R_work is being defined, as well as other
>> > >> > > > similarly
>> > >> > > > defined
>> > >> > > > variables.  I think if I know how these are being
defined,
>> > >> > > > then
>> > >> > > > I'll
>> > >> > > > be
>> > >> > > > able to pipe in my information and process the data
using met
>> > >> > > > and
>> > >> > > > produce
>> > >> > > > results in batch mode.  Am I right on that?  Also, if I
want
>> > >> > > > to use
>> > >> > > > obs for
>> > >> > > > verification, do I add another line in the XML file that
>> > >> > > > designates
>> > >> > > > file
>> > >> > > > location of obs?  I assume that other steps have to be
run
>> > >> > > > before
>> > >> > > > all
>> > >> > > > this.  If data is in prepbufr form, for example, I would
have
>> > >> > > > to
>> > >> > > > separately
>> > >> > > > run PB2NC.  If I want grid2obs, then I would want to do
that
>> > >> > > > as
>> > >> > > > well.
>> > >> > > >
>> > >> > > > 3. Communicating results to metviewer online
>> > >> > > >
>> > >> > > > Lastly, I see a lot of people/projects listed here - see
>> > >> > > > attachment
>> > >> > > >
>> > >> > > > How do I get the results I generate to be uploaded to
this
>> > >> > > > location?
>> > >> > > > It's
>> > >> > > > not clear how others are setting this up.
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >>
>> > >>
>> > >>
>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH 2020-
05-12
>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00
>> > >> 2020-05-12
>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00
>> > >> 2020-05-13
>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00
>> > >> 2020-05-13
>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00
>> > >> 2020-05-13
>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00
>> > >> 2020-05-13
>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00
>> > >> true
>> > >> median plot_20200612_173626.data plot_20200612_173626.png
>> > >> plot_20200612_173626.R test x_label test y_label
>> > >> false list() list() false false false false false false true
false
>> > >> false
>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4) c(1,1,0)
1 2
>> > >> 1.4 -2
>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3 #cccccc
1
>> > >> listX 1
>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8
1
>> > >> #333333
>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
>> > >> c(1,2) c("","") c() c() 0.04 c()
>> > >
>> > >
>> > >
>> > > --
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/
>> > > IMSG Contractor
>> > > Cubicle#: 2029
>> > > 301-683-3717
>> > >
>>
>>
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>


--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 07:13:37 2020

It seems that those options must be treated separately outside of
plot_spec.  Is it acceptable to create both a load_spec and plot_spec
in
the same xml file?  Would the definitions in load_spec be able to
carry
over into plot_spec?  It appears that data loading and plotting are
treated
separately.

On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> I do plan to use date_range; however, in an example provided by the
link I
> see this:
>
> <load_spec>
>   <connection>
>     <host>kemosabe:3306</host>
>     <database>metvdb_hwt</database>
>     <user>pgoldenb</user>
>     <password>pgoldenb</password>
>   </connection>
>
>   <date_list name="folder_dates">
>     <start>2010051914V</start>
>     <end><date_offset><day_offset>0</day_offset><hour>6
> </hour></date_offset></end>
>     <inc>3600</inc>
>     <format>yyyyMMddHH'V'</format>
>   </date_list>
>
>   <met_version>V3.0</met_version>
>
>   <verbose>false</verbose>
>   <insert_size>1</insert_size>
>   <mode_header_db_check>true</mode_header_db_check>
>   <drop_indexes>false</drop_indexes>
>   <apply_indexes>true</apply_indexes>
>   <group>Group name</group>
>   <load_stat>true</load_stat>
>   <load_mode>true</load_mode>
>   <load_mtd>true</load_mtd>
>   <load_mpr>false</load_mpr>
>
>   <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>   <load_val>
>     <field name="model">
>       <val>arw</val>
>       <val>nmm</val>
>     </field>
>
>     <field name="valid_time">
>       <date_list name="folder_dates"/>
>     </field>
>
>     <field name="vx_mask">
>       <val>FULL</val>
>       <val>SWC</val>
>     </field>
>   </load_val>
> </load_spec>
>
> Are similar settings allowed with <plot_spec>?  I see <connection>,
which
> seems carried over.  I'm looking to specify my folder_tmpl in
<plot_spec>,
> which according to Tara, should be possible.  Since my directory
structure
> looks like this:
>
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>
> I would do something like this I would think:
>
> .......
>
>   <date_list name="folder_dates">
>     <start>20200612/start>
>     <end><date_offset><day_offset>0</day_offset></end>
>     <inc>86400</inc>
>     <format>yyyyMMdd</format>
>   </date_list>
>
>   <folder_tmpl>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> </folder_tmpl>
>   <load_val>
>     <field name="model">
>       <val>prod</val>
>     </field>
>
>     <field name="valid_time">
>       <date_list name="folder_dates"/>
>     </field>
>
> ......
>
> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> Thank you.  This is a very helpful example
>>
>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
>> wrote:
>>
>>> Hi,
>>> Here is some examples how to use <date_range> and
<date_range_list> in
>>> batch XML.
>>> First, you need to defined your range. You can have more then one
>>> <date_range>:
>>> <date_range name="may">
>>>         <start>2019-05-01 00:00:00</start>
>>>         <end>2019-05-31 23:59:59</end>
>>> </date_range>
>>> <date_range name="march">
>>>         <start>2019-03-01 00:00:00</start>
>>>         <end>2019-03-31 23:59:59</end>
>>> </date_range>
>>>
>>> and then use one or all date_ranges:
>>> <plot_fix>
>>>     <field name="fcst_init_beg">
>>>         <date_range name="march"/>
>>>         <date_range name="may"/>
>>>    </field>...
>>> This configuration would include all dates for March and May of
2019.
>>>
>>> If you need more flexible configuration you can use
date_range_list.
>>> For example, you want to select first 6 hours every day beetween
>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
>>> <date_range_list name = 'all'>
>>>        <range_start>2020-01-01 00:00:00</range_start>
>>>         <range_end>2020-01-05 00:00:00</range_end>
>>>         <range_length>360</range_length>
>>>         <inc>1440</inc>
>>> </date_range_list>
>>> where the 1st and 2nd values are the start and end,
>>> range_length - the length of a range in minutes(6 houres)
>>> inc - lenght beetween ranges in minutes (24 houres)
>>> This example would create following ranges:
>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>>>
>>> Also you can add more parameters to <plot_fix> to filter your
data. For
>>> example, if you want to select only dates with 00:00 you can
specify
>>>  <field name="valid_hour">
>>>         <val>00</val>
>>>  </field>
>>>
>>> Tatiana
>>>
>>>
>>>
>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
>>> > and if there isn't a better way, I wonder the whereabouts of a
>>> > separate
>>> > script that pipes dates (and other relevant information) into
the XML
>>> > file
>>> > used to generate the results.
>>> >
>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
Affiliate <
>>> > edward.strobach at noaa.gov> wrote:
>>> >
>>> > > Thanks Tatiana.  These resources are helpful.
>>> > >
>>> > > One last question.  Simply typing lines like this in can take
>>> > > forever:
>>> > >
>>> > > <plot_fix>
>>> > >     <field equalize="false" name="fcst_init_beg">
>>> > >         <set name="fcst_init_beg_0">
>>> > >             <val>2019-05-01 12:00:00</val>
>>> > >             <val>2019-05-02 06:00:00</val>
>>> > >             <val>2019-05-02 12:00:00</val>
>>> > >             <val>2019-05-03 06:00:00</val>
>>> > >             <val>2019-05-03 12:00:00</val>
>>> > >
>>> > >
>>> > > Is there a better way?   I see date_range, date_range_list, or
>>> > > date_list.
>>> > > Can this be used instead of typing individual values?  This
could
>>> > > prove
>>> > > pretty daunting if one would have to type in 700 entries each
time.
>>> > >
>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>>> > > <met_help at ucar.edu>
>>> > > wrote:
>>> > >
>>> > >> I found a few problems in your XML.
>>> > >> The database you are using doesn't have data with vx_mask
>>> > >> 'abcREGabc'.
>>> > >> That is why METviewer returns an empty dataset and can't
create a
>>> > >> plot.
>>> > >> The other problem is how you define fcst_lead. This database
has
>>> > >> values
>>> > >> like 10000, 20000 and not 010000, 020000. Since you include
all
>>> > >> fcst_leads
>>> > >> this variable can be omitted.
>>> > >> To minimize problems with variables values, I would suggest
to
>>> > >> create XML
>>> > >> using GUI and then edit it to fit batch standards.
>>> > >> I created a plot using you configurations with GUI (without
vx_mack
>>> > >> and
>>> > >> plot and axis titles) #20200612_173626. You can load it back
to the
>>> > >> page by
>>> > >> using the "History" tab (left bar on GUI). Find the plot,
open XML
>>> > >> and
>>> > >> click on "Load to page". After that you can recreate a plot
and edit
>>> > >> the
>>> > >> configurations. You can also download a new XML and edit it
for
>>> > >> batch.
>>> > >> I attached the example of load XML to this message.
>>> > >>
>>> > >> Description of rds parameters:
>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>>> > >> rds_plots - temporary directory where the plots will be
generated
>>> > >> rds_data - temporary directory where the data retrieved from
the
>>> > >> database
>>> > >> and other intermediate files will be stored
>>> > >> rds_scripts - temporary directory where R scripts for this
run will
>>> > >> be
>>> > >> stored
>>> > >> After the execution is done all the files from temporary
directories
>>> > >> would be removed.
>>> > >>
>>> > >>
>>> > >> We don't have the offitial user guide for METviewer. There
are some
>>> > >> instructions on how to crate XML config files:
>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>> > >>
>>> > >> Tatiana
>>> > >>
>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
>>> > >> > Hi,
>>> > >> >
>>> > >> > I'm attaching the xml below, but am having hard time
following how
>>> > >> > to
>>> > >> > go
>>> > >> > about the other steps since I have zero experience on how
to do
>>> > >> > these
>>> > >> > things.
>>> > >> >
>>> > >> > #2. All parameters in <folders> section should NOT be
changed and
>>> > >> > will
>>> > >> > be
>>> > >> > replaced by other scripts during the batch execution. The
data for
>>> > >> > the
>>> > >> > batch mode is coming from the database and not from files.
The
>>> > >> > name of
>>> > >> > the
>>> > >> > database is in <database> tag.
>>> > >> >
>>> > >> > I understand that nothing should be changed in the folders
>>> > >> > section,
>>> > >> > but
>>> > >> > there is no indication how and where these variables are
being
>>> > >> > defined.
>>> > >> > Grepping these names of the files available does not reveal
formal
>>> > >> > definitions of these variables.  If I understood exactly
how these
>>> > >> > were
>>> > >> > being defined, then I would be able connect this better.
>>> > >> >
>>> > >> > #3. Users rum MET and produce MET output files. They use
METviewer
>>> > >> > load
>>> > >> > mode to load these files to the specific database. In the
config
>>> > >> > file
>>> > >> > for
>>> > >> > this process they also can specify the database group. For
example
>>> > >> > database
>>> > >> > mv_dev is in Test12 group.
>>> > >> > After the data has been loaded to the database it is
visible and
>>> > >> > available
>>> > >> > in the GUI and also can be used by the batch mode.
>>> > >> > You can always add more data to the database using the same
load
>>> > >> > mode.
>>> > >> >
>>> > >> > Is there a guide for me to understand the process that you
are
>>> > >> > referring to?
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>>> > >> > <met_help at ucar.edu>
>>> > >> > wrote:
>>> > >> >
>>> > >> > > Hi Edward,
>>> > >> > > #1. I don't have an access to NOAA servers and can't view
your
>>> > >> > > XML
>>> > >> > > for the
>>> > >> > > batch mode. Please, attache it to this issue and I will
take a
>>> > >> > > look.
>>> > >> > >
>>> > >> > > #2. All parameters in <folders> section should NOT be
changed
>>> > >> > > and
>>> > >> > > will be
>>> > >> > > replaced by other scripts during the batch execution. The
data
>>> > >> > > for
>>> > >> > > the
>>> > >> > > batch mode is coming from the database and not from
files. The
>>> > >> > > name
>>> > >> > > of the
>>> > >> > > database is in <database> tag.
>>> > >> > >
>>> > >> > > #3. Users rum MET and produce MET output files. They use
>>> > >> > > METviewer
>>> > >> > > load
>>> > >> > > mode to load these files to the specific database. In the
config
>>> > >> > > file
>>> > >> > > for
>>> > >> > > this process they also can specify the database group.
For
>>> > >> > > example
>>> > >> > > database
>>> > >> > > mv_dev is in Test12 group.
>>> > >> > > After the data has been loaded to the database it is
visible and
>>> > >> > > available
>>> > >> > > in the GUI and also can be used by the batch mode.
>>> > >> > > You can always add more data to the database using the
same load
>>> > >> > > mode.
>>> > >> > >
>>> > >> > > Tatiana
>>> > >> > >
>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
>>> > >> > > > Good morning,
>>> > >> > > >
>>> > >> > > > Over the last two weeks I've made an effort to
understand
>>> > >> > > > metviewer,
>>> > >> > > > test
>>> > >> > > > different cases using the history tab, changing
information to
>>> > >> > > > see
>>> > >> > > > what
>>> > >> > > > works and what does not, and make connections between
the GUI
>>> > >> > > > settings
>>> > >> > > > and
>>> > >> > > > the XML file.  Although my comfort level with metviewer
has
>>> > >> > > > significantly
>>> > >> > > > increased during this time, there are some remaining
issues
>>> > >> > > > that
>>> > >> > > > I'm
>>> > >> > > > hoping
>>> > >> > > > to clear up.
>>> > >> > > >
>>> > >> > > > 1.  Problem changing XML file and running with
different
>>> > >> > > > statistic
>>> > >> > > > and
>>> > >> > > > plot
>>> > >> > > > type:
>>> > >> > > >
>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
>>> > >> > > > directory
>>> > >> > > > to get
>>> > >> > > > started which I copied over:
>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>> > >> > > >
>>> > >> > > > He has a test.sh script that I adjusted and used to run
his
>>> > >> > > > example.
>>> > >> > > > running of the script "bash test.sh" produced a CSI
plot that
>>> > >> > > > was
>>> > >> > > > stored
>>> > >> > > > here:
>>> > >> > > >
>>> > >> > >
>>> > >>
>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>> > >> > > >
>>> > >> > > > Running the script produced the expected result.  The
next
>>> > >> > > > natural
>>> > >> > > > step was
>>> > >> > > > to copy hochum_xml.template and create a new xml file
called
>>> > >> > > > Strobach_xml.template in order to create a time series
where
>>> > >> > > > the
>>> > >> > > > independent time is fcst_valid_beg instead of
obs_thresh.
>>> > >> > > > Accordingly,
>>> > >> > > > I
>>> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg) and
>>> > >> > > > changed
>>> > >> > > > the
>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).  The
error
>>> > >> > > > that
>>> > >> > > > resulted
>>> > >> > > > is as follows:
>>> > >> > > >
>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
13:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 01
>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
16:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 01
>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
19:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 01
>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
22:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 01
>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
01:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 02
>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
04:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 02
>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
07:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 02
>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
10:00:00',
>>> > >> > > > '2020-05-
>>> > >> > > > 02
>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>> > >> > > > AND ld.fmean != -9999;
>>> > >> > > > No data was returned from database
>>> > >> > > > 2020-06-09
>>> > >> > > >
>>> > >> > >
>>> > >>
>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR: Caught
class
>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>>> > >> > > > printFormattedTable(ResultSet
>>> > >> > > > res): result set contained no data
>>> > >> > > > **  ERROR: Caught class
>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>> > >> > > > in
>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>> > >> > > > data
>>> > >> > > > **  ERROR: Caught class
>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>> > >> > > > in
>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>> > >> > > > data
>>> > >> > > >
>>> > >> > > > Apparently there was no data during this time.  I tried
>>> > >> > > > changing
>>> > >> > > > the
>>> > >> > > > range
>>> > >> > > > of valid times and changed the valid time to different
days
>>> > >> > > > just in
>>> > >> > > > case
>>> > >> > > > data was missing.  The issue was the same.  To confirm
my
>>> > >> > > > logic, I
>>> > >> > > > reproduced my attempted approach on metviewer, which I
found
>>> > >> > > > to
>>> > >> > > > work.
>>> > >> > > >
>>> > >> > > > 2.  Running in batch mode
>>> > >> > > >
>>> > >> > > > I think I lack an understanding of how to set up batch
mode
>>> > >> > > > because
>>> > >> > > > I
>>> > >> > > > don't
>>> > >> > > > see where these definitions are being explicitly
defined:
>>> > >> > > >
>>> > >> > > > <plot_spec>
>>> > >> > > >     <connection>
>>> > >> > > >         <host>rds_host:3306</host>
>>> > >> > > >
>>>  <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>> > >> > > >         <user>rds_user</user>
>>> > >> > > >         <password>rds_pwd</password>
>>> > >> > > >     </connection>
>>> > >> > > >     <rscript>Rscript</rscript>
>>> > >> > > >     <folders>
>>> > >> > > >         <r_tmpl>rds_R_tmpl</r_tmpl>
>>> > >> > > >         <r_work>rds_R_work</r_work>
>>> > >> > > >         <plots>rds_plots</plots>
>>> > >> > > >         <data>rds_data</data>
>>> > >> > > >         <scripts>rds_scripts</scripts>
>>> > >> > > >     </folders>
>>> > >> > > >     <plot>
>>> > >> > > >         <template>series_plot.R_tmpl</template>
>>> > >> > > >
>>> > >> > > > I see the <folders> section, but grepping all available
files
>>> > >> > > > does
>>> > >> > > > not
>>> > >> > > > reveal how rds_R_work is being defined, as well as
other
>>> > >> > > > similarly
>>> > >> > > > defined
>>> > >> > > > variables.  I think if I know how these are being
defined,
>>> > >> > > > then
>>> > >> > > > I'll
>>> > >> > > > be
>>> > >> > > > able to pipe in my information and process the data
using met
>>> > >> > > > and
>>> > >> > > > produce
>>> > >> > > > results in batch mode.  Am I right on that?  Also, if I
want
>>> > >> > > > to use
>>> > >> > > > obs for
>>> > >> > > > verification, do I add another line in the XML file
that
>>> > >> > > > designates
>>> > >> > > > file
>>> > >> > > > location of obs?  I assume that other steps have to be
run
>>> > >> > > > before
>>> > >> > > > all
>>> > >> > > > this.  If data is in prepbufr form, for example, I
would have
>>> > >> > > > to
>>> > >> > > > separately
>>> > >> > > > run PB2NC.  If I want grid2obs, then I would want to do
that
>>> > >> > > > as
>>> > >> > > > well.
>>> > >> > > >
>>> > >> > > > 3. Communicating results to metviewer online
>>> > >> > > >
>>> > >> > > > Lastly, I see a lot of people/projects listed here -
see
>>> > >> > > > attachment
>>> > >> > > >
>>> > >> > > > How do I get the results I generate to be uploaded to
this
>>> > >> > > > location?
>>> > >> > > > It's
>>> > >> > > > not clear how others are setting this up.
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >>
>>> > >>
>>> > >>
>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
2020-05-12
>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
20:00:00
>>> > >> 2020-05-12
>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
10:00:00
>>> > >> 2020-05-13
>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
14:00:00
>>> > >> 2020-05-13
>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
18:00:00
>>> > >> 2020-05-13
>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
22:00:00
>>> > >> 2020-05-13
>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
12:00:00
>>> > >> true
>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
>>> > >> plot_20200612_173626.R test x_label test y_label
>>> > >> false list() list() false false false false false false true
false
>>> > >> false
>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
c(1,1,0) 1 2
>>> > >> 1.4 -2
>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
#cccccc 1
>>> > >> listX 1
>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3 0.8
1
>>> > >> #333333
>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Edward Strobach
>>> > > EMC/NCEP/NWS/
>>> > > IMSG Contractor
>>> > > Cubicle#: 2029
>>> > > 301-683-3717
>>> > >
>>>
>>>
>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>


--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 07:35:03 2020

Actually, it appears that I need to run the database loading module
first
if I want to change/add a data source.  From there I can run the
plotting
batch module.  Is that right?   So if I run the database loading
module,
then I can specify my dates and other related info specific to the
dataset.  If I create my database with a unique name, then it should
be
recognized on the metviewer gui, presumably.

On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> It seems that those options must be treated separately outside of
> plot_spec.  Is it acceptable to create both a load_spec and
plot_spec in
> the same xml file?  Would the definitions in load_spec be able to
carry
> over into plot_spec?  It appears that data loading and plotting are
treated
> separately.
>
> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> I do plan to use date_range; however, in an example provided by the
link
>> I see this:
>>
>> <load_spec>
>>   <connection>
>>     <host>kemosabe:3306</host>
>>     <database>metvdb_hwt</database>
>>     <user>pgoldenb</user>
>>     <password>pgoldenb</password>
>>   </connection>
>>
>>   <date_list name="folder_dates">
>>     <start>2010051914V</start>
>>     <end><date_offset><day_offset>0</day_offset><hour>6
>> </hour></date_offset></end>
>>     <inc>3600</inc>
>>     <format>yyyyMMddHH'V'</format>
>>   </date_list>
>>
>>   <met_version>V3.0</met_version>
>>
>>   <verbose>false</verbose>
>>   <insert_size>1</insert_size>
>>   <mode_header_db_check>true</mode_header_db_check>
>>   <drop_indexes>false</drop_indexes>
>>   <apply_indexes>true</apply_indexes>
>>   <group>Group name</group>
>>   <load_stat>true</load_stat>
>>   <load_mode>true</load_mode>
>>   <load_mtd>true</load_mtd>
>>   <load_mpr>false</load_mpr>
>>
>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>   <load_val>
>>     <field name="model">
>>       <val>arw</val>
>>       <val>nmm</val>
>>     </field>
>>
>>     <field name="valid_time">
>>       <date_list name="folder_dates"/>
>>     </field>
>>
>>     <field name="vx_mask">
>>       <val>FULL</val>
>>       <val>SWC</val>
>>     </field>
>>   </load_val>
>> </load_spec>
>>
>> Are similar settings allowed with <plot_spec>?  I see <connection>,
which
>> seems carried over.  I'm looking to specify my folder_tmpl in
<plot_spec>,
>> which according to Tara, should be possible.  Since my directory
structure
>> looks like this:
>>
>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>
>> I would do something like this I would think:
>>
>> .......
>>
>>   <date_list name="folder_dates">
>>     <start>20200612/start>
>>     <end><date_offset><day_offset>0</day_offset></end>
>>     <inc>86400</inc>
>>     <format>yyyyMMdd</format>
>>   </date_list>
>>
>>   <folder_tmpl>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>> </folder_tmpl>
>>   <load_val>
>>     <field name="model">
>>       <val>prod</val>
>>     </field>
>>
>>     <field name="valid_time">
>>       <date_list name="folder_dates"/>
>>     </field>
>>
>> ......
>>
>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> Thank you.  This is a very helpful example
>>>
>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
>>> wrote:
>>>
>>>> Hi,
>>>> Here is some examples how to use <date_range> and
<date_range_list> in
>>>> batch XML.
>>>> First, you need to defined your range. You can have more then one
>>>> <date_range>:
>>>> <date_range name="may">
>>>>         <start>2019-05-01 00:00:00</start>
>>>>         <end>2019-05-31 23:59:59</end>
>>>> </date_range>
>>>> <date_range name="march">
>>>>         <start>2019-03-01 00:00:00</start>
>>>>         <end>2019-03-31 23:59:59</end>
>>>> </date_range>
>>>>
>>>> and then use one or all date_ranges:
>>>> <plot_fix>
>>>>     <field name="fcst_init_beg">
>>>>         <date_range name="march"/>
>>>>         <date_range name="may"/>
>>>>    </field>...
>>>> This configuration would include all dates for March and May of
2019.
>>>>
>>>> If you need more flexible configuration you can use
date_range_list.
>>>> For example, you want to select first 6 hours every day beetween
>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
>>>> <date_range_list name = 'all'>
>>>>        <range_start>2020-01-01 00:00:00</range_start>
>>>>         <range_end>2020-01-05 00:00:00</range_end>
>>>>         <range_length>360</range_length>
>>>>         <inc>1440</inc>
>>>> </date_range_list>
>>>> where the 1st and 2nd values are the start and end,
>>>> range_length - the length of a range in minutes(6 houres)
>>>> inc - lenght beetween ranges in minutes (24 houres)
>>>> This example would create following ranges:
>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>>>>
>>>> Also you can add more parameters to <plot_fix> to filter your
data. For
>>>> example, if you want to select only dates with 00:00 you can
specify
>>>>  <field name="valid_hour">
>>>>         <val>00</val>
>>>>  </field>
>>>>
>>>> Tatiana
>>>>
>>>>
>>>>
>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
>>>> > and if there isn't a better way, I wonder the whereabouts of a
>>>> > separate
>>>> > script that pipes dates (and other relevant information) into
the XML
>>>> > file
>>>> > used to generate the results.
>>>> >
>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
Affiliate <
>>>> > edward.strobach at noaa.gov> wrote:
>>>> >
>>>> > > Thanks Tatiana.  These resources are helpful.
>>>> > >
>>>> > > One last question.  Simply typing lines like this in can take
>>>> > > forever:
>>>> > >
>>>> > > <plot_fix>
>>>> > >     <field equalize="false" name="fcst_init_beg">
>>>> > >         <set name="fcst_init_beg_0">
>>>> > >             <val>2019-05-01 12:00:00</val>
>>>> > >             <val>2019-05-02 06:00:00</val>
>>>> > >             <val>2019-05-02 12:00:00</val>
>>>> > >             <val>2019-05-03 06:00:00</val>
>>>> > >             <val>2019-05-03 12:00:00</val>
>>>> > >
>>>> > >
>>>> > > Is there a better way?   I see date_range, date_range_list,
or
>>>> > > date_list.
>>>> > > Can this be used instead of typing individual values?  This
could
>>>> > > prove
>>>> > > pretty daunting if one would have to type in 700 entries each
time.
>>>> > >
>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>>>> > > <met_help at ucar.edu>
>>>> > > wrote:
>>>> > >
>>>> > >> I found a few problems in your XML.
>>>> > >> The database you are using doesn't have data with vx_mask
>>>> > >> 'abcREGabc'.
>>>> > >> That is why METviewer returns an empty dataset and can't
create a
>>>> > >> plot.
>>>> > >> The other problem is how you define fcst_lead. This database
has
>>>> > >> values
>>>> > >> like 10000, 20000 and not 010000, 020000. Since you include
all
>>>> > >> fcst_leads
>>>> > >> this variable can be omitted.
>>>> > >> To minimize problems with variables values, I would suggest
to
>>>> > >> create XML
>>>> > >> using GUI and then edit it to fit batch standards.
>>>> > >> I created a plot using you configurations with GUI (without
vx_mack
>>>> > >> and
>>>> > >> plot and axis titles) #20200612_173626. You can load it back
to the
>>>> > >> page by
>>>> > >> using the "History" tab (left bar on GUI). Find the plot,
open XML
>>>> > >> and
>>>> > >> click on "Load to page". After that you can recreate a plot
and
>>>> edit
>>>> > >> the
>>>> > >> configurations. You can also download a new XML and edit it
for
>>>> > >> batch.
>>>> > >> I attached the example of load XML to this message.
>>>> > >>
>>>> > >> Description of rds parameters:
>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>>>> > >> rds_plots - temporary directory where the plots will be
generated
>>>> > >> rds_data - temporary directory where the data retrieved from
the
>>>> > >> database
>>>> > >> and other intermediate files will be stored
>>>> > >> rds_scripts - temporary directory where R scripts for this
run will
>>>> > >> be
>>>> > >> stored
>>>> > >> After the execution is done all the files from temporary
>>>> directories
>>>> > >> would be removed.
>>>> > >>
>>>> > >>
>>>> > >> We don't have the offitial user guide for METviewer. There
are some
>>>> > >> instructions on how to crate XML config files:
>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>>> > >>
>>>> > >> Tatiana
>>>> > >>
>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov wrote:
>>>> > >> > Hi,
>>>> > >> >
>>>> > >> > I'm attaching the xml below, but am having hard time
following
>>>> how
>>>> > >> > to
>>>> > >> > go
>>>> > >> > about the other steps since I have zero experience on how
to do
>>>> > >> > these
>>>> > >> > things.
>>>> > >> >
>>>> > >> > #2. All parameters in <folders> section should NOT be
changed and
>>>> > >> > will
>>>> > >> > be
>>>> > >> > replaced by other scripts during the batch execution. The
data
>>>> for
>>>> > >> > the
>>>> > >> > batch mode is coming from the database and not from files.
The
>>>> > >> > name of
>>>> > >> > the
>>>> > >> > database is in <database> tag.
>>>> > >> >
>>>> > >> > I understand that nothing should be changed in the folders
>>>> > >> > section,
>>>> > >> > but
>>>> > >> > there is no indication how and where these variables are
being
>>>> > >> > defined.
>>>> > >> > Grepping these names of the files available does not
reveal
>>>> formal
>>>> > >> > definitions of these variables.  If I understood exactly
how
>>>> these
>>>> > >> > were
>>>> > >> > being defined, then I would be able connect this better.
>>>> > >> >
>>>> > >> > #3. Users rum MET and produce MET output files. They use
>>>> METviewer
>>>> > >> > load
>>>> > >> > mode to load these files to the specific database. In the
config
>>>> > >> > file
>>>> > >> > for
>>>> > >> > this process they also can specify the database group. For
>>>> example
>>>> > >> > database
>>>> > >> > mv_dev is in Test12 group.
>>>> > >> > After the data has been loaded to the database it is
visible and
>>>> > >> > available
>>>> > >> > in the GUI and also can be used by the batch mode.
>>>> > >> > You can always add more data to the database using the
same load
>>>> > >> > mode.
>>>> > >> >
>>>> > >> > Is there a guide for me to understand the process that you
are
>>>> > >> > referring to?
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>>>> > >> > <met_help at ucar.edu>
>>>> > >> > wrote:
>>>> > >> >
>>>> > >> > > Hi Edward,
>>>> > >> > > #1. I don't have an access to NOAA servers and can't
view your
>>>> > >> > > XML
>>>> > >> > > for the
>>>> > >> > > batch mode. Please, attache it to this issue and I will
take a
>>>> > >> > > look.
>>>> > >> > >
>>>> > >> > > #2. All parameters in <folders> section should NOT be
changed
>>>> > >> > > and
>>>> > >> > > will be
>>>> > >> > > replaced by other scripts during the batch execution.
The data
>>>> > >> > > for
>>>> > >> > > the
>>>> > >> > > batch mode is coming from the database and not from
files. The
>>>> > >> > > name
>>>> > >> > > of the
>>>> > >> > > database is in <database> tag.
>>>> > >> > >
>>>> > >> > > #3. Users rum MET and produce MET output files. They use
>>>> > >> > > METviewer
>>>> > >> > > load
>>>> > >> > > mode to load these files to the specific database. In
the
>>>> config
>>>> > >> > > file
>>>> > >> > > for
>>>> > >> > > this process they also can specify the database group.
For
>>>> > >> > > example
>>>> > >> > > database
>>>> > >> > > mv_dev is in Test12 group.
>>>> > >> > > After the data has been loaded to the database it is
visible
>>>> and
>>>> > >> > > available
>>>> > >> > > in the GUI and also can be used by the batch mode.
>>>> > >> > > You can always add more data to the database using the
same
>>>> load
>>>> > >> > > mode.
>>>> > >> > >
>>>> > >> > > Tatiana
>>>> > >> > >
>>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
>>>> > >> > > > Good morning,
>>>> > >> > > >
>>>> > >> > > > Over the last two weeks I've made an effort to
understand
>>>> > >> > > > metviewer,
>>>> > >> > > > test
>>>> > >> > > > different cases using the history tab, changing
information
>>>> to
>>>> > >> > > > see
>>>> > >> > > > what
>>>> > >> > > > works and what does not, and make connections between
the GUI
>>>> > >> > > > settings
>>>> > >> > > > and
>>>> > >> > > > the XML file.  Although my comfort level with
metviewer has
>>>> > >> > > > significantly
>>>> > >> > > > increased during this time, there are some remaining
issues
>>>> > >> > > > that
>>>> > >> > > > I'm
>>>> > >> > > > hoping
>>>> > >> > > > to clear up.
>>>> > >> > > >
>>>> > >> > > > 1.  Problem changing XML file and running with
different
>>>> > >> > > > statistic
>>>> > >> > > > and
>>>> > >> > > > plot
>>>> > >> > > > type:
>>>> > >> > > >
>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
>>>> > >> > > > directory
>>>> > >> > > > to get
>>>> > >> > > > started which I copied over:
>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>> > >> > > >
>>>> > >> > > > He has a test.sh script that I adjusted and used to
run his
>>>> > >> > > > example.
>>>> > >> > > > running of the script "bash test.sh" produced a CSI
plot that
>>>> > >> > > > was
>>>> > >> > > > stored
>>>> > >> > > > here:
>>>> > >> > > >
>>>> > >> > >
>>>> > >>
>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>> > >> > > >
>>>> > >> > > > Running the script produced the expected result.  The
next
>>>> > >> > > > natural
>>>> > >> > > > step was
>>>> > >> > > > to copy hochum_xml.template and create a new xml file
called
>>>> > >> > > > Strobach_xml.template in order to create a time series
where
>>>> > >> > > > the
>>>> > >> > > > independent time is fcst_valid_beg instead of
obs_thresh.
>>>> > >> > > > Accordingly,
>>>> > >> > > > I
>>>> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg)
>>>> and
>>>> > >> > > > changed
>>>> > >> > > > the
>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).
The error
>>>> > >> > > > that
>>>> > >> > > > resulted
>>>> > >> > > > is as follows:
>>>> > >> > > >
>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
13:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 01
>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
16:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 01
>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
19:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 01
>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
22:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 01
>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
01:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 02
>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
04:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 02
>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
07:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 02
>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
10:00:00',
>>>> > >> > > > '2020-05-
>>>> > >> > > > 02
>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>>> > >> > > > AND ld.fmean != -9999;
>>>> > >> > > > No data was returned from database
>>>> > >> > > > 2020-06-09
>>>> > >> > > >
>>>> > >> > >
>>>> > >>
>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR: Caught
class
>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>>>> > >> > > > printFormattedTable(ResultSet
>>>> > >> > > > res): result set contained no data
>>>> > >> > > > **  ERROR: Caught class
>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>>> > >> > > > in
>>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>>> > >> > > > data
>>>> > >> > > > **  ERROR: Caught class
>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>>> > >> > > > in
>>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>>> > >> > > > data
>>>> > >> > > >
>>>> > >> > > > Apparently there was no data during this time.  I
tried
>>>> > >> > > > changing
>>>> > >> > > > the
>>>> > >> > > > range
>>>> > >> > > > of valid times and changed the valid time to different
days
>>>> > >> > > > just in
>>>> > >> > > > case
>>>> > >> > > > data was missing.  The issue was the same.  To confirm
my
>>>> > >> > > > logic, I
>>>> > >> > > > reproduced my attempted approach on metviewer, which I
found
>>>> > >> > > > to
>>>> > >> > > > work.
>>>> > >> > > >
>>>> > >> > > > 2.  Running in batch mode
>>>> > >> > > >
>>>> > >> > > > I think I lack an understanding of how to set up batch
mode
>>>> > >> > > > because
>>>> > >> > > > I
>>>> > >> > > > don't
>>>> > >> > > > see where these definitions are being explicitly
defined:
>>>> > >> > > >
>>>> > >> > > > <plot_spec>
>>>> > >> > > >     <connection>
>>>> > >> > > >         <host>rds_host:3306</host>
>>>> > >> > > >
>>>>  <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>> > >> > > >         <user>rds_user</user>
>>>> > >> > > >         <password>rds_pwd</password>
>>>> > >> > > >     </connection>
>>>> > >> > > >     <rscript>Rscript</rscript>
>>>> > >> > > >     <folders>
>>>> > >> > > >         <r_tmpl>rds_R_tmpl</r_tmpl>
>>>> > >> > > >         <r_work>rds_R_work</r_work>
>>>> > >> > > >         <plots>rds_plots</plots>
>>>> > >> > > >         <data>rds_data</data>
>>>> > >> > > >         <scripts>rds_scripts</scripts>
>>>> > >> > > >     </folders>
>>>> > >> > > >     <plot>
>>>> > >> > > >         <template>series_plot.R_tmpl</template>
>>>> > >> > > >
>>>> > >> > > > I see the <folders> section, but grepping all
available files
>>>> > >> > > > does
>>>> > >> > > > not
>>>> > >> > > > reveal how rds_R_work is being defined, as well as
other
>>>> > >> > > > similarly
>>>> > >> > > > defined
>>>> > >> > > > variables.  I think if I know how these are being
defined,
>>>> > >> > > > then
>>>> > >> > > > I'll
>>>> > >> > > > be
>>>> > >> > > > able to pipe in my information and process the data
using met
>>>> > >> > > > and
>>>> > >> > > > produce
>>>> > >> > > > results in batch mode.  Am I right on that?  Also, if
I want
>>>> > >> > > > to use
>>>> > >> > > > obs for
>>>> > >> > > > verification, do I add another line in the XML file
that
>>>> > >> > > > designates
>>>> > >> > > > file
>>>> > >> > > > location of obs?  I assume that other steps have to be
run
>>>> > >> > > > before
>>>> > >> > > > all
>>>> > >> > > > this.  If data is in prepbufr form, for example, I
would have
>>>> > >> > > > to
>>>> > >> > > > separately
>>>> > >> > > > run PB2NC.  If I want grid2obs, then I would want to
do that
>>>> > >> > > > as
>>>> > >> > > > well.
>>>> > >> > > >
>>>> > >> > > > 3. Communicating results to metviewer online
>>>> > >> > > >
>>>> > >> > > > Lastly, I see a lot of people/projects listed here -
see
>>>> > >> > > > attachment
>>>> > >> > > >
>>>> > >> > > > How do I get the results I generate to be uploaded to
this
>>>> > >> > > > location?
>>>> > >> > > > It's
>>>> > >> > > > not clear how others are setting this up.
>>>> > >> > >
>>>> > >> > >
>>>> > >> > >
>>>> > >> > >
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
rds_pwd
>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
rds_scripts
>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
2020-05-12
>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
>>>> 20:00:00
>>>> > >> 2020-05-12
>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
>>>> 10:00:00
>>>> > >> 2020-05-13
>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
>>>> 14:00:00
>>>> > >> 2020-05-13
>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
>>>> 18:00:00
>>>> > >> 2020-05-13
>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
>>>> 22:00:00
>>>> > >> 2020-05-13
>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
>>>> 12:00:00
>>>> > >> true
>>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
>>>> > >> plot_20200612_173626.R test x_label test y_label
>>>> > >> false list() list() false false false false false false true
false
>>>> > >> false
>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
c(1,1,0) 1 2
>>>> > >> 1.4 -2
>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
#cccccc 1
>>>> > >> listX 1
>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3
0.8 1
>>>> > >> #333333
>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1) c(1,1)
c(1,1)
>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Edward Strobach
>>>> > > EMC/NCEP/NWS/
>>>> > > IMSG Contractor
>>>> > > Cubicle#: 2029
>>>> > > 301-683-3717
>>>> > >
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>


--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 08:26:32 2020

lastly, it does seem thatmysql -u[db_username] -p[db_password]
-e'create
database [db_name]

would need to be used to create a database.  I assume that an sql
module is
available on wcoss; I know you don't have access to those machines
Tatiana.  If I create my own database with my own name, then I should
be
able to proceed with project specific tasks under that tab. Something
like
mv_EdwardStrobach.

I can then create mv_cmaq_prod_v_para which I add data to using
load_spec.
Once that step is done, then I can run plot_spec.  That's my takeaway
currently

On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> Actually, it appears that I need to run the database loading module
first
> if I want to change/add a data source.  From there I can run the
plotting
> batch module.  Is that right?   So if I run the database loading
module,
> then I can specify my dates and other related info specific to the
> dataset.  If I create my database with a unique name, then it should
be
> recognized on the metviewer gui, presumably.
>
> On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> It seems that those options must be treated separately outside of
>> plot_spec.  Is it acceptable to create both a load_spec and
plot_spec in
>> the same xml file?  Would the definitions in load_spec be able to
carry
>> over into plot_spec?  It appears that data loading and plotting are
treated
>> separately.
>>
>> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> I do plan to use date_range; however, in an example provided by
the link
>>> I see this:
>>>
>>> <load_spec>
>>>   <connection>
>>>     <host>kemosabe:3306</host>
>>>     <database>metvdb_hwt</database>
>>>     <user>pgoldenb</user>
>>>     <password>pgoldenb</password>
>>>   </connection>
>>>
>>>   <date_list name="folder_dates">
>>>     <start>2010051914V</start>
>>>     <end><date_offset><day_offset>0</day_offset><hour>6
>>> </hour></date_offset></end>
>>>     <inc>3600</inc>
>>>     <format>yyyyMMddHH'V'</format>
>>>   </date_list>
>>>
>>>   <met_version>V3.0</met_version>
>>>
>>>   <verbose>false</verbose>
>>>   <insert_size>1</insert_size>
>>>   <mode_header_db_check>true</mode_header_db_check>
>>>   <drop_indexes>false</drop_indexes>
>>>   <apply_indexes>true</apply_indexes>
>>>   <group>Group name</group>
>>>   <load_stat>true</load_stat>
>>>   <load_mode>true</load_mode>
>>>   <load_mtd>true</load_mtd>
>>>   <load_mpr>false</load_mpr>
>>>
>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>   <load_val>
>>>     <field name="model">
>>>       <val>arw</val>
>>>       <val>nmm</val>
>>>     </field>
>>>
>>>     <field name="valid_time">
>>>       <date_list name="folder_dates"/>
>>>     </field>
>>>
>>>     <field name="vx_mask">
>>>       <val>FULL</val>
>>>       <val>SWC</val>
>>>     </field>
>>>   </load_val>
>>> </load_spec>
>>>
>>> Are similar settings allowed with <plot_spec>?  I see
<connection>,
>>> which seems carried over.  I'm looking to specify my folder_tmpl
in
>>> <plot_spec>, which according to Tara, should be possible.  Since
my
>>> directory structure looks like this:
>>>
>>>
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>
>>> I would do something like this I would think:
>>>
>>> .......
>>>
>>>   <date_list name="folder_dates">
>>>     <start>20200612/start>
>>>     <end><date_offset><day_offset>0</day_offset></end>
>>>     <inc>86400</inc>
>>>     <format>yyyyMMdd</format>
>>>   </date_list>
>>>
>>>   <folder_tmpl>
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>> </folder_tmpl>
>>>   <load_val>
>>>     <field name="model">
>>>       <val>prod</val>
>>>     </field>
>>>
>>>     <field name="valid_time">
>>>       <date_list name="folder_dates"/>
>>>     </field>
>>>
>>> ......
>>>
>>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate <
>>> edward.strobach at noaa.gov> wrote:
>>>
>>>> Thank you.  This is a very helpful example
>>>>
>>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> Here is some examples how to use <date_range> and
<date_range_list> in
>>>>> batch XML.
>>>>> First, you need to defined your range. You can have more then
one
>>>>> <date_range>:
>>>>> <date_range name="may">
>>>>>         <start>2019-05-01 00:00:00</start>
>>>>>         <end>2019-05-31 23:59:59</end>
>>>>> </date_range>
>>>>> <date_range name="march">
>>>>>         <start>2019-03-01 00:00:00</start>
>>>>>         <end>2019-03-31 23:59:59</end>
>>>>> </date_range>
>>>>>
>>>>> and then use one or all date_ranges:
>>>>> <plot_fix>
>>>>>     <field name="fcst_init_beg">
>>>>>         <date_range name="march"/>
>>>>>         <date_range name="may"/>
>>>>>    </field>...
>>>>> This configuration would include all dates for March and May of
2019.
>>>>>
>>>>> If you need more flexible configuration you can use
date_range_list.
>>>>> For example, you want to select first 6 hours every day beetween
>>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would look
like this:
>>>>> <date_range_list name = 'all'>
>>>>>        <range_start>2020-01-01 00:00:00</range_start>
>>>>>         <range_end>2020-01-05 00:00:00</range_end>
>>>>>         <range_length>360</range_length>
>>>>>         <inc>1440</inc>
>>>>> </date_range_list>
>>>>> where the 1st and 2nd values are the start and end,
>>>>> range_length - the length of a range in minutes(6 houres)
>>>>> inc - lenght beetween ranges in minutes (24 houres)
>>>>> This example would create following ranges:
>>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>>>>>
>>>>> Also you can add more parameters to <plot_fix> to filter your
data.
>>>>> For example, if you want to select only dates with 00:00 you can
specify
>>>>>  <field name="valid_hour">
>>>>>         <val>00</val>
>>>>>  </field>
>>>>>
>>>>> Tatiana
>>>>>
>>>>>
>>>>>
>>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
>>>>> > and if there isn't a better way, I wonder the whereabouts of a
>>>>> > separate
>>>>> > script that pipes dates (and other relevant information) into
the XML
>>>>> > file
>>>>> > used to generate the results.
>>>>> >
>>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
Affiliate <
>>>>> > edward.strobach at noaa.gov> wrote:
>>>>> >
>>>>> > > Thanks Tatiana.  These resources are helpful.
>>>>> > >
>>>>> > > One last question.  Simply typing lines like this in can
take
>>>>> > > forever:
>>>>> > >
>>>>> > > <plot_fix>
>>>>> > >     <field equalize="false" name="fcst_init_beg">
>>>>> > >         <set name="fcst_init_beg_0">
>>>>> > >             <val>2019-05-01 12:00:00</val>
>>>>> > >             <val>2019-05-02 06:00:00</val>
>>>>> > >             <val>2019-05-02 12:00:00</val>
>>>>> > >             <val>2019-05-03 06:00:00</val>
>>>>> > >             <val>2019-05-03 12:00:00</val>
>>>>> > >
>>>>> > >
>>>>> > > Is there a better way?   I see date_range, date_range_list,
or
>>>>> > > date_list.
>>>>> > > Can this be used instead of typing individual values?  This
could
>>>>> > > prove
>>>>> > > pretty daunting if one would have to type in 700 entries
each time.
>>>>> > >
>>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>>>>> > > <met_help at ucar.edu>
>>>>> > > wrote:
>>>>> > >
>>>>> > >> I found a few problems in your XML.
>>>>> > >> The database you are using doesn't have data with vx_mask
>>>>> > >> 'abcREGabc'.
>>>>> > >> That is why METviewer returns an empty dataset and can't
create a
>>>>> > >> plot.
>>>>> > >> The other problem is how you define fcst_lead. This
database has
>>>>> > >> values
>>>>> > >> like 10000, 20000 and not 010000, 020000. Since you include
all
>>>>> > >> fcst_leads
>>>>> > >> this variable can be omitted.
>>>>> > >> To minimize problems with variables values, I would suggest
to
>>>>> > >> create XML
>>>>> > >> using GUI and then edit it to fit batch standards.
>>>>> > >> I created a plot using you configurations with GUI (without
>>>>> vx_mack
>>>>> > >> and
>>>>> > >> plot and axis titles) #20200612_173626. You can load it
back to
>>>>> the
>>>>> > >> page by
>>>>> > >> using the "History" tab (left bar on GUI). Find the plot,
open XML
>>>>> > >> and
>>>>> > >> click on "Load to page". After that you can recreate a plot
and
>>>>> edit
>>>>> > >> the
>>>>> > >> configurations. You can also download a new XML and edit it
for
>>>>> > >> batch.
>>>>> > >> I attached the example of load XML to this message.
>>>>> > >>
>>>>> > >> Description of rds parameters:
>>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R scripts
>>>>> > >> rds_plots - temporary directory where the plots will be
generated
>>>>> > >> rds_data - temporary directory where the data retrieved
from the
>>>>> > >> database
>>>>> > >> and other intermediate files will be stored
>>>>> > >> rds_scripts - temporary directory where R scripts for this
run
>>>>> will
>>>>> > >> be
>>>>> > >> stored
>>>>> > >> After the execution is done all the files from temporary
>>>>> directories
>>>>> > >> would be removed.
>>>>> > >>
>>>>> > >>
>>>>> > >> We don't have the offitial user guide for METviewer. There
are
>>>>> some
>>>>> > >> instructions on how to crate XML config files:
>>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>>>> > >>
>>>>> > >> Tatiana
>>>>> > >>
>>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
>>>>> > >> > Hi,
>>>>> > >> >
>>>>> > >> > I'm attaching the xml below, but am having hard time
following
>>>>> how
>>>>> > >> > to
>>>>> > >> > go
>>>>> > >> > about the other steps since I have zero experience on how
to do
>>>>> > >> > these
>>>>> > >> > things.
>>>>> > >> >
>>>>> > >> > #2. All parameters in <folders> section should NOT be
changed
>>>>> and
>>>>> > >> > will
>>>>> > >> > be
>>>>> > >> > replaced by other scripts during the batch execution. The
data
>>>>> for
>>>>> > >> > the
>>>>> > >> > batch mode is coming from the database and not from
files. The
>>>>> > >> > name of
>>>>> > >> > the
>>>>> > >> > database is in <database> tag.
>>>>> > >> >
>>>>> > >> > I understand that nothing should be changed in the
folders
>>>>> > >> > section,
>>>>> > >> > but
>>>>> > >> > there is no indication how and where these variables are
being
>>>>> > >> > defined.
>>>>> > >> > Grepping these names of the files available does not
reveal
>>>>> formal
>>>>> > >> > definitions of these variables.  If I understood exactly
how
>>>>> these
>>>>> > >> > were
>>>>> > >> > being defined, then I would be able connect this better.
>>>>> > >> >
>>>>> > >> > #3. Users rum MET and produce MET output files. They use
>>>>> METviewer
>>>>> > >> > load
>>>>> > >> > mode to load these files to the specific database. In the
config
>>>>> > >> > file
>>>>> > >> > for
>>>>> > >> > this process they also can specify the database group.
For
>>>>> example
>>>>> > >> > database
>>>>> > >> > mv_dev is in Test12 group.
>>>>> > >> > After the data has been loaded to the database it is
visible and
>>>>> > >> > available
>>>>> > >> > in the GUI and also can be used by the batch mode.
>>>>> > >> > You can always add more data to the database using the
same load
>>>>> > >> > mode.
>>>>> > >> >
>>>>> > >> > Is there a guide for me to understand the process that
you are
>>>>> > >> > referring to?
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
>>>>> > >> > <met_help at ucar.edu>
>>>>> > >> > wrote:
>>>>> > >> >
>>>>> > >> > > Hi Edward,
>>>>> > >> > > #1. I don't have an access to NOAA servers and can't
view your
>>>>> > >> > > XML
>>>>> > >> > > for the
>>>>> > >> > > batch mode. Please, attache it to this issue and I will
take a
>>>>> > >> > > look.
>>>>> > >> > >
>>>>> > >> > > #2. All parameters in <folders> section should NOT be
changed
>>>>> > >> > > and
>>>>> > >> > > will be
>>>>> > >> > > replaced by other scripts during the batch execution.
The data
>>>>> > >> > > for
>>>>> > >> > > the
>>>>> > >> > > batch mode is coming from the database and not from
files. The
>>>>> > >> > > name
>>>>> > >> > > of the
>>>>> > >> > > database is in <database> tag.
>>>>> > >> > >
>>>>> > >> > > #3. Users rum MET and produce MET output files. They
use
>>>>> > >> > > METviewer
>>>>> > >> > > load
>>>>> > >> > > mode to load these files to the specific database. In
the
>>>>> config
>>>>> > >> > > file
>>>>> > >> > > for
>>>>> > >> > > this process they also can specify the database group.
For
>>>>> > >> > > example
>>>>> > >> > > database
>>>>> > >> > > mv_dev is in Test12 group.
>>>>> > >> > > After the data has been loaded to the database it is
visible
>>>>> and
>>>>> > >> > > available
>>>>> > >> > > in the GUI and also can be used by the batch mode.
>>>>> > >> > > You can always add more data to the database using the
same
>>>>> load
>>>>> > >> > > mode.
>>>>> > >> > >
>>>>> > >> > > Tatiana
>>>>> > >> > >
>>>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
wrote:
>>>>> > >> > > > Good morning,
>>>>> > >> > > >
>>>>> > >> > > > Over the last two weeks I've made an effort to
understand
>>>>> > >> > > > metviewer,
>>>>> > >> > > > test
>>>>> > >> > > > different cases using the history tab, changing
information
>>>>> to
>>>>> > >> > > > see
>>>>> > >> > > > what
>>>>> > >> > > > works and what does not, and make connections between
the
>>>>> GUI
>>>>> > >> > > > settings
>>>>> > >> > > > and
>>>>> > >> > > > the XML file.  Although my comfort level with
metviewer has
>>>>> > >> > > > significantly
>>>>> > >> > > > increased during this time, there are some remaining
issues
>>>>> > >> > > > that
>>>>> > >> > > > I'm
>>>>> > >> > > > hoping
>>>>> > >> > > > to clear up.
>>>>> > >> > > >
>>>>> > >> > > > 1.  Problem changing XML file and running with
different
>>>>> > >> > > > statistic
>>>>> > >> > > > and
>>>>> > >> > > > plot
>>>>> > >> > > > type:
>>>>> > >> > > >
>>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
working
>>>>> > >> > > > directory
>>>>> > >> > > > to get
>>>>> > >> > > > started which I copied over:
>>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>>>> > >> > > >
>>>>> > >> > > > He has a test.sh script that I adjusted and used to
run his
>>>>> > >> > > > example.
>>>>> > >> > > > running of the script "bash test.sh" produced a CSI
plot
>>>>> that
>>>>> > >> > > > was
>>>>> > >> > > > stored
>>>>> > >> > > > here:
>>>>> > >> > > >
>>>>> > >> > >
>>>>> > >>
>>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>>>> > >> > > >
>>>>> > >> > > > Running the script produced the expected result.  The
next
>>>>> > >> > > > natural
>>>>> > >> > > > step was
>>>>> > >> > > > to copy hochum_xml.template and create a new xml file
called
>>>>> > >> > > > Strobach_xml.template in order to create a time
series where
>>>>> > >> > > > the
>>>>> > >> > > > independent time is fcst_valid_beg instead of
obs_thresh.
>>>>> > >> > > > Accordingly,
>>>>> > >> > > > I
>>>>> > >> > > > made changes to the fixed fields (removing
fcst_valid_beg)
>>>>> and
>>>>> > >> > > > changed
>>>>> > >> > > > the
>>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).
The
>>>>> error
>>>>> > >> > > > that
>>>>> > >> > > > resulted
>>>>> > >> > > > is as follows:
>>>>> > >> > > >
>>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
13:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 01
>>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
16:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 01
>>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
19:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 01
>>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
22:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 01
>>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
01:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 02
>>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
04:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 02
>>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
07:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 02
>>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
10:00:00',
>>>>> > >> > > > '2020-05-
>>>>> > >> > > > 02
>>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
>>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>>>> > >> > > > AND ld.fmean != -9999;
>>>>> > >> > > > No data was returned from database
>>>>> > >> > > > 2020-06-09
>>>>> > >> > > >
>>>>> > >> > >
>>>>> > >>
>>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR: Caught
class
>>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>>>>> > >> > > > printFormattedTable(ResultSet
>>>>> > >> > > > res): result set contained no data
>>>>> > >> > > > **  ERROR: Caught class
>>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>>>> > >> > > > in
>>>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>>>> > >> > > > data
>>>>> > >> > > > **  ERROR: Caught class
>>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>>>> > >> > > > in
>>>>> > >> > > > printFormattedTable(ResultSet res): result set
contained no
>>>>> > >> > > > data
>>>>> > >> > > >
>>>>> > >> > > > Apparently there was no data during this time.  I
tried
>>>>> > >> > > > changing
>>>>> > >> > > > the
>>>>> > >> > > > range
>>>>> > >> > > > of valid times and changed the valid time to
different days
>>>>> > >> > > > just in
>>>>> > >> > > > case
>>>>> > >> > > > data was missing.  The issue was the same.  To
confirm my
>>>>> > >> > > > logic, I
>>>>> > >> > > > reproduced my attempted approach on metviewer, which
I found
>>>>> > >> > > > to
>>>>> > >> > > > work.
>>>>> > >> > > >
>>>>> > >> > > > 2.  Running in batch mode
>>>>> > >> > > >
>>>>> > >> > > > I think I lack an understanding of how to set up
batch mode
>>>>> > >> > > > because
>>>>> > >> > > > I
>>>>> > >> > > > don't
>>>>> > >> > > > see where these definitions are being explicitly
defined:
>>>>> > >> > > >
>>>>> > >> > > > <plot_spec>
>>>>> > >> > > >     <connection>
>>>>> > >> > > >         <host>rds_host:3306</host>
>>>>> > >> > > >
>>>>>  <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>>>> > >> > > >         <user>rds_user</user>
>>>>> > >> > > >         <password>rds_pwd</password>
>>>>> > >> > > >     </connection>
>>>>> > >> > > >     <rscript>Rscript</rscript>
>>>>> > >> > > >     <folders>
>>>>> > >> > > >         <r_tmpl>rds_R_tmpl</r_tmpl>
>>>>> > >> > > >         <r_work>rds_R_work</r_work>
>>>>> > >> > > >         <plots>rds_plots</plots>
>>>>> > >> > > >         <data>rds_data</data>
>>>>> > >> > > >         <scripts>rds_scripts</scripts>
>>>>> > >> > > >     </folders>
>>>>> > >> > > >     <plot>
>>>>> > >> > > >         <template>series_plot.R_tmpl</template>
>>>>> > >> > > >
>>>>> > >> > > > I see the <folders> section, but grepping all
available
>>>>> files
>>>>> > >> > > > does
>>>>> > >> > > > not
>>>>> > >> > > > reveal how rds_R_work is being defined, as well as
other
>>>>> > >> > > > similarly
>>>>> > >> > > > defined
>>>>> > >> > > > variables.  I think if I know how these are being
defined,
>>>>> > >> > > > then
>>>>> > >> > > > I'll
>>>>> > >> > > > be
>>>>> > >> > > > able to pipe in my information and process the data
using
>>>>> met
>>>>> > >> > > > and
>>>>> > >> > > > produce
>>>>> > >> > > > results in batch mode.  Am I right on that?  Also, if
I want
>>>>> > >> > > > to use
>>>>> > >> > > > obs for
>>>>> > >> > > > verification, do I add another line in the XML file
that
>>>>> > >> > > > designates
>>>>> > >> > > > file
>>>>> > >> > > > location of obs?  I assume that other steps have to
be run
>>>>> > >> > > > before
>>>>> > >> > > > all
>>>>> > >> > > > this.  If data is in prepbufr form, for example, I
would
>>>>> have
>>>>> > >> > > > to
>>>>> > >> > > > separately
>>>>> > >> > > > run PB2NC.  If I want grid2obs, then I would want to
do that
>>>>> > >> > > > as
>>>>> > >> > > > well.
>>>>> > >> > > >
>>>>> > >> > > > 3. Communicating results to metviewer online
>>>>> > >> > > >
>>>>> > >> > > > Lastly, I see a lot of people/projects listed here -
see
>>>>> > >> > > > attachment
>>>>> > >> > > >
>>>>> > >> > > > How do I get the results I generate to be uploaded to
this
>>>>> > >> > > > location?
>>>>> > >> > > > It's
>>>>> > >> > > > not clear how others are setting this up.
>>>>> > >> > >
>>>>> > >> > >
>>>>> > >> > >
>>>>> > >> > >
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus rds_user
>>>>> rds_pwd
>>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
>>>>> rds_scripts
>>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
>>>>> 2020-05-12
>>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-12
>>>>> 20:00:00
>>>>> > >> 2020-05-12
>>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-13
>>>>> 10:00:00
>>>>> > >> 2020-05-13
>>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-13
>>>>> 14:00:00
>>>>> > >> 2020-05-13
>>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-13
>>>>> 18:00:00
>>>>> > >> 2020-05-13
>>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-13
>>>>> 22:00:00
>>>>> > >> 2020-05-13
>>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-14
>>>>> 12:00:00
>>>>> > >> true
>>>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
>>>>> > >> plot_20200612_173626.R test x_label test y_label
>>>>> > >> false list() list() false false false false false false
true false
>>>>> > >> false
>>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
c(1,1,0) 1 2
>>>>> > >> 1.4 -2
>>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
#cccccc 1
>>>>> > >> listX 1
>>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3
0.8 1
>>>>> > >> #333333
>>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
>>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
>>>>> c(1,1)
>>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > Edward Strobach
>>>>> > > EMC/NCEP/NWS/
>>>>> > > IMSG Contractor
>>>>> > > Cubicle#: 2029
>>>>> > > 301-683-3717
>>>>> > >
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Edward Strobach
>>>> EMC/NCEP/NWS/
>>>> IMSG Contractor
>>>> Cubicle#: 2029
>>>> 301-683-3717
>>>>
>>>
>>>
>>> --
>>> Edward Strobach
>>> EMC/NCEP/NWS/
>>> IMSG Contractor
>>> Cubicle#: 2029
>>> 301-683-3717
>>>
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>


--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Tue Jun 16 14:32:04 2020

You are correct. You need to run the data loading first and then the
plotting.
You do need to have two separate XML for each stage. XML for the
loading and plotting look similar but they describe different
information and can not be reused.
Loading XML describes the location and the directory structure of the
data. The XML doesn't  describes the data itself. Plotting XML
describes the data needed for the particular plot and the potting
specifics.
If you create a new database an load data to it you should be able to
view that data and use it for potting.

Tatiana

On Tue Jun 16 07:35:03 2020, edward.strobach at noaa.gov wrote:
> Actually, it appears that I need to run the database loading module
> first
> if I want to change/add a data source.  From there I can run the
> plotting
> batch module.  Is that right?   So if I run the database loading
> module,
> then I can specify my dates and other related info specific to the
> dataset.  If I create my database with a unique name, then it should
> be
> recognized on the metviewer gui, presumably.
>
> On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > It seems that those options must be treated separately outside of
> > plot_spec.  Is it acceptable to create both a load_spec and
plot_spec
> > in
> > the same xml file?  Would the definitions in load_spec be able to
> > carry
> > over into plot_spec?  It appears that data loading and plotting
are
> > treated
> > separately.
> >
> > On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> >> I do plan to use date_range; however, in an example provided by
the
> >> link
> >> I see this:
> >>
> >> <load_spec>
> >>   <connection>
> >>     <host>kemosabe:3306</host>
> >>     <database>metvdb_hwt</database>
> >>     <user>pgoldenb</user>
> >>     <password>pgoldenb</password>
> >>   </connection>
> >>
> >> <date_list name="folder_dates">
> >>   <start>2010051914V</start>
> >>   <end><date_offset><day_offset>0</day_offset><hour>6
> >> </hour></date_offset></end>
> >>   <inc>3600</inc>
> >>   <format>yyyyMMddHH'V'</format>
> >> </date_list>
> >>
> >> <met_version>V3.0</met_version>
> >>
> >> <verbose>false</verbose>
> >> <insert_size>1</insert_size>
> >> <mode_header_db_check>true</mode_header_db_check>
> >> <drop_indexes>false</drop_indexes>
> >> <apply_indexes>true</apply_indexes>
> >> <group>Group name</group>
> >> <load_stat>true</load_stat>
> >> <load_mode>true</load_mode>
> >> <load_mtd>true</load_mtd>
> >> <load_mpr>false</load_mpr>
> >>
> >>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> >> <load_val>
> >>   <field name="model">
> >>     <val>arw</val>
> >>     <val>nmm</val>
> >>   </field>
> >>
> >> <field name="valid_time">
> >>   <date_list name="folder_dates"/>
> >> </field>
> >>
> >> <field name="vx_mask">
> >>   <val>FULL</val>
> >>   <val>SWC</val>
> >> </field>
> >> </load_val>
> >> </load_spec>
> >>
> >> Are similar settings allowed with <plot_spec>?  I see
<connection>,
> >> which
> >> seems carried over.  I'm looking to specify my folder_tmpl in
> >> <plot_spec>,
> >> which according to Tara, should be possible.  Since my directory
> >> structure
> >> looks like this:
> >>
> >>
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> >>
> >> I would do something like this I would think:
> >>
> >> .......
> >>
> >> <date_list name="folder_dates">
> >>   <start>20200612/start>
> >>   <end><date_offset><day_offset>0</day_offset></end>
> >>   <inc>86400</inc>
> >>   <format>yyyyMMdd</format>
> >> </date_list>
> >>
> >> <folder_tmpl>
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> >> </folder_tmpl>
> >> <load_val>
> >>   <field name="model">
> >>     <val>prod</val>
> >>   </field>
> >>
> >> <field name="valid_time">
> >>   <date_list name="folder_dates"/>
> >> </field>
> >>
> >> ......
> >>
> >> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate
<
> >> edward.strobach at noaa.gov> wrote:
> >>
> >>> Thank you.  This is a very helpful example
> >>>
> >>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> >>> <met_help at ucar.edu>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>> Here is some examples how to use <date_range> and
> >>>> <date_range_list> in
> >>>> batch XML.
> >>>> First, you need to defined your range. You can have more then
one
> >>>> <date_range>:
> >>>> <date_range name="may">
> >>>>         <start>2019-05-01 00:00:00</start>
> >>>>         <end>2019-05-31 23:59:59</end>
> >>>> </date_range>
> >>>> <date_range name="march">
> >>>>         <start>2019-03-01 00:00:00</start>
> >>>>         <end>2019-03-31 23:59:59</end>
> >>>> </date_range>
> >>>>
> >>>> and then use one or all date_ranges:
> >>>> <plot_fix>
> >>>>     <field name="fcst_init_beg">
> >>>>         <date_range name="march"/>
> >>>>         <date_range name="may"/>
> >>>>    </field>...
> >>>> This configuration would include all dates for March and May of
> >>>> 2019.
> >>>>
> >>>> If you need more flexible configuration you can use
> >>>> date_range_list.
> >>>> For example, you want to select first 6 hours every day
beetween
> >>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would
look
> >>>> like this:
> >>>> <date_range_list name = 'all'>
> >>>>        <range_start>2020-01-01 00:00:00</range_start>
> >>>>         <range_end>2020-01-05 00:00:00</range_end>
> >>>>         <range_length>360</range_length>
> >>>>         <inc>1440</inc>
> >>>> </date_range_list>
> >>>> where the 1st and 2nd values are the start and end,
> >>>> range_length - the length of a range in minutes(6 houres)
> >>>> inc - lenght beetween ranges in minutes (24 houres)
> >>>> This example would create following ranges:
> >>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> >>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> >>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> >>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> >>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> >>>>
> >>>> Also you can add more parameters to <plot_fix> to filter your
> >>>> data. For
> >>>> example, if you want to select only dates with 00:00 you can
> >>>> specify
> >>>>  <field name="valid_hour">
> >>>>         <val>00</val>
> >>>>  </field>
> >>>>
> >>>> Tatiana
> >>>>
> >>>>
> >>>>
> >>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> >>>> > and if there isn't a better way, I wonder the whereabouts of
a
> >>>> > separate
> >>>> > script that pipes dates (and other relevant information) into
> >>>> > the XML
> >>>> > file
> >>>> > used to generate the results.
> >>>> >
> >>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> >>>> > Affiliate <
> >>>> > edward.strobach at noaa.gov> wrote:
> >>>> >
> >>>> > > Thanks Tatiana.  These resources are helpful.
> >>>> > >
> >>>> > > One last question.  Simply typing lines like this in can
take
> >>>> > > forever:
> >>>> > >
> >>>> > > <plot_fix>
> >>>> > >     <field equalize="false" name="fcst_init_beg">
> >>>> > >         <set name="fcst_init_beg_0">
> >>>> > >             <val>2019-05-01 12:00:00</val>
> >>>> > >             <val>2019-05-02 06:00:00</val>
> >>>> > >             <val>2019-05-02 12:00:00</val>
> >>>> > >             <val>2019-05-03 06:00:00</val>
> >>>> > >             <val>2019-05-03 12:00:00</val>
> >>>> > >
> >>>> > >
> >>>> > > Is there a better way?   I see date_range, date_range_list,
or
> >>>> > > date_list.
> >>>> > > Can this be used instead of typing individual values?  This
> >>>> > > could
> >>>> > > prove
> >>>> > > pretty daunting if one would have to type in 700 entries
each
> >>>> > > time.
> >>>> > >
> >>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> >>>> > > <met_help at ucar.edu>
> >>>> > > wrote:
> >>>> > >
> >>>> > >> I found a few problems in your XML.
> >>>> > >> The database you are using doesn't have data with vx_mask
> >>>> > >> 'abcREGabc'.
> >>>> > >> That is why METviewer returns an empty dataset and can't
> >>>> > >> create a
> >>>> > >> plot.
> >>>> > >> The other problem is how you define fcst_lead. This
database
> >>>> > >> has
> >>>> > >> values
> >>>> > >> like 10000, 20000 and not 010000, 020000. Since you
include
> >>>> > >> all
> >>>> > >> fcst_leads
> >>>> > >> this variable can be omitted.
> >>>> > >> To minimize problems with variables values, I would
suggest
> >>>> > >> to
> >>>> > >> create XML
> >>>> > >> using GUI and then edit it to fit batch standards.
> >>>> > >> I created a plot using you configurations with GUI
(without
> >>>> > >> vx_mack
> >>>> > >> and
> >>>> > >> plot and axis titles) #20200612_173626. You can load it
back
> >>>> > >> to the
> >>>> > >> page by
> >>>> > >> using the "History" tab (left bar on GUI). Find the plot,
> >>>> > >> open XML
> >>>> > >> and
> >>>> > >> click on "Load to page". After that you can recreate a
plot
> >>>> > >> and
> >>>> edit
> >>>> > >> the
> >>>> > >> configurations. You can also download a new XML and edit
it
> >>>> > >> for
> >>>> > >> batch.
> >>>> > >> I attached the example of load XML to this message.
> >>>> > >>
> >>>> > >> Description of rds parameters:
> >>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
scripts
> >>>> > >> rds_plots - temporary directory where the plots will be
> >>>> > >> generated
> >>>> > >> rds_data - temporary directory where the data retrieved
from
> >>>> > >> the
> >>>> > >> database
> >>>> > >> and other intermediate files will be stored
> >>>> > >> rds_scripts - temporary directory where R scripts for this
> >>>> > >> run will
> >>>> > >> be
> >>>> > >> stored
> >>>> > >> After the execution is done all the files from temporary
> >>>> directories
> >>>> > >> would be removed.
> >>>> > >>
> >>>> > >>
> >>>> > >> We don't have the offitial user guide for METviewer. There
> >>>> > >> are some
> >>>> > >> instructions on how to crate XML config files:
> >>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> >>>> > >>
> >>>> > >> Tatiana
> >>>> > >>
> >>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
> >>>> > >> > Hi,
> >>>> > >> >
> >>>> > >> > I'm attaching the xml below, but am having hard time
> >>>> > >> > following
> >>>> how
> >>>> > >> > to
> >>>> > >> > go
> >>>> > >> > about the other steps since I have zero experience on
how
> >>>> > >> > to do
> >>>> > >> > these
> >>>> > >> > things.
> >>>> > >> >
> >>>> > >> > #2. All parameters in <folders> section should NOT be
> >>>> > >> > changed and
> >>>> > >> > will
> >>>> > >> > be
> >>>> > >> > replaced by other scripts during the batch execution.
The
> >>>> > >> > data
> >>>> for
> >>>> > >> > the
> >>>> > >> > batch mode is coming from the database and not from
files.
> >>>> > >> > The
> >>>> > >> > name of
> >>>> > >> > the
> >>>> > >> > database is in <database> tag.
> >>>> > >> >
> >>>> > >> > I understand that nothing should be changed in the
folders
> >>>> > >> > section,
> >>>> > >> > but
> >>>> > >> > there is no indication how and where these variables are
> >>>> > >> > being
> >>>> > >> > defined.
> >>>> > >> > Grepping these names of the files available does not
reveal
> >>>> formal
> >>>> > >> > definitions of these variables.  If I understood exactly
> >>>> > >> > how
> >>>> these
> >>>> > >> > were
> >>>> > >> > being defined, then I would be able connect this better.
> >>>> > >> >
> >>>> > >> > #3. Users rum MET and produce MET output files. They use
> >>>> METviewer
> >>>> > >> > load
> >>>> > >> > mode to load these files to the specific database. In
the
> >>>> > >> > config
> >>>> > >> > file
> >>>> > >> > for
> >>>> > >> > this process they also can specify the database group.
For
> >>>> example
> >>>> > >> > database
> >>>> > >> > mv_dev is in Test12 group.
> >>>> > >> > After the data has been loaded to the database it is
> >>>> > >> > visible and
> >>>> > >> > available
> >>>> > >> > in the GUI and also can be used by the batch mode.
> >>>> > >> > You can always add more data to the database using the
same
> >>>> > >> > load
> >>>> > >> > mode.
> >>>> > >> >
> >>>> > >> > Is there a guide for me to understand the process that
you
> >>>> > >> > are
> >>>> > >> > referring to?
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> >
> >>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> >>>> > >> > <met_help at ucar.edu>
> >>>> > >> > wrote:
> >>>> > >> >
> >>>> > >> > > Hi Edward,
> >>>> > >> > > #1. I don't have an access to NOAA servers and can't
view
> >>>> > >> > > your
> >>>> > >> > > XML
> >>>> > >> > > for the
> >>>> > >> > > batch mode. Please, attache it to this issue and I
will
> >>>> > >> > > take a
> >>>> > >> > > look.
> >>>> > >> > >
> >>>> > >> > > #2. All parameters in <folders> section should NOT be
> >>>> > >> > > changed
> >>>> > >> > > and
> >>>> > >> > > will be
> >>>> > >> > > replaced by other scripts during the batch execution.
The
> >>>> > >> > > data
> >>>> > >> > > for
> >>>> > >> > > the
> >>>> > >> > > batch mode is coming from the database and not from
> >>>> > >> > > files. The
> >>>> > >> > > name
> >>>> > >> > > of the
> >>>> > >> > > database is in <database> tag.
> >>>> > >> > >
> >>>> > >> > > #3. Users rum MET and produce MET output files. They
use
> >>>> > >> > > METviewer
> >>>> > >> > > load
> >>>> > >> > > mode to load these files to the specific database. In
the
> >>>> config
> >>>> > >> > > file
> >>>> > >> > > for
> >>>> > >> > > this process they also can specify the database group.
> >>>> > >> > > For
> >>>> > >> > > example
> >>>> > >> > > database
> >>>> > >> > > mv_dev is in Test12 group.
> >>>> > >> > > After the data has been loaded to the database it is
> >>>> > >> > > visible
> >>>> and
> >>>> > >> > > available
> >>>> > >> > > in the GUI and also can be used by the batch mode.
> >>>> > >> > > You can always add more data to the database using the
> >>>> > >> > > same
> >>>> load
> >>>> > >> > > mode.
> >>>> > >> > >
> >>>> > >> > > Tatiana
> >>>> > >> > >
> >>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
> >>>> > >> > > wrote:
> >>>> > >> > > > Good morning,
> >>>> > >> > > >
> >>>> > >> > > > Over the last two weeks I've made an effort to
> >>>> > >> > > > understand
> >>>> > >> > > > metviewer,
> >>>> > >> > > > test
> >>>> > >> > > > different cases using the history tab, changing
> >>>> > >> > > > information
> >>>> to
> >>>> > >> > > > see
> >>>> > >> > > > what
> >>>> > >> > > > works and what does not, and make connections
between
> >>>> > >> > > > the GUI
> >>>> > >> > > > settings
> >>>> > >> > > > and
> >>>> > >> > > > the XML file.  Although my comfort level with
metviewer
> >>>> > >> > > > has
> >>>> > >> > > > significantly
> >>>> > >> > > > increased during this time, there are some remaining
> >>>> > >> > > > issues
> >>>> > >> > > > that
> >>>> > >> > > > I'm
> >>>> > >> > > > hoping
> >>>> > >> > > > to clear up.
> >>>> > >> > > >
> >>>> > >> > > > 1.  Problem changing XML file and running with
> >>>> > >> > > > different
> >>>> > >> > > > statistic
> >>>> > >> > > > and
> >>>> > >> > > > plot
> >>>> > >> > > > type:
> >>>> > >> > > >
> >>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
> >>>> > >> > > > working
> >>>> > >> > > > directory
> >>>> > >> > > > to get
> >>>> > >> > > > started which I copied over:
> >>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >>>> > >> > > >
> >>>> > >> > > > He has a test.sh script that I adjusted and used to
run
> >>>> > >> > > > his
> >>>> > >> > > > example.
> >>>> > >> > > > running of the script "bash test.sh" produced a CSI
> >>>> > >> > > > plot that
> >>>> > >> > > > was
> >>>> > >> > > > stored
> >>>> > >> > > > here:
> >>>> > >> > > >
> >>>> > >> > >
> >>>> > >>
> >>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >>>> > >> > > >
> >>>> > >> > > > Running the script produced the expected result.
The
> >>>> > >> > > > next
> >>>> > >> > > > natural
> >>>> > >> > > > step was
> >>>> > >> > > > to copy hochum_xml.template and create a new xml
file
> >>>> > >> > > > called
> >>>> > >> > > > Strobach_xml.template in order to create a time
series
> >>>> > >> > > > where
> >>>> > >> > > > the
> >>>> > >> > > > independent time is fcst_valid_beg instead of
> >>>> > >> > > > obs_thresh.
> >>>> > >> > > > Accordingly,
> >>>> > >> > > > I
> >>>> > >> > > > made changes to the fixed fields (removing
> >>>> > >> > > > fcst_valid_beg)
> >>>> and
> >>>> > >> > > > changed
> >>>> > >> > > > the
> >>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).
The
> >>>> > >> > > > error
> >>>> > >> > > > that
> >>>> > >> > > > resulted
> >>>> > >> > > > is as follows:
> >>>> > >> > > >
> >>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
> >>>> > >> > > > 13:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 01
> >>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> >>>> > >> > > > 16:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 01
> >>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> >>>> > >> > > > 19:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 01
> >>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> >>>> > >> > > > 22:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 01
> >>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> >>>> > >> > > > 01:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 02
> >>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> >>>> > >> > > > 04:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 02
> >>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> >>>> > >> > > > 07:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 02
> >>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> >>>> > >> > > > 10:00:00',
> >>>> > >> > > > '2020-05-
> >>>> > >> > > > 02
> >>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> >>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
> >>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> >>>> > >> > > > AND ld.fmean != -9999;
> >>>> > >> > > > No data was returned from database
> >>>> > >> > > > 2020-06-09
> >>>> > >> > > >
> >>>> > >> > >
> >>>> > >>
> >>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR: Caught
> >>>> > >> > > > class
> >>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> >>>> > >> > > > printFormattedTable(ResultSet
> >>>> > >> > > > res): result set contained no data
> >>>> > >> > > > **  ERROR: Caught class
> >>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >>>> > >> > > > in
> >>>> > >> > > > printFormattedTable(ResultSet res): result set
> >>>> > >> > > > contained no
> >>>> > >> > > > data
> >>>> > >> > > > **  ERROR: Caught class
> >>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >>>> > >> > > > in
> >>>> > >> > > > printFormattedTable(ResultSet res): result set
> >>>> > >> > > > contained no
> >>>> > >> > > > data
> >>>> > >> > > >
> >>>> > >> > > > Apparently there was no data during this time.  I
tried
> >>>> > >> > > > changing
> >>>> > >> > > > the
> >>>> > >> > > > range
> >>>> > >> > > > of valid times and changed the valid time to
different
> >>>> > >> > > > days
> >>>> > >> > > > just in
> >>>> > >> > > > case
> >>>> > >> > > > data was missing.  The issue was the same.  To
confirm
> >>>> > >> > > > my
> >>>> > >> > > > logic, I
> >>>> > >> > > > reproduced my attempted approach on metviewer, which
I
> >>>> > >> > > > found
> >>>> > >> > > > to
> >>>> > >> > > > work.
> >>>> > >> > > >
> >>>> > >> > > > 2.  Running in batch mode
> >>>> > >> > > >
> >>>> > >> > > > I think I lack an understanding of how to set up
batch
> >>>> > >> > > > mode
> >>>> > >> > > > because
> >>>> > >> > > > I
> >>>> > >> > > > don't
> >>>> > >> > > > see where these definitions are being explicitly
> >>>> > >> > > > defined:
> >>>> > >> > > >
> >>>> > >> > > > <plot_spec>
> >>>> > >> > > >     <connection>
> >>>> > >> > > >         <host>rds_host:3306</host>
> >>>> > >> > > >
> >>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >>>> > >> > > > <user>rds_user</user>
> >>>> > >> > > > <password>rds_pwd</password>
> >>>> > >> > > > </connection>
> >>>> > >> > > > <rscript>Rscript</rscript>
> >>>> > >> > > > <folders>
> >>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >>>> > >> > > > <r_work>rds_R_work</r_work>
> >>>> > >> > > > <plots>rds_plots</plots>
> >>>> > >> > > > <data>rds_data</data>
> >>>> > >> > > > <scripts>rds_scripts</scripts>
> >>>> > >> > > > </folders>
> >>>> > >> > > > <plot>
> >>>> > >> > > > <template>series_plot.R_tmpl</template>
> >>>> > >> > > >
> >>>> > >> > > > I see the <folders> section, but grepping all
available
> >>>> > >> > > > files
> >>>> > >> > > > does
> >>>> > >> > > > not
> >>>> > >> > > > reveal how rds_R_work is being defined, as well as
> >>>> > >> > > > other
> >>>> > >> > > > similarly
> >>>> > >> > > > defined
> >>>> > >> > > > variables.  I think if I know how these are being
> >>>> > >> > > > defined,
> >>>> > >> > > > then
> >>>> > >> > > > I'll
> >>>> > >> > > > be
> >>>> > >> > > > able to pipe in my information and process the data
> >>>> > >> > > > using met
> >>>> > >> > > > and
> >>>> > >> > > > produce
> >>>> > >> > > > results in batch mode.  Am I right on that?  Also,
if I
> >>>> > >> > > > want
> >>>> > >> > > > to use
> >>>> > >> > > > obs for
> >>>> > >> > > > verification, do I add another line in the XML file
> >>>> > >> > > > that
> >>>> > >> > > > designates
> >>>> > >> > > > file
> >>>> > >> > > > location of obs?  I assume that other steps have to
be
> >>>> > >> > > > run
> >>>> > >> > > > before
> >>>> > >> > > > all
> >>>> > >> > > > this.  If data is in prepbufr form, for example, I
> >>>> > >> > > > would have
> >>>> > >> > > > to
> >>>> > >> > > > separately
> >>>> > >> > > > run PB2NC.  If I want grid2obs, then I would want to
do
> >>>> > >> > > > that
> >>>> > >> > > > as
> >>>> > >> > > > well.
> >>>> > >> > > >
> >>>> > >> > > > 3. Communicating results to metviewer online
> >>>> > >> > > >
> >>>> > >> > > > Lastly, I see a lot of people/projects listed here -
> >>>> > >> > > > see
> >>>> > >> > > > attachment
> >>>> > >> > > >
> >>>> > >> > > > How do I get the results I generate to be uploaded
to
> >>>> > >> > > > this
> >>>> > >> > > > location?
> >>>> > >> > > > It's
> >>>> > >> > > > not clear how others are setting this up.
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
rds_user
> >>>> > >> rds_pwd
> >>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
> >>>> > >> rds_scripts
> >>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
> >>>> > >> 2020-05-12
> >>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-
12
> >>>> 20:00:00
> >>>> > >> 2020-05-12
> >>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-
13
> >>>> 10:00:00
> >>>> > >> 2020-05-13
> >>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-
13
> >>>> 14:00:00
> >>>> > >> 2020-05-13
> >>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-
13
> >>>> 18:00:00
> >>>> > >> 2020-05-13
> >>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-
13
> >>>> 22:00:00
> >>>> > >> 2020-05-13
> >>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-
14
> >>>> 12:00:00
> >>>> > >> true
> >>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
> >>>> > >> plot_20200612_173626.R test x_label test y_label
> >>>> > >> false list() list() false false false false false false
true
> >>>> > >> false
> >>>> > >> false
> >>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
> >>>> > >> c(1,1,0) 1 2
> >>>> > >> 1.4 -2
> >>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
> >>>> > >> #cccccc 1
> >>>> > >> listX 1
> >>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3
0.8
> >>>> > >> 1
> >>>> > >> #333333
> >>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> >>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
> >>>> > >> c(1,1)
> >>>> > >> c(1,2) c("","") c() c() 0.04 c()
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > --
> >>>> > > Edward Strobach
> >>>> > > EMC/NCEP/NWS/
> >>>> > > IMSG Contractor
> >>>> > > Cubicle#: 2029
> >>>> > > 301-683-3717
> >>>> > >
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> Edward Strobach
> >>> EMC/NCEP/NWS/
> >>> IMSG Contractor
> >>> Cubicle#: 2029
> >>> 301-683-3717
> >>>
> >>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >



------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Tue Jun 16 14:45:34 2020

We don't have database and METviewer running on wcoss.
Instead, users use METviewer-wcoss scripts that help to connect to and
use METviewer that is running on AWS. That is why XML connection and
folder sections look somewhat different than once from the Docs page.
To create a database on AWS you need to run mv_create_db_on_aws.sh
mv_create_db_on_aws.sh <user_name> <database_name>
For data loading :
mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
And for batch:
mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>

These scripts copy XML and data files if needed to AWS, adjust data
path and users credentials, run METviewer and copy results back to
wcoss.

Tatiana
On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> lastly, it does seem thatmysql -u[db_username] -p[db_password]
> -e'create
> database [db_name]
>
> would need to be used to create a database.  I assume that an sql
> module is
> available on wcoss; I know you don't have access to those machines
> Tatiana.  If I create my own database with my own name, then I
should
> be
> able to proceed with project specific tasks under that tab.
Something
> like
> mv_EdwardStrobach.
>
> I can then create mv_cmaq_prod_v_para which I add data to using
> load_spec.
> Once that step is done, then I can run plot_spec.  That's my
takeaway
> currently
>
> On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Actually, it appears that I need to run the database loading
module
> > first
> > if I want to change/add a data source.  From there I can run the
> > plotting
> > batch module.  Is that right?   So if I run the database loading
> > module,
> > then I can specify my dates and other related info specific to the
> > dataset.  If I create my database with a unique name, then it
should
> > be
> > recognized on the metviewer gui, presumably.
> >
> > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> >> It seems that those options must be treated separately outside of
> >> plot_spec.  Is it acceptable to create both a load_spec and
> >> plot_spec in
> >> the same xml file?  Would the definitions in load_spec be able to
> >> carry
> >> over into plot_spec?  It appears that data loading and plotting
are
> >> treated
> >> separately.
> >>
> >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA Affiliate
<
> >> edward.strobach at noaa.gov> wrote:
> >>
> >>> I do plan to use date_range; however, in an example provided by
the
> >>> link
> >>> I see this:
> >>>
> >>> <load_spec>
> >>>   <connection>
> >>>     <host>kemosabe:3306</host>
> >>>     <database>metvdb_hwt</database>
> >>>     <user>pgoldenb</user>
> >>>     <password>pgoldenb</password>
> >>>   </connection>
> >>>
> >>> <date_list name="folder_dates">
> >>>   <start>2010051914V</start>
> >>>   <end><date_offset><day_offset>0</day_offset><hour>6
> >>> </hour></date_offset></end>
> >>>   <inc>3600</inc>
> >>>   <format>yyyyMMddHH'V'</format>
> >>> </date_list>
> >>>
> >>> <met_version>V3.0</met_version>
> >>>
> >>> <verbose>false</verbose>
> >>> <insert_size>1</insert_size>
> >>> <mode_header_db_check>true</mode_header_db_check>
> >>> <drop_indexes>false</drop_indexes>
> >>> <apply_indexes>true</apply_indexes>
> >>> <group>Group name</group>
> >>> <load_stat>true</load_stat>
> >>> <load_mode>true</load_mode>
> >>> <load_mtd>true</load_mtd>
> >>> <load_mpr>false</load_mpr>
> >>>
> >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> >>> <load_val>
> >>>   <field name="model">
> >>>     <val>arw</val>
> >>>     <val>nmm</val>
> >>>   </field>
> >>>
> >>> <field name="valid_time">
> >>>   <date_list name="folder_dates"/>
> >>> </field>
> >>>
> >>> <field name="vx_mask">
> >>>   <val>FULL</val>
> >>>   <val>SWC</val>
> >>> </field>
> >>> </load_val>
> >>> </load_spec>
> >>>
> >>> Are similar settings allowed with <plot_spec>?  I see
<connection>,
> >>> which seems carried over.  I'm looking to specify my folder_tmpl
in
> >>> <plot_spec>, which according to Tara, should be possible.  Since
my
> >>> directory structure looks like this:
> >>>
> >>>
> >>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> >>>
> >>> I would do something like this I would think:
> >>>
> >>> .......
> >>>
> >>> <date_list name="folder_dates">
> >>>   <start>20200612/start>
> >>>   <end><date_offset><day_offset>0</day_offset></end>
> >>>   <inc>86400</inc>
> >>>   <format>yyyyMMdd</format>
> >>> </date_list>
> >>>
> >>> <folder_tmpl>
> >>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> >>> </folder_tmpl>
> >>> <load_val>
> >>>   <field name="model">
> >>>     <val>prod</val>
> >>>   </field>
> >>>
> >>> <field name="valid_time">
> >>>   <date_list name="folder_dates"/>
> >>> </field>
> >>>
> >>> ......
> >>>
> >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA Affiliate
<
> >>> edward.strobach at noaa.gov> wrote:
> >>>
> >>>> Thank you.  This is a very helpful example
> >>>>
> >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> >>>> <met_help at ucar.edu>
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>> Here is some examples how to use <date_range> and
> >>>>> <date_range_list> in
> >>>>> batch XML.
> >>>>> First, you need to defined your range. You can have more then
one
> >>>>> <date_range>:
> >>>>> <date_range name="may">
> >>>>>         <start>2019-05-01 00:00:00</start>
> >>>>>         <end>2019-05-31 23:59:59</end>
> >>>>> </date_range>
> >>>>> <date_range name="march">
> >>>>>         <start>2019-03-01 00:00:00</start>
> >>>>>         <end>2019-03-31 23:59:59</end>
> >>>>> </date_range>
> >>>>>
> >>>>> and then use one or all date_ranges:
> >>>>> <plot_fix>
> >>>>>     <field name="fcst_init_beg">
> >>>>>         <date_range name="march"/>
> >>>>>         <date_range name="may"/>
> >>>>>    </field>...
> >>>>> This configuration would include all dates for March and May
of
> >>>>> 2019.
> >>>>>
> >>>>> If you need more flexible configuration you can use
> >>>>> date_range_list.
> >>>>> For example, you want to select first 6 hours every day
beetween
> >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would
look
> >>>>> like this:
> >>>>> <date_range_list name = 'all'>
> >>>>>        <range_start>2020-01-01 00:00:00</range_start>
> >>>>>         <range_end>2020-01-05 00:00:00</range_end>
> >>>>>         <range_length>360</range_length>
> >>>>>         <inc>1440</inc>
> >>>>> </date_range_list>
> >>>>> where the 1st and 2nd values are the start and end,
> >>>>> range_length - the length of a range in minutes(6 houres)
> >>>>> inc - lenght beetween ranges in minutes (24 houres)
> >>>>> This example would create following ranges:
> >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> >>>>>
> >>>>> Also you can add more parameters to <plot_fix> to filter your
> >>>>> data.
> >>>>> For example, if you want to select only dates with 00:00 you
can
> >>>>> specify
> >>>>>  <field name="valid_hour">
> >>>>>         <val>00</val>
> >>>>>  </field>
> >>>>>
> >>>>> Tatiana
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> >>>>> > and if there isn't a better way, I wonder the whereabouts of
a
> >>>>> > separate
> >>>>> > script that pipes dates (and other relevant information)
into
> >>>>> > the XML
> >>>>> > file
> >>>>> > used to generate the results.
> >>>>> >
> >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> >>>>> > Affiliate <
> >>>>> > edward.strobach at noaa.gov> wrote:
> >>>>> >
> >>>>> > > Thanks Tatiana.  These resources are helpful.
> >>>>> > >
> >>>>> > > One last question.  Simply typing lines like this in can
take
> >>>>> > > forever:
> >>>>> > >
> >>>>> > > <plot_fix>
> >>>>> > >     <field equalize="false" name="fcst_init_beg">
> >>>>> > >         <set name="fcst_init_beg_0">
> >>>>> > >             <val>2019-05-01 12:00:00</val>
> >>>>> > >             <val>2019-05-02 06:00:00</val>
> >>>>> > >             <val>2019-05-02 12:00:00</val>
> >>>>> > >             <val>2019-05-03 06:00:00</val>
> >>>>> > >             <val>2019-05-03 12:00:00</val>
> >>>>> > >
> >>>>> > >
> >>>>> > > Is there a better way?   I see date_range,
date_range_list,
> >>>>> > > or
> >>>>> > > date_list.
> >>>>> > > Can this be used instead of typing individual values?
This
> >>>>> > > could
> >>>>> > > prove
> >>>>> > > pretty daunting if one would have to type in 700 entries
each
> >>>>> > > time.
> >>>>> > >
> >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> >>>>> > > <met_help at ucar.edu>
> >>>>> > > wrote:
> >>>>> > >
> >>>>> > >> I found a few problems in your XML.
> >>>>> > >> The database you are using doesn't have data with vx_mask
> >>>>> > >> 'abcREGabc'.
> >>>>> > >> That is why METviewer returns an empty dataset and can't
> >>>>> > >> create a
> >>>>> > >> plot.
> >>>>> > >> The other problem is how you define fcst_lead. This
database
> >>>>> > >> has
> >>>>> > >> values
> >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
include
> >>>>> > >> all
> >>>>> > >> fcst_leads
> >>>>> > >> this variable can be omitted.
> >>>>> > >> To minimize problems with variables values, I would
suggest
> >>>>> > >> to
> >>>>> > >> create XML
> >>>>> > >> using GUI and then edit it to fit batch standards.
> >>>>> > >> I created a plot using you configurations with GUI
(without
> >>>>> vx_mack
> >>>>> > >> and
> >>>>> > >> plot and axis titles) #20200612_173626. You can load it
back
> >>>>> > >> to
> >>>>> the
> >>>>> > >> page by
> >>>>> > >> using the "History" tab (left bar on GUI). Find the plot,
> >>>>> > >> open XML
> >>>>> > >> and
> >>>>> > >> click on "Load to page". After that you can recreate a
plot
> >>>>> > >> and
> >>>>> edit
> >>>>> > >> the
> >>>>> > >> configurations. You can also download a new XML and edit
it
> >>>>> > >> for
> >>>>> > >> batch.
> >>>>> > >> I attached the example of load XML to this message.
> >>>>> > >>
> >>>>> > >> Description of rds parameters:
> >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
scripts
> >>>>> > >> rds_plots - temporary directory where the plots will be
> >>>>> > >> generated
> >>>>> > >> rds_data - temporary directory where the data retrieved
from
> >>>>> > >> the
> >>>>> > >> database
> >>>>> > >> and other intermediate files will be stored
> >>>>> > >> rds_scripts - temporary directory where R scripts for
this
> >>>>> > >> run
> >>>>> will
> >>>>> > >> be
> >>>>> > >> stored
> >>>>> > >> After the execution is done all the files from temporary
> >>>>> directories
> >>>>> > >> would be removed.
> >>>>> > >>
> >>>>> > >>
> >>>>> > >> We don't have the offitial user guide for METviewer.
There
> >>>>> > >> are
> >>>>> some
> >>>>> > >> instructions on how to crate XML config files:
> >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> >>>>> > >>
> >>>>> > >> Tatiana
> >>>>> > >>
> >>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
> >>>>> > >> > Hi,
> >>>>> > >> >
> >>>>> > >> > I'm attaching the xml below, but am having hard time
> >>>>> > >> > following
> >>>>> how
> >>>>> > >> > to
> >>>>> > >> > go
> >>>>> > >> > about the other steps since I have zero experience on
how
> >>>>> > >> > to do
> >>>>> > >> > these
> >>>>> > >> > things.
> >>>>> > >> >
> >>>>> > >> > #2. All parameters in <folders> section should NOT be
> >>>>> > >> > changed
> >>>>> and
> >>>>> > >> > will
> >>>>> > >> > be
> >>>>> > >> > replaced by other scripts during the batch execution.
The
> >>>>> > >> > data
> >>>>> for
> >>>>> > >> > the
> >>>>> > >> > batch mode is coming from the database and not from
files.
> >>>>> > >> > The
> >>>>> > >> > name of
> >>>>> > >> > the
> >>>>> > >> > database is in <database> tag.
> >>>>> > >> >
> >>>>> > >> > I understand that nothing should be changed in the
folders
> >>>>> > >> > section,
> >>>>> > >> > but
> >>>>> > >> > there is no indication how and where these variables
are
> >>>>> > >> > being
> >>>>> > >> > defined.
> >>>>> > >> > Grepping these names of the files available does not
> >>>>> > >> > reveal
> >>>>> formal
> >>>>> > >> > definitions of these variables.  If I understood
exactly
> >>>>> > >> > how
> >>>>> these
> >>>>> > >> > were
> >>>>> > >> > being defined, then I would be able connect this
better.
> >>>>> > >> >
> >>>>> > >> > #3. Users rum MET and produce MET output files. They
use
> >>>>> METviewer
> >>>>> > >> > load
> >>>>> > >> > mode to load these files to the specific database. In
the
> >>>>> > >> > config
> >>>>> > >> > file
> >>>>> > >> > for
> >>>>> > >> > this process they also can specify the database group.
For
> >>>>> example
> >>>>> > >> > database
> >>>>> > >> > mv_dev is in Test12 group.
> >>>>> > >> > After the data has been loaded to the database it is
> >>>>> > >> > visible and
> >>>>> > >> > available
> >>>>> > >> > in the GUI and also can be used by the batch mode.
> >>>>> > >> > You can always add more data to the database using the
> >>>>> > >> > same load
> >>>>> > >> > mode.
> >>>>> > >> >
> >>>>> > >> > Is there a guide for me to understand the process that
you
> >>>>> > >> > are
> >>>>> > >> > referring to?
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> >
> >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> >>>>> > >> > <met_help at ucar.edu>
> >>>>> > >> > wrote:
> >>>>> > >> >
> >>>>> > >> > > Hi Edward,
> >>>>> > >> > > #1. I don't have an access to NOAA servers and can't
> >>>>> > >> > > view your
> >>>>> > >> > > XML
> >>>>> > >> > > for the
> >>>>> > >> > > batch mode. Please, attache it to this issue and I
will
> >>>>> > >> > > take a
> >>>>> > >> > > look.
> >>>>> > >> > >
> >>>>> > >> > > #2. All parameters in <folders> section should NOT be
> >>>>> > >> > > changed
> >>>>> > >> > > and
> >>>>> > >> > > will be
> >>>>> > >> > > replaced by other scripts during the batch execution.
> >>>>> > >> > > The data
> >>>>> > >> > > for
> >>>>> > >> > > the
> >>>>> > >> > > batch mode is coming from the database and not from
> >>>>> > >> > > files. The
> >>>>> > >> > > name
> >>>>> > >> > > of the
> >>>>> > >> > > database is in <database> tag.
> >>>>> > >> > >
> >>>>> > >> > > #3. Users rum MET and produce MET output files. They
use
> >>>>> > >> > > METviewer
> >>>>> > >> > > load
> >>>>> > >> > > mode to load these files to the specific database. In
> >>>>> > >> > > the
> >>>>> config
> >>>>> > >> > > file
> >>>>> > >> > > for
> >>>>> > >> > > this process they also can specify the database
group.
> >>>>> > >> > > For
> >>>>> > >> > > example
> >>>>> > >> > > database
> >>>>> > >> > > mv_dev is in Test12 group.
> >>>>> > >> > > After the data has been loaded to the database it is
> >>>>> > >> > > visible
> >>>>> and
> >>>>> > >> > > available
> >>>>> > >> > > in the GUI and also can be used by the batch mode.
> >>>>> > >> > > You can always add more data to the database using
the
> >>>>> > >> > > same
> >>>>> load
> >>>>> > >> > > mode.
> >>>>> > >> > >
> >>>>> > >> > > Tatiana
> >>>>> > >> > >
> >>>>> > >> > > On Fri Jun 12 09:08:29 2020, edward.strobach at noaa.gov
> >>>>> > >> > > wrote:
> >>>>> > >> > > > Good morning,
> >>>>> > >> > > >
> >>>>> > >> > > > Over the last two weeks I've made an effort to
> >>>>> > >> > > > understand
> >>>>> > >> > > > metviewer,
> >>>>> > >> > > > test
> >>>>> > >> > > > different cases using the history tab, changing
> >>>>> > >> > > > information
> >>>>> to
> >>>>> > >> > > > see
> >>>>> > >> > > > what
> >>>>> > >> > > > works and what does not, and make connections
between
> >>>>> > >> > > > the
> >>>>> GUI
> >>>>> > >> > > > settings
> >>>>> > >> > > > and
> >>>>> > >> > > > the XML file.  Although my comfort level with
> >>>>> > >> > > > metviewer has
> >>>>> > >> > > > significantly
> >>>>> > >> > > > increased during this time, there are some
remaining
> >>>>> > >> > > > issues
> >>>>> > >> > > > that
> >>>>> > >> > > > I'm
> >>>>> > >> > > > hoping
> >>>>> > >> > > > to clear up.
> >>>>> > >> > > >
> >>>>> > >> > > > 1.  Problem changing XML file and running with
> >>>>> > >> > > > different
> >>>>> > >> > > > statistic
> >>>>> > >> > > > and
> >>>>> > >> > > > plot
> >>>>> > >> > > > type:
> >>>>> > >> > > >
> >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me the
> >>>>> > >> > > > working
> >>>>> > >> > > > directory
> >>>>> > >> > > > to get
> >>>>> > >> > > > started which I copied over:
> >>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >>>>> > >> > > >
> >>>>> > >> > > > He has a test.sh script that I adjusted and used to
> >>>>> > >> > > > run his
> >>>>> > >> > > > example.
> >>>>> > >> > > > running of the script "bash test.sh" produced a CSI
> >>>>> > >> > > > plot
> >>>>> that
> >>>>> > >> > > > was
> >>>>> > >> > > > stored
> >>>>> > >> > > > here:
> >>>>> > >> > > >
> >>>>> > >> > >
> >>>>> > >>
> >>>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >>>>> > >> > > >
> >>>>> > >> > > > Running the script produced the expected result.
The
> >>>>> > >> > > > next
> >>>>> > >> > > > natural
> >>>>> > >> > > > step was
> >>>>> > >> > > > to copy hochum_xml.template and create a new xml
file
> >>>>> > >> > > > called
> >>>>> > >> > > > Strobach_xml.template in order to create a time
series
> >>>>> > >> > > > where
> >>>>> > >> > > > the
> >>>>> > >> > > > independent time is fcst_valid_beg instead of
> >>>>> > >> > > > obs_thresh.
> >>>>> > >> > > > Accordingly,
> >>>>> > >> > > > I
> >>>>> > >> > > > made changes to the fixed fields (removing
> >>>>> > >> > > > fcst_valid_beg)
> >>>>> and
> >>>>> > >> > > > changed
> >>>>> > >> > > > the
> >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also tried).
> >>>>> > >> > > > The
> >>>>> error
> >>>>> > >> > > > that
> >>>>> > >> > > > resulted
> >>>>> > >> > > > is as follows:
> >>>>> > >> > > >
> >>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
> >>>>> > >> > > > 13:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 01
> >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> >>>>> > >> > > > 16:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 01
> >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> >>>>> > >> > > > 19:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 01
> >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> >>>>> > >> > > > 22:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 01
> >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> >>>>> > >> > > > 01:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 02
> >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> >>>>> > >> > > > 04:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 02
> >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> >>>>> > >> > > > 07:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 02
> >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> >>>>> > >> > > > 10:00:00',
> >>>>> > >> > > > '2020-05-
> >>>>> > >> > > > 02
> >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> >>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
> >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> >>>>> > >> > > > AND ld.fmean != -9999;
> >>>>> > >> > > > No data was returned from database
> >>>>> > >> > > > 2020-06-09
> >>>>> > >> > > >
> >>>>> > >> > >
> >>>>> > >>
> >>>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >>>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR:
Caught
> >>>>> > >> > > > class
> >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> >>>>> > >> > > > printFormattedTable(ResultSet
> >>>>> > >> > > > res): result set contained no data
> >>>>> > >> > > > **  ERROR: Caught class
> >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >>>>> > >> > > > in
> >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> >>>>> > >> > > > contained no
> >>>>> > >> > > > data
> >>>>> > >> > > > **  ERROR: Caught class
> >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >>>>> > >> > > > in
> >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> >>>>> > >> > > > contained no
> >>>>> > >> > > > data
> >>>>> > >> > > >
> >>>>> > >> > > > Apparently there was no data during this time.  I
> >>>>> > >> > > > tried
> >>>>> > >> > > > changing
> >>>>> > >> > > > the
> >>>>> > >> > > > range
> >>>>> > >> > > > of valid times and changed the valid time to
different
> >>>>> > >> > > > days
> >>>>> > >> > > > just in
> >>>>> > >> > > > case
> >>>>> > >> > > > data was missing.  The issue was the same.  To
confirm
> >>>>> > >> > > > my
> >>>>> > >> > > > logic, I
> >>>>> > >> > > > reproduced my attempted approach on metviewer,
which I
> >>>>> > >> > > > found
> >>>>> > >> > > > to
> >>>>> > >> > > > work.
> >>>>> > >> > > >
> >>>>> > >> > > > 2.  Running in batch mode
> >>>>> > >> > > >
> >>>>> > >> > > > I think I lack an understanding of how to set up
batch
> >>>>> > >> > > > mode
> >>>>> > >> > > > because
> >>>>> > >> > > > I
> >>>>> > >> > > > don't
> >>>>> > >> > > > see where these definitions are being explicitly
> >>>>> > >> > > > defined:
> >>>>> > >> > > >
> >>>>> > >> > > > <plot_spec>
> >>>>> > >> > > >     <connection>
> >>>>> > >> > > >         <host>rds_host:3306</host>
> >>>>> > >> > > >
> >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >>>>> > >> > > > <user>rds_user</user>
> >>>>> > >> > > > <password>rds_pwd</password>
> >>>>> > >> > > > </connection>
> >>>>> > >> > > > <rscript>Rscript</rscript>
> >>>>> > >> > > > <folders>
> >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >>>>> > >> > > > <r_work>rds_R_work</r_work>
> >>>>> > >> > > > <plots>rds_plots</plots>
> >>>>> > >> > > > <data>rds_data</data>
> >>>>> > >> > > > <scripts>rds_scripts</scripts>
> >>>>> > >> > > > </folders>
> >>>>> > >> > > > <plot>
> >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> >>>>> > >> > > >
> >>>>> > >> > > > I see the <folders> section, but grepping all
> >>>>> > >> > > > available
> >>>>> files
> >>>>> > >> > > > does
> >>>>> > >> > > > not
> >>>>> > >> > > > reveal how rds_R_work is being defined, as well as
> >>>>> > >> > > > other
> >>>>> > >> > > > similarly
> >>>>> > >> > > > defined
> >>>>> > >> > > > variables.  I think if I know how these are being
> >>>>> > >> > > > defined,
> >>>>> > >> > > > then
> >>>>> > >> > > > I'll
> >>>>> > >> > > > be
> >>>>> > >> > > > able to pipe in my information and process the data
> >>>>> > >> > > > using
> >>>>> met
> >>>>> > >> > > > and
> >>>>> > >> > > > produce
> >>>>> > >> > > > results in batch mode.  Am I right on that?  Also,
if
> >>>>> > >> > > > I want
> >>>>> > >> > > > to use
> >>>>> > >> > > > obs for
> >>>>> > >> > > > verification, do I add another line in the XML file
> >>>>> > >> > > > that
> >>>>> > >> > > > designates
> >>>>> > >> > > > file
> >>>>> > >> > > > location of obs?  I assume that other steps have to
be
> >>>>> > >> > > > run
> >>>>> > >> > > > before
> >>>>> > >> > > > all
> >>>>> > >> > > > this.  If data is in prepbufr form, for example, I
> >>>>> > >> > > > would
> >>>>> have
> >>>>> > >> > > > to
> >>>>> > >> > > > separately
> >>>>> > >> > > > run PB2NC.  If I want grid2obs, then I would want
to
> >>>>> > >> > > > do that
> >>>>> > >> > > > as
> >>>>> > >> > > > well.
> >>>>> > >> > > >
> >>>>> > >> > > > 3. Communicating results to metviewer online
> >>>>> > >> > > >
> >>>>> > >> > > > Lastly, I see a lot of people/projects listed here
-
> >>>>> > >> > > > see
> >>>>> > >> > > > attachment
> >>>>> > >> > > >
> >>>>> > >> > > > How do I get the results I generate to be uploaded
to
> >>>>> > >> > > > this
> >>>>> > >> > > > location?
> >>>>> > >> > > > It's
> >>>>> > >> > > > not clear how others are setting this up.
> >>>>> > >> > >
> >>>>> > >> > >
> >>>>> > >> > >
> >>>>> > >> > >
> >>>>> > >>
> >>>>> > >>
> >>>>> > >>
> >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
rds_user
> >>>>> rds_pwd
> >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
> >>>>> rds_scripts
> >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH CMAQAODPRODHIGH
> >>>>> 2020-05-12
> >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-05-
12
> >>>>> 20:00:00
> >>>>> > >> 2020-05-12
> >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-05-
13
> >>>>> 10:00:00
> >>>>> > >> 2020-05-13
> >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-05-
13
> >>>>> 14:00:00
> >>>>> > >> 2020-05-13
> >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-05-
13
> >>>>> 18:00:00
> >>>>> > >> 2020-05-13
> >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-05-
13
> >>>>> 22:00:00
> >>>>> > >> 2020-05-13
> >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-05-
14
> >>>>> 12:00:00
> >>>>> > >> true
> >>>>> > >> median plot_20200612_173626.data plot_20200612_173626.png
> >>>>> > >> plot_20200612_173626.R test x_label test y_label
> >>>>> > >> false list() list() false false false false false false
true
> >>>>> > >> false
> >>>>> > >> false
> >>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
> >>>>> > >> c(1,1,0) 1 2
> >>>>> > >> 1.4 -2
> >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
> >>>>> > >> #cccccc 1
> >>>>> > >> listX 1
> >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25) 3
> >>>>> > >> 0.8 1
> >>>>> > >> #333333
> >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE) c(TRUE,TRUE)
> >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
> >>>>> c(1,1)
> >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> >>>>> > >
> >>>>> > >
> >>>>> > >
> >>>>> > > --
> >>>>> > > Edward Strobach
> >>>>> > > EMC/NCEP/NWS/
> >>>>> > > IMSG Contractor
> >>>>> > > Cubicle#: 2029
> >>>>> > > 301-683-3717
> >>>>> > >
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Edward Strobach
> >>>> EMC/NCEP/NWS/
> >>>> IMSG Contractor
> >>>> Cubicle#: 2029
> >>>> 301-683-3717
> >>>>
> >>>
> >>>
> >>> --
> >>> Edward Strobach
> >>> EMC/NCEP/NWS/
> >>> IMSG Contractor
> >>> Cubicle#: 2029
> >>> 301-683-3717
> >>>
> >>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >



------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 15:40:50 2020

Thanks Tatiana.  Yes, I've created a database which seems different
from
mv_group. How can one create their own group so that I can store and
access
the database.  The database exists but cannot be viewed on the
metviewer
GUI.  The list starts with EMC HREFv3 and ends with vhagerty.  I named
my
database Ed_Strobach, but that was really intended for the group name.

On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> We don't have database and METviewer running on wcoss.
> Instead, users use METviewer-wcoss scripts that help to connect to
and use
> METviewer that is running on AWS. That is why XML connection and
folder
> sections look somewhat different than once from the Docs page.
> To create a database on AWS you need to run mv_create_db_on_aws.sh
> mv_create_db_on_aws.sh <user_name> <database_name>
> For data loading :
> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> And for batch:
> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>
> These scripts copy XML and data files if needed to AWS, adjust data
path
> and users credentials, run METviewer and copy results back to wcoss.
>
> Tatiana
> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> > lastly, it does seem thatmysql -u[db_username] -p[db_password]
> > -e'create
> > database [db_name]
> >
> > would need to be used to create a database.  I assume that an sql
> > module is
> > available on wcoss; I know you don't have access to those machines
> > Tatiana.  If I create my own database with my own name, then I
should
> > be
> > able to proceed with project specific tasks under that tab.
Something
> > like
> > mv_EdwardStrobach.
> >
> > I can then create mv_cmaq_prod_v_para which I add data to using
> > load_spec.
> > Once that step is done, then I can run plot_spec.  That's my
takeaway
> > currently
> >
> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > Actually, it appears that I need to run the database loading
module
> > > first
> > > if I want to change/add a data source.  From there I can run the
> > > plotting
> > > batch module.  Is that right?   So if I run the database loading
> > > module,
> > > then I can specify my dates and other related info specific to
the
> > > dataset.  If I create my database with a unique name, then it
should
> > > be
> > > recognized on the metviewer gui, presumably.
> > >
> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA Affiliate
<
> > > edward.strobach at noaa.gov> wrote:
> > >
> > >> It seems that those options must be treated separately outside
of
> > >> plot_spec.  Is it acceptable to create both a load_spec and
> > >> plot_spec in
> > >> the same xml file?  Would the definitions in load_spec be able
to
> > >> carry
> > >> over into plot_spec?  It appears that data loading and plotting
are
> > >> treated
> > >> separately.
> > >>
> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
Affiliate <
> > >> edward.strobach at noaa.gov> wrote:
> > >>
> > >>> I do plan to use date_range; however, in an example provided
by the
> > >>> link
> > >>> I see this:
> > >>>
> > >>> <load_spec>
> > >>>   <connection>
> > >>>     <host>kemosabe:3306</host>
> > >>>     <database>metvdb_hwt</database>
> > >>>     <user>pgoldenb</user>
> > >>>     <password>pgoldenb</password>
> > >>>   </connection>
> > >>>
> > >>> <date_list name="folder_dates">
> > >>>   <start>2010051914V</start>
> > >>>   <end><date_offset><day_offset>0</day_offset><hour>6
> > >>> </hour></date_offset></end>
> > >>>   <inc>3600</inc>
> > >>>   <format>yyyyMMddHH'V'</format>
> > >>> </date_list>
> > >>>
> > >>> <met_version>V3.0</met_version>
> > >>>
> > >>> <verbose>false</verbose>
> > >>> <insert_size>1</insert_size>
> > >>> <mode_header_db_check>true</mode_header_db_check>
> > >>> <drop_indexes>false</drop_indexes>
> > >>> <apply_indexes>true</apply_indexes>
> > >>> <group>Group name</group>
> > >>> <load_stat>true</load_stat>
> > >>> <load_mode>true</load_mode>
> > >>> <load_mtd>true</load_mtd>
> > >>> <load_mpr>false</load_mpr>
> > >>>
> > >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > >>> <load_val>
> > >>>   <field name="model">
> > >>>     <val>arw</val>
> > >>>     <val>nmm</val>
> > >>>   </field>
> > >>>
> > >>> <field name="valid_time">
> > >>>   <date_list name="folder_dates"/>
> > >>> </field>
> > >>>
> > >>> <field name="vx_mask">
> > >>>   <val>FULL</val>
> > >>>   <val>SWC</val>
> > >>> </field>
> > >>> </load_val>
> > >>> </load_spec>
> > >>>
> > >>> Are similar settings allowed with <plot_spec>?  I see
<connection>,
> > >>> which seems carried over.  I'm looking to specify my
folder_tmpl in
> > >>> <plot_spec>, which according to Tara, should be possible.
Since my
> > >>> directory structure looks like this:
> > >>>
> > >>>
> > >>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > >>>
> > >>> I would do something like this I would think:
> > >>>
> > >>> .......
> > >>>
> > >>> <date_list name="folder_dates">
> > >>>   <start>20200612/start>
> > >>>   <end><date_offset><day_offset>0</day_offset></end>
> > >>>   <inc>86400</inc>
> > >>>   <format>yyyyMMdd</format>
> > >>> </date_list>
> > >>>
> > >>> <folder_tmpl>
> > >>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > >>> </folder_tmpl>
> > >>> <load_val>
> > >>>   <field name="model">
> > >>>     <val>prod</val>
> > >>>   </field>
> > >>>
> > >>> <field name="valid_time">
> > >>>   <date_list name="folder_dates"/>
> > >>> </field>
> > >>>
> > >>> ......
> > >>>
> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
Affiliate <
> > >>> edward.strobach at noaa.gov> wrote:
> > >>>
> > >>>> Thank you.  This is a very helpful example
> > >>>>
> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> > >>>> <met_help at ucar.edu>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>> Here is some examples how to use <date_range> and
> > >>>>> <date_range_list> in
> > >>>>> batch XML.
> > >>>>> First, you need to defined your range. You can have more
then one
> > >>>>> <date_range>:
> > >>>>> <date_range name="may">
> > >>>>>         <start>2019-05-01 00:00:00</start>
> > >>>>>         <end>2019-05-31 23:59:59</end>
> > >>>>> </date_range>
> > >>>>> <date_range name="march">
> > >>>>>         <start>2019-03-01 00:00:00</start>
> > >>>>>         <end>2019-03-31 23:59:59</end>
> > >>>>> </date_range>
> > >>>>>
> > >>>>> and then use one or all date_ranges:
> > >>>>> <plot_fix>
> > >>>>>     <field name="fcst_init_beg">
> > >>>>>         <date_range name="march"/>
> > >>>>>         <date_range name="may"/>
> > >>>>>    </field>...
> > >>>>> This configuration would include all dates for March and May
of
> > >>>>> 2019.
> > >>>>>
> > >>>>> If you need more flexible configuration you can use
> > >>>>> date_range_list.
> > >>>>> For example, you want to select first 6 hours every day
beetween
> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would
look
> > >>>>> like this:
> > >>>>> <date_range_list name = 'all'>
> > >>>>>        <range_start>2020-01-01 00:00:00</range_start>
> > >>>>>         <range_end>2020-01-05 00:00:00</range_end>
> > >>>>>         <range_length>360</range_length>
> > >>>>>         <inc>1440</inc>
> > >>>>> </date_range_list>
> > >>>>> where the 1st and 2nd values are the start and end,
> > >>>>> range_length - the length of a range in minutes(6 houres)
> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> > >>>>> This example would create following ranges:
> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> > >>>>>
> > >>>>> Also you can add more parameters to <plot_fix> to filter
your
> > >>>>> data.
> > >>>>> For example, if you want to select only dates with 00:00 you
can
> > >>>>> specify
> > >>>>>  <field name="valid_hour">
> > >>>>>         <val>00</val>
> > >>>>>  </field>
> > >>>>>
> > >>>>> Tatiana
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov wrote:
> > >>>>> > and if there isn't a better way, I wonder the whereabouts
of a
> > >>>>> > separate
> > >>>>> > script that pipes dates (and other relevant information)
into
> > >>>>> > the XML
> > >>>>> > file
> > >>>>> > used to generate the results.
> > >>>>> >
> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> > >>>>> > Affiliate <
> > >>>>> > edward.strobach at noaa.gov> wrote:
> > >>>>> >
> > >>>>> > > Thanks Tatiana.  These resources are helpful.
> > >>>>> > >
> > >>>>> > > One last question.  Simply typing lines like this in can
take
> > >>>>> > > forever:
> > >>>>> > >
> > >>>>> > > <plot_fix>
> > >>>>> > >     <field equalize="false" name="fcst_init_beg">
> > >>>>> > >         <set name="fcst_init_beg_0">
> > >>>>> > >             <val>2019-05-01 12:00:00</val>
> > >>>>> > >             <val>2019-05-02 06:00:00</val>
> > >>>>> > >             <val>2019-05-02 12:00:00</val>
> > >>>>> > >             <val>2019-05-03 06:00:00</val>
> > >>>>> > >             <val>2019-05-03 12:00:00</val>
> > >>>>> > >
> > >>>>> > >
> > >>>>> > > Is there a better way?   I see date_range,
date_range_list,
> > >>>>> > > or
> > >>>>> > > date_list.
> > >>>>> > > Can this be used instead of typing individual values?
This
> > >>>>> > > could
> > >>>>> > > prove
> > >>>>> > > pretty daunting if one would have to type in 700 entries
each
> > >>>>> > > time.
> > >>>>> > >
> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > >>>>> > > <met_help at ucar.edu>
> > >>>>> > > wrote:
> > >>>>> > >
> > >>>>> > >> I found a few problems in your XML.
> > >>>>> > >> The database you are using doesn't have data with
vx_mask
> > >>>>> > >> 'abcREGabc'.
> > >>>>> > >> That is why METviewer returns an empty dataset and
can't
> > >>>>> > >> create a
> > >>>>> > >> plot.
> > >>>>> > >> The other problem is how you define fcst_lead. This
database
> > >>>>> > >> has
> > >>>>> > >> values
> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
include
> > >>>>> > >> all
> > >>>>> > >> fcst_leads
> > >>>>> > >> this variable can be omitted.
> > >>>>> > >> To minimize problems with variables values, I would
suggest
> > >>>>> > >> to
> > >>>>> > >> create XML
> > >>>>> > >> using GUI and then edit it to fit batch standards.
> > >>>>> > >> I created a plot using you configurations with GUI
(without
> > >>>>> vx_mack
> > >>>>> > >> and
> > >>>>> > >> plot and axis titles) #20200612_173626. You can load it
back
> > >>>>> > >> to
> > >>>>> the
> > >>>>> > >> page by
> > >>>>> > >> using the "History" tab (left bar on GUI). Find the
plot,
> > >>>>> > >> open XML
> > >>>>> > >> and
> > >>>>> > >> click on "Load to page". After that you can recreate a
plot
> > >>>>> > >> and
> > >>>>> edit
> > >>>>> > >> the
> > >>>>> > >> configurations. You can also download a new XML and
edit it
> > >>>>> > >> for
> > >>>>> > >> batch.
> > >>>>> > >> I attached the example of load XML to this message.
> > >>>>> > >>
> > >>>>> > >> Description of rds parameters:
> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
scripts
> > >>>>> > >> rds_plots - temporary directory where the plots will be
> > >>>>> > >> generated
> > >>>>> > >> rds_data - temporary directory where the data retrieved
from
> > >>>>> > >> the
> > >>>>> > >> database
> > >>>>> > >> and other intermediate files will be stored
> > >>>>> > >> rds_scripts - temporary directory where R scripts for
this
> > >>>>> > >> run
> > >>>>> will
> > >>>>> > >> be
> > >>>>> > >> stored
> > >>>>> > >> After the execution is done all the files from
temporary
> > >>>>> directories
> > >>>>> > >> would be removed.
> > >>>>> > >>
> > >>>>> > >>
> > >>>>> > >> We don't have the offitial user guide for METviewer.
There
> > >>>>> > >> are
> > >>>>> some
> > >>>>> > >> instructions on how to crate XML config files:
> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > >>>>> > >>
> > >>>>> > >> Tatiana
> > >>>>> > >>
> > >>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
> > >>>>> > >> > Hi,
> > >>>>> > >> >
> > >>>>> > >> > I'm attaching the xml below, but am having hard time
> > >>>>> > >> > following
> > >>>>> how
> > >>>>> > >> > to
> > >>>>> > >> > go
> > >>>>> > >> > about the other steps since I have zero experience on
how
> > >>>>> > >> > to do
> > >>>>> > >> > these
> > >>>>> > >> > things.
> > >>>>> > >> >
> > >>>>> > >> > #2. All parameters in <folders> section should NOT be
> > >>>>> > >> > changed
> > >>>>> and
> > >>>>> > >> > will
> > >>>>> > >> > be
> > >>>>> > >> > replaced by other scripts during the batch execution.
The
> > >>>>> > >> > data
> > >>>>> for
> > >>>>> > >> > the
> > >>>>> > >> > batch mode is coming from the database and not from
files.
> > >>>>> > >> > The
> > >>>>> > >> > name of
> > >>>>> > >> > the
> > >>>>> > >> > database is in <database> tag.
> > >>>>> > >> >
> > >>>>> > >> > I understand that nothing should be changed in the
folders
> > >>>>> > >> > section,
> > >>>>> > >> > but
> > >>>>> > >> > there is no indication how and where these variables
are
> > >>>>> > >> > being
> > >>>>> > >> > defined.
> > >>>>> > >> > Grepping these names of the files available does not
> > >>>>> > >> > reveal
> > >>>>> formal
> > >>>>> > >> > definitions of these variables.  If I understood
exactly
> > >>>>> > >> > how
> > >>>>> these
> > >>>>> > >> > were
> > >>>>> > >> > being defined, then I would be able connect this
better.
> > >>>>> > >> >
> > >>>>> > >> > #3. Users rum MET and produce MET output files. They
use
> > >>>>> METviewer
> > >>>>> > >> > load
> > >>>>> > >> > mode to load these files to the specific database. In
the
> > >>>>> > >> > config
> > >>>>> > >> > file
> > >>>>> > >> > for
> > >>>>> > >> > this process they also can specify the database
group. For
> > >>>>> example
> > >>>>> > >> > database
> > >>>>> > >> > mv_dev is in Test12 group.
> > >>>>> > >> > After the data has been loaded to the database it is
> > >>>>> > >> > visible and
> > >>>>> > >> > available
> > >>>>> > >> > in the GUI and also can be used by the batch mode.
> > >>>>> > >> > You can always add more data to the database using
the
> > >>>>> > >> > same load
> > >>>>> > >> > mode.
> > >>>>> > >> >
> > >>>>> > >> > Is there a guide for me to understand the process
that you
> > >>>>> > >> > are
> > >>>>> > >> > referring to?
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> >
> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via RT
> > >>>>> > >> > <met_help at ucar.edu>
> > >>>>> > >> > wrote:
> > >>>>> > >> >
> > >>>>> > >> > > Hi Edward,
> > >>>>> > >> > > #1. I don't have an access to NOAA servers and
can't
> > >>>>> > >> > > view your
> > >>>>> > >> > > XML
> > >>>>> > >> > > for the
> > >>>>> > >> > > batch mode. Please, attache it to this issue and I
will
> > >>>>> > >> > > take a
> > >>>>> > >> > > look.
> > >>>>> > >> > >
> > >>>>> > >> > > #2. All parameters in <folders> section should NOT
be
> > >>>>> > >> > > changed
> > >>>>> > >> > > and
> > >>>>> > >> > > will be
> > >>>>> > >> > > replaced by other scripts during the batch
execution.
> > >>>>> > >> > > The data
> > >>>>> > >> > > for
> > >>>>> > >> > > the
> > >>>>> > >> > > batch mode is coming from the database and not from
> > >>>>> > >> > > files. The
> > >>>>> > >> > > name
> > >>>>> > >> > > of the
> > >>>>> > >> > > database is in <database> tag.
> > >>>>> > >> > >
> > >>>>> > >> > > #3. Users rum MET and produce MET output files.
They use
> > >>>>> > >> > > METviewer
> > >>>>> > >> > > load
> > >>>>> > >> > > mode to load these files to the specific database.
In
> > >>>>> > >> > > the
> > >>>>> config
> > >>>>> > >> > > file
> > >>>>> > >> > > for
> > >>>>> > >> > > this process they also can specify the database
group.
> > >>>>> > >> > > For
> > >>>>> > >> > > example
> > >>>>> > >> > > database
> > >>>>> > >> > > mv_dev is in Test12 group.
> > >>>>> > >> > > After the data has been loaded to the database it
is
> > >>>>> > >> > > visible
> > >>>>> and
> > >>>>> > >> > > available
> > >>>>> > >> > > in the GUI and also can be used by the batch mode.
> > >>>>> > >> > > You can always add more data to the database using
the
> > >>>>> > >> > > same
> > >>>>> load
> > >>>>> > >> > > mode.
> > >>>>> > >> > >
> > >>>>> > >> > > Tatiana
> > >>>>> > >> > >
> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
edward.strobach at noaa.gov
> > >>>>> > >> > > wrote:
> > >>>>> > >> > > > Good morning,
> > >>>>> > >> > > >
> > >>>>> > >> > > > Over the last two weeks I've made an effort to
> > >>>>> > >> > > > understand
> > >>>>> > >> > > > metviewer,
> > >>>>> > >> > > > test
> > >>>>> > >> > > > different cases using the history tab, changing
> > >>>>> > >> > > > information
> > >>>>> to
> > >>>>> > >> > > > see
> > >>>>> > >> > > > what
> > >>>>> > >> > > > works and what does not, and make connections
between
> > >>>>> > >> > > > the
> > >>>>> GUI
> > >>>>> > >> > > > settings
> > >>>>> > >> > > > and
> > >>>>> > >> > > > the XML file.  Although my comfort level with
> > >>>>> > >> > > > metviewer has
> > >>>>> > >> > > > significantly
> > >>>>> > >> > > > increased during this time, there are some
remaining
> > >>>>> > >> > > > issues
> > >>>>> > >> > > > that
> > >>>>> > >> > > > I'm
> > >>>>> > >> > > > hoping
> > >>>>> > >> > > > to clear up.
> > >>>>> > >> > > >
> > >>>>> > >> > > > 1.  Problem changing XML file and running with
> > >>>>> > >> > > > different
> > >>>>> > >> > > > statistic
> > >>>>> > >> > > > and
> > >>>>> > >> > > > plot
> > >>>>> > >> > > > type:
> > >>>>> > >> > > >
> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me
the
> > >>>>> > >> > > > working
> > >>>>> > >> > > > directory
> > >>>>> > >> > > > to get
> > >>>>> > >> > > > started which I copied over:
> > >>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >>>>> > >> > > >
> > >>>>> > >> > > > He has a test.sh script that I adjusted and used
to
> > >>>>> > >> > > > run his
> > >>>>> > >> > > > example.
> > >>>>> > >> > > > running of the script "bash test.sh" produced a
CSI
> > >>>>> > >> > > > plot
> > >>>>> that
> > >>>>> > >> > > > was
> > >>>>> > >> > > > stored
> > >>>>> > >> > > > here:
> > >>>>> > >> > > >
> > >>>>> > >> > >
> > >>>>> > >>
> > >>>>>
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >>>>> > >> > > >
> > >>>>> > >> > > > Running the script produced the expected result.
The
> > >>>>> > >> > > > next
> > >>>>> > >> > > > natural
> > >>>>> > >> > > > step was
> > >>>>> > >> > > > to copy hochum_xml.template and create a new xml
file
> > >>>>> > >> > > > called
> > >>>>> > >> > > > Strobach_xml.template in order to create a time
series
> > >>>>> > >> > > > where
> > >>>>> > >> > > > the
> > >>>>> > >> > > > independent time is fcst_valid_beg instead of
> > >>>>> > >> > > > obs_thresh.
> > >>>>> > >> > > > Accordingly,
> > >>>>> > >> > > > I
> > >>>>> > >> > > > made changes to the fixed fields (removing
> > >>>>> > >> > > > fcst_valid_beg)
> > >>>>> and
> > >>>>> > >> > > > changed
> > >>>>> > >> > > > the
> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
tried).
> > >>>>> > >> > > > The
> > >>>>> error
> > >>>>> > >> > > > that
> > >>>>> > >> > > > resulted
> > >>>>> > >> > > > is as follows:
> > >>>>> > >> > > >
> > >>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
> > >>>>> > >> > > > 13:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 01
> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> > >>>>> > >> > > > 16:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 01
> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> > >>>>> > >> > > > 19:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 01
> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> > >>>>> > >> > > > 22:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 01
> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> > >>>>> > >> > > > 01:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 02
> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> > >>>>> > >> > > > 04:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 02
> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> > >>>>> > >> > > > 07:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 02
> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> > >>>>> > >> > > > 10:00:00',
> > >>>>> > >> > > > '2020-05-
> > >>>>> > >> > > > 02
> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > >>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > >>>>> > >> > > > AND ld.fmean != -9999;
> > >>>>> > >> > > > No data was returned from database
> > >>>>> > >> > > > 2020-06-09
> > >>>>> > >> > > >
> > >>>>> > >> > >
> > >>>>> > >>
> > >>>>>
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > >>>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR:
Caught
> > >>>>> > >> > > > class
> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> > >>>>> > >> > > > printFormattedTable(ResultSet
> > >>>>> > >> > > > res): result set contained no data
> > >>>>> > >> > > > **  ERROR: Caught class
> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >>>>> > >> > > > in
> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> > >>>>> > >> > > > contained no
> > >>>>> > >> > > > data
> > >>>>> > >> > > > **  ERROR: Caught class
> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >>>>> > >> > > > in
> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> > >>>>> > >> > > > contained no
> > >>>>> > >> > > > data
> > >>>>> > >> > > >
> > >>>>> > >> > > > Apparently there was no data during this time.  I
> > >>>>> > >> > > > tried
> > >>>>> > >> > > > changing
> > >>>>> > >> > > > the
> > >>>>> > >> > > > range
> > >>>>> > >> > > > of valid times and changed the valid time to
different
> > >>>>> > >> > > > days
> > >>>>> > >> > > > just in
> > >>>>> > >> > > > case
> > >>>>> > >> > > > data was missing.  The issue was the same.  To
confirm
> > >>>>> > >> > > > my
> > >>>>> > >> > > > logic, I
> > >>>>> > >> > > > reproduced my attempted approach on metviewer,
which I
> > >>>>> > >> > > > found
> > >>>>> > >> > > > to
> > >>>>> > >> > > > work.
> > >>>>> > >> > > >
> > >>>>> > >> > > > 2.  Running in batch mode
> > >>>>> > >> > > >
> > >>>>> > >> > > > I think I lack an understanding of how to set up
batch
> > >>>>> > >> > > > mode
> > >>>>> > >> > > > because
> > >>>>> > >> > > > I
> > >>>>> > >> > > > don't
> > >>>>> > >> > > > see where these definitions are being explicitly
> > >>>>> > >> > > > defined:
> > >>>>> > >> > > >
> > >>>>> > >> > > > <plot_spec>
> > >>>>> > >> > > >     <connection>
> > >>>>> > >> > > >         <host>rds_host:3306</host>
> > >>>>> > >> > > >
> > >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > >>>>> > >> > > > <user>rds_user</user>
> > >>>>> > >> > > > <password>rds_pwd</password>
> > >>>>> > >> > > > </connection>
> > >>>>> > >> > > > <rscript>Rscript</rscript>
> > >>>>> > >> > > > <folders>
> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > >>>>> > >> > > > <plots>rds_plots</plots>
> > >>>>> > >> > > > <data>rds_data</data>
> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > >>>>> > >> > > > </folders>
> > >>>>> > >> > > > <plot>
> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> > >>>>> > >> > > >
> > >>>>> > >> > > > I see the <folders> section, but grepping all
> > >>>>> > >> > > > available
> > >>>>> files
> > >>>>> > >> > > > does
> > >>>>> > >> > > > not
> > >>>>> > >> > > > reveal how rds_R_work is being defined, as well
as
> > >>>>> > >> > > > other
> > >>>>> > >> > > > similarly
> > >>>>> > >> > > > defined
> > >>>>> > >> > > > variables.  I think if I know how these are being
> > >>>>> > >> > > > defined,
> > >>>>> > >> > > > then
> > >>>>> > >> > > > I'll
> > >>>>> > >> > > > be
> > >>>>> > >> > > > able to pipe in my information and process the
data
> > >>>>> > >> > > > using
> > >>>>> met
> > >>>>> > >> > > > and
> > >>>>> > >> > > > produce
> > >>>>> > >> > > > results in batch mode.  Am I right on that?
Also, if
> > >>>>> > >> > > > I want
> > >>>>> > >> > > > to use
> > >>>>> > >> > > > obs for
> > >>>>> > >> > > > verification, do I add another line in the XML
file
> > >>>>> > >> > > > that
> > >>>>> > >> > > > designates
> > >>>>> > >> > > > file
> > >>>>> > >> > > > location of obs?  I assume that other steps have
to be
> > >>>>> > >> > > > run
> > >>>>> > >> > > > before
> > >>>>> > >> > > > all
> > >>>>> > >> > > > this.  If data is in prepbufr form, for example,
I
> > >>>>> > >> > > > would
> > >>>>> have
> > >>>>> > >> > > > to
> > >>>>> > >> > > > separately
> > >>>>> > >> > > > run PB2NC.  If I want grid2obs, then I would want
to
> > >>>>> > >> > > > do that
> > >>>>> > >> > > > as
> > >>>>> > >> > > > well.
> > >>>>> > >> > > >
> > >>>>> > >> > > > 3. Communicating results to metviewer online
> > >>>>> > >> > > >
> > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
here -
> > >>>>> > >> > > > see
> > >>>>> > >> > > > attachment
> > >>>>> > >> > > >
> > >>>>> > >> > > > How do I get the results I generate to be
uploaded to
> > >>>>> > >> > > > this
> > >>>>> > >> > > > location?
> > >>>>> > >> > > > It's
> > >>>>> > >> > > > not clear how others are setting this up.
> > >>>>> > >> > >
> > >>>>> > >> > >
> > >>>>> > >> > >
> > >>>>> > >> > >
> > >>>>> > >>
> > >>>>> > >>
> > >>>>> > >>
> > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
rds_user
> > >>>>> rds_pwd
> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots rds_data
> > >>>>> rds_scripts
> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
CMAQAODPRODHIGH
> > >>>>> 2020-05-12
> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-
05-12
> > >>>>> 20:00:00
> > >>>>> > >> 2020-05-12
> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-
05-13
> > >>>>> 10:00:00
> > >>>>> > >> 2020-05-13
> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-
05-13
> > >>>>> 14:00:00
> > >>>>> > >> 2020-05-13
> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-
05-13
> > >>>>> 18:00:00
> > >>>>> > >> 2020-05-13
> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-
05-13
> > >>>>> 22:00:00
> > >>>>> > >> 2020-05-13
> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-
05-14
> > >>>>> 12:00:00
> > >>>>> > >> true
> > >>>>> > >> median plot_20200612_173626.data
plot_20200612_173626.png
> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> > >>>>> > >> false list() list() false false false false false false
true
> > >>>>> > >> false
> > >>>>> > >> false
> > >>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
> > >>>>> > >> c(1,1,0) 1 2
> > >>>>> > >> 1.4 -2
> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
> > >>>>> > >> #cccccc 1
> > >>>>> > >> listX 1
> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0, -.25)
3
> > >>>>> > >> 0.8 1
> > >>>>> > >> #333333
> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
c(TRUE,TRUE)
> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
> > >>>>> c(1,1)
> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > >>>>> > >
> > >>>>> > >
> > >>>>> > >
> > >>>>> > > --
> > >>>>> > > Edward Strobach
> > >>>>> > > EMC/NCEP/NWS/
> > >>>>> > > IMSG Contractor
> > >>>>> > > Cubicle#: 2029
> > >>>>> > > 301-683-3717
> > >>>>> > >
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>> --
> > >>>> Edward Strobach
> > >>>> EMC/NCEP/NWS/
> > >>>> IMSG Contractor
> > >>>> Cubicle#: 2029
> > >>>> 301-683-3717
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Edward Strobach
> > >>> EMC/NCEP/NWS/
> > >>> IMSG Contractor
> > >>> Cubicle#: 2029
> > >>> 301-683-3717
> > >>>
> > >>
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
>
>
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Jun 16 18:41:10 2020

I was basically able to build my xml file but it says I have no
available
space.  I'm not sure how that can be since I just started adding data
to my
account

scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space left
on
device
scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space left
on
device
ERROR: Command returned with non-zero status (1): scp -r ./*
edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
4) Check METviewer AWS database list using mv_db_size_on_aws.sh
TABLE_SCHEMA                              USER             SIZE_MB
mv_met_g2o_gefs                           binbin.zhou      140027.31
mv_met_system_hrefv2_v3                   binbin.zhou      71986.11

On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> Thanks Tatiana.  Yes, I've created a database which seems different
from
> mv_group. How can one create their own group so that I can store and
access
> the database.  The database exists but cannot be viewed on the
metviewer
> GUI.  The list starts with EMC HREFv3 and ends with vhagerty.  I
named my
> database Ed_Strobach, but that was really intended for the group
name.
>
> On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> We don't have database and METviewer running on wcoss.
>> Instead, users use METviewer-wcoss scripts that help to connect to
and
>> use METviewer that is running on AWS. That is why XML connection
and folder
>> sections look somewhat different than once from the Docs page.
>> To create a database on AWS you need to run mv_create_db_on_aws.sh
>> mv_create_db_on_aws.sh <user_name> <database_name>
>> For data loading :
>> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>> And for batch:
>> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>
>> These scripts copy XML and data files if needed to AWS, adjust data
path
>> and users credentials, run METviewer and copy results back to
wcoss.
>>
>> Tatiana
>> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
>> > lastly, it does seem thatmysql -u[db_username] -p[db_password]
>> > -e'create
>> > database [db_name]
>> >
>> > would need to be used to create a database.  I assume that an sql
>> > module is
>> > available on wcoss; I know you don't have access to those
machines
>> > Tatiana.  If I create my own database with my own name, then I
should
>> > be
>> > able to proceed with project specific tasks under that tab.
Something
>> > like
>> > mv_EdwardStrobach.
>> >
>> > I can then create mv_cmaq_prod_v_para which I add data to using
>> > load_spec.
>> > Once that step is done, then I can run plot_spec.  That's my
takeaway
>> > currently
>> >
>> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate
<
>> > edward.strobach at noaa.gov> wrote:
>> >
>> > > Actually, it appears that I need to run the database loading
module
>> > > first
>> > > if I want to change/add a data source.  From there I can run
the
>> > > plotting
>> > > batch module.  Is that right?   So if I run the database
loading
>> > > module,
>> > > then I can specify my dates and other related info specific to
the
>> > > dataset.  If I create my database with a unique name, then it
should
>> > > be
>> > > recognized on the metviewer gui, presumably.
>> > >
>> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate <
>> > > edward.strobach at noaa.gov> wrote:
>> > >
>> > >> It seems that those options must be treated separately outside
of
>> > >> plot_spec.  Is it acceptable to create both a load_spec and
>> > >> plot_spec in
>> > >> the same xml file?  Would the definitions in load_spec be able
to
>> > >> carry
>> > >> over into plot_spec?  It appears that data loading and
plotting are
>> > >> treated
>> > >> separately.
>> > >>
>> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
Affiliate <
>> > >> edward.strobach at noaa.gov> wrote:
>> > >>
>> > >>> I do plan to use date_range; however, in an example provided
by the
>> > >>> link
>> > >>> I see this:
>> > >>>
>> > >>> <load_spec>
>> > >>>   <connection>
>> > >>>     <host>kemosabe:3306</host>
>> > >>>     <database>metvdb_hwt</database>
>> > >>>     <user>pgoldenb</user>
>> > >>>     <password>pgoldenb</password>
>> > >>>   </connection>
>> > >>>
>> > >>> <date_list name="folder_dates">
>> > >>>   <start>2010051914V</start>
>> > >>>   <end><date_offset><day_offset>0</day_offset><hour>6
>> > >>> </hour></date_offset></end>
>> > >>>   <inc>3600</inc>
>> > >>>   <format>yyyyMMddHH'V'</format>
>> > >>> </date_list>
>> > >>>
>> > >>> <met_version>V3.0</met_version>
>> > >>>
>> > >>> <verbose>false</verbose>
>> > >>> <insert_size>1</insert_size>
>> > >>> <mode_header_db_check>true</mode_header_db_check>
>> > >>> <drop_indexes>false</drop_indexes>
>> > >>> <apply_indexes>true</apply_indexes>
>> > >>> <group>Group name</group>
>> > >>> <load_stat>true</load_stat>
>> > >>> <load_mode>true</load_mode>
>> > >>> <load_mtd>true</load_mtd>
>> > >>> <load_mpr>false</load_mpr>
>> > >>>
>> > >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>> > >>> <load_val>
>> > >>>   <field name="model">
>> > >>>     <val>arw</val>
>> > >>>     <val>nmm</val>
>> > >>>   </field>
>> > >>>
>> > >>> <field name="valid_time">
>> > >>>   <date_list name="folder_dates"/>
>> > >>> </field>
>> > >>>
>> > >>> <field name="vx_mask">
>> > >>>   <val>FULL</val>
>> > >>>   <val>SWC</val>
>> > >>> </field>
>> > >>> </load_val>
>> > >>> </load_spec>
>> > >>>
>> > >>> Are similar settings allowed with <plot_spec>?  I see
<connection>,
>> > >>> which seems carried over.  I'm looking to specify my
folder_tmpl in
>> > >>> <plot_spec>, which according to Tara, should be possible.
Since my
>> > >>> directory structure looks like this:
>> > >>>
>> > >>>
>> > >>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>> > >>>
>> > >>> I would do something like this I would think:
>> > >>>
>> > >>> .......
>> > >>>
>> > >>> <date_list name="folder_dates">
>> > >>>   <start>20200612/start>
>> > >>>   <end><date_offset><day_offset>0</day_offset></end>
>> > >>>   <inc>86400</inc>
>> > >>>   <format>yyyyMMdd</format>
>> > >>> </date_list>
>> > >>>
>> > >>> <folder_tmpl>
>> > >>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>> > >>> </folder_tmpl>
>> > >>> <load_val>
>> > >>>   <field name="model">
>> > >>>     <val>prod</val>
>> > >>>   </field>
>> > >>>
>> > >>> <field name="valid_time">
>> > >>>   <date_list name="folder_dates"/>
>> > >>> </field>
>> > >>>
>> > >>> ......
>> > >>>
>> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
Affiliate <
>> > >>> edward.strobach at noaa.gov> wrote:
>> > >>>
>> > >>>> Thank you.  This is a very helpful example
>> > >>>>
>> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
>> > >>>> <met_help at ucar.edu>
>> > >>>> wrote:
>> > >>>>
>> > >>>>> Hi,
>> > >>>>> Here is some examples how to use <date_range> and
>> > >>>>> <date_range_list> in
>> > >>>>> batch XML.
>> > >>>>> First, you need to defined your range. You can have more
then one
>> > >>>>> <date_range>:
>> > >>>>> <date_range name="may">
>> > >>>>>         <start>2019-05-01 00:00:00</start>
>> > >>>>>         <end>2019-05-31 23:59:59</end>
>> > >>>>> </date_range>
>> > >>>>> <date_range name="march">
>> > >>>>>         <start>2019-03-01 00:00:00</start>
>> > >>>>>         <end>2019-03-31 23:59:59</end>
>> > >>>>> </date_range>
>> > >>>>>
>> > >>>>> and then use one or all date_ranges:
>> > >>>>> <plot_fix>
>> > >>>>>     <field name="fcst_init_beg">
>> > >>>>>         <date_range name="march"/>
>> > >>>>>         <date_range name="may"/>
>> > >>>>>    </field>...
>> > >>>>> This configuration would include all dates for March and
May of
>> > >>>>> 2019.
>> > >>>>>
>> > >>>>> If you need more flexible configuration you can use
>> > >>>>> date_range_list.
>> > >>>>> For example, you want to select first 6 hours every day
beetween
>> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML would
look
>> > >>>>> like this:
>> > >>>>> <date_range_list name = 'all'>
>> > >>>>>        <range_start>2020-01-01 00:00:00</range_start>
>> > >>>>>         <range_end>2020-01-05 00:00:00</range_end>
>> > >>>>>         <range_length>360</range_length>
>> > >>>>>         <inc>1440</inc>
>> > >>>>> </date_range_list>
>> > >>>>> where the 1st and 2nd values are the start and end,
>> > >>>>> range_length - the length of a range in minutes(6 houres)
>> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
>> > >>>>> This example would create following ranges:
>> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
>> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
>> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
>> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
>> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
>> > >>>>>
>> > >>>>> Also you can add more parameters to <plot_fix> to filter
your
>> > >>>>> data.
>> > >>>>> For example, if you want to select only dates with 00:00
you can
>> > >>>>> specify
>> > >>>>>  <field name="valid_hour">
>> > >>>>>         <val>00</val>
>> > >>>>>  </field>
>> > >>>>>
>> > >>>>> Tatiana
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
>> > >>>>> > and if there isn't a better way, I wonder the whereabouts
of a
>> > >>>>> > separate
>> > >>>>> > script that pipes dates (and other relevant information)
into
>> > >>>>> > the XML
>> > >>>>> > file
>> > >>>>> > used to generate the results.
>> > >>>>> >
>> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
>> > >>>>> > Affiliate <
>> > >>>>> > edward.strobach at noaa.gov> wrote:
>> > >>>>> >
>> > >>>>> > > Thanks Tatiana.  These resources are helpful.
>> > >>>>> > >
>> > >>>>> > > One last question.  Simply typing lines like this in
can take
>> > >>>>> > > forever:
>> > >>>>> > >
>> > >>>>> > > <plot_fix>
>> > >>>>> > >     <field equalize="false" name="fcst_init_beg">
>> > >>>>> > >         <set name="fcst_init_beg_0">
>> > >>>>> > >             <val>2019-05-01 12:00:00</val>
>> > >>>>> > >             <val>2019-05-02 06:00:00</val>
>> > >>>>> > >             <val>2019-05-02 12:00:00</val>
>> > >>>>> > >             <val>2019-05-03 06:00:00</val>
>> > >>>>> > >             <val>2019-05-03 12:00:00</val>
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > > Is there a better way?   I see date_range,
date_range_list,
>> > >>>>> > > or
>> > >>>>> > > date_list.
>> > >>>>> > > Can this be used instead of typing individual values?
This
>> > >>>>> > > could
>> > >>>>> > > prove
>> > >>>>> > > pretty daunting if one would have to type in 700
entries each
>> > >>>>> > > time.
>> > >>>>> > >
>> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
>> > >>>>> > > <met_help at ucar.edu>
>> > >>>>> > > wrote:
>> > >>>>> > >
>> > >>>>> > >> I found a few problems in your XML.
>> > >>>>> > >> The database you are using doesn't have data with
vx_mask
>> > >>>>> > >> 'abcREGabc'.
>> > >>>>> > >> That is why METviewer returns an empty dataset and
can't
>> > >>>>> > >> create a
>> > >>>>> > >> plot.
>> > >>>>> > >> The other problem is how you define fcst_lead. This
database
>> > >>>>> > >> has
>> > >>>>> > >> values
>> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
include
>> > >>>>> > >> all
>> > >>>>> > >> fcst_leads
>> > >>>>> > >> this variable can be omitted.
>> > >>>>> > >> To minimize problems with variables values, I would
suggest
>> > >>>>> > >> to
>> > >>>>> > >> create XML
>> > >>>>> > >> using GUI and then edit it to fit batch standards.
>> > >>>>> > >> I created a plot using you configurations with GUI
(without
>> > >>>>> vx_mack
>> > >>>>> > >> and
>> > >>>>> > >> plot and axis titles) #20200612_173626. You can load
it back
>> > >>>>> > >> to
>> > >>>>> the
>> > >>>>> > >> page by
>> > >>>>> > >> using the "History" tab (left bar on GUI). Find the
plot,
>> > >>>>> > >> open XML
>> > >>>>> > >> and
>> > >>>>> > >> click on "Load to page". After that you can recreate a
plot
>> > >>>>> > >> and
>> > >>>>> edit
>> > >>>>> > >> the
>> > >>>>> > >> configurations. You can also download a new XML and
edit it
>> > >>>>> > >> for
>> > >>>>> > >> batch.
>> > >>>>> > >> I attached the example of load XML to this message.
>> > >>>>> > >>
>> > >>>>> > >> Description of rds parameters:
>> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
scripts
>> > >>>>> > >> rds_plots - temporary directory where the plots will
be
>> > >>>>> > >> generated
>> > >>>>> > >> rds_data - temporary directory where the data
retrieved from
>> > >>>>> > >> the
>> > >>>>> > >> database
>> > >>>>> > >> and other intermediate files will be stored
>> > >>>>> > >> rds_scripts - temporary directory where R scripts for
this
>> > >>>>> > >> run
>> > >>>>> will
>> > >>>>> > >> be
>> > >>>>> > >> stored
>> > >>>>> > >> After the execution is done all the files from
temporary
>> > >>>>> directories
>> > >>>>> > >> would be removed.
>> > >>>>> > >>
>> > >>>>> > >>
>> > >>>>> > >> We don't have the offitial user guide for METviewer.
There
>> > >>>>> > >> are
>> > >>>>> some
>> > >>>>> > >> instructions on how to crate XML config files:
>> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>> > >>>>> > >>
>> > >>>>> > >> Tatiana
>> > >>>>> > >>
>> > >>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
wrote:
>> > >>>>> > >> > Hi,
>> > >>>>> > >> >
>> > >>>>> > >> > I'm attaching the xml below, but am having hard time
>> > >>>>> > >> > following
>> > >>>>> how
>> > >>>>> > >> > to
>> > >>>>> > >> > go
>> > >>>>> > >> > about the other steps since I have zero experience
on how
>> > >>>>> > >> > to do
>> > >>>>> > >> > these
>> > >>>>> > >> > things.
>> > >>>>> > >> >
>> > >>>>> > >> > #2. All parameters in <folders> section should NOT
be
>> > >>>>> > >> > changed
>> > >>>>> and
>> > >>>>> > >> > will
>> > >>>>> > >> > be
>> > >>>>> > >> > replaced by other scripts during the batch
execution. The
>> > >>>>> > >> > data
>> > >>>>> for
>> > >>>>> > >> > the
>> > >>>>> > >> > batch mode is coming from the database and not from
files.
>> > >>>>> > >> > The
>> > >>>>> > >> > name of
>> > >>>>> > >> > the
>> > >>>>> > >> > database is in <database> tag.
>> > >>>>> > >> >
>> > >>>>> > >> > I understand that nothing should be changed in the
folders
>> > >>>>> > >> > section,
>> > >>>>> > >> > but
>> > >>>>> > >> > there is no indication how and where these variables
are
>> > >>>>> > >> > being
>> > >>>>> > >> > defined.
>> > >>>>> > >> > Grepping these names of the files available does not
>> > >>>>> > >> > reveal
>> > >>>>> formal
>> > >>>>> > >> > definitions of these variables.  If I understood
exactly
>> > >>>>> > >> > how
>> > >>>>> these
>> > >>>>> > >> > were
>> > >>>>> > >> > being defined, then I would be able connect this
better.
>> > >>>>> > >> >
>> > >>>>> > >> > #3. Users rum MET and produce MET output files. They
use
>> > >>>>> METviewer
>> > >>>>> > >> > load
>> > >>>>> > >> > mode to load these files to the specific database.
In the
>> > >>>>> > >> > config
>> > >>>>> > >> > file
>> > >>>>> > >> > for
>> > >>>>> > >> > this process they also can specify the database
group. For
>> > >>>>> example
>> > >>>>> > >> > database
>> > >>>>> > >> > mv_dev is in Test12 group.
>> > >>>>> > >> > After the data has been loaded to the database it is
>> > >>>>> > >> > visible and
>> > >>>>> > >> > available
>> > >>>>> > >> > in the GUI and also can be used by the batch mode.
>> > >>>>> > >> > You can always add more data to the database using
the
>> > >>>>> > >> > same load
>> > >>>>> > >> > mode.
>> > >>>>> > >> >
>> > >>>>> > >> > Is there a guide for me to understand the process
that you
>> > >>>>> > >> > are
>> > >>>>> > >> > referring to?
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> >
>> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via
RT
>> > >>>>> > >> > <met_help at ucar.edu>
>> > >>>>> > >> > wrote:
>> > >>>>> > >> >
>> > >>>>> > >> > > Hi Edward,
>> > >>>>> > >> > > #1. I don't have an access to NOAA servers and
can't
>> > >>>>> > >> > > view your
>> > >>>>> > >> > > XML
>> > >>>>> > >> > > for the
>> > >>>>> > >> > > batch mode. Please, attache it to this issue and I
will
>> > >>>>> > >> > > take a
>> > >>>>> > >> > > look.
>> > >>>>> > >> > >
>> > >>>>> > >> > > #2. All parameters in <folders> section should NOT
be
>> > >>>>> > >> > > changed
>> > >>>>> > >> > > and
>> > >>>>> > >> > > will be
>> > >>>>> > >> > > replaced by other scripts during the batch
execution.
>> > >>>>> > >> > > The data
>> > >>>>> > >> > > for
>> > >>>>> > >> > > the
>> > >>>>> > >> > > batch mode is coming from the database and not
from
>> > >>>>> > >> > > files. The
>> > >>>>> > >> > > name
>> > >>>>> > >> > > of the
>> > >>>>> > >> > > database is in <database> tag.
>> > >>>>> > >> > >
>> > >>>>> > >> > > #3. Users rum MET and produce MET output files.
They use
>> > >>>>> > >> > > METviewer
>> > >>>>> > >> > > load
>> > >>>>> > >> > > mode to load these files to the specific database.
In
>> > >>>>> > >> > > the
>> > >>>>> config
>> > >>>>> > >> > > file
>> > >>>>> > >> > > for
>> > >>>>> > >> > > this process they also can specify the database
group.
>> > >>>>> > >> > > For
>> > >>>>> > >> > > example
>> > >>>>> > >> > > database
>> > >>>>> > >> > > mv_dev is in Test12 group.
>> > >>>>> > >> > > After the data has been loaded to the database it
is
>> > >>>>> > >> > > visible
>> > >>>>> and
>> > >>>>> > >> > > available
>> > >>>>> > >> > > in the GUI and also can be used by the batch mode.
>> > >>>>> > >> > > You can always add more data to the database using
the
>> > >>>>> > >> > > same
>> > >>>>> load
>> > >>>>> > >> > > mode.
>> > >>>>> > >> > >
>> > >>>>> > >> > > Tatiana
>> > >>>>> > >> > >
>> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
edward.strobach at noaa.gov
>> > >>>>> > >> > > wrote:
>> > >>>>> > >> > > > Good morning,
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Over the last two weeks I've made an effort to
>> > >>>>> > >> > > > understand
>> > >>>>> > >> > > > metviewer,
>> > >>>>> > >> > > > test
>> > >>>>> > >> > > > different cases using the history tab, changing
>> > >>>>> > >> > > > information
>> > >>>>> to
>> > >>>>> > >> > > > see
>> > >>>>> > >> > > > what
>> > >>>>> > >> > > > works and what does not, and make connections
between
>> > >>>>> > >> > > > the
>> > >>>>> GUI
>> > >>>>> > >> > > > settings
>> > >>>>> > >> > > > and
>> > >>>>> > >> > > > the XML file.  Although my comfort level with
>> > >>>>> > >> > > > metviewer has
>> > >>>>> > >> > > > significantly
>> > >>>>> > >> > > > increased during this time, there are some
remaining
>> > >>>>> > >> > > > issues
>> > >>>>> > >> > > > that
>> > >>>>> > >> > > > I'm
>> > >>>>> > >> > > > hoping
>> > >>>>> > >> > > > to clear up.
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > 1.  Problem changing XML file and running with
>> > >>>>> > >> > > > different
>> > >>>>> > >> > > > statistic
>> > >>>>> > >> > > > and
>> > >>>>> > >> > > > plot
>> > >>>>> > >> > > > type:
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me
the
>> > >>>>> > >> > > > working
>> > >>>>> > >> > > > directory
>> > >>>>> > >> > > > to get
>> > >>>>> > >> > > > started which I copied over:
>> > >>>>> > >> > > >
>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > He has a test.sh script that I adjusted and used
to
>> > >>>>> > >> > > > run his
>> > >>>>> > >> > > > example.
>> > >>>>> > >> > > > running of the script "bash test.sh" produced a
CSI
>> > >>>>> > >> > > > plot
>> > >>>>> that
>> > >>>>> > >> > > > was
>> > >>>>> > >> > > > stored
>> > >>>>> > >> > > > here:
>> > >>>>> > >> > > >
>> > >>>>> > >> > >
>> > >>>>> > >>
>> > >>>>>
>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Running the script produced the expected result.
The
>> > >>>>> > >> > > > next
>> > >>>>> > >> > > > natural
>> > >>>>> > >> > > > step was
>> > >>>>> > >> > > > to copy hochum_xml.template and create a new xml
file
>> > >>>>> > >> > > > called
>> > >>>>> > >> > > > Strobach_xml.template in order to create a time
series
>> > >>>>> > >> > > > where
>> > >>>>> > >> > > > the
>> > >>>>> > >> > > > independent time is fcst_valid_beg instead of
>> > >>>>> > >> > > > obs_thresh.
>> > >>>>> > >> > > > Accordingly,
>> > >>>>> > >> > > > I
>> > >>>>> > >> > > > made changes to the fixed fields (removing
>> > >>>>> > >> > > > fcst_valid_beg)
>> > >>>>> and
>> > >>>>> > >> > > > changed
>> > >>>>> > >> > > > the
>> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
tried).
>> > >>>>> > >> > > > The
>> > >>>>> error
>> > >>>>> > >> > > > that
>> > >>>>> > >> > > > resulted
>> > >>>>> > >> > > > is as follows:
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
>> > >>>>> > >> > > > 13:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 01
>> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
>> > >>>>> > >> > > > 16:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 01
>> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
>> > >>>>> > >> > > > 19:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 01
>> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
>> > >>>>> > >> > > > 22:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 01
>> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
>> > >>>>> > >> > > > 01:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 02
>> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
>> > >>>>> > >> > > > 04:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 02
>> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
>> > >>>>> > >> > > > 07:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 02
>> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
>> > >>>>> > >> > > > 10:00:00',
>> > >>>>> > >> > > > '2020-05-
>> > >>>>> > >> > > > 02
>> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>> > >>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
>> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>> > >>>>> > >> > > > AND ld.fmean != -9999;
>> > >>>>> > >> > > > No data was returned from database
>> > >>>>> > >> > > > 2020-06-09
>> > >>>>> > >> > > >
>> > >>>>> > >> > >
>> > >>>>> > >>
>> > >>>>>
>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>> > >>>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR:
Caught
>> > >>>>> > >> > > > class
>> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
>> > >>>>> > >> > > > printFormattedTable(ResultSet
>> > >>>>> > >> > > > res): result set contained no data
>> > >>>>> > >> > > > **  ERROR: Caught class
>> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >>>>> > >> > > > in
>> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
>> > >>>>> > >> > > > contained no
>> > >>>>> > >> > > > data
>> > >>>>> > >> > > > **  ERROR: Caught class
>> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >>>>> > >> > > > in
>> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
>> > >>>>> > >> > > > contained no
>> > >>>>> > >> > > > data
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Apparently there was no data during this time.
I
>> > >>>>> > >> > > > tried
>> > >>>>> > >> > > > changing
>> > >>>>> > >> > > > the
>> > >>>>> > >> > > > range
>> > >>>>> > >> > > > of valid times and changed the valid time to
different
>> > >>>>> > >> > > > days
>> > >>>>> > >> > > > just in
>> > >>>>> > >> > > > case
>> > >>>>> > >> > > > data was missing.  The issue was the same.  To
confirm
>> > >>>>> > >> > > > my
>> > >>>>> > >> > > > logic, I
>> > >>>>> > >> > > > reproduced my attempted approach on metviewer,
which I
>> > >>>>> > >> > > > found
>> > >>>>> > >> > > > to
>> > >>>>> > >> > > > work.
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > 2.  Running in batch mode
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > I think I lack an understanding of how to set up
batch
>> > >>>>> > >> > > > mode
>> > >>>>> > >> > > > because
>> > >>>>> > >> > > > I
>> > >>>>> > >> > > > don't
>> > >>>>> > >> > > > see where these definitions are being explicitly
>> > >>>>> > >> > > > defined:
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > <plot_spec>
>> > >>>>> > >> > > >     <connection>
>> > >>>>> > >> > > >         <host>rds_host:3306</host>
>> > >>>>> > >> > > >
>> > >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>> > >>>>> > >> > > > <user>rds_user</user>
>> > >>>>> > >> > > > <password>rds_pwd</password>
>> > >>>>> > >> > > > </connection>
>> > >>>>> > >> > > > <rscript>Rscript</rscript>
>> > >>>>> > >> > > > <folders>
>> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>> > >>>>> > >> > > > <plots>rds_plots</plots>
>> > >>>>> > >> > > > <data>rds_data</data>
>> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>> > >>>>> > >> > > > </folders>
>> > >>>>> > >> > > > <plot>
>> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > I see the <folders> section, but grepping all
>> > >>>>> > >> > > > available
>> > >>>>> files
>> > >>>>> > >> > > > does
>> > >>>>> > >> > > > not
>> > >>>>> > >> > > > reveal how rds_R_work is being defined, as well
as
>> > >>>>> > >> > > > other
>> > >>>>> > >> > > > similarly
>> > >>>>> > >> > > > defined
>> > >>>>> > >> > > > variables.  I think if I know how these are
being
>> > >>>>> > >> > > > defined,
>> > >>>>> > >> > > > then
>> > >>>>> > >> > > > I'll
>> > >>>>> > >> > > > be
>> > >>>>> > >> > > > able to pipe in my information and process the
data
>> > >>>>> > >> > > > using
>> > >>>>> met
>> > >>>>> > >> > > > and
>> > >>>>> > >> > > > produce
>> > >>>>> > >> > > > results in batch mode.  Am I right on that?
Also, if
>> > >>>>> > >> > > > I want
>> > >>>>> > >> > > > to use
>> > >>>>> > >> > > > obs for
>> > >>>>> > >> > > > verification, do I add another line in the XML
file
>> > >>>>> > >> > > > that
>> > >>>>> > >> > > > designates
>> > >>>>> > >> > > > file
>> > >>>>> > >> > > > location of obs?  I assume that other steps have
to be
>> > >>>>> > >> > > > run
>> > >>>>> > >> > > > before
>> > >>>>> > >> > > > all
>> > >>>>> > >> > > > this.  If data is in prepbufr form, for example,
I
>> > >>>>> > >> > > > would
>> > >>>>> have
>> > >>>>> > >> > > > to
>> > >>>>> > >> > > > separately
>> > >>>>> > >> > > > run PB2NC.  If I want grid2obs, then I would
want to
>> > >>>>> > >> > > > do that
>> > >>>>> > >> > > > as
>> > >>>>> > >> > > > well.
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > 3. Communicating results to metviewer online
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
here -
>> > >>>>> > >> > > > see
>> > >>>>> > >> > > > attachment
>> > >>>>> > >> > > >
>> > >>>>> > >> > > > How do I get the results I generate to be
uploaded to
>> > >>>>> > >> > > > this
>> > >>>>> > >> > > > location?
>> > >>>>> > >> > > > It's
>> > >>>>> > >> > > > not clear how others are setting this up.
>> > >>>>> > >> > >
>> > >>>>> > >> > >
>> > >>>>> > >> > >
>> > >>>>> > >> > >
>> > >>>>> > >>
>> > >>>>> > >>
>> > >>>>> > >>
>> > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
rds_user
>> > >>>>> rds_pwd
>> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
>> > >>>>> rds_scripts
>> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
CMAQAODPRODHIGH
>> > >>>>> 2020-05-12
>> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00 2020-
05-12
>> > >>>>> 20:00:00
>> > >>>>> > >> 2020-05-12
>> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00 2020-
05-13
>> > >>>>> 10:00:00
>> > >>>>> > >> 2020-05-13
>> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00 2020-
05-13
>> > >>>>> 14:00:00
>> > >>>>> > >> 2020-05-13
>> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00 2020-
05-13
>> > >>>>> 18:00:00
>> > >>>>> > >> 2020-05-13
>> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00 2020-
05-13
>> > >>>>> 22:00:00
>> > >>>>> > >> 2020-05-13
>> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00 2020-
05-14
>> > >>>>> 12:00:00
>> > >>>>> > >> true
>> > >>>>> > >> median plot_20200612_173626.data
plot_20200612_173626.png
>> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
>> > >>>>> > >> false list() list() false false false false false
false true
>> > >>>>> > >> false
>> > >>>>> > >> false
>> > >>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
>> > >>>>> > >> c(1,1,0) 1 2
>> > >>>>> > >> 1.4 -2
>> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5 3
>> > >>>>> > >> #cccccc 1
>> > >>>>> > >> listX 1
>> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25) 3
>> > >>>>> > >> 0.8 1
>> > >>>>> > >> #333333
>> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
c(TRUE,TRUE)
>> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
c(1,1)
>> > >>>>> c(1,1)
>> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > > --
>> > >>>>> > > Edward Strobach
>> > >>>>> > > EMC/NCEP/NWS/
>> > >>>>> > > IMSG Contractor
>> > >>>>> > > Cubicle#: 2029
>> > >>>>> > > 301-683-3717
>> > >>>>> > >
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>
>> > >>>> --
>> > >>>> Edward Strobach
>> > >>>> EMC/NCEP/NWS/
>> > >>>> IMSG Contractor
>> > >>>> Cubicle#: 2029
>> > >>>> 301-683-3717
>> > >>>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Edward Strobach
>> > >>> EMC/NCEP/NWS/
>> > >>> IMSG Contractor
>> > >>> Cubicle#: 2029
>> > >>> 301-683-3717
>> > >>>
>> > >>
>> > >>
>> > >> --
>> > >> Edward Strobach
>> > >> EMC/NCEP/NWS/
>> > >> IMSG Contractor
>> > >> Cubicle#: 2029
>> > >> 301-683-3717
>> > >>
>> > >
>> > >
>> > > --
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/
>> > > IMSG Contractor
>> > > Cubicle#: 2029
>> > > 301-683-3717
>> > >
>>
>>
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>


--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Wed Jun 17 09:40:52 2020

Only databases that have a name that starts with 'mv_'are visible in
GUI. That is why 'Ed_Strobach' is not on the list.
If database doesn't have a group it will end up in "NO GROUP" group.
You specify the name for the group you want to see your database in
when you load data.
There is a tag <group> where you put the group name.

Tatiana

On Tue Jun 16 15:40:50 2020, edward.strobach at noaa.gov wrote:
> Thanks Tatiana.  Yes, I've created a database which seems different
> from
> mv_group. How can one create their own group so that I can store and
> access
> the database.  The database exists but cannot be viewed on the
> metviewer
> GUI.  The list starts with EMC HREFv3 and ends with vhagerty.  I
named
> my
> database Ed_Strobach, but that was really intended for the group
name.
>
> On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > We don't have database and METviewer running on wcoss.
> > Instead, users use METviewer-wcoss scripts that help to connect to
> > and use
> > METviewer that is running on AWS. That is why XML connection and
> > folder
> > sections look somewhat different than once from the Docs page.
> > To create a database on AWS you need to run mv_create_db_on_aws.sh
> > mv_create_db_on_aws.sh <user_name> <database_name>
> > For data loading :
> > mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> > And for batch:
> > mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> >
> > These scripts copy XML and data files if needed to AWS, adjust
data
> > path
> > and users credentials, run METviewer and copy results back to
wcoss.
> >
> > Tatiana
> > On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> > > lastly, it does seem thatmysql -u[db_username] -p[db_password]
> > > -e'create
> > > database [db_name]
> > >
> > > would need to be used to create a database.  I assume that an
sql
> > > module is
> > > available on wcoss; I know you don't have access to those
machines
> > > Tatiana.  If I create my own database with my own name, then I
> > > should
> > > be
> > > able to proceed with project specific tasks under that tab.
> > > Something
> > > like
> > > mv_EdwardStrobach.
> > >
> > > I can then create mv_cmaq_prod_v_para which I add data to using
> > > load_spec.
> > > Once that step is done, then I can run plot_spec.  That's my
> > > takeaway
> > > currently
> > >
> > > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA Affiliate
<
> > > edward.strobach at noaa.gov> wrote:
> > >
> > > > Actually, it appears that I need to run the database loading
> > > > module
> > > > first
> > > > if I want to change/add a data source.  From there I can run
the
> > > > plotting
> > > > batch module.  Is that right?   So if I run the database
loading
> > > > module,
> > > > then I can specify my dates and other related info specific to
> > > > the
> > > > dataset.  If I create my database with a unique name, then it
> > > > should
> > > > be
> > > > recognized on the metviewer gui, presumably.
> > > >
> > > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
> > > > <
> > > > edward.strobach at noaa.gov> wrote:
> > > >
> > > >> It seems that those options must be treated separately
outside
> > > >> of
> > > >> plot_spec.  Is it acceptable to create both a load_spec and
> > > >> plot_spec in
> > > >> the same xml file?  Would the definitions in load_spec be
able
> > > >> to
> > > >> carry
> > > >> over into plot_spec?  It appears that data loading and
plotting
> > > >> are
> > > >> treated
> > > >> separately.
> > > >>
> > > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
Affiliate
> > > >> <
> > > >> edward.strobach at noaa.gov> wrote:
> > > >>
> > > >>> I do plan to use date_range; however, in an example provided
by
> > > >>> the
> > > >>> link
> > > >>> I see this:
> > > >>>
> > > >>> <load_spec>
> > > >>>   <connection>
> > > >>>     <host>kemosabe:3306</host>
> > > >>>     <database>metvdb_hwt</database>
> > > >>>     <user>pgoldenb</user>
> > > >>>     <password>pgoldenb</password>
> > > >>>   </connection>
> > > >>>
> > > >>> <date_list name="folder_dates">
> > > >>>   <start>2010051914V</start>
> > > >>>   <end><date_offset><day_offset>0</day_offset><hour>6
> > > >>> </hour></date_offset></end>
> > > >>>   <inc>3600</inc>
> > > >>>   <format>yyyyMMddHH'V'</format>
> > > >>> </date_list>
> > > >>>
> > > >>> <met_version>V3.0</met_version>
> > > >>>
> > > >>> <verbose>false</verbose>
> > > >>> <insert_size>1</insert_size>
> > > >>> <mode_header_db_check>true</mode_header_db_check>
> > > >>> <drop_indexes>false</drop_indexes>
> > > >>> <apply_indexes>true</apply_indexes>
> > > >>> <group>Group name</group>
> > > >>> <load_stat>true</load_stat>
> > > >>> <load_mode>true</load_mode>
> > > >>> <load_mtd>true</load_mtd>
> > > >>> <load_mpr>false</load_mpr>
> > > >>>
> > > >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > > >>> <load_val>
> > > >>>   <field name="model">
> > > >>>     <val>arw</val>
> > > >>>     <val>nmm</val>
> > > >>>   </field>
> > > >>>
> > > >>> <field name="valid_time">
> > > >>>   <date_list name="folder_dates"/>
> > > >>> </field>
> > > >>>
> > > >>> <field name="vx_mask">
> > > >>>   <val>FULL</val>
> > > >>>   <val>SWC</val>
> > > >>> </field>
> > > >>> </load_val>
> > > >>> </load_spec>
> > > >>>
> > > >>> Are similar settings allowed with <plot_spec>?  I see
> > > >>> <connection>,
> > > >>> which seems carried over.  I'm looking to specify my
> > > >>> folder_tmpl in
> > > >>> <plot_spec>, which according to Tara, should be possible.
> > > >>> Since my
> > > >>> directory structure looks like this:
> > > >>>
> > > >>>
> > > >>>
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > > >>>
> > > >>> I would do something like this I would think:
> > > >>>
> > > >>> .......
> > > >>>
> > > >>> <date_list name="folder_dates">
> > > >>>   <start>20200612/start>
> > > >>>   <end><date_offset><day_offset>0</day_offset></end>
> > > >>>   <inc>86400</inc>
> > > >>>   <format>yyyyMMdd</format>
> > > >>> </date_list>
> > > >>>
> > > >>> <folder_tmpl>
> > > >>>
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > > >>> </folder_tmpl>
> > > >>> <load_val>
> > > >>>   <field name="model">
> > > >>>     <val>prod</val>
> > > >>>   </field>
> > > >>>
> > > >>> <field name="valid_time">
> > > >>>   <date_list name="folder_dates"/>
> > > >>> </field>
> > > >>>
> > > >>> ......
> > > >>>
> > > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
> > > >>> Affiliate <
> > > >>> edward.strobach at noaa.gov> wrote:
> > > >>>
> > > >>>> Thank you.  This is a very helpful example
> > > >>>>
> > > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> > > >>>> <met_help at ucar.edu>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi,
> > > >>>>> Here is some examples how to use <date_range> and
> > > >>>>> <date_range_list> in
> > > >>>>> batch XML.
> > > >>>>> First, you need to defined your range. You can have more
then
> > > >>>>> one
> > > >>>>> <date_range>:
> > > >>>>> <date_range name="may">
> > > >>>>>         <start>2019-05-01 00:00:00</start>
> > > >>>>>         <end>2019-05-31 23:59:59</end>
> > > >>>>> </date_range>
> > > >>>>> <date_range name="march">
> > > >>>>>         <start>2019-03-01 00:00:00</start>
> > > >>>>>         <end>2019-03-31 23:59:59</end>
> > > >>>>> </date_range>
> > > >>>>>
> > > >>>>> and then use one or all date_ranges:
> > > >>>>> <plot_fix>
> > > >>>>>     <field name="fcst_init_beg">
> > > >>>>>         <date_range name="march"/>
> > > >>>>>         <date_range name="may"/>
> > > >>>>>    </field>...
> > > >>>>> This configuration would include all dates for March and
May
> > > >>>>> of
> > > >>>>> 2019.
> > > >>>>>
> > > >>>>> If you need more flexible configuration you can use
> > > >>>>> date_range_list.
> > > >>>>> For example, you want to select first 6 hours every day
> > > >>>>> beetween
> > > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
> > > >>>>> look
> > > >>>>> like this:
> > > >>>>> <date_range_list name = 'all'>
> > > >>>>>        <range_start>2020-01-01 00:00:00</range_start>
> > > >>>>>         <range_end>2020-01-05 00:00:00</range_end>
> > > >>>>>         <range_length>360</range_length>
> > > >>>>>         <inc>1440</inc>
> > > >>>>> </date_range_list>
> > > >>>>> where the 1st and 2nd values are the start and end,
> > > >>>>> range_length - the length of a range in minutes(6 houres)
> > > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> > > >>>>> This example would create following ranges:
> > > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> > > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> > > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> > > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> > > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> > > >>>>>
> > > >>>>> Also you can add more parameters to <plot_fix> to filter
your
> > > >>>>> data.
> > > >>>>> For example, if you want to select only dates with 00:00
you
> > > >>>>> can
> > > >>>>> specify
> > > >>>>>  <field name="valid_hour">
> > > >>>>>         <val>00</val>
> > > >>>>>  </field>
> > > >>>>>
> > > >>>>> Tatiana
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
> > > >>>>> > and if there isn't a better way, I wonder the
whereabouts
> > > >>>>> > of a
> > > >>>>> > separate
> > > >>>>> > script that pipes dates (and other relevant information)
> > > >>>>> > into
> > > >>>>> > the XML
> > > >>>>> > file
> > > >>>>> > used to generate the results.
> > > >>>>> >
> > > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> > > >>>>> > Affiliate <
> > > >>>>> > edward.strobach at noaa.gov> wrote:
> > > >>>>> >
> > > >>>>> > > Thanks Tatiana.  These resources are helpful.
> > > >>>>> > >
> > > >>>>> > > One last question.  Simply typing lines like this in
can
> > > >>>>> > > take
> > > >>>>> > > forever:
> > > >>>>> > >
> > > >>>>> > > <plot_fix>
> > > >>>>> > >     <field equalize="false" name="fcst_init_beg">
> > > >>>>> > >         <set name="fcst_init_beg_0">
> > > >>>>> > >             <val>2019-05-01 12:00:00</val>
> > > >>>>> > >             <val>2019-05-02 06:00:00</val>
> > > >>>>> > >             <val>2019-05-02 12:00:00</val>
> > > >>>>> > >             <val>2019-05-03 06:00:00</val>
> > > >>>>> > >             <val>2019-05-03 12:00:00</val>
> > > >>>>> > >
> > > >>>>> > >
> > > >>>>> > > Is there a better way?   I see date_range,
> > > >>>>> > > date_range_list,
> > > >>>>> > > or
> > > >>>>> > > date_list.
> > > >>>>> > > Can this be used instead of typing individual values?
> > > >>>>> > > This
> > > >>>>> > > could
> > > >>>>> > > prove
> > > >>>>> > > pretty daunting if one would have to type in 700
entries
> > > >>>>> > > each
> > > >>>>> > > time.
> > > >>>>> > >
> > > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > > >>>>> > > <met_help at ucar.edu>
> > > >>>>> > > wrote:
> > > >>>>> > >
> > > >>>>> > >> I found a few problems in your XML.
> > > >>>>> > >> The database you are using doesn't have data with
> > > >>>>> > >> vx_mask
> > > >>>>> > >> 'abcREGabc'.
> > > >>>>> > >> That is why METviewer returns an empty dataset and
can't
> > > >>>>> > >> create a
> > > >>>>> > >> plot.
> > > >>>>> > >> The other problem is how you define fcst_lead. This
> > > >>>>> > >> database
> > > >>>>> > >> has
> > > >>>>> > >> values
> > > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
> > > >>>>> > >> include
> > > >>>>> > >> all
> > > >>>>> > >> fcst_leads
> > > >>>>> > >> this variable can be omitted.
> > > >>>>> > >> To minimize problems with variables values, I would
> > > >>>>> > >> suggest
> > > >>>>> > >> to
> > > >>>>> > >> create XML
> > > >>>>> > >> using GUI and then edit it to fit batch standards.
> > > >>>>> > >> I created a plot using you configurations with GUI
> > > >>>>> > >> (without
> > > >>>>> vx_mack
> > > >>>>> > >> and
> > > >>>>> > >> plot and axis titles) #20200612_173626. You can load
it
> > > >>>>> > >> back
> > > >>>>> > >> to
> > > >>>>> the
> > > >>>>> > >> page by
> > > >>>>> > >> using the "History" tab (left bar on GUI). Find the
> > > >>>>> > >> plot,
> > > >>>>> > >> open XML
> > > >>>>> > >> and
> > > >>>>> > >> click on "Load to page". After that you can recreate
a
> > > >>>>> > >> plot
> > > >>>>> > >> and
> > > >>>>> edit
> > > >>>>> > >> the
> > > >>>>> > >> configurations. You can also download a new XML and
edit
> > > >>>>> > >> it
> > > >>>>> > >> for
> > > >>>>> > >> batch.
> > > >>>>> > >> I attached the example of load XML to this message.
> > > >>>>> > >>
> > > >>>>> > >> Description of rds parameters:
> > > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
> > > >>>>> > >> scripts
> > > >>>>> > >> rds_plots - temporary directory where the plots will
be
> > > >>>>> > >> generated
> > > >>>>> > >> rds_data - temporary directory where the data
retrieved
> > > >>>>> > >> from
> > > >>>>> > >> the
> > > >>>>> > >> database
> > > >>>>> > >> and other intermediate files will be stored
> > > >>>>> > >> rds_scripts - temporary directory where R scripts for
> > > >>>>> > >> this
> > > >>>>> > >> run
> > > >>>>> will
> > > >>>>> > >> be
> > > >>>>> > >> stored
> > > >>>>> > >> After the execution is done all the files from
temporary
> > > >>>>> directories
> > > >>>>> > >> would be removed.
> > > >>>>> > >>
> > > >>>>> > >>
> > > >>>>> > >> We don't have the offitial user guide for METviewer.
> > > >>>>> > >> There
> > > >>>>> > >> are
> > > >>>>> some
> > > >>>>> > >> instructions on how to crate XML config files:
> > > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > > >>>>> > >>
> > > >>>>> > >> Tatiana
> > > >>>>> > >>
> > > >>>>> > >> On Fri Jun 12 11:08:43 2020, edward.strobach at noaa.gov
> > > >>>>> > >> wrote:
> > > >>>>> > >> > Hi,
> > > >>>>> > >> >
> > > >>>>> > >> > I'm attaching the xml below, but am having hard
time
> > > >>>>> > >> > following
> > > >>>>> how
> > > >>>>> > >> > to
> > > >>>>> > >> > go
> > > >>>>> > >> > about the other steps since I have zero experience
on
> > > >>>>> > >> > how
> > > >>>>> > >> > to do
> > > >>>>> > >> > these
> > > >>>>> > >> > things.
> > > >>>>> > >> >
> > > >>>>> > >> > #2. All parameters in <folders> section should NOT
be
> > > >>>>> > >> > changed
> > > >>>>> and
> > > >>>>> > >> > will
> > > >>>>> > >> > be
> > > >>>>> > >> > replaced by other scripts during the batch
execution.
> > > >>>>> > >> > The
> > > >>>>> > >> > data
> > > >>>>> for
> > > >>>>> > >> > the
> > > >>>>> > >> > batch mode is coming from the database and not from
> > > >>>>> > >> > files.
> > > >>>>> > >> > The
> > > >>>>> > >> > name of
> > > >>>>> > >> > the
> > > >>>>> > >> > database is in <database> tag.
> > > >>>>> > >> >
> > > >>>>> > >> > I understand that nothing should be changed in the
> > > >>>>> > >> > folders
> > > >>>>> > >> > section,
> > > >>>>> > >> > but
> > > >>>>> > >> > there is no indication how and where these
variables
> > > >>>>> > >> > are
> > > >>>>> > >> > being
> > > >>>>> > >> > defined.
> > > >>>>> > >> > Grepping these names of the files available does
not
> > > >>>>> > >> > reveal
> > > >>>>> formal
> > > >>>>> > >> > definitions of these variables.  If I understood
> > > >>>>> > >> > exactly
> > > >>>>> > >> > how
> > > >>>>> these
> > > >>>>> > >> > were
> > > >>>>> > >> > being defined, then I would be able connect this
> > > >>>>> > >> > better.
> > > >>>>> > >> >
> > > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
> > > >>>>> > >> > use
> > > >>>>> METviewer
> > > >>>>> > >> > load
> > > >>>>> > >> > mode to load these files to the specific database.
In
> > > >>>>> > >> > the
> > > >>>>> > >> > config
> > > >>>>> > >> > file
> > > >>>>> > >> > for
> > > >>>>> > >> > this process they also can specify the database
group.
> > > >>>>> > >> > For
> > > >>>>> example
> > > >>>>> > >> > database
> > > >>>>> > >> > mv_dev is in Test12 group.
> > > >>>>> > >> > After the data has been loaded to the database it
is
> > > >>>>> > >> > visible and
> > > >>>>> > >> > available
> > > >>>>> > >> > in the GUI and also can be used by the batch mode.
> > > >>>>> > >> > You can always add more data to the database using
the
> > > >>>>> > >> > same load
> > > >>>>> > >> > mode.
> > > >>>>> > >> >
> > > >>>>> > >> > Is there a guide for me to understand the process
that
> > > >>>>> > >> > you
> > > >>>>> > >> > are
> > > >>>>> > >> > referring to?
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> >
> > > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via
RT
> > > >>>>> > >> > <met_help at ucar.edu>
> > > >>>>> > >> > wrote:
> > > >>>>> > >> >
> > > >>>>> > >> > > Hi Edward,
> > > >>>>> > >> > > #1. I don't have an access to NOAA servers and
can't
> > > >>>>> > >> > > view your
> > > >>>>> > >> > > XML
> > > >>>>> > >> > > for the
> > > >>>>> > >> > > batch mode. Please, attache it to this issue and
I
> > > >>>>> > >> > > will
> > > >>>>> > >> > > take a
> > > >>>>> > >> > > look.
> > > >>>>> > >> > >
> > > >>>>> > >> > > #2. All parameters in <folders> section should
NOT
> > > >>>>> > >> > > be
> > > >>>>> > >> > > changed
> > > >>>>> > >> > > and
> > > >>>>> > >> > > will be
> > > >>>>> > >> > > replaced by other scripts during the batch
> > > >>>>> > >> > > execution.
> > > >>>>> > >> > > The data
> > > >>>>> > >> > > for
> > > >>>>> > >> > > the
> > > >>>>> > >> > > batch mode is coming from the database and not
from
> > > >>>>> > >> > > files. The
> > > >>>>> > >> > > name
> > > >>>>> > >> > > of the
> > > >>>>> > >> > > database is in <database> tag.
> > > >>>>> > >> > >
> > > >>>>> > >> > > #3. Users rum MET and produce MET output files.
They
> > > >>>>> > >> > > use
> > > >>>>> > >> > > METviewer
> > > >>>>> > >> > > load
> > > >>>>> > >> > > mode to load these files to the specific
database.
> > > >>>>> > >> > > In
> > > >>>>> > >> > > the
> > > >>>>> config
> > > >>>>> > >> > > file
> > > >>>>> > >> > > for
> > > >>>>> > >> > > this process they also can specify the database
> > > >>>>> > >> > > group.
> > > >>>>> > >> > > For
> > > >>>>> > >> > > example
> > > >>>>> > >> > > database
> > > >>>>> > >> > > mv_dev is in Test12 group.
> > > >>>>> > >> > > After the data has been loaded to the database it
is
> > > >>>>> > >> > > visible
> > > >>>>> and
> > > >>>>> > >> > > available
> > > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
> > > >>>>> > >> > > You can always add more data to the database
using
> > > >>>>> > >> > > the
> > > >>>>> > >> > > same
> > > >>>>> load
> > > >>>>> > >> > > mode.
> > > >>>>> > >> > >
> > > >>>>> > >> > > Tatiana
> > > >>>>> > >> > >
> > > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> > > >>>>> > >> > > edward.strobach at noaa.gov
> > > >>>>> > >> > > wrote:
> > > >>>>> > >> > > > Good morning,
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Over the last two weeks I've made an effort to
> > > >>>>> > >> > > > understand
> > > >>>>> > >> > > > metviewer,
> > > >>>>> > >> > > > test
> > > >>>>> > >> > > > different cases using the history tab, changing
> > > >>>>> > >> > > > information
> > > >>>>> to
> > > >>>>> > >> > > > see
> > > >>>>> > >> > > > what
> > > >>>>> > >> > > > works and what does not, and make connections
> > > >>>>> > >> > > > between
> > > >>>>> > >> > > > the
> > > >>>>> GUI
> > > >>>>> > >> > > > settings
> > > >>>>> > >> > > > and
> > > >>>>> > >> > > > the XML file.  Although my comfort level with
> > > >>>>> > >> > > > metviewer has
> > > >>>>> > >> > > > significantly
> > > >>>>> > >> > > > increased during this time, there are some
> > > >>>>> > >> > > > remaining
> > > >>>>> > >> > > > issues
> > > >>>>> > >> > > > that
> > > >>>>> > >> > > > I'm
> > > >>>>> > >> > > > hoping
> > > >>>>> > >> > > > to clear up.
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > 1.  Problem changing XML file and running with
> > > >>>>> > >> > > > different
> > > >>>>> > >> > > > statistic
> > > >>>>> > >> > > > and
> > > >>>>> > >> > > > plot
> > > >>>>> > >> > > > type:
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me
the
> > > >>>>> > >> > > > working
> > > >>>>> > >> > > > directory
> > > >>>>> > >> > > > to get
> > > >>>>> > >> > > > started which I copied over:
> > > >>>>> > >> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > run his
> > > >>>>> > >> > > > example.
> > > >>>>> > >> > > > running of the script "bash test.sh" produced a
> > > >>>>> > >> > > > CSI
> > > >>>>> > >> > > > plot
> > > >>>>> that
> > > >>>>> > >> > > > was
> > > >>>>> > >> > > > stored
> > > >>>>> > >> > > > here:
> > > >>>>> > >> > > >
> > > >>>>> > >> > >
> > > >>>>> > >>
> > > >>>>>
> >
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Running the script produced the expected
result.
> > > >>>>> > >> > > > The
> > > >>>>> > >> > > > next
> > > >>>>> > >> > > > natural
> > > >>>>> > >> > > > step was
> > > >>>>> > >> > > > to copy hochum_xml.template and create a new
xml
> > > >>>>> > >> > > > file
> > > >>>>> > >> > > > called
> > > >>>>> > >> > > > Strobach_xml.template in order to create a time
> > > >>>>> > >> > > > series
> > > >>>>> > >> > > > where
> > > >>>>> > >> > > > the
> > > >>>>> > >> > > > independent time is fcst_valid_beg instead of
> > > >>>>> > >> > > > obs_thresh.
> > > >>>>> > >> > > > Accordingly,
> > > >>>>> > >> > > > I
> > > >>>>> > >> > > > made changes to the fixed fields (removing
> > > >>>>> > >> > > > fcst_valid_beg)
> > > >>>>> and
> > > >>>>> > >> > > > changed
> > > >>>>> > >> > > > the
> > > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
tried).
> > > >>>>> > >> > > > The
> > > >>>>> error
> > > >>>>> > >> > > > that
> > > >>>>> > >> > > > resulted
> > > >>>>> > >> > > > is as follows:
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
> > > >>>>> > >> > > > 13:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 01
> > > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> > > >>>>> > >> > > > 16:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 01
> > > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> > > >>>>> > >> > > > 19:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 01
> > > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> > > >>>>> > >> > > > 22:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 01
> > > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> > > >>>>> > >> > > > 01:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 02
> > > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> > > >>>>> > >> > > > 04:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 02
> > > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> > > >>>>> > >> > > > 07:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 02
> > > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> > > >>>>> > >> > > > 10:00:00',
> > > >>>>> > >> > > > '2020-05-
> > > >>>>> > >> > > > 02
> > > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > > >>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
> > > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > > >>>>> > >> > > > AND ld.fmean != -9999;
> > > >>>>> > >> > > > No data was returned from database
> > > >>>>> > >> > > > 2020-06-09
> > > >>>>> > >> > > >
> > > >>>>> > >> > >
> > > >>>>> > >>
> > > >>>>>
> >
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > > >>>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR:
> > > >>>>> > >> > > > Caught
> > > >>>>> > >> > > > class
> > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> > > >>>>> > >> > > > printFormattedTable(ResultSet
> > > >>>>> > >> > > > res): result set contained no data
> > > >>>>> > >> > > > **  ERROR: Caught class
> > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > > >>>>> > >> > > > in
> > > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> > > >>>>> > >> > > > contained no
> > > >>>>> > >> > > > data
> > > >>>>> > >> > > > **  ERROR: Caught class
> > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > > >>>>> > >> > > > in
> > > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> > > >>>>> > >> > > > contained no
> > > >>>>> > >> > > > data
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Apparently there was no data during this time.
I
> > > >>>>> > >> > > > tried
> > > >>>>> > >> > > > changing
> > > >>>>> > >> > > > the
> > > >>>>> > >> > > > range
> > > >>>>> > >> > > > of valid times and changed the valid time to
> > > >>>>> > >> > > > different
> > > >>>>> > >> > > > days
> > > >>>>> > >> > > > just in
> > > >>>>> > >> > > > case
> > > >>>>> > >> > > > data was missing.  The issue was the same.  To
> > > >>>>> > >> > > > confirm
> > > >>>>> > >> > > > my
> > > >>>>> > >> > > > logic, I
> > > >>>>> > >> > > > reproduced my attempted approach on metviewer,
> > > >>>>> > >> > > > which I
> > > >>>>> > >> > > > found
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > work.
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > 2.  Running in batch mode
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > I think I lack an understanding of how to set
up
> > > >>>>> > >> > > > batch
> > > >>>>> > >> > > > mode
> > > >>>>> > >> > > > because
> > > >>>>> > >> > > > I
> > > >>>>> > >> > > > don't
> > > >>>>> > >> > > > see where these definitions are being
explicitly
> > > >>>>> > >> > > > defined:
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > <plot_spec>
> > > >>>>> > >> > > >     <connection>
> > > >>>>> > >> > > >         <host>rds_host:3306</host>
> > > >>>>> > >> > > >
> > > >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > > >>>>> > >> > > > <user>rds_user</user>
> > > >>>>> > >> > > > <password>rds_pwd</password>
> > > >>>>> > >> > > > </connection>
> > > >>>>> > >> > > > <rscript>Rscript</rscript>
> > > >>>>> > >> > > > <folders>
> > > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > > >>>>> > >> > > > <plots>rds_plots</plots>
> > > >>>>> > >> > > > <data>rds_data</data>
> > > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > > >>>>> > >> > > > </folders>
> > > >>>>> > >> > > > <plot>
> > > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > I see the <folders> section, but grepping all
> > > >>>>> > >> > > > available
> > > >>>>> files
> > > >>>>> > >> > > > does
> > > >>>>> > >> > > > not
> > > >>>>> > >> > > > reveal how rds_R_work is being defined, as well
as
> > > >>>>> > >> > > > other
> > > >>>>> > >> > > > similarly
> > > >>>>> > >> > > > defined
> > > >>>>> > >> > > > variables.  I think if I know how these are
being
> > > >>>>> > >> > > > defined,
> > > >>>>> > >> > > > then
> > > >>>>> > >> > > > I'll
> > > >>>>> > >> > > > be
> > > >>>>> > >> > > > able to pipe in my information and process the
> > > >>>>> > >> > > > data
> > > >>>>> > >> > > > using
> > > >>>>> met
> > > >>>>> > >> > > > and
> > > >>>>> > >> > > > produce
> > > >>>>> > >> > > > results in batch mode.  Am I right on that?
Also,
> > > >>>>> > >> > > > if
> > > >>>>> > >> > > > I want
> > > >>>>> > >> > > > to use
> > > >>>>> > >> > > > obs for
> > > >>>>> > >> > > > verification, do I add another line in the XML
> > > >>>>> > >> > > > file
> > > >>>>> > >> > > > that
> > > >>>>> > >> > > > designates
> > > >>>>> > >> > > > file
> > > >>>>> > >> > > > location of obs?  I assume that other steps
have
> > > >>>>> > >> > > > to be
> > > >>>>> > >> > > > run
> > > >>>>> > >> > > > before
> > > >>>>> > >> > > > all
> > > >>>>> > >> > > > this.  If data is in prepbufr form, for
example, I
> > > >>>>> > >> > > > would
> > > >>>>> have
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > separately
> > > >>>>> > >> > > > run PB2NC.  If I want grid2obs, then I would
want
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > do that
> > > >>>>> > >> > > > as
> > > >>>>> > >> > > > well.
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > 3. Communicating results to metviewer online
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
here
> > > >>>>> > >> > > > -
> > > >>>>> > >> > > > see
> > > >>>>> > >> > > > attachment
> > > >>>>> > >> > > >
> > > >>>>> > >> > > > How do I get the results I generate to be
uploaded
> > > >>>>> > >> > > > to
> > > >>>>> > >> > > > this
> > > >>>>> > >> > > > location?
> > > >>>>> > >> > > > It's
> > > >>>>> > >> > > > not clear how others are setting this up.
> > > >>>>> > >> > >
> > > >>>>> > >> > >
> > > >>>>> > >> > >
> > > >>>>> > >> > >
> > > >>>>> > >>
> > > >>>>> > >>
> > > >>>>> > >>
> > > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
> > > >>>>> > >> rds_user
> > > >>>>> rds_pwd
> > > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
> > > >>>>> rds_scripts
> > > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
CMAQAODPRODHIGH
> > > >>>>> 2020-05-12
> > > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
> > > >>>>> > >> 05-12
> > > >>>>> 20:00:00
> > > >>>>> > >> 2020-05-12
> > > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
> > > >>>>> > >> 05-13
> > > >>>>> 10:00:00
> > > >>>>> > >> 2020-05-13
> > > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
> > > >>>>> > >> 05-13
> > > >>>>> 14:00:00
> > > >>>>> > >> 2020-05-13
> > > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
> > > >>>>> > >> 05-13
> > > >>>>> 18:00:00
> > > >>>>> > >> 2020-05-13
> > > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
> > > >>>>> > >> 05-13
> > > >>>>> 22:00:00
> > > >>>>> > >> 2020-05-13
> > > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
> > > >>>>> > >> 05-14
> > > >>>>> 12:00:00
> > > >>>>> > >> true
> > > >>>>> > >> median plot_20200612_173626.data
> > > >>>>> > >> plot_20200612_173626.png
> > > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> > > >>>>> > >> false list() list() false false false false false
false
> > > >>>>> > >> true
> > > >>>>> > >> false
> > > >>>>> > >> false
> > > >>>>> > >> false false false true png16m 8.5 11 72 in c(8,4,5,4)
> > > >>>>> > >> c(1,1,0) 1 2
> > > >>>>> > >> 1.4 -2
> > > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5
3
> > > >>>>> > >> #cccccc 1
> > > >>>>> > >> listX 1
> > > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
> > > >>>>> > >> 3
> > > >>>>> > >> 0.8 1
> > > >>>>> > >> #333333
> > > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
> > > >>>>> > >> c(TRUE,TRUE)
> > > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b") c(1,1)
> > > >>>>> > >> c(1,1)
> > > >>>>> c(1,1)
> > > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > > >>>>> > >
> > > >>>>> > >
> > > >>>>> > >
> > > >>>>> > > --
> > > >>>>> > > Edward Strobach
> > > >>>>> > > EMC/NCEP/NWS/
> > > >>>>> > > IMSG Contractor
> > > >>>>> > > Cubicle#: 2029
> > > >>>>> > > 301-683-3717
> > > >>>>> > >
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>> --
> > > >>>> Edward Strobach
> > > >>>> EMC/NCEP/NWS/
> > > >>>> IMSG Contractor
> > > >>>> Cubicle#: 2029
> > > >>>> 301-683-3717
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Edward Strobach
> > > >>> EMC/NCEP/NWS/
> > > >>> IMSG Contractor
> > > >>> Cubicle#: 2029
> > > >>> 301-683-3717
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Edward Strobach
> > > >> EMC/NCEP/NWS/
> > > >> IMSG Contractor
> > > >> Cubicle#: 2029
> > > >> 301-683-3717
> > > >>
> > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> >
> >
> >
> >



------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Tatiana Burek
Time: Wed Jun 17 10:00:26 2020

Ed, now you should have enough space for the data loading.

Tatiana

On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
> I was basically able to build my xml file but it says I have no
> available
> space.  I'm not sure how that can be since I just started adding
data
> to my
> account
>
> scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space
left
> on
> device
> scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space
left
> on
> device
> ERROR: Command returned with non-zero status (1): scp -r ./*
> edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
> 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
> TABLE_SCHEMA                              USER             SIZE_MB
> mv_met_g2o_gefs                           binbin.zhou      140027.31
> mv_met_system_hrefv2_v3                   binbin.zhou      71986.11
>
> On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Thanks Tatiana.  Yes, I've created a database which seems
different
> > from
> > mv_group. How can one create their own group so that I can store
and
> > access
> > the database.  The database exists but cannot be viewed on the
> > metviewer
> > GUI.  The list starts with EMC HREFv3 and ends with vhagerty.  I
> > named my
> > database Ed_Strobach, but that was really intended for the group
> > name.
> >
> > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> >> We don't have database and METviewer running on wcoss.
> >> Instead, users use METviewer-wcoss scripts that help to connect
to
> >> and
> >> use METviewer that is running on AWS. That is why XML connection
and
> >> folder
> >> sections look somewhat different than once from the Docs page.
> >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
> >> mv_create_db_on_aws.sh <user_name> <database_name>
> >> For data loading :
> >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> >> And for batch:
> >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> >>
> >> These scripts copy XML and data files if needed to AWS, adjust
data
> >> path
> >> and users credentials, run METviewer and copy results back to
wcoss.
> >>
> >> Tatiana
> >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> >> > lastly, it does seem thatmysql -u[db_username] -p[db_password]
> >> > -e'create
> >> > database [db_name]
> >> >
> >> > would need to be used to create a database.  I assume that an
sql
> >> > module is
> >> > available on wcoss; I know you don't have access to those
machines
> >> > Tatiana.  If I create my own database with my own name, then I
> >> > should
> >> > be
> >> > able to proceed with project specific tasks under that tab.
> >> > Something
> >> > like
> >> > mv_EdwardStrobach.
> >> >
> >> > I can then create mv_cmaq_prod_v_para which I add data to using
> >> > load_spec.
> >> > Once that step is done, then I can run plot_spec.  That's my
> >> > takeaway
> >> > currently
> >> >
> >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
> >> > edward.strobach at noaa.gov> wrote:
> >> >
> >> > > Actually, it appears that I need to run the database loading
> >> > > module
> >> > > first
> >> > > if I want to change/add a data source.  From there I can run
the
> >> > > plotting
> >> > > batch module.  Is that right?   So if I run the database
loading
> >> > > module,
> >> > > then I can specify my dates and other related info specific
to
> >> > > the
> >> > > dataset.  If I create my database with a unique name, then it
> >> > > should
> >> > > be
> >> > > recognized on the metviewer gui, presumably.
> >> > >
> >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
> >> > > <
> >> > > edward.strobach at noaa.gov> wrote:
> >> > >
> >> > >> It seems that those options must be treated separately
outside
> >> > >> of
> >> > >> plot_spec.  Is it acceptable to create both a load_spec and
> >> > >> plot_spec in
> >> > >> the same xml file?  Would the definitions in load_spec be
able
> >> > >> to
> >> > >> carry
> >> > >> over into plot_spec?  It appears that data loading and
plotting
> >> > >> are
> >> > >> treated
> >> > >> separately.
> >> > >>
> >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
> >> > >> Affiliate <
> >> > >> edward.strobach at noaa.gov> wrote:
> >> > >>
> >> > >>> I do plan to use date_range; however, in an example
provided
> >> > >>> by the
> >> > >>> link
> >> > >>> I see this:
> >> > >>>
> >> > >>> <load_spec>
> >> > >>>   <connection>
> >> > >>>     <host>kemosabe:3306</host>
> >> > >>>     <database>metvdb_hwt</database>
> >> > >>>     <user>pgoldenb</user>
> >> > >>>     <password>pgoldenb</password>
> >> > >>>   </connection>
> >> > >>>
> >> > >>> <date_list name="folder_dates">
> >> > >>>   <start>2010051914V</start>
> >> > >>>   <end><date_offset><day_offset>0</day_offset><hour>6
> >> > >>> </hour></date_offset></end>
> >> > >>>   <inc>3600</inc>
> >> > >>>   <format>yyyyMMddHH'V'</format>
> >> > >>> </date_list>
> >> > >>>
> >> > >>> <met_version>V3.0</met_version>
> >> > >>>
> >> > >>> <verbose>false</verbose>
> >> > >>> <insert_size>1</insert_size>
> >> > >>> <mode_header_db_check>true</mode_header_db_check>
> >> > >>> <drop_indexes>false</drop_indexes>
> >> > >>> <apply_indexes>true</apply_indexes>
> >> > >>> <group>Group name</group>
> >> > >>> <load_stat>true</load_stat>
> >> > >>> <load_mode>true</load_mode>
> >> > >>> <load_mtd>true</load_mtd>
> >> > >>> <load_mpr>false</load_mpr>
> >> > >>>
> >> > >>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> >> > >>> <load_val>
> >> > >>>   <field name="model">
> >> > >>>     <val>arw</val>
> >> > >>>     <val>nmm</val>
> >> > >>>   </field>
> >> > >>>
> >> > >>> <field name="valid_time">
> >> > >>>   <date_list name="folder_dates"/>
> >> > >>> </field>
> >> > >>>
> >> > >>> <field name="vx_mask">
> >> > >>>   <val>FULL</val>
> >> > >>>   <val>SWC</val>
> >> > >>> </field>
> >> > >>> </load_val>
> >> > >>> </load_spec>
> >> > >>>
> >> > >>> Are similar settings allowed with <plot_spec>?  I see
> >> > >>> <connection>,
> >> > >>> which seems carried over.  I'm looking to specify my
> >> > >>> folder_tmpl in
> >> > >>> <plot_spec>, which according to Tara, should be possible.
> >> > >>> Since my
> >> > >>> directory structure looks like this:
> >> > >>>
> >> > >>>
> >> > >>>
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> >> > >>>
> >> > >>> I would do something like this I would think:
> >> > >>>
> >> > >>> .......
> >> > >>>
> >> > >>> <date_list name="folder_dates">
> >> > >>>   <start>20200612/start>
> >> > >>>   <end><date_offset><day_offset>0</day_offset></end>
> >> > >>>   <inc>86400</inc>
> >> > >>>   <format>yyyyMMdd</format>
> >> > >>> </date_list>
> >> > >>>
> >> > >>> <folder_tmpl>
> >> > >>>
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> >> > >>> </folder_tmpl>
> >> > >>> <load_val>
> >> > >>>   <field name="model">
> >> > >>>     <val>prod</val>
> >> > >>>   </field>
> >> > >>>
> >> > >>> <field name="valid_time">
> >> > >>>   <date_list name="folder_dates"/>
> >> > >>> </field>
> >> > >>>
> >> > >>> ......
> >> > >>>
> >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
> >> > >>> Affiliate <
> >> > >>> edward.strobach at noaa.gov> wrote:
> >> > >>>
> >> > >>>> Thank you.  This is a very helpful example
> >> > >>>>
> >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> >> > >>>> <met_help at ucar.edu>
> >> > >>>> wrote:
> >> > >>>>
> >> > >>>>> Hi,
> >> > >>>>> Here is some examples how to use <date_range> and
> >> > >>>>> <date_range_list> in
> >> > >>>>> batch XML.
> >> > >>>>> First, you need to defined your range. You can have more
> >> > >>>>> then one
> >> > >>>>> <date_range>:
> >> > >>>>> <date_range name="may">
> >> > >>>>>         <start>2019-05-01 00:00:00</start>
> >> > >>>>>         <end>2019-05-31 23:59:59</end>
> >> > >>>>> </date_range>
> >> > >>>>> <date_range name="march">
> >> > >>>>>         <start>2019-03-01 00:00:00</start>
> >> > >>>>>         <end>2019-03-31 23:59:59</end>
> >> > >>>>> </date_range>
> >> > >>>>>
> >> > >>>>> and then use one or all date_ranges:
> >> > >>>>> <plot_fix>
> >> > >>>>>     <field name="fcst_init_beg">
> >> > >>>>>         <date_range name="march"/>
> >> > >>>>>         <date_range name="may"/>
> >> > >>>>>    </field>...
> >> > >>>>> This configuration would include all dates for March and
May
> >> > >>>>> of
> >> > >>>>> 2019.
> >> > >>>>>
> >> > >>>>> If you need more flexible configuration you can use
> >> > >>>>> date_range_list.
> >> > >>>>> For example, you want to select first 6 hours every day
> >> > >>>>> beetween
> >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
> >> > >>>>> look
> >> > >>>>> like this:
> >> > >>>>> <date_range_list name = 'all'>
> >> > >>>>>        <range_start>2020-01-01 00:00:00</range_start>
> >> > >>>>>         <range_end>2020-01-05 00:00:00</range_end>
> >> > >>>>>         <range_length>360</range_length>
> >> > >>>>>         <inc>1440</inc>
> >> > >>>>> </date_range_list>
> >> > >>>>> where the 1st and 2nd values are the start and end,
> >> > >>>>> range_length - the length of a range in minutes(6 houres)
> >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> >> > >>>>> This example would create following ranges:
> >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> >> > >>>>>
> >> > >>>>> Also you can add more parameters to <plot_fix> to filter
> >> > >>>>> your
> >> > >>>>> data.
> >> > >>>>> For example, if you want to select only dates with 00:00
you
> >> > >>>>> can
> >> > >>>>> specify
> >> > >>>>>  <field name="valid_hour">
> >> > >>>>>         <val>00</val>
> >> > >>>>>  </field>
> >> > >>>>>
> >> > >>>>> Tatiana
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
> >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
> >> > >>>>> > of a
> >> > >>>>> > separate
> >> > >>>>> > script that pipes dates (and other relevant
information)
> >> > >>>>> > into
> >> > >>>>> > the XML
> >> > >>>>> > file
> >> > >>>>> > used to generate the results.
> >> > >>>>> >
> >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach - NOAA
> >> > >>>>> > Affiliate <
> >> > >>>>> > edward.strobach at noaa.gov> wrote:
> >> > >>>>> >
> >> > >>>>> > > Thanks Tatiana.  These resources are helpful.
> >> > >>>>> > >
> >> > >>>>> > > One last question.  Simply typing lines like this in
can
> >> > >>>>> > > take
> >> > >>>>> > > forever:
> >> > >>>>> > >
> >> > >>>>> > > <plot_fix>
> >> > >>>>> > >     <field equalize="false" name="fcst_init_beg">
> >> > >>>>> > >         <set name="fcst_init_beg_0">
> >> > >>>>> > >             <val>2019-05-01 12:00:00</val>
> >> > >>>>> > >             <val>2019-05-02 06:00:00</val>
> >> > >>>>> > >             <val>2019-05-02 12:00:00</val>
> >> > >>>>> > >             <val>2019-05-03 06:00:00</val>
> >> > >>>>> > >             <val>2019-05-03 12:00:00</val>
> >> > >>>>> > >
> >> > >>>>> > >
> >> > >>>>> > > Is there a better way?   I see date_range,
> >> > >>>>> > > date_range_list,
> >> > >>>>> > > or
> >> > >>>>> > > date_list.
> >> > >>>>> > > Can this be used instead of typing individual values?
> >> > >>>>> > > This
> >> > >>>>> > > could
> >> > >>>>> > > prove
> >> > >>>>> > > pretty daunting if one would have to type in 700
entries
> >> > >>>>> > > each
> >> > >>>>> > > time.
> >> > >>>>> > >
> >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> >> > >>>>> > > <met_help at ucar.edu>
> >> > >>>>> > > wrote:
> >> > >>>>> > >
> >> > >>>>> > >> I found a few problems in your XML.
> >> > >>>>> > >> The database you are using doesn't have data with
> >> > >>>>> > >> vx_mask
> >> > >>>>> > >> 'abcREGabc'.
> >> > >>>>> > >> That is why METviewer returns an empty dataset and
> >> > >>>>> > >> can't
> >> > >>>>> > >> create a
> >> > >>>>> > >> plot.
> >> > >>>>> > >> The other problem is how you define fcst_lead. This
> >> > >>>>> > >> database
> >> > >>>>> > >> has
> >> > >>>>> > >> values
> >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
> >> > >>>>> > >> include
> >> > >>>>> > >> all
> >> > >>>>> > >> fcst_leads
> >> > >>>>> > >> this variable can be omitted.
> >> > >>>>> > >> To minimize problems with variables values, I would
> >> > >>>>> > >> suggest
> >> > >>>>> > >> to
> >> > >>>>> > >> create XML
> >> > >>>>> > >> using GUI and then edit it to fit batch standards.
> >> > >>>>> > >> I created a plot using you configurations with GUI
> >> > >>>>> > >> (without
> >> > >>>>> vx_mack
> >> > >>>>> > >> and
> >> > >>>>> > >> plot and axis titles) #20200612_173626. You can load
it
> >> > >>>>> > >> back
> >> > >>>>> > >> to
> >> > >>>>> the
> >> > >>>>> > >> page by
> >> > >>>>> > >> using the "History" tab (left bar on GUI). Find the
> >> > >>>>> > >> plot,
> >> > >>>>> > >> open XML
> >> > >>>>> > >> and
> >> > >>>>> > >> click on "Load to page". After that you can recreate
a
> >> > >>>>> > >> plot
> >> > >>>>> > >> and
> >> > >>>>> edit
> >> > >>>>> > >> the
> >> > >>>>> > >> configurations. You can also download a new XML and
> >> > >>>>> > >> edit it
> >> > >>>>> > >> for
> >> > >>>>> > >> batch.
> >> > >>>>> > >> I attached the example of load XML to this message.
> >> > >>>>> > >>
> >> > >>>>> > >> Description of rds parameters:
> >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
> >> > >>>>> > >> scripts
> >> > >>>>> > >> rds_plots - temporary directory where the plots will
be
> >> > >>>>> > >> generated
> >> > >>>>> > >> rds_data - temporary directory where the data
retrieved
> >> > >>>>> > >> from
> >> > >>>>> > >> the
> >> > >>>>> > >> database
> >> > >>>>> > >> and other intermediate files will be stored
> >> > >>>>> > >> rds_scripts - temporary directory where R scripts
for
> >> > >>>>> > >> this
> >> > >>>>> > >> run
> >> > >>>>> will
> >> > >>>>> > >> be
> >> > >>>>> > >> stored
> >> > >>>>> > >> After the execution is done all the files from
> >> > >>>>> > >> temporary
> >> > >>>>> directories
> >> > >>>>> > >> would be removed.
> >> > >>>>> > >>
> >> > >>>>> > >>
> >> > >>>>> > >> We don't have the offitial user guide for METviewer.
> >> > >>>>> > >> There
> >> > >>>>> > >> are
> >> > >>>>> some
> >> > >>>>> > >> instructions on how to crate XML config files:
> >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> >> > >>>>> > >>
> >> > >>>>> > >> Tatiana
> >> > >>>>> > >>
> >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
> >> > >>>>> > >> wrote:
> >> > >>>>> > >> > Hi,
> >> > >>>>> > >> >
> >> > >>>>> > >> > I'm attaching the xml below, but am having hard
time
> >> > >>>>> > >> > following
> >> > >>>>> how
> >> > >>>>> > >> > to
> >> > >>>>> > >> > go
> >> > >>>>> > >> > about the other steps since I have zero experience
on
> >> > >>>>> > >> > how
> >> > >>>>> > >> > to do
> >> > >>>>> > >> > these
> >> > >>>>> > >> > things.
> >> > >>>>> > >> >
> >> > >>>>> > >> > #2. All parameters in <folders> section should NOT
be
> >> > >>>>> > >> > changed
> >> > >>>>> and
> >> > >>>>> > >> > will
> >> > >>>>> > >> > be
> >> > >>>>> > >> > replaced by other scripts during the batch
execution.
> >> > >>>>> > >> > The
> >> > >>>>> > >> > data
> >> > >>>>> for
> >> > >>>>> > >> > the
> >> > >>>>> > >> > batch mode is coming from the database and not
from
> >> > >>>>> > >> > files.
> >> > >>>>> > >> > The
> >> > >>>>> > >> > name of
> >> > >>>>> > >> > the
> >> > >>>>> > >> > database is in <database> tag.
> >> > >>>>> > >> >
> >> > >>>>> > >> > I understand that nothing should be changed in the
> >> > >>>>> > >> > folders
> >> > >>>>> > >> > section,
> >> > >>>>> > >> > but
> >> > >>>>> > >> > there is no indication how and where these
variables
> >> > >>>>> > >> > are
> >> > >>>>> > >> > being
> >> > >>>>> > >> > defined.
> >> > >>>>> > >> > Grepping these names of the files available does
not
> >> > >>>>> > >> > reveal
> >> > >>>>> formal
> >> > >>>>> > >> > definitions of these variables.  If I understood
> >> > >>>>> > >> > exactly
> >> > >>>>> > >> > how
> >> > >>>>> these
> >> > >>>>> > >> > were
> >> > >>>>> > >> > being defined, then I would be able connect this
> >> > >>>>> > >> > better.
> >> > >>>>> > >> >
> >> > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
> >> > >>>>> > >> > use
> >> > >>>>> METviewer
> >> > >>>>> > >> > load
> >> > >>>>> > >> > mode to load these files to the specific database.
In
> >> > >>>>> > >> > the
> >> > >>>>> > >> > config
> >> > >>>>> > >> > file
> >> > >>>>> > >> > for
> >> > >>>>> > >> > this process they also can specify the database
> >> > >>>>> > >> > group. For
> >> > >>>>> example
> >> > >>>>> > >> > database
> >> > >>>>> > >> > mv_dev is in Test12 group.
> >> > >>>>> > >> > After the data has been loaded to the database it
is
> >> > >>>>> > >> > visible and
> >> > >>>>> > >> > available
> >> > >>>>> > >> > in the GUI and also can be used by the batch mode.
> >> > >>>>> > >> > You can always add more data to the database using
> >> > >>>>> > >> > the
> >> > >>>>> > >> > same load
> >> > >>>>> > >> > mode.
> >> > >>>>> > >> >
> >> > >>>>> > >> > Is there a guide for me to understand the process
> >> > >>>>> > >> > that you
> >> > >>>>> > >> > are
> >> > >>>>> > >> > referring to?
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> >
> >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek via
RT
> >> > >>>>> > >> > <met_help at ucar.edu>
> >> > >>>>> > >> > wrote:
> >> > >>>>> > >> >
> >> > >>>>> > >> > > Hi Edward,
> >> > >>>>> > >> > > #1. I don't have an access to NOAA servers and
> >> > >>>>> > >> > > can't
> >> > >>>>> > >> > > view your
> >> > >>>>> > >> > > XML
> >> > >>>>> > >> > > for the
> >> > >>>>> > >> > > batch mode. Please, attache it to this issue and
I
> >> > >>>>> > >> > > will
> >> > >>>>> > >> > > take a
> >> > >>>>> > >> > > look.
> >> > >>>>> > >> > >
> >> > >>>>> > >> > > #2. All parameters in <folders> section should
NOT
> >> > >>>>> > >> > > be
> >> > >>>>> > >> > > changed
> >> > >>>>> > >> > > and
> >> > >>>>> > >> > > will be
> >> > >>>>> > >> > > replaced by other scripts during the batch
> >> > >>>>> > >> > > execution.
> >> > >>>>> > >> > > The data
> >> > >>>>> > >> > > for
> >> > >>>>> > >> > > the
> >> > >>>>> > >> > > batch mode is coming from the database and not
from
> >> > >>>>> > >> > > files. The
> >> > >>>>> > >> > > name
> >> > >>>>> > >> > > of the
> >> > >>>>> > >> > > database is in <database> tag.
> >> > >>>>> > >> > >
> >> > >>>>> > >> > > #3. Users rum MET and produce MET output files.
> >> > >>>>> > >> > > They use
> >> > >>>>> > >> > > METviewer
> >> > >>>>> > >> > > load
> >> > >>>>> > >> > > mode to load these files to the specific
database.
> >> > >>>>> > >> > > In
> >> > >>>>> > >> > > the
> >> > >>>>> config
> >> > >>>>> > >> > > file
> >> > >>>>> > >> > > for
> >> > >>>>> > >> > > this process they also can specify the database
> >> > >>>>> > >> > > group.
> >> > >>>>> > >> > > For
> >> > >>>>> > >> > > example
> >> > >>>>> > >> > > database
> >> > >>>>> > >> > > mv_dev is in Test12 group.
> >> > >>>>> > >> > > After the data has been loaded to the database
it
> >> > >>>>> > >> > > is
> >> > >>>>> > >> > > visible
> >> > >>>>> and
> >> > >>>>> > >> > > available
> >> > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
> >> > >>>>> > >> > > You can always add more data to the database
using
> >> > >>>>> > >> > > the
> >> > >>>>> > >> > > same
> >> > >>>>> load
> >> > >>>>> > >> > > mode.
> >> > >>>>> > >> > >
> >> > >>>>> > >> > > Tatiana
> >> > >>>>> > >> > >
> >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> >> > >>>>> > >> > > edward.strobach at noaa.gov
> >> > >>>>> > >> > > wrote:
> >> > >>>>> > >> > > > Good morning,
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Over the last two weeks I've made an effort to
> >> > >>>>> > >> > > > understand
> >> > >>>>> > >> > > > metviewer,
> >> > >>>>> > >> > > > test
> >> > >>>>> > >> > > > different cases using the history tab,
changing
> >> > >>>>> > >> > > > information
> >> > >>>>> to
> >> > >>>>> > >> > > > see
> >> > >>>>> > >> > > > what
> >> > >>>>> > >> > > > works and what does not, and make connections
> >> > >>>>> > >> > > > between
> >> > >>>>> > >> > > > the
> >> > >>>>> GUI
> >> > >>>>> > >> > > > settings
> >> > >>>>> > >> > > > and
> >> > >>>>> > >> > > > the XML file.  Although my comfort level with
> >> > >>>>> > >> > > > metviewer has
> >> > >>>>> > >> > > > significantly
> >> > >>>>> > >> > > > increased during this time, there are some
> >> > >>>>> > >> > > > remaining
> >> > >>>>> > >> > > > issues
> >> > >>>>> > >> > > > that
> >> > >>>>> > >> > > > I'm
> >> > >>>>> > >> > > > hoping
> >> > >>>>> > >> > > > to clear up.
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > 1.  Problem changing XML file and running with
> >> > >>>>> > >> > > > different
> >> > >>>>> > >> > > > statistic
> >> > >>>>> > >> > > > and
> >> > >>>>> > >> > > > plot
> >> > >>>>> > >> > > > type:
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided me
> >> > >>>>> > >> > > > the
> >> > >>>>> > >> > > > working
> >> > >>>>> > >> > > > directory
> >> > >>>>> > >> > > > to get
> >> > >>>>> > >> > > > started which I copied over:
> >> > >>>>> > >> > > >
> >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
> >> > >>>>> > >> > > > to
> >> > >>>>> > >> > > > run his
> >> > >>>>> > >> > > > example.
> >> > >>>>> > >> > > > running of the script "bash test.sh" produced
a
> >> > >>>>> > >> > > > CSI
> >> > >>>>> > >> > > > plot
> >> > >>>>> that
> >> > >>>>> > >> > > > was
> >> > >>>>> > >> > > > stored
> >> > >>>>> > >> > > > here:
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > >
> >> > >>>>> > >>
> >> > >>>>>
> >>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Running the script produced the expected
result.
> >> > >>>>> > >> > > > The
> >> > >>>>> > >> > > > next
> >> > >>>>> > >> > > > natural
> >> > >>>>> > >> > > > step was
> >> > >>>>> > >> > > > to copy hochum_xml.template and create a new
xml
> >> > >>>>> > >> > > > file
> >> > >>>>> > >> > > > called
> >> > >>>>> > >> > > > Strobach_xml.template in order to create a
time
> >> > >>>>> > >> > > > series
> >> > >>>>> > >> > > > where
> >> > >>>>> > >> > > > the
> >> > >>>>> > >> > > > independent time is fcst_valid_beg instead of
> >> > >>>>> > >> > > > obs_thresh.
> >> > >>>>> > >> > > > Accordingly,
> >> > >>>>> > >> > > > I
> >> > >>>>> > >> > > > made changes to the fixed fields (removing
> >> > >>>>> > >> > > > fcst_valid_beg)
> >> > >>>>> and
> >> > >>>>> > >> > > > changed
> >> > >>>>> > >> > > > the
> >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
> >> > >>>>> > >> > > > tried).
> >> > >>>>> > >> > > > The
> >> > >>>>> error
> >> > >>>>> > >> > > > that
> >> > >>>>> > >> > > > resulted
> >> > >>>>> > >> > > > is as follows:
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-01
> >> > >>>>> > >> > > > 13:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 01
> >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> >> > >>>>> > >> > > > 16:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 01
> >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> >> > >>>>> > >> > > > 19:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 01
> >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> >> > >>>>> > >> > > > 22:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 01
> >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> >> > >>>>> > >> > > > 01:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 02
> >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> >> > >>>>> > >> > > > 04:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 02
> >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> >> > >>>>> > >> > > > 07:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 02
> >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> >> > >>>>> > >> > > > 10:00:00',
> >> > >>>>> > >> > > > '2020-05-
> >> > >>>>> > >> > > > 02
> >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> >> > >>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
> >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> >> > >>>>> > >> > > > AND ld.fmean != -9999;
> >> > >>>>> > >> > > > No data was returned from database
> >> > >>>>> > >> > > > 2020-06-09
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > >
> >> > >>>>> > >>
> >> > >>>>>
> >>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> >> > >>>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR:
> >> > >>>>> > >> > > > Caught
> >> > >>>>> > >> > > > class
> >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> >> > >>>>> > >> > > > printFormattedTable(ResultSet
> >> > >>>>> > >> > > > res): result set contained no data
> >> > >>>>> > >> > > > **  ERROR: Caught class
> >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >> > >>>>> > >> > > > in
> >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> >> > >>>>> > >> > > > contained no
> >> > >>>>> > >> > > > data
> >> > >>>>> > >> > > > **  ERROR: Caught class
> >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> >> > >>>>> > >> > > > in
> >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result set
> >> > >>>>> > >> > > > contained no
> >> > >>>>> > >> > > > data
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Apparently there was no data during this time.
I
> >> > >>>>> > >> > > > tried
> >> > >>>>> > >> > > > changing
> >> > >>>>> > >> > > > the
> >> > >>>>> > >> > > > range
> >> > >>>>> > >> > > > of valid times and changed the valid time to
> >> > >>>>> > >> > > > different
> >> > >>>>> > >> > > > days
> >> > >>>>> > >> > > > just in
> >> > >>>>> > >> > > > case
> >> > >>>>> > >> > > > data was missing.  The issue was the same.  To
> >> > >>>>> > >> > > > confirm
> >> > >>>>> > >> > > > my
> >> > >>>>> > >> > > > logic, I
> >> > >>>>> > >> > > > reproduced my attempted approach on metviewer,
> >> > >>>>> > >> > > > which I
> >> > >>>>> > >> > > > found
> >> > >>>>> > >> > > > to
> >> > >>>>> > >> > > > work.
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > 2.  Running in batch mode
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > I think I lack an understanding of how to set
up
> >> > >>>>> > >> > > > batch
> >> > >>>>> > >> > > > mode
> >> > >>>>> > >> > > > because
> >> > >>>>> > >> > > > I
> >> > >>>>> > >> > > > don't
> >> > >>>>> > >> > > > see where these definitions are being
explicitly
> >> > >>>>> > >> > > > defined:
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > <plot_spec>
> >> > >>>>> > >> > > >     <connection>
> >> > >>>>> > >> > > >         <host>rds_host:3306</host>
> >> > >>>>> > >> > > >
> >> > >>>>> <database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> >> > >>>>> > >> > > > <user>rds_user</user>
> >> > >>>>> > >> > > > <password>rds_pwd</password>
> >> > >>>>> > >> > > > </connection>
> >> > >>>>> > >> > > > <rscript>Rscript</rscript>
> >> > >>>>> > >> > > > <folders>
> >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> >> > >>>>> > >> > > > <plots>rds_plots</plots>
> >> > >>>>> > >> > > > <data>rds_data</data>
> >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> >> > >>>>> > >> > > > </folders>
> >> > >>>>> > >> > > > <plot>
> >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > I see the <folders> section, but grepping all
> >> > >>>>> > >> > > > available
> >> > >>>>> files
> >> > >>>>> > >> > > > does
> >> > >>>>> > >> > > > not
> >> > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well
> >> > >>>>> > >> > > > as
> >> > >>>>> > >> > > > other
> >> > >>>>> > >> > > > similarly
> >> > >>>>> > >> > > > defined
> >> > >>>>> > >> > > > variables.  I think if I know how these are
being
> >> > >>>>> > >> > > > defined,
> >> > >>>>> > >> > > > then
> >> > >>>>> > >> > > > I'll
> >> > >>>>> > >> > > > be
> >> > >>>>> > >> > > > able to pipe in my information and process the
> >> > >>>>> > >> > > > data
> >> > >>>>> > >> > > > using
> >> > >>>>> met
> >> > >>>>> > >> > > > and
> >> > >>>>> > >> > > > produce
> >> > >>>>> > >> > > > results in batch mode.  Am I right on that?
> >> > >>>>> > >> > > > Also, if
> >> > >>>>> > >> > > > I want
> >> > >>>>> > >> > > > to use
> >> > >>>>> > >> > > > obs for
> >> > >>>>> > >> > > > verification, do I add another line in the XML
> >> > >>>>> > >> > > > file
> >> > >>>>> > >> > > > that
> >> > >>>>> > >> > > > designates
> >> > >>>>> > >> > > > file
> >> > >>>>> > >> > > > location of obs?  I assume that other steps
have
> >> > >>>>> > >> > > > to be
> >> > >>>>> > >> > > > run
> >> > >>>>> > >> > > > before
> >> > >>>>> > >> > > > all
> >> > >>>>> > >> > > > this.  If data is in prepbufr form, for
example,
> >> > >>>>> > >> > > > I
> >> > >>>>> > >> > > > would
> >> > >>>>> have
> >> > >>>>> > >> > > > to
> >> > >>>>> > >> > > > separately
> >> > >>>>> > >> > > > run PB2NC.  If I want grid2obs, then I would
want
> >> > >>>>> > >> > > > to
> >> > >>>>> > >> > > > do that
> >> > >>>>> > >> > > > as
> >> > >>>>> > >> > > > well.
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > 3. Communicating results to metviewer online
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
> >> > >>>>> > >> > > > here -
> >> > >>>>> > >> > > > see
> >> > >>>>> > >> > > > attachment
> >> > >>>>> > >> > > >
> >> > >>>>> > >> > > > How do I get the results I generate to be
> >> > >>>>> > >> > > > uploaded to
> >> > >>>>> > >> > > > this
> >> > >>>>> > >> > > > location?
> >> > >>>>> > >> > > > It's
> >> > >>>>> > >> > > > not clear how others are setting this up.
> >> > >>>>> > >> > >
> >> > >>>>> > >> > >
> >> > >>>>> > >> > >
> >> > >>>>> > >> > >
> >> > >>>>> > >>
> >> > >>>>> > >>
> >> > >>>>> > >>
> >> > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
> >> > >>>>> > >> rds_user
> >> > >>>>> rds_pwd
> >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
> >> > >>>>> rds_scripts
> >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
> >> > >>>>> > >> CMAQAODPRODHIGH
> >> > >>>>> 2020-05-12
> >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
> >> > >>>>> > >> 05-12
> >> > >>>>> 20:00:00
> >> > >>>>> > >> 2020-05-12
> >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
> >> > >>>>> > >> 05-13
> >> > >>>>> 10:00:00
> >> > >>>>> > >> 2020-05-13
> >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
> >> > >>>>> > >> 05-13
> >> > >>>>> 14:00:00
> >> > >>>>> > >> 2020-05-13
> >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
> >> > >>>>> > >> 05-13
> >> > >>>>> 18:00:00
> >> > >>>>> > >> 2020-05-13
> >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
> >> > >>>>> > >> 05-13
> >> > >>>>> 22:00:00
> >> > >>>>> > >> 2020-05-13
> >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
> >> > >>>>> > >> 05-14
> >> > >>>>> 12:00:00
> >> > >>>>> > >> true
> >> > >>>>> > >> median plot_20200612_173626.data
> >> > >>>>> > >> plot_20200612_173626.png
> >> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> >> > >>>>> > >> false list() list() false false false false false
false
> >> > >>>>> > >> true
> >> > >>>>> > >> false
> >> > >>>>> > >> false
> >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
> >> > >>>>> > >> c(1,1,0) 1 2
> >> > >>>>> > >> 1.4 -2
> >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2 0.5
3
> >> > >>>>> > >> #cccccc 1
> >> > >>>>> > >> listX 1
> >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
> >> > >>>>> > >> 3
> >> > >>>>> > >> 0.8 1
> >> > >>>>> > >> #333333
> >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
> >> > >>>>> > >> c(TRUE,TRUE)
> >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
> >> > >>>>> > >> c(1,1)
> >> > >>>>> c(1,1)
> >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> >> > >>>>> > >
> >> > >>>>> > >
> >> > >>>>> > >
> >> > >>>>> > > --
> >> > >>>>> > > Edward Strobach
> >> > >>>>> > > EMC/NCEP/NWS/
> >> > >>>>> > > IMSG Contractor
> >> > >>>>> > > Cubicle#: 2029
> >> > >>>>> > > 301-683-3717
> >> > >>>>> > >
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>
> >> > >>>> --
> >> > >>>> Edward Strobach
> >> > >>>> EMC/NCEP/NWS/
> >> > >>>> IMSG Contractor
> >> > >>>> Cubicle#: 2029
> >> > >>>> 301-683-3717
> >> > >>>>
> >> > >>>
> >> > >>>
> >> > >>> --
> >> > >>> Edward Strobach
> >> > >>> EMC/NCEP/NWS/
> >> > >>> IMSG Contractor
> >> > >>> Cubicle#: 2029
> >> > >>> 301-683-3717
> >> > >>>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Edward Strobach
> >> > >> EMC/NCEP/NWS/
> >> > >> IMSG Contractor
> >> > >> Cubicle#: 2029
> >> > >> 301-683-3717
> >> > >>
> >> > >
> >> > >
> >> > > --
> >> > > Edward Strobach
> >> > > EMC/NCEP/NWS/
> >> > > IMSG Contractor
> >> > > Cubicle#: 2029
> >> > > 301-683-3717
> >> > >
> >>
> >>
> >>
> >>
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >



------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 10:00:27 2020

I understand, but I made mv_cmaq_para_metplus, and that is not listed.
I
ran a script called check_my_db.sh and it was part of the list.  I see
that
many people have their own groups, and with that group they have
separate
databases.  As you can see, none of the groups have mv_ in front of
them.
This is reserved for the database.  The group does not need mv_.
Please
see attached example.

On Wed, Jun 17, 2020 at 11:41 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Only databases that have a name that starts with 'mv_'are visible in
GUI.
> That is why 'Ed_Strobach' is not on the list.
> If database doesn't have a group it will end up in "NO GROUP" group.
> You specify the name for the group you want to see your database in
when
> you load data.
> There is a tag <group> where you put the group name.
>
> Tatiana
>
> On Tue Jun 16 15:40:50 2020, edward.strobach at noaa.gov wrote:
> > Thanks Tatiana.  Yes, I've created a database which seems
different
> > from
> > mv_group. How can one create their own group so that I can store
and
> > access
> > the database.  The database exists but cannot be viewed on the
> > metviewer
> > GUI.  The list starts with EMC HREFv3 and ends with vhagerty.  I
named
> > my
> > database Ed_Strobach, but that was really intended for the group
name.
> >
> > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > We don't have database and METviewer running on wcoss.
> > > Instead, users use METviewer-wcoss scripts that help to connect
to
> > > and use
> > > METviewer that is running on AWS. That is why XML connection and
> > > folder
> > > sections look somewhat different than once from the Docs page.
> > > To create a database on AWS you need to run
mv_create_db_on_aws.sh
> > > mv_create_db_on_aws.sh <user_name> <database_name>
> > > For data loading :
> > > mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> > > And for batch:
> > > mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> > >
> > > These scripts copy XML and data files if needed to AWS, adjust
data
> > > path
> > > and users credentials, run METviewer and copy results back to
wcoss.
> > >
> > > Tatiana
> > > On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> > > > lastly, it does seem thatmysql -u[db_username] -p[db_password]
> > > > -e'create
> > > > database [db_name]
> > > >
> > > > would need to be used to create a database.  I assume that an
sql
> > > > module is
> > > > available on wcoss; I know you don't have access to those
machines
> > > > Tatiana.  If I create my own database with my own name, then I
> > > > should
> > > > be
> > > > able to proceed with project specific tasks under that tab.
> > > > Something
> > > > like
> > > > mv_EdwardStrobach.
> > > >
> > > > I can then create mv_cmaq_prod_v_para which I add data to
using
> > > > load_spec.
> > > > Once that step is done, then I can run plot_spec.  That's my
> > > > takeaway
> > > > currently
> > > >
> > > > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
> > > > edward.strobach at noaa.gov> wrote:
> > > >
> > > > > Actually, it appears that I need to run the database loading
> > > > > module
> > > > > first
> > > > > if I want to change/add a data source.  From there I can run
the
> > > > > plotting
> > > > > batch module.  Is that right?   So if I run the database
loading
> > > > > module,
> > > > > then I can specify my dates and other related info specific
to
> > > > > the
> > > > > dataset.  If I create my database with a unique name, then
it
> > > > > should
> > > > > be
> > > > > recognized on the metviewer gui, presumably.
> > > > >
> > > > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
> > > > > <
> > > > > edward.strobach at noaa.gov> wrote:
> > > > >
> > > > >> It seems that those options must be treated separately
outside
> > > > >> of
> > > > >> plot_spec.  Is it acceptable to create both a load_spec and
> > > > >> plot_spec in
> > > > >> the same xml file?  Would the definitions in load_spec be
able
> > > > >> to
> > > > >> carry
> > > > >> over into plot_spec?  It appears that data loading and
plotting
> > > > >> are
> > > > >> treated
> > > > >> separately.
> > > > >>
> > > > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
Affiliate
> > > > >> <
> > > > >> edward.strobach at noaa.gov> wrote:
> > > > >>
> > > > >>> I do plan to use date_range; however, in an example
provided by
> > > > >>> the
> > > > >>> link
> > > > >>> I see this:
> > > > >>>
> > > > >>> <load_spec>
> > > > >>>   <connection>
> > > > >>>     <host>kemosabe:3306</host>
> > > > >>>     <database>metvdb_hwt</database>
> > > > >>>     <user>pgoldenb</user>
> > > > >>>     <password>pgoldenb</password>
> > > > >>>   </connection>
> > > > >>>
> > > > >>> <date_list name="folder_dates">
> > > > >>>   <start>2010051914V</start>
> > > > >>>   <end><date_offset><day_offset>0</day_offset><hour>6
> > > > >>> </hour></date_offset></end>
> > > > >>>   <inc>3600</inc>
> > > > >>>   <format>yyyyMMddHH'V'</format>
> > > > >>> </date_list>
> > > > >>>
> > > > >>> <met_version>V3.0</met_version>
> > > > >>>
> > > > >>> <verbose>false</verbose>
> > > > >>> <insert_size>1</insert_size>
> > > > >>> <mode_header_db_check>true</mode_header_db_check>
> > > > >>> <drop_indexes>false</drop_indexes>
> > > > >>> <apply_indexes>true</apply_indexes>
> > > > >>> <group>Group name</group>
> > > > >>> <load_stat>true</load_stat>
> > > > >>> <load_mode>true</load_mode>
> > > > >>> <load_mtd>true</load_mtd>
> > > > >>> <load_mpr>false</load_mpr>
> > > > >>>
> > > > >>>
> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > > > >>> <load_val>
> > > > >>>   <field name="model">
> > > > >>>     <val>arw</val>
> > > > >>>     <val>nmm</val>
> > > > >>>   </field>
> > > > >>>
> > > > >>> <field name="valid_time">
> > > > >>>   <date_list name="folder_dates"/>
> > > > >>> </field>
> > > > >>>
> > > > >>> <field name="vx_mask">
> > > > >>>   <val>FULL</val>
> > > > >>>   <val>SWC</val>
> > > > >>> </field>
> > > > >>> </load_val>
> > > > >>> </load_spec>
> > > > >>>
> > > > >>> Are similar settings allowed with <plot_spec>?  I see
> > > > >>> <connection>,
> > > > >>> which seems carried over.  I'm looking to specify my
> > > > >>> folder_tmpl in
> > > > >>> <plot_spec>, which according to Tara, should be possible.
> > > > >>> Since my
> > > > >>> directory structure looks like this:
> > > > >>>
> > > > >>>
> > > > >>>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > > > >>>
> > > > >>> I would do something like this I would think:
> > > > >>>
> > > > >>> .......
> > > > >>>
> > > > >>> <date_list name="folder_dates">
> > > > >>>   <start>20200612/start>
> > > > >>>   <end><date_offset><day_offset>0</day_offset></end>
> > > > >>>   <inc>86400</inc>
> > > > >>>   <format>yyyyMMdd</format>
> > > > >>> </date_list>
> > > > >>>
> > > > >>> <folder_tmpl>
> > > > >>>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > > > >>> </folder_tmpl>
> > > > >>> <load_val>
> > > > >>>   <field name="model">
> > > > >>>     <val>prod</val>
> > > > >>>   </field>
> > > > >>>
> > > > >>> <field name="valid_time">
> > > > >>>   <date_list name="folder_dates"/>
> > > > >>> </field>
> > > > >>>
> > > > >>> ......
> > > > >>>
> > > > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
> > > > >>> Affiliate <
> > > > >>> edward.strobach at noaa.gov> wrote:
> > > > >>>
> > > > >>>> Thank you.  This is a very helpful example
> > > > >>>>
> > > > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> > > > >>>> <met_help at ucar.edu>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Hi,
> > > > >>>>> Here is some examples how to use <date_range> and
> > > > >>>>> <date_range_list> in
> > > > >>>>> batch XML.
> > > > >>>>> First, you need to defined your range. You can have more
then
> > > > >>>>> one
> > > > >>>>> <date_range>:
> > > > >>>>> <date_range name="may">
> > > > >>>>>         <start>2019-05-01 00:00:00</start>
> > > > >>>>>         <end>2019-05-31 23:59:59</end>
> > > > >>>>> </date_range>
> > > > >>>>> <date_range name="march">
> > > > >>>>>         <start>2019-03-01 00:00:00</start>
> > > > >>>>>         <end>2019-03-31 23:59:59</end>
> > > > >>>>> </date_range>
> > > > >>>>>
> > > > >>>>> and then use one or all date_ranges:
> > > > >>>>> <plot_fix>
> > > > >>>>>     <field name="fcst_init_beg">
> > > > >>>>>         <date_range name="march"/>
> > > > >>>>>         <date_range name="may"/>
> > > > >>>>>    </field>...
> > > > >>>>> This configuration would include all dates for March and
May
> > > > >>>>> of
> > > > >>>>> 2019.
> > > > >>>>>
> > > > >>>>> If you need more flexible configuration you can use
> > > > >>>>> date_range_list.
> > > > >>>>> For example, you want to select first 6 hours every day
> > > > >>>>> beetween
> > > > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
> > > > >>>>> look
> > > > >>>>> like this:
> > > > >>>>> <date_range_list name = 'all'>
> > > > >>>>>        <range_start>2020-01-01 00:00:00</range_start>
> > > > >>>>>         <range_end>2020-01-05 00:00:00</range_end>
> > > > >>>>>         <range_length>360</range_length>
> > > > >>>>>         <inc>1440</inc>
> > > > >>>>> </date_range_list>
> > > > >>>>> where the 1st and 2nd values are the start and end,
> > > > >>>>> range_length - the length of a range in minutes(6
houres)
> > > > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> > > > >>>>> This example would create following ranges:
> > > > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> > > > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> > > > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> > > > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> > > > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> > > > >>>>>
> > > > >>>>> Also you can add more parameters to <plot_fix> to filter
your
> > > > >>>>> data.
> > > > >>>>> For example, if you want to select only dates with 00:00
you
> > > > >>>>> can
> > > > >>>>> specify
> > > > >>>>>  <field name="valid_hour">
> > > > >>>>>         <val>00</val>
> > > > >>>>>  </field>
> > > > >>>>>
> > > > >>>>> Tatiana
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
> > > > >>>>> > and if there isn't a better way, I wonder the
whereabouts
> > > > >>>>> > of a
> > > > >>>>> > separate
> > > > >>>>> > script that pipes dates (and other relevant
information)
> > > > >>>>> > into
> > > > >>>>> > the XML
> > > > >>>>> > file
> > > > >>>>> > used to generate the results.
> > > > >>>>> >
> > > > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
> > > > >>>>> > Affiliate <
> > > > >>>>> > edward.strobach at noaa.gov> wrote:
> > > > >>>>> >
> > > > >>>>> > > Thanks Tatiana.  These resources are helpful.
> > > > >>>>> > >
> > > > >>>>> > > One last question.  Simply typing lines like this in
can
> > > > >>>>> > > take
> > > > >>>>> > > forever:
> > > > >>>>> > >
> > > > >>>>> > > <plot_fix>
> > > > >>>>> > >     <field equalize="false" name="fcst_init_beg">
> > > > >>>>> > >         <set name="fcst_init_beg_0">
> > > > >>>>> > >             <val>2019-05-01 12:00:00</val>
> > > > >>>>> > >             <val>2019-05-02 06:00:00</val>
> > > > >>>>> > >             <val>2019-05-02 12:00:00</val>
> > > > >>>>> > >             <val>2019-05-03 06:00:00</val>
> > > > >>>>> > >             <val>2019-05-03 12:00:00</val>
> > > > >>>>> > >
> > > > >>>>> > >
> > > > >>>>> > > Is there a better way?   I see date_range,
> > > > >>>>> > > date_range_list,
> > > > >>>>> > > or
> > > > >>>>> > > date_list.
> > > > >>>>> > > Can this be used instead of typing individual
values?
> > > > >>>>> > > This
> > > > >>>>> > > could
> > > > >>>>> > > prove
> > > > >>>>> > > pretty daunting if one would have to type in 700
entries
> > > > >>>>> > > each
> > > > >>>>> > > time.
> > > > >>>>> > >
> > > > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via RT
> > > > >>>>> > > <met_help at ucar.edu>
> > > > >>>>> > > wrote:
> > > > >>>>> > >
> > > > >>>>> > >> I found a few problems in your XML.
> > > > >>>>> > >> The database you are using doesn't have data with
> > > > >>>>> > >> vx_mask
> > > > >>>>> > >> 'abcREGabc'.
> > > > >>>>> > >> That is why METviewer returns an empty dataset and
can't
> > > > >>>>> > >> create a
> > > > >>>>> > >> plot.
> > > > >>>>> > >> The other problem is how you define fcst_lead. This
> > > > >>>>> > >> database
> > > > >>>>> > >> has
> > > > >>>>> > >> values
> > > > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since you
> > > > >>>>> > >> include
> > > > >>>>> > >> all
> > > > >>>>> > >> fcst_leads
> > > > >>>>> > >> this variable can be omitted.
> > > > >>>>> > >> To minimize problems with variables values, I would
> > > > >>>>> > >> suggest
> > > > >>>>> > >> to
> > > > >>>>> > >> create XML
> > > > >>>>> > >> using GUI and then edit it to fit batch standards.
> > > > >>>>> > >> I created a plot using you configurations with GUI
> > > > >>>>> > >> (without
> > > > >>>>> vx_mack
> > > > >>>>> > >> and
> > > > >>>>> > >> plot and axis titles) #20200612_173626. You can
load it
> > > > >>>>> > >> back
> > > > >>>>> > >> to
> > > > >>>>> the
> > > > >>>>> > >> page by
> > > > >>>>> > >> using the "History" tab (left bar on GUI). Find the
> > > > >>>>> > >> plot,
> > > > >>>>> > >> open XML
> > > > >>>>> > >> and
> > > > >>>>> > >> click on "Load to page". After that you can
recreate a
> > > > >>>>> > >> plot
> > > > >>>>> > >> and
> > > > >>>>> edit
> > > > >>>>> > >> the
> > > > >>>>> > >> configurations. You can also download a new XML and
edit
> > > > >>>>> > >> it
> > > > >>>>> > >> for
> > > > >>>>> > >> batch.
> > > > >>>>> > >> I attached the example of load XML to this message.
> > > > >>>>> > >>
> > > > >>>>> > >> Description of rds parameters:
> > > > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer R
> > > > >>>>> > >> scripts
> > > > >>>>> > >> rds_plots - temporary directory where the plots
will be
> > > > >>>>> > >> generated
> > > > >>>>> > >> rds_data - temporary directory where the data
retrieved
> > > > >>>>> > >> from
> > > > >>>>> > >> the
> > > > >>>>> > >> database
> > > > >>>>> > >> and other intermediate files will be stored
> > > > >>>>> > >> rds_scripts - temporary directory where R scripts
for
> > > > >>>>> > >> this
> > > > >>>>> > >> run
> > > > >>>>> will
> > > > >>>>> > >> be
> > > > >>>>> > >> stored
> > > > >>>>> > >> After the execution is done all the files from
temporary
> > > > >>>>> directories
> > > > >>>>> > >> would be removed.
> > > > >>>>> > >>
> > > > >>>>> > >>
> > > > >>>>> > >> We don't have the offitial user guide for
METviewer.
> > > > >>>>> > >> There
> > > > >>>>> > >> are
> > > > >>>>> some
> > > > >>>>> > >> instructions on how to crate XML config files:
> > > > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > > > >>>>> > >>
> > > > >>>>> > >> Tatiana
> > > > >>>>> > >>
> > > > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
> > > > >>>>> > >> wrote:
> > > > >>>>> > >> > Hi,
> > > > >>>>> > >> >
> > > > >>>>> > >> > I'm attaching the xml below, but am having hard
time
> > > > >>>>> > >> > following
> > > > >>>>> how
> > > > >>>>> > >> > to
> > > > >>>>> > >> > go
> > > > >>>>> > >> > about the other steps since I have zero
experience on
> > > > >>>>> > >> > how
> > > > >>>>> > >> > to do
> > > > >>>>> > >> > these
> > > > >>>>> > >> > things.
> > > > >>>>> > >> >
> > > > >>>>> > >> > #2. All parameters in <folders> section should
NOT be
> > > > >>>>> > >> > changed
> > > > >>>>> and
> > > > >>>>> > >> > will
> > > > >>>>> > >> > be
> > > > >>>>> > >> > replaced by other scripts during the batch
execution.
> > > > >>>>> > >> > The
> > > > >>>>> > >> > data
> > > > >>>>> for
> > > > >>>>> > >> > the
> > > > >>>>> > >> > batch mode is coming from the database and not
from
> > > > >>>>> > >> > files.
> > > > >>>>> > >> > The
> > > > >>>>> > >> > name of
> > > > >>>>> > >> > the
> > > > >>>>> > >> > database is in <database> tag.
> > > > >>>>> > >> >
> > > > >>>>> > >> > I understand that nothing should be changed in
the
> > > > >>>>> > >> > folders
> > > > >>>>> > >> > section,
> > > > >>>>> > >> > but
> > > > >>>>> > >> > there is no indication how and where these
variables
> > > > >>>>> > >> > are
> > > > >>>>> > >> > being
> > > > >>>>> > >> > defined.
> > > > >>>>> > >> > Grepping these names of the files available does
not
> > > > >>>>> > >> > reveal
> > > > >>>>> formal
> > > > >>>>> > >> > definitions of these variables.  If I understood
> > > > >>>>> > >> > exactly
> > > > >>>>> > >> > how
> > > > >>>>> these
> > > > >>>>> > >> > were
> > > > >>>>> > >> > being defined, then I would be able connect this
> > > > >>>>> > >> > better.
> > > > >>>>> > >> >
> > > > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
> > > > >>>>> > >> > use
> > > > >>>>> METviewer
> > > > >>>>> > >> > load
> > > > >>>>> > >> > mode to load these files to the specific
database. In
> > > > >>>>> > >> > the
> > > > >>>>> > >> > config
> > > > >>>>> > >> > file
> > > > >>>>> > >> > for
> > > > >>>>> > >> > this process they also can specify the database
group.
> > > > >>>>> > >> > For
> > > > >>>>> example
> > > > >>>>> > >> > database
> > > > >>>>> > >> > mv_dev is in Test12 group.
> > > > >>>>> > >> > After the data has been loaded to the database it
is
> > > > >>>>> > >> > visible and
> > > > >>>>> > >> > available
> > > > >>>>> > >> > in the GUI and also can be used by the batch
mode.
> > > > >>>>> > >> > You can always add more data to the database
using the
> > > > >>>>> > >> > same load
> > > > >>>>> > >> > mode.
> > > > >>>>> > >> >
> > > > >>>>> > >> > Is there a guide for me to understand the process
that
> > > > >>>>> > >> > you
> > > > >>>>> > >> > are
> > > > >>>>> > >> > referring to?
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> >
> > > > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek
via RT
> > > > >>>>> > >> > <met_help at ucar.edu>
> > > > >>>>> > >> > wrote:
> > > > >>>>> > >> >
> > > > >>>>> > >> > > Hi Edward,
> > > > >>>>> > >> > > #1. I don't have an access to NOAA servers and
can't
> > > > >>>>> > >> > > view your
> > > > >>>>> > >> > > XML
> > > > >>>>> > >> > > for the
> > > > >>>>> > >> > > batch mode. Please, attache it to this issue
and I
> > > > >>>>> > >> > > will
> > > > >>>>> > >> > > take a
> > > > >>>>> > >> > > look.
> > > > >>>>> > >> > >
> > > > >>>>> > >> > > #2. All parameters in <folders> section should
NOT
> > > > >>>>> > >> > > be
> > > > >>>>> > >> > > changed
> > > > >>>>> > >> > > and
> > > > >>>>> > >> > > will be
> > > > >>>>> > >> > > replaced by other scripts during the batch
> > > > >>>>> > >> > > execution.
> > > > >>>>> > >> > > The data
> > > > >>>>> > >> > > for
> > > > >>>>> > >> > > the
> > > > >>>>> > >> > > batch mode is coming from the database and not
from
> > > > >>>>> > >> > > files. The
> > > > >>>>> > >> > > name
> > > > >>>>> > >> > > of the
> > > > >>>>> > >> > > database is in <database> tag.
> > > > >>>>> > >> > >
> > > > >>>>> > >> > > #3. Users rum MET and produce MET output files.
They
> > > > >>>>> > >> > > use
> > > > >>>>> > >> > > METviewer
> > > > >>>>> > >> > > load
> > > > >>>>> > >> > > mode to load these files to the specific
database.
> > > > >>>>> > >> > > In
> > > > >>>>> > >> > > the
> > > > >>>>> config
> > > > >>>>> > >> > > file
> > > > >>>>> > >> > > for
> > > > >>>>> > >> > > this process they also can specify the database
> > > > >>>>> > >> > > group.
> > > > >>>>> > >> > > For
> > > > >>>>> > >> > > example
> > > > >>>>> > >> > > database
> > > > >>>>> > >> > > mv_dev is in Test12 group.
> > > > >>>>> > >> > > After the data has been loaded to the database
it is
> > > > >>>>> > >> > > visible
> > > > >>>>> and
> > > > >>>>> > >> > > available
> > > > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
> > > > >>>>> > >> > > You can always add more data to the database
using
> > > > >>>>> > >> > > the
> > > > >>>>> > >> > > same
> > > > >>>>> load
> > > > >>>>> > >> > > mode.
> > > > >>>>> > >> > >
> > > > >>>>> > >> > > Tatiana
> > > > >>>>> > >> > >
> > > > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> > > > >>>>> > >> > > edward.strobach at noaa.gov
> > > > >>>>> > >> > > wrote:
> > > > >>>>> > >> > > > Good morning,
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Over the last two weeks I've made an effort
to
> > > > >>>>> > >> > > > understand
> > > > >>>>> > >> > > > metviewer,
> > > > >>>>> > >> > > > test
> > > > >>>>> > >> > > > different cases using the history tab,
changing
> > > > >>>>> > >> > > > information
> > > > >>>>> to
> > > > >>>>> > >> > > > see
> > > > >>>>> > >> > > > what
> > > > >>>>> > >> > > > works and what does not, and make connections
> > > > >>>>> > >> > > > between
> > > > >>>>> > >> > > > the
> > > > >>>>> GUI
> > > > >>>>> > >> > > > settings
> > > > >>>>> > >> > > > and
> > > > >>>>> > >> > > > the XML file.  Although my comfort level with
> > > > >>>>> > >> > > > metviewer has
> > > > >>>>> > >> > > > significantly
> > > > >>>>> > >> > > > increased during this time, there are some
> > > > >>>>> > >> > > > remaining
> > > > >>>>> > >> > > > issues
> > > > >>>>> > >> > > > that
> > > > >>>>> > >> > > > I'm
> > > > >>>>> > >> > > > hoping
> > > > >>>>> > >> > > > to clear up.
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > 1.  Problem changing XML file and running
with
> > > > >>>>> > >> > > > different
> > > > >>>>> > >> > > > statistic
> > > > >>>>> > >> > > > and
> > > > >>>>> > >> > > > plot
> > > > >>>>> > >> > > > type:
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided
me the
> > > > >>>>> > >> > > > working
> > > > >>>>> > >> > > > directory
> > > > >>>>> > >> > > > to get
> > > > >>>>> > >> > > > started which I copied over:
> > > > >>>>> > >> > > >
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > run his
> > > > >>>>> > >> > > > example.
> > > > >>>>> > >> > > > running of the script "bash test.sh" produced
a
> > > > >>>>> > >> > > > CSI
> > > > >>>>> > >> > > > plot
> > > > >>>>> that
> > > > >>>>> > >> > > > was
> > > > >>>>> > >> > > > stored
> > > > >>>>> > >> > > > here:
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > >
> > > > >>>>> > >>
> > > > >>>>>
> > >
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Running the script produced the expected
result.
> > > > >>>>> > >> > > > The
> > > > >>>>> > >> > > > next
> > > > >>>>> > >> > > > natural
> > > > >>>>> > >> > > > step was
> > > > >>>>> > >> > > > to copy hochum_xml.template and create a new
xml
> > > > >>>>> > >> > > > file
> > > > >>>>> > >> > > > called
> > > > >>>>> > >> > > > Strobach_xml.template in order to create a
time
> > > > >>>>> > >> > > > series
> > > > >>>>> > >> > > > where
> > > > >>>>> > >> > > > the
> > > > >>>>> > >> > > > independent time is fcst_valid_beg instead of
> > > > >>>>> > >> > > > obs_thresh.
> > > > >>>>> > >> > > > Accordingly,
> > > > >>>>> > >> > > > I
> > > > >>>>> > >> > > > made changes to the fixed fields (removing
> > > > >>>>> > >> > > > fcst_valid_beg)
> > > > >>>>> and
> > > > >>>>> > >> > > > changed
> > > > >>>>> > >> > > > the
> > > > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
tried).
> > > > >>>>> > >> > > > The
> > > > >>>>> error
> > > > >>>>> > >> > > > that
> > > > >>>>> > >> > > > resulted
> > > > >>>>> > >> > > > is as follows:
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-
01
> > > > >>>>> > >> > > > 13:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 01
> > > > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-01
> > > > >>>>> > >> > > > 16:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 01
> > > > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-01
> > > > >>>>> > >> > > > 19:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 01
> > > > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-01
> > > > >>>>> > >> > > > 22:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 01
> > > > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-02
> > > > >>>>> > >> > > > 01:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 02
> > > > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-02
> > > > >>>>> > >> > > > 04:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 02
> > > > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-02
> > > > >>>>> > >> > > > 07:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 02
> > > > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-02
> > > > >>>>> > >> > > > 10:00:00',
> > > > >>>>> > >> > > > '2020-05-
> > > > >>>>> > >> > > > 02
> > > > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > > > >>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
> > > > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > > > >>>>> > >> > > > AND ld.fmean != -9999;
> > > > >>>>> > >> > > > No data was returned from database
> > > > >>>>> > >> > > > 2020-06-09
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > >
> > > > >>>>> > >>
> > > > >>>>>
> > >
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > > > >>>>> > >> > > > Source) -1 printFormattedTable -   **  ERROR:
> > > > >>>>> > >> > > > Caught
> > > > >>>>> > >> > > > class
> > > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException in
> > > > >>>>> > >> > > > printFormattedTable(ResultSet
> > > > >>>>> > >> > > > res): result set contained no data
> > > > >>>>> > >> > > > **  ERROR: Caught class
> > > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > > > >>>>> > >> > > > in
> > > > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
> > > > >>>>> > >> > > > contained no
> > > > >>>>> > >> > > > data
> > > > >>>>> > >> > > > **  ERROR: Caught class
> > > > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > > > >>>>> > >> > > > in
> > > > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
> > > > >>>>> > >> > > > contained no
> > > > >>>>> > >> > > > data
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Apparently there was no data during this
time.  I
> > > > >>>>> > >> > > > tried
> > > > >>>>> > >> > > > changing
> > > > >>>>> > >> > > > the
> > > > >>>>> > >> > > > range
> > > > >>>>> > >> > > > of valid times and changed the valid time to
> > > > >>>>> > >> > > > different
> > > > >>>>> > >> > > > days
> > > > >>>>> > >> > > > just in
> > > > >>>>> > >> > > > case
> > > > >>>>> > >> > > > data was missing.  The issue was the same.
To
> > > > >>>>> > >> > > > confirm
> > > > >>>>> > >> > > > my
> > > > >>>>> > >> > > > logic, I
> > > > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
> > > > >>>>> > >> > > > which I
> > > > >>>>> > >> > > > found
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > work.
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > 2.  Running in batch mode
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > I think I lack an understanding of how to set
up
> > > > >>>>> > >> > > > batch
> > > > >>>>> > >> > > > mode
> > > > >>>>> > >> > > > because
> > > > >>>>> > >> > > > I
> > > > >>>>> > >> > > > don't
> > > > >>>>> > >> > > > see where these definitions are being
explicitly
> > > > >>>>> > >> > > > defined:
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > <plot_spec>
> > > > >>>>> > >> > > >     <connection>
> > > > >>>>> > >> > > >         <host>rds_host:3306</host>
> > > > >>>>> > >> > > >
> > > > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > > > >>>>> > >> > > > <user>rds_user</user>
> > > > >>>>> > >> > > > <password>rds_pwd</password>
> > > > >>>>> > >> > > > </connection>
> > > > >>>>> > >> > > > <rscript>Rscript</rscript>
> > > > >>>>> > >> > > > <folders>
> > > > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > > > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > > > >>>>> > >> > > > <plots>rds_plots</plots>
> > > > >>>>> > >> > > > <data>rds_data</data>
> > > > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > > > >>>>> > >> > > > </folders>
> > > > >>>>> > >> > > > <plot>
> > > > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > I see the <folders> section, but grepping all
> > > > >>>>> > >> > > > available
> > > > >>>>> files
> > > > >>>>> > >> > > > does
> > > > >>>>> > >> > > > not
> > > > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well as
> > > > >>>>> > >> > > > other
> > > > >>>>> > >> > > > similarly
> > > > >>>>> > >> > > > defined
> > > > >>>>> > >> > > > variables.  I think if I know how these are
being
> > > > >>>>> > >> > > > defined,
> > > > >>>>> > >> > > > then
> > > > >>>>> > >> > > > I'll
> > > > >>>>> > >> > > > be
> > > > >>>>> > >> > > > able to pipe in my information and process
the
> > > > >>>>> > >> > > > data
> > > > >>>>> > >> > > > using
> > > > >>>>> met
> > > > >>>>> > >> > > > and
> > > > >>>>> > >> > > > produce
> > > > >>>>> > >> > > > results in batch mode.  Am I right on that?
Also,
> > > > >>>>> > >> > > > if
> > > > >>>>> > >> > > > I want
> > > > >>>>> > >> > > > to use
> > > > >>>>> > >> > > > obs for
> > > > >>>>> > >> > > > verification, do I add another line in the
XML
> > > > >>>>> > >> > > > file
> > > > >>>>> > >> > > > that
> > > > >>>>> > >> > > > designates
> > > > >>>>> > >> > > > file
> > > > >>>>> > >> > > > location of obs?  I assume that other steps
have
> > > > >>>>> > >> > > > to be
> > > > >>>>> > >> > > > run
> > > > >>>>> > >> > > > before
> > > > >>>>> > >> > > > all
> > > > >>>>> > >> > > > this.  If data is in prepbufr form, for
example, I
> > > > >>>>> > >> > > > would
> > > > >>>>> have
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > separately
> > > > >>>>> > >> > > > run PB2NC.  If I want grid2obs, then I would
want
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > do that
> > > > >>>>> > >> > > > as
> > > > >>>>> > >> > > > well.
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > 3. Communicating results to metviewer online
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > Lastly, I see a lot of people/projects listed
here
> > > > >>>>> > >> > > > -
> > > > >>>>> > >> > > > see
> > > > >>>>> > >> > > > attachment
> > > > >>>>> > >> > > >
> > > > >>>>> > >> > > > How do I get the results I generate to be
uploaded
> > > > >>>>> > >> > > > to
> > > > >>>>> > >> > > > this
> > > > >>>>> > >> > > > location?
> > > > >>>>> > >> > > > It's
> > > > >>>>> > >> > > > not clear how others are setting this up.
> > > > >>>>> > >> > >
> > > > >>>>> > >> > >
> > > > >>>>> > >> > >
> > > > >>>>> > >> > >
> > > > >>>>> > >>
> > > > >>>>> > >>
> > > > >>>>> > >>
> > > > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
> > > > >>>>> > >> rds_user
> > > > >>>>> rds_pwd
> > > > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
> > > > >>>>> rds_scripts
> > > > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
CMAQAODPRODHIGH
> > > > >>>>> 2020-05-12
> > > > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
> > > > >>>>> > >> 05-12
> > > > >>>>> 20:00:00
> > > > >>>>> > >> 2020-05-12
> > > > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
> > > > >>>>> > >> 05-13
> > > > >>>>> 10:00:00
> > > > >>>>> > >> 2020-05-13
> > > > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
> > > > >>>>> > >> 05-13
> > > > >>>>> 14:00:00
> > > > >>>>> > >> 2020-05-13
> > > > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
> > > > >>>>> > >> 05-13
> > > > >>>>> 18:00:00
> > > > >>>>> > >> 2020-05-13
> > > > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
> > > > >>>>> > >> 05-13
> > > > >>>>> 22:00:00
> > > > >>>>> > >> 2020-05-13
> > > > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
> > > > >>>>> > >> 05-14
> > > > >>>>> 12:00:00
> > > > >>>>> > >> true
> > > > >>>>> > >> median plot_20200612_173626.data
> > > > >>>>> > >> plot_20200612_173626.png
> > > > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> > > > >>>>> > >> false list() list() false false false false false
false
> > > > >>>>> > >> true
> > > > >>>>> > >> false
> > > > >>>>> > >> false
> > > > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
> > > > >>>>> > >> c(1,1,0) 1 2
> > > > >>>>> > >> 1.4 -2
> > > > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2
0.5 3
> > > > >>>>> > >> #cccccc 1
> > > > >>>>> > >> listX 1
> > > > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
> > > > >>>>> > >> 3
> > > > >>>>> > >> 0.8 1
> > > > >>>>> > >> #333333
> > > > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
> > > > >>>>> > >> c(TRUE,TRUE)
> > > > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
> > > > >>>>> > >> c(1,1)
> > > > >>>>> c(1,1)
> > > > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > > > >>>>> > >
> > > > >>>>> > >
> > > > >>>>> > >
> > > > >>>>> > > --
> > > > >>>>> > > Edward Strobach
> > > > >>>>> > > EMC/NCEP/NWS/
> > > > >>>>> > > IMSG Contractor
> > > > >>>>> > > Cubicle#: 2029
> > > > >>>>> > > 301-683-3717
> > > > >>>>> > >
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Edward Strobach
> > > > >>>> EMC/NCEP/NWS/
> > > > >>>> IMSG Contractor
> > > > >>>> Cubicle#: 2029
> > > > >>>> 301-683-3717
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Edward Strobach
> > > > >>> EMC/NCEP/NWS/
> > > > >>> IMSG Contractor
> > > > >>> Cubicle#: 2029
> > > > >>> 301-683-3717
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Edward Strobach
> > > > >> EMC/NCEP/NWS/
> > > > >> IMSG Contractor
> > > > >> Cubicle#: 2029
> > > > >> 301-683-3717
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 10:05:58 2020

I'm not following you.  How do I now have enough space?  What changed?

On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Ed, now you should have enough space for the data loading.
>
> Tatiana
>
> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
> > I was basically able to build my xml file but it says I have no
> > available
> > space.  I'm not sure how that can be since I just started adding
data
> > to my
> > account
> >
> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space
left
> > on
> > device
> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space
left
> > on
> > device
> > ERROR: Command returned with non-zero status (1): scp -r ./*
> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
> > 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
> > TABLE_SCHEMA                              USER             SIZE_MB
> > mv_met_g2o_gefs                           binbin.zhou
140027.31
> > mv_met_system_hrefv2_v3                   binbin.zhou
71986.11
> >
> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > Thanks Tatiana.  Yes, I've created a database which seems
different
> > > from
> > > mv_group. How can one create their own group so that I can store
and
> > > access
> > > the database.  The database exists but cannot be viewed on the
> > > metviewer
> > > GUI.  The list starts with EMC HREFv3 and ends with vhagerty.  I
> > > named my
> > > database Ed_Strobach, but that was really intended for the group
> > > name.
> > >
> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > >> We don't have database and METviewer running on wcoss.
> > >> Instead, users use METviewer-wcoss scripts that help to connect
to
> > >> and
> > >> use METviewer that is running on AWS. That is why XML
connection and
> > >> folder
> > >> sections look somewhat different than once from the Docs page.
> > >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
> > >> mv_create_db_on_aws.sh <user_name> <database_name>
> > >> For data loading :
> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
> > >> And for batch:
> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
> > >>
> > >> These scripts copy XML and data files if needed to AWS, adjust
data
> > >> path
> > >> and users credentials, run METviewer and copy results back to
wcoss.
> > >>
> > >> Tatiana
> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
> > >> > -e'create
> > >> > database [db_name]
> > >> >
> > >> > would need to be used to create a database.  I assume that an
sql
> > >> > module is
> > >> > available on wcoss; I know you don't have access to those
machines
> > >> > Tatiana.  If I create my own database with my own name, then
I
> > >> > should
> > >> > be
> > >> > able to proceed with project specific tasks under that tab.
> > >> > Something
> > >> > like
> > >> > mv_EdwardStrobach.
> > >> >
> > >> > I can then create mv_cmaq_prod_v_para which I add data to
using
> > >> > load_spec.
> > >> > Once that step is done, then I can run plot_spec.  That's my
> > >> > takeaway
> > >> > currently
> > >> >
> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
> > >> > edward.strobach at noaa.gov> wrote:
> > >> >
> > >> > > Actually, it appears that I need to run the database
loading
> > >> > > module
> > >> > > first
> > >> > > if I want to change/add a data source.  From there I can
run the
> > >> > > plotting
> > >> > > batch module.  Is that right?   So if I run the database
loading
> > >> > > module,
> > >> > > then I can specify my dates and other related info specific
to
> > >> > > the
> > >> > > dataset.  If I create my database with a unique name, then
it
> > >> > > should
> > >> > > be
> > >> > > recognized on the metviewer gui, presumably.
> > >> > >
> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
> > >> > > <
> > >> > > edward.strobach at noaa.gov> wrote:
> > >> > >
> > >> > >> It seems that those options must be treated separately
outside
> > >> > >> of
> > >> > >> plot_spec.  Is it acceptable to create both a load_spec
and
> > >> > >> plot_spec in
> > >> > >> the same xml file?  Would the definitions in load_spec be
able
> > >> > >> to
> > >> > >> carry
> > >> > >> over into plot_spec?  It appears that data loading and
plotting
> > >> > >> are
> > >> > >> treated
> > >> > >> separately.
> > >> > >>
> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
> > >> > >> Affiliate <
> > >> > >> edward.strobach at noaa.gov> wrote:
> > >> > >>
> > >> > >>> I do plan to use date_range; however, in an example
provided
> > >> > >>> by the
> > >> > >>> link
> > >> > >>> I see this:
> > >> > >>>
> > >> > >>> <load_spec>
> > >> > >>>   <connection>
> > >> > >>>     <host>kemosabe:3306</host>
> > >> > >>>     <database>metvdb_hwt</database>
> > >> > >>>     <user>pgoldenb</user>
> > >> > >>>     <password>pgoldenb</password>
> > >> > >>>   </connection>
> > >> > >>>
> > >> > >>> <date_list name="folder_dates">
> > >> > >>>   <start>2010051914V</start>
> > >> > >>>   <end><date_offset><day_offset>0</day_offset><hour>6
> > >> > >>> </hour></date_offset></end>
> > >> > >>>   <inc>3600</inc>
> > >> > >>>   <format>yyyyMMddHH'V'</format>
> > >> > >>> </date_list>
> > >> > >>>
> > >> > >>> <met_version>V3.0</met_version>
> > >> > >>>
> > >> > >>> <verbose>false</verbose>
> > >> > >>> <insert_size>1</insert_size>
> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
> > >> > >>> <drop_indexes>false</drop_indexes>
> > >> > >>> <apply_indexes>true</apply_indexes>
> > >> > >>> <group>Group name</group>
> > >> > >>> <load_stat>true</load_stat>
> > >> > >>> <load_mode>true</load_mode>
> > >> > >>> <load_mtd>true</load_mtd>
> > >> > >>> <load_mpr>false</load_mpr>
> > >> > >>>
> > >> > >>>
> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
> > >> > >>> <load_val>
> > >> > >>>   <field name="model">
> > >> > >>>     <val>arw</val>
> > >> > >>>     <val>nmm</val>
> > >> > >>>   </field>
> > >> > >>>
> > >> > >>> <field name="valid_time">
> > >> > >>>   <date_list name="folder_dates"/>
> > >> > >>> </field>
> > >> > >>>
> > >> > >>> <field name="vx_mask">
> > >> > >>>   <val>FULL</val>
> > >> > >>>   <val>SWC</val>
> > >> > >>> </field>
> > >> > >>> </load_val>
> > >> > >>> </load_spec>
> > >> > >>>
> > >> > >>> Are similar settings allowed with <plot_spec>?  I see
> > >> > >>> <connection>,
> > >> > >>> which seems carried over.  I'm looking to specify my
> > >> > >>> folder_tmpl in
> > >> > >>> <plot_spec>, which according to Tara, should be possible.
> > >> > >>> Since my
> > >> > >>> directory structure looks like this:
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
> > >> > >>>
> > >> > >>> I would do something like this I would think:
> > >> > >>>
> > >> > >>> .......
> > >> > >>>
> > >> > >>> <date_list name="folder_dates">
> > >> > >>>   <start>20200612/start>
> > >> > >>>   <end><date_offset><day_offset>0</day_offset></end>
> > >> > >>>   <inc>86400</inc>
> > >> > >>>   <format>yyyyMMdd</format>
> > >> > >>> </date_list>
> > >> > >>>
> > >> > >>> <folder_tmpl>
> > >> > >>>
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
> > >> > >>> </folder_tmpl>
> > >> > >>> <load_val>
> > >> > >>>   <field name="model">
> > >> > >>>     <val>prod</val>
> > >> > >>>   </field>
> > >> > >>>
> > >> > >>> <field name="valid_time">
> > >> > >>>   <date_list name="folder_dates"/>
> > >> > >>> </field>
> > >> > >>>
> > >> > >>> ......
> > >> > >>>
> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
> > >> > >>> Affiliate <
> > >> > >>> edward.strobach at noaa.gov> wrote:
> > >> > >>>
> > >> > >>>> Thank you.  This is a very helpful example
> > >> > >>>>
> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
> > >> > >>>> <met_help at ucar.edu>
> > >> > >>>> wrote:
> > >> > >>>>
> > >> > >>>>> Hi,
> > >> > >>>>> Here is some examples how to use <date_range> and
> > >> > >>>>> <date_range_list> in
> > >> > >>>>> batch XML.
> > >> > >>>>> First, you need to defined your range. You can have
more
> > >> > >>>>> then one
> > >> > >>>>> <date_range>:
> > >> > >>>>> <date_range name="may">
> > >> > >>>>>         <start>2019-05-01 00:00:00</start>
> > >> > >>>>>         <end>2019-05-31 23:59:59</end>
> > >> > >>>>> </date_range>
> > >> > >>>>> <date_range name="march">
> > >> > >>>>>         <start>2019-03-01 00:00:00</start>
> > >> > >>>>>         <end>2019-03-31 23:59:59</end>
> > >> > >>>>> </date_range>
> > >> > >>>>>
> > >> > >>>>> and then use one or all date_ranges:
> > >> > >>>>> <plot_fix>
> > >> > >>>>>     <field name="fcst_init_beg">
> > >> > >>>>>         <date_range name="march"/>
> > >> > >>>>>         <date_range name="may"/>
> > >> > >>>>>    </field>...
> > >> > >>>>> This configuration would include all dates for March
and May
> > >> > >>>>> of
> > >> > >>>>> 2019.
> > >> > >>>>>
> > >> > >>>>> If you need more flexible configuration you can use
> > >> > >>>>> date_range_list.
> > >> > >>>>> For example, you want to select first 6 hours every day
> > >> > >>>>> beetween
> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
> > >> > >>>>> look
> > >> > >>>>> like this:
> > >> > >>>>> <date_range_list name = 'all'>
> > >> > >>>>>        <range_start>2020-01-01 00:00:00</range_start>
> > >> > >>>>>         <range_end>2020-01-05 00:00:00</range_end>
> > >> > >>>>>         <range_length>360</range_length>
> > >> > >>>>>         <inc>1440</inc>
> > >> > >>>>> </date_range_list>
> > >> > >>>>> where the 1st and 2nd values are the start and end,
> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
> > >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
> > >> > >>>>> This example would create following ranges:
> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01 06:00:00'
> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02 06:00:00'
> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03 06:00:00'
> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04 06:00:00'
> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05 06:00:00'
> > >> > >>>>>
> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
> > >> > >>>>> your
> > >> > >>>>> data.
> > >> > >>>>> For example, if you want to select only dates with
00:00 you
> > >> > >>>>> can
> > >> > >>>>> specify
> > >> > >>>>>  <field name="valid_hour">
> > >> > >>>>>         <val>00</val>
> > >> > >>>>>  </field>
> > >> > >>>>>
> > >> > >>>>> Tatiana
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
> > >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
> > >> > >>>>> > of a
> > >> > >>>>> > separate
> > >> > >>>>> > script that pipes dates (and other relevant
information)
> > >> > >>>>> > into
> > >> > >>>>> > the XML
> > >> > >>>>> > file
> > >> > >>>>> > used to generate the results.
> > >> > >>>>> >
> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
> > >> > >>>>> > Affiliate <
> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
> > >> > >>>>> >
> > >> > >>>>> > > Thanks Tatiana.  These resources are helpful.
> > >> > >>>>> > >
> > >> > >>>>> > > One last question.  Simply typing lines like this
in can
> > >> > >>>>> > > take
> > >> > >>>>> > > forever:
> > >> > >>>>> > >
> > >> > >>>>> > > <plot_fix>
> > >> > >>>>> > >     <field equalize="false" name="fcst_init_beg">
> > >> > >>>>> > >         <set name="fcst_init_beg_0">
> > >> > >>>>> > >             <val>2019-05-01 12:00:00</val>
> > >> > >>>>> > >             <val>2019-05-02 06:00:00</val>
> > >> > >>>>> > >             <val>2019-05-02 12:00:00</val>
> > >> > >>>>> > >             <val>2019-05-03 06:00:00</val>
> > >> > >>>>> > >             <val>2019-05-03 12:00:00</val>
> > >> > >>>>> > >
> > >> > >>>>> > >
> > >> > >>>>> > > Is there a better way?   I see date_range,
> > >> > >>>>> > > date_range_list,
> > >> > >>>>> > > or
> > >> > >>>>> > > date_list.
> > >> > >>>>> > > Can this be used instead of typing individual
values?
> > >> > >>>>> > > This
> > >> > >>>>> > > could
> > >> > >>>>> > > prove
> > >> > >>>>> > > pretty daunting if one would have to type in 700
entries
> > >> > >>>>> > > each
> > >> > >>>>> > > time.
> > >> > >>>>> > >
> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via
RT
> > >> > >>>>> > > <met_help at ucar.edu>
> > >> > >>>>> > > wrote:
> > >> > >>>>> > >
> > >> > >>>>> > >> I found a few problems in your XML.
> > >> > >>>>> > >> The database you are using doesn't have data with
> > >> > >>>>> > >> vx_mask
> > >> > >>>>> > >> 'abcREGabc'.
> > >> > >>>>> > >> That is why METviewer returns an empty dataset and
> > >> > >>>>> > >> can't
> > >> > >>>>> > >> create a
> > >> > >>>>> > >> plot.
> > >> > >>>>> > >> The other problem is how you define fcst_lead.
This
> > >> > >>>>> > >> database
> > >> > >>>>> > >> has
> > >> > >>>>> > >> values
> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since
you
> > >> > >>>>> > >> include
> > >> > >>>>> > >> all
> > >> > >>>>> > >> fcst_leads
> > >> > >>>>> > >> this variable can be omitted.
> > >> > >>>>> > >> To minimize problems with variables values, I
would
> > >> > >>>>> > >> suggest
> > >> > >>>>> > >> to
> > >> > >>>>> > >> create XML
> > >> > >>>>> > >> using GUI and then edit it to fit batch standards.
> > >> > >>>>> > >> I created a plot using you configurations with GUI
> > >> > >>>>> > >> (without
> > >> > >>>>> vx_mack
> > >> > >>>>> > >> and
> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You can
load it
> > >> > >>>>> > >> back
> > >> > >>>>> > >> to
> > >> > >>>>> the
> > >> > >>>>> > >> page by
> > >> > >>>>> > >> using the "History" tab (left bar on GUI). Find
the
> > >> > >>>>> > >> plot,
> > >> > >>>>> > >> open XML
> > >> > >>>>> > >> and
> > >> > >>>>> > >> click on "Load to page". After that you can
recreate a
> > >> > >>>>> > >> plot
> > >> > >>>>> > >> and
> > >> > >>>>> edit
> > >> > >>>>> > >> the
> > >> > >>>>> > >> configurations. You can also download a new XML
and
> > >> > >>>>> > >> edit it
> > >> > >>>>> > >> for
> > >> > >>>>> > >> batch.
> > >> > >>>>> > >> I attached the example of load XML to this
message.
> > >> > >>>>> > >>
> > >> > >>>>> > >> Description of rds parameters:
> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer
R
> > >> > >>>>> > >> scripts
> > >> > >>>>> > >> rds_plots - temporary directory where the plots
will be
> > >> > >>>>> > >> generated
> > >> > >>>>> > >> rds_data - temporary directory where the data
retrieved
> > >> > >>>>> > >> from
> > >> > >>>>> > >> the
> > >> > >>>>> > >> database
> > >> > >>>>> > >> and other intermediate files will be stored
> > >> > >>>>> > >> rds_scripts - temporary directory where R scripts
for
> > >> > >>>>> > >> this
> > >> > >>>>> > >> run
> > >> > >>>>> will
> > >> > >>>>> > >> be
> > >> > >>>>> > >> stored
> > >> > >>>>> > >> After the execution is done all the files from
> > >> > >>>>> > >> temporary
> > >> > >>>>> directories
> > >> > >>>>> > >> would be removed.
> > >> > >>>>> > >>
> > >> > >>>>> > >>
> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
> > >> > >>>>> > >> There
> > >> > >>>>> > >> are
> > >> > >>>>> some
> > >> > >>>>> > >> instructions on how to crate XML config files:
> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
> > >> > >>>>> > >>
> > >> > >>>>> > >> Tatiana
> > >> > >>>>> > >>
> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
> > >> > >>>>> > >> wrote:
> > >> > >>>>> > >> > Hi,
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > I'm attaching the xml below, but am having hard
time
> > >> > >>>>> > >> > following
> > >> > >>>>> how
> > >> > >>>>> > >> > to
> > >> > >>>>> > >> > go
> > >> > >>>>> > >> > about the other steps since I have zero
experience on
> > >> > >>>>> > >> > how
> > >> > >>>>> > >> > to do
> > >> > >>>>> > >> > these
> > >> > >>>>> > >> > things.
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > #2. All parameters in <folders> section should
NOT be
> > >> > >>>>> > >> > changed
> > >> > >>>>> and
> > >> > >>>>> > >> > will
> > >> > >>>>> > >> > be
> > >> > >>>>> > >> > replaced by other scripts during the batch
execution.
> > >> > >>>>> > >> > The
> > >> > >>>>> > >> > data
> > >> > >>>>> for
> > >> > >>>>> > >> > the
> > >> > >>>>> > >> > batch mode is coming from the database and not
from
> > >> > >>>>> > >> > files.
> > >> > >>>>> > >> > The
> > >> > >>>>> > >> > name of
> > >> > >>>>> > >> > the
> > >> > >>>>> > >> > database is in <database> tag.
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > I understand that nothing should be changed in
the
> > >> > >>>>> > >> > folders
> > >> > >>>>> > >> > section,
> > >> > >>>>> > >> > but
> > >> > >>>>> > >> > there is no indication how and where these
variables
> > >> > >>>>> > >> > are
> > >> > >>>>> > >> > being
> > >> > >>>>> > >> > defined.
> > >> > >>>>> > >> > Grepping these names of the files available does
not
> > >> > >>>>> > >> > reveal
> > >> > >>>>> formal
> > >> > >>>>> > >> > definitions of these variables.  If I understood
> > >> > >>>>> > >> > exactly
> > >> > >>>>> > >> > how
> > >> > >>>>> these
> > >> > >>>>> > >> > were
> > >> > >>>>> > >> > being defined, then I would be able connect this
> > >> > >>>>> > >> > better.
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
> > >> > >>>>> > >> > use
> > >> > >>>>> METviewer
> > >> > >>>>> > >> > load
> > >> > >>>>> > >> > mode to load these files to the specific
database. In
> > >> > >>>>> > >> > the
> > >> > >>>>> > >> > config
> > >> > >>>>> > >> > file
> > >> > >>>>> > >> > for
> > >> > >>>>> > >> > this process they also can specify the database
> > >> > >>>>> > >> > group. For
> > >> > >>>>> example
> > >> > >>>>> > >> > database
> > >> > >>>>> > >> > mv_dev is in Test12 group.
> > >> > >>>>> > >> > After the data has been loaded to the database
it is
> > >> > >>>>> > >> > visible and
> > >> > >>>>> > >> > available
> > >> > >>>>> > >> > in the GUI and also can be used by the batch
mode.
> > >> > >>>>> > >> > You can always add more data to the database
using
> > >> > >>>>> > >> > the
> > >> > >>>>> > >> > same load
> > >> > >>>>> > >> > mode.
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > Is there a guide for me to understand the
process
> > >> > >>>>> > >> > that you
> > >> > >>>>> > >> > are
> > >> > >>>>> > >> > referring to?
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek
via RT
> > >> > >>>>> > >> > <met_help at ucar.edu>
> > >> > >>>>> > >> > wrote:
> > >> > >>>>> > >> >
> > >> > >>>>> > >> > > Hi Edward,
> > >> > >>>>> > >> > > #1. I don't have an access to NOAA servers and
> > >> > >>>>> > >> > > can't
> > >> > >>>>> > >> > > view your
> > >> > >>>>> > >> > > XML
> > >> > >>>>> > >> > > for the
> > >> > >>>>> > >> > > batch mode. Please, attache it to this issue
and I
> > >> > >>>>> > >> > > will
> > >> > >>>>> > >> > > take a
> > >> > >>>>> > >> > > look.
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > > #2. All parameters in <folders> section should
NOT
> > >> > >>>>> > >> > > be
> > >> > >>>>> > >> > > changed
> > >> > >>>>> > >> > > and
> > >> > >>>>> > >> > > will be
> > >> > >>>>> > >> > > replaced by other scripts during the batch
> > >> > >>>>> > >> > > execution.
> > >> > >>>>> > >> > > The data
> > >> > >>>>> > >> > > for
> > >> > >>>>> > >> > > the
> > >> > >>>>> > >> > > batch mode is coming from the database and not
from
> > >> > >>>>> > >> > > files. The
> > >> > >>>>> > >> > > name
> > >> > >>>>> > >> > > of the
> > >> > >>>>> > >> > > database is in <database> tag.
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
> > >> > >>>>> > >> > > They use
> > >> > >>>>> > >> > > METviewer
> > >> > >>>>> > >> > > load
> > >> > >>>>> > >> > > mode to load these files to the specific
database.
> > >> > >>>>> > >> > > In
> > >> > >>>>> > >> > > the
> > >> > >>>>> config
> > >> > >>>>> > >> > > file
> > >> > >>>>> > >> > > for
> > >> > >>>>> > >> > > this process they also can specify the
database
> > >> > >>>>> > >> > > group.
> > >> > >>>>> > >> > > For
> > >> > >>>>> > >> > > example
> > >> > >>>>> > >> > > database
> > >> > >>>>> > >> > > mv_dev is in Test12 group.
> > >> > >>>>> > >> > > After the data has been loaded to the database
it
> > >> > >>>>> > >> > > is
> > >> > >>>>> > >> > > visible
> > >> > >>>>> and
> > >> > >>>>> > >> > > available
> > >> > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
> > >> > >>>>> > >> > > You can always add more data to the database
using
> > >> > >>>>> > >> > > the
> > >> > >>>>> > >> > > same
> > >> > >>>>> load
> > >> > >>>>> > >> > > mode.
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > > Tatiana
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
> > >> > >>>>> > >> > > edward.strobach at noaa.gov
> > >> > >>>>> > >> > > wrote:
> > >> > >>>>> > >> > > > Good morning,
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Over the last two weeks I've made an effort
to
> > >> > >>>>> > >> > > > understand
> > >> > >>>>> > >> > > > metviewer,
> > >> > >>>>> > >> > > > test
> > >> > >>>>> > >> > > > different cases using the history tab,
changing
> > >> > >>>>> > >> > > > information
> > >> > >>>>> to
> > >> > >>>>> > >> > > > see
> > >> > >>>>> > >> > > > what
> > >> > >>>>> > >> > > > works and what does not, and make
connections
> > >> > >>>>> > >> > > > between
> > >> > >>>>> > >> > > > the
> > >> > >>>>> GUI
> > >> > >>>>> > >> > > > settings
> > >> > >>>>> > >> > > > and
> > >> > >>>>> > >> > > > the XML file.  Although my comfort level
with
> > >> > >>>>> > >> > > > metviewer has
> > >> > >>>>> > >> > > > significantly
> > >> > >>>>> > >> > > > increased during this time, there are some
> > >> > >>>>> > >> > > > remaining
> > >> > >>>>> > >> > > > issues
> > >> > >>>>> > >> > > > that
> > >> > >>>>> > >> > > > I'm
> > >> > >>>>> > >> > > > hoping
> > >> > >>>>> > >> > > > to clear up.
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > 1.  Problem changing XML file and running
with
> > >> > >>>>> > >> > > > different
> > >> > >>>>> > >> > > > statistic
> > >> > >>>>> > >> > > > and
> > >> > >>>>> > >> > > > plot
> > >> > >>>>> > >> > > > type:
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided
me
> > >> > >>>>> > >> > > > the
> > >> > >>>>> > >> > > > working
> > >> > >>>>> > >> > > > directory
> > >> > >>>>> > >> > > > to get
> > >> > >>>>> > >> > > > started which I copied over:
> > >> > >>>>> > >> > > >
> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
> > >> > >>>>> > >> > > > to
> > >> > >>>>> > >> > > > run his
> > >> > >>>>> > >> > > > example.
> > >> > >>>>> > >> > > > running of the script "bash test.sh"
produced a
> > >> > >>>>> > >> > > > CSI
> > >> > >>>>> > >> > > > plot
> > >> > >>>>> that
> > >> > >>>>> > >> > > > was
> > >> > >>>>> > >> > > > stored
> > >> > >>>>> > >> > > > here:
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >>
> > >> > >>>>>
> > >>
>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Running the script produced the expected
result.
> > >> > >>>>> > >> > > > The
> > >> > >>>>> > >> > > > next
> > >> > >>>>> > >> > > > natural
> > >> > >>>>> > >> > > > step was
> > >> > >>>>> > >> > > > to copy hochum_xml.template and create a new
xml
> > >> > >>>>> > >> > > > file
> > >> > >>>>> > >> > > > called
> > >> > >>>>> > >> > > > Strobach_xml.template in order to create a
time
> > >> > >>>>> > >> > > > series
> > >> > >>>>> > >> > > > where
> > >> > >>>>> > >> > > > the
> > >> > >>>>> > >> > > > independent time is fcst_valid_beg instead
of
> > >> > >>>>> > >> > > > obs_thresh.
> > >> > >>>>> > >> > > > Accordingly,
> > >> > >>>>> > >> > > > I
> > >> > >>>>> > >> > > > made changes to the fixed fields (removing
> > >> > >>>>> > >> > > > fcst_valid_beg)
> > >> > >>>>> and
> > >> > >>>>> > >> > > > changed
> > >> > >>>>> > >> > > > the
> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
> > >> > >>>>> > >> > > > tried).
> > >> > >>>>> > >> > > > The
> > >> > >>>>> error
> > >> > >>>>> > >> > > > that
> > >> > >>>>> > >> > > > resulted
> > >> > >>>>> > >> > > > is as follows:
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-05-
01
> > >> > >>>>> > >> > > > 13:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 01
> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-
01
> > >> > >>>>> > >> > > > 16:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 01
> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-
01
> > >> > >>>>> > >> > > > 19:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 01
> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-
01
> > >> > >>>>> > >> > > > 22:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 01
> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-
02
> > >> > >>>>> > >> > > > 01:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 02
> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-
02
> > >> > >>>>> > >> > > > 04:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 02
> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-
02
> > >> > >>>>> > >> > > > 07:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 02
> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-
02
> > >> > >>>>> > >> > > > 10:00:00',
> > >> > >>>>> > >> > > > '2020-05-
> > >> > >>>>> > >> > > > 02
> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
> > >> > >>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
> > >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
> > >> > >>>>> > >> > > > No data was returned from database
> > >> > >>>>> > >> > > > 2020-06-09
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >>
> > >> > >>>>>
> > >>
>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
> > >> > >>>>> > >> > > > Source) -1 printFormattedTable -   **
ERROR:
> > >> > >>>>> > >> > > > Caught
> > >> > >>>>> > >> > > > class
> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
in
> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
> > >> > >>>>> > >> > > > res): result set contained no data
> > >> > >>>>> > >> > > > **  ERROR: Caught class
> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >> > >>>>> > >> > > > in
> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
> > >> > >>>>> > >> > > > contained no
> > >> > >>>>> > >> > > > data
> > >> > >>>>> > >> > > > **  ERROR: Caught class
> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
> > >> > >>>>> > >> > > > in
> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
> > >> > >>>>> > >> > > > contained no
> > >> > >>>>> > >> > > > data
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Apparently there was no data during this
time.  I
> > >> > >>>>> > >> > > > tried
> > >> > >>>>> > >> > > > changing
> > >> > >>>>> > >> > > > the
> > >> > >>>>> > >> > > > range
> > >> > >>>>> > >> > > > of valid times and changed the valid time to
> > >> > >>>>> > >> > > > different
> > >> > >>>>> > >> > > > days
> > >> > >>>>> > >> > > > just in
> > >> > >>>>> > >> > > > case
> > >> > >>>>> > >> > > > data was missing.  The issue was the same.
To
> > >> > >>>>> > >> > > > confirm
> > >> > >>>>> > >> > > > my
> > >> > >>>>> > >> > > > logic, I
> > >> > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
> > >> > >>>>> > >> > > > which I
> > >> > >>>>> > >> > > > found
> > >> > >>>>> > >> > > > to
> > >> > >>>>> > >> > > > work.
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > 2.  Running in batch mode
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > I think I lack an understanding of how to
set up
> > >> > >>>>> > >> > > > batch
> > >> > >>>>> > >> > > > mode
> > >> > >>>>> > >> > > > because
> > >> > >>>>> > >> > > > I
> > >> > >>>>> > >> > > > don't
> > >> > >>>>> > >> > > > see where these definitions are being
explicitly
> > >> > >>>>> > >> > > > defined:
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > <plot_spec>
> > >> > >>>>> > >> > > >     <connection>
> > >> > >>>>> > >> > > >         <host>rds_host:3306</host>
> > >> > >>>>> > >> > > >
> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
> > >> > >>>>> > >> > > > <user>rds_user</user>
> > >> > >>>>> > >> > > > <password>rds_pwd</password>
> > >> > >>>>> > >> > > > </connection>
> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
> > >> > >>>>> > >> > > > <folders>
> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
> > >> > >>>>> > >> > > > <data>rds_data</data>
> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
> > >> > >>>>> > >> > > > </folders>
> > >> > >>>>> > >> > > > <plot>
> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > I see the <folders> section, but grepping
all
> > >> > >>>>> > >> > > > available
> > >> > >>>>> files
> > >> > >>>>> > >> > > > does
> > >> > >>>>> > >> > > > not
> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well
> > >> > >>>>> > >> > > > as
> > >> > >>>>> > >> > > > other
> > >> > >>>>> > >> > > > similarly
> > >> > >>>>> > >> > > > defined
> > >> > >>>>> > >> > > > variables.  I think if I know how these are
being
> > >> > >>>>> > >> > > > defined,
> > >> > >>>>> > >> > > > then
> > >> > >>>>> > >> > > > I'll
> > >> > >>>>> > >> > > > be
> > >> > >>>>> > >> > > > able to pipe in my information and process
the
> > >> > >>>>> > >> > > > data
> > >> > >>>>> > >> > > > using
> > >> > >>>>> met
> > >> > >>>>> > >> > > > and
> > >> > >>>>> > >> > > > produce
> > >> > >>>>> > >> > > > results in batch mode.  Am I right on that?
> > >> > >>>>> > >> > > > Also, if
> > >> > >>>>> > >> > > > I want
> > >> > >>>>> > >> > > > to use
> > >> > >>>>> > >> > > > obs for
> > >> > >>>>> > >> > > > verification, do I add another line in the
XML
> > >> > >>>>> > >> > > > file
> > >> > >>>>> > >> > > > that
> > >> > >>>>> > >> > > > designates
> > >> > >>>>> > >> > > > file
> > >> > >>>>> > >> > > > location of obs?  I assume that other steps
have
> > >> > >>>>> > >> > > > to be
> > >> > >>>>> > >> > > > run
> > >> > >>>>> > >> > > > before
> > >> > >>>>> > >> > > > all
> > >> > >>>>> > >> > > > this.  If data is in prepbufr form, for
example,
> > >> > >>>>> > >> > > > I
> > >> > >>>>> > >> > > > would
> > >> > >>>>> have
> > >> > >>>>> > >> > > > to
> > >> > >>>>> > >> > > > separately
> > >> > >>>>> > >> > > > run PB2NC.  If I want grid2obs, then I would
want
> > >> > >>>>> > >> > > > to
> > >> > >>>>> > >> > > > do that
> > >> > >>>>> > >> > > > as
> > >> > >>>>> > >> > > > well.
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > 3. Communicating results to metviewer online
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
> > >> > >>>>> > >> > > > here -
> > >> > >>>>> > >> > > > see
> > >> > >>>>> > >> > > > attachment
> > >> > >>>>> > >> > > >
> > >> > >>>>> > >> > > > How do I get the results I generate to be
> > >> > >>>>> > >> > > > uploaded to
> > >> > >>>>> > >> > > > this
> > >> > >>>>> > >> > > > location?
> > >> > >>>>> > >> > > > It's
> > >> > >>>>> > >> > > > not clear how others are setting this up.
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >> > >
> > >> > >>>>> > >>
> > >> > >>>>> > >>
> > >> > >>>>> > >>
> > >> > >>>>> > >> rds_host:3306 mv_cmaqv5para8_aod_grid2grid_metplus
> > >> > >>>>> > >> rds_user
> > >> > >>>>> rds_pwd
> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
> > >> > >>>>> rds_scripts
> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
> > >> > >>>>> > >> CMAQAODPRODHIGH
> > >> > >>>>> 2020-05-12
> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
> > >> > >>>>> > >> 05-12
> > >> > >>>>> 20:00:00
> > >> > >>>>> > >> 2020-05-12
> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
> > >> > >>>>> > >> 05-13
> > >> > >>>>> 10:00:00
> > >> > >>>>> > >> 2020-05-13
> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
> > >> > >>>>> > >> 05-13
> > >> > >>>>> 14:00:00
> > >> > >>>>> > >> 2020-05-13
> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
> > >> > >>>>> > >> 05-13
> > >> > >>>>> 18:00:00
> > >> > >>>>> > >> 2020-05-13
> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
> > >> > >>>>> > >> 05-13
> > >> > >>>>> 22:00:00
> > >> > >>>>> > >> 2020-05-13
> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
> > >> > >>>>> > >> 05-14
> > >> > >>>>> 12:00:00
> > >> > >>>>> > >> true
> > >> > >>>>> > >> median plot_20200612_173626.data
> > >> > >>>>> > >> plot_20200612_173626.png
> > >> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
> > >> > >>>>> > >> false list() list() false false false false false
false
> > >> > >>>>> > >> true
> > >> > >>>>> > >> false
> > >> > >>>>> > >> false
> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
> > >> > >>>>> > >> c(1,1,0) 1 2
> > >> > >>>>> > >> 1.4 -2
> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2
0.5 3
> > >> > >>>>> > >> #cccccc 1
> > >> > >>>>> > >> listX 1
> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
> > >> > >>>>> > >> 3
> > >> > >>>>> > >> 0.8 1
> > >> > >>>>> > >> #333333
> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
> > >> > >>>>> > >> c(TRUE,TRUE)
> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
> > >> > >>>>> > >> c(1,1)
> > >> > >>>>> c(1,1)
> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
> > >> > >>>>> > >
> > >> > >>>>> > >
> > >> > >>>>> > >
> > >> > >>>>> > > --
> > >> > >>>>> > > Edward Strobach
> > >> > >>>>> > > EMC/NCEP/NWS/
> > >> > >>>>> > > IMSG Contractor
> > >> > >>>>> > > Cubicle#: 2029
> > >> > >>>>> > > 301-683-3717
> > >> > >>>>> > >
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>
> > >> > >>>> --
> > >> > >>>> Edward Strobach
> > >> > >>>> EMC/NCEP/NWS/
> > >> > >>>> IMSG Contractor
> > >> > >>>> Cubicle#: 2029
> > >> > >>>> 301-683-3717
> > >> > >>>>
> > >> > >>>
> > >> > >>>
> > >> > >>> --
> > >> > >>> Edward Strobach
> > >> > >>> EMC/NCEP/NWS/
> > >> > >>> IMSG Contractor
> > >> > >>> Cubicle#: 2029
> > >> > >>> 301-683-3717
> > >> > >>>
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> Edward Strobach
> > >> > >> EMC/NCEP/NWS/
> > >> > >> IMSG Contractor
> > >> > >> Cubicle#: 2029
> > >> > >> 301-683-3717
> > >> > >>
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Edward Strobach
> > >> > > EMC/NCEP/NWS/
> > >> > > IMSG Contractor
> > >> > > Cubicle#: 2029
> > >> > > 301-683-3717
> > >> > >
> > >>
> > >>
> > >>
> > >>
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
>
>
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 10:19:02 2020

you're right.  I didn't have the storage problem this time.  I'm not
sure
what changed.  All I know is that I have now created my load xml which
now
should allow me to move on to the next step of plotting

On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> I'm not following you.  How do I now have enough space?  What
changed?
>
> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> Ed, now you should have enough space for the data loading.
>>
>> Tatiana
>>
>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>> > I was basically able to build my xml file but it says I have no
>> > available
>> > space.  I'm not sure how that can be since I just started adding
data
>> > to my
>> > account
>> >
>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space
left
>> > on
>> > device
>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space
left
>> > on
>> > device
>> > ERROR: Command returned with non-zero status (1): scp -r ./*
>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>> > 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
>> > TABLE_SCHEMA                              USER
SIZE_MB
>> > mv_met_g2o_gefs                           binbin.zhou
140027.31
>> > mv_met_system_hrefv2_v3                   binbin.zhou
71986.11
>> >
>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate
<
>> > edward.strobach at noaa.gov> wrote:
>> >
>> > > Thanks Tatiana.  Yes, I've created a database which seems
different
>> > > from
>> > > mv_group. How can one create their own group so that I can
store and
>> > > access
>> > > the database.  The database exists but cannot be viewed on the
>> > > metviewer
>> > > GUI.  The list starts with EMC HREFv3 and ends with vhagerty.
I
>> > > named my
>> > > database Ed_Strobach, but that was really intended for the
group
>> > > name.
>> > >
>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>> > > <met_help at ucar.edu>
>> > > wrote:
>> > >
>> > >> We don't have database and METviewer running on wcoss.
>> > >> Instead, users use METviewer-wcoss scripts that help to
connect to
>> > >> and
>> > >> use METviewer that is running on AWS. That is why XML
connection and
>> > >> folder
>> > >> sections look somewhat different than once from the Docs page.
>> > >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>> > >> For data loading :
>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>> > >> And for batch:
>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>> > >>
>> > >> These scripts copy XML and data files if needed to AWS, adjust
data
>> > >> path
>> > >> and users credentials, run METviewer and copy results back to
wcoss.
>> > >>
>> > >> Tatiana
>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
>> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
>> > >> > -e'create
>> > >> > database [db_name]
>> > >> >
>> > >> > would need to be used to create a database.  I assume that
an sql
>> > >> > module is
>> > >> > available on wcoss; I know you don't have access to those
machines
>> > >> > Tatiana.  If I create my own database with my own name, then
I
>> > >> > should
>> > >> > be
>> > >> > able to proceed with project specific tasks under that tab.
>> > >> > Something
>> > >> > like
>> > >> > mv_EdwardStrobach.
>> > >> >
>> > >> > I can then create mv_cmaq_prod_v_para which I add data to
using
>> > >> > load_spec.
>> > >> > Once that step is done, then I can run plot_spec.  That's my
>> > >> > takeaway
>> > >> > currently
>> > >> >
>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
>> > >> > edward.strobach at noaa.gov> wrote:
>> > >> >
>> > >> > > Actually, it appears that I need to run the database
loading
>> > >> > > module
>> > >> > > first
>> > >> > > if I want to change/add a data source.  From there I can
run the
>> > >> > > plotting
>> > >> > > batch module.  Is that right?   So if I run the database
loading
>> > >> > > module,
>> > >> > > then I can specify my dates and other related info
specific to
>> > >> > > the
>> > >> > > dataset.  If I create my database with a unique name, then
it
>> > >> > > should
>> > >> > > be
>> > >> > > recognized on the metviewer gui, presumably.
>> > >> > >
>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
>> > >> > > <
>> > >> > > edward.strobach at noaa.gov> wrote:
>> > >> > >
>> > >> > >> It seems that those options must be treated separately
outside
>> > >> > >> of
>> > >> > >> plot_spec.  Is it acceptable to create both a load_spec
and
>> > >> > >> plot_spec in
>> > >> > >> the same xml file?  Would the definitions in load_spec be
able
>> > >> > >> to
>> > >> > >> carry
>> > >> > >> over into plot_spec?  It appears that data loading and
plotting
>> > >> > >> are
>> > >> > >> treated
>> > >> > >> separately.
>> > >> > >>
>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
>> > >> > >> Affiliate <
>> > >> > >> edward.strobach at noaa.gov> wrote:
>> > >> > >>
>> > >> > >>> I do plan to use date_range; however, in an example
provided
>> > >> > >>> by the
>> > >> > >>> link
>> > >> > >>> I see this:
>> > >> > >>>
>> > >> > >>> <load_spec>
>> > >> > >>>   <connection>
>> > >> > >>>     <host>kemosabe:3306</host>
>> > >> > >>>     <database>metvdb_hwt</database>
>> > >> > >>>     <user>pgoldenb</user>
>> > >> > >>>     <password>pgoldenb</password>
>> > >> > >>>   </connection>
>> > >> > >>>
>> > >> > >>> <date_list name="folder_dates">
>> > >> > >>>   <start>2010051914V</start>
>> > >> > >>>   <end><date_offset><day_offset>0</day_offset><hour>6
>> > >> > >>> </hour></date_offset></end>
>> > >> > >>>   <inc>3600</inc>
>> > >> > >>>   <format>yyyyMMddHH'V'</format>
>> > >> > >>> </date_list>
>> > >> > >>>
>> > >> > >>> <met_version>V3.0</met_version>
>> > >> > >>>
>> > >> > >>> <verbose>false</verbose>
>> > >> > >>> <insert_size>1</insert_size>
>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>> > >> > >>> <drop_indexes>false</drop_indexes>
>> > >> > >>> <apply_indexes>true</apply_indexes>
>> > >> > >>> <group>Group name</group>
>> > >> > >>> <load_stat>true</load_stat>
>> > >> > >>> <load_mode>true</load_mode>
>> > >> > >>> <load_mtd>true</load_mtd>
>> > >> > >>> <load_mpr>false</load_mpr>
>> > >> > >>>
>> > >> > >>>
>> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>> > >> > >>> <load_val>
>> > >> > >>>   <field name="model">
>> > >> > >>>     <val>arw</val>
>> > >> > >>>     <val>nmm</val>
>> > >> > >>>   </field>
>> > >> > >>>
>> > >> > >>> <field name="valid_time">
>> > >> > >>>   <date_list name="folder_dates"/>
>> > >> > >>> </field>
>> > >> > >>>
>> > >> > >>> <field name="vx_mask">
>> > >> > >>>   <val>FULL</val>
>> > >> > >>>   <val>SWC</val>
>> > >> > >>> </field>
>> > >> > >>> </load_val>
>> > >> > >>> </load_spec>
>> > >> > >>>
>> > >> > >>> Are similar settings allowed with <plot_spec>?  I see
>> > >> > >>> <connection>,
>> > >> > >>> which seems carried over.  I'm looking to specify my
>> > >> > >>> folder_tmpl in
>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>> > >> > >>> Since my
>> > >> > >>> directory structure looks like this:
>> > >> > >>>
>> > >> > >>>
>> > >> > >>>
>> > >>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>> > >> > >>>
>> > >> > >>> I would do something like this I would think:
>> > >> > >>>
>> > >> > >>> .......
>> > >> > >>>
>> > >> > >>> <date_list name="folder_dates">
>> > >> > >>>   <start>20200612/start>
>> > >> > >>>   <end><date_offset><day_offset>0</day_offset></end>
>> > >> > >>>   <inc>86400</inc>
>> > >> > >>>   <format>yyyyMMdd</format>
>> > >> > >>> </date_list>
>> > >> > >>>
>> > >> > >>> <folder_tmpl>
>> > >> > >>>
>> > >>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>> > >> > >>> </folder_tmpl>
>> > >> > >>> <load_val>
>> > >> > >>>   <field name="model">
>> > >> > >>>     <val>prod</val>
>> > >> > >>>   </field>
>> > >> > >>>
>> > >> > >>> <field name="valid_time">
>> > >> > >>>   <date_list name="folder_dates"/>
>> > >> > >>> </field>
>> > >> > >>>
>> > >> > >>> ......
>> > >> > >>>
>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
>> > >> > >>> Affiliate <
>> > >> > >>> edward.strobach at noaa.gov> wrote:
>> > >> > >>>
>> > >> > >>>> Thank you.  This is a very helpful example
>> > >> > >>>>
>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
>> > >> > >>>> <met_help at ucar.edu>
>> > >> > >>>> wrote:
>> > >> > >>>>
>> > >> > >>>>> Hi,
>> > >> > >>>>> Here is some examples how to use <date_range> and
>> > >> > >>>>> <date_range_list> in
>> > >> > >>>>> batch XML.
>> > >> > >>>>> First, you need to defined your range. You can have
more
>> > >> > >>>>> then one
>> > >> > >>>>> <date_range>:
>> > >> > >>>>> <date_range name="may">
>> > >> > >>>>>         <start>2019-05-01 00:00:00</start>
>> > >> > >>>>>         <end>2019-05-31 23:59:59</end>
>> > >> > >>>>> </date_range>
>> > >> > >>>>> <date_range name="march">
>> > >> > >>>>>         <start>2019-03-01 00:00:00</start>
>> > >> > >>>>>         <end>2019-03-31 23:59:59</end>
>> > >> > >>>>> </date_range>
>> > >> > >>>>>
>> > >> > >>>>> and then use one or all date_ranges:
>> > >> > >>>>> <plot_fix>
>> > >> > >>>>>     <field name="fcst_init_beg">
>> > >> > >>>>>         <date_range name="march"/>
>> > >> > >>>>>         <date_range name="may"/>
>> > >> > >>>>>    </field>...
>> > >> > >>>>> This configuration would include all dates for March
and May
>> > >> > >>>>> of
>> > >> > >>>>> 2019.
>> > >> > >>>>>
>> > >> > >>>>> If you need more flexible configuration you can use
>> > >> > >>>>> date_range_list.
>> > >> > >>>>> For example, you want to select first 6 hours every
day
>> > >> > >>>>> beetween
>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
>> > >> > >>>>> look
>> > >> > >>>>> like this:
>> > >> > >>>>> <date_range_list name = 'all'>
>> > >> > >>>>>        <range_start>2020-01-01 00:00:00</range_start>
>> > >> > >>>>>         <range_end>2020-01-05 00:00:00</range_end>
>> > >> > >>>>>         <range_length>360</range_length>
>> > >> > >>>>>         <inc>1440</inc>
>> > >> > >>>>> </date_range_list>
>> > >> > >>>>> where the 1st and 2nd values are the start and end,
>> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
>> > >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
>> > >> > >>>>> This example would create following ranges:
>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>> > >> > >>>>>
>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
>> > >> > >>>>> your
>> > >> > >>>>> data.
>> > >> > >>>>> For example, if you want to select only dates with
00:00 you
>> > >> > >>>>> can
>> > >> > >>>>> specify
>> > >> > >>>>>  <field name="valid_hour">
>> > >> > >>>>>         <val>00</val>
>> > >> > >>>>>  </field>
>> > >> > >>>>>
>> > >> > >>>>> Tatiana
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
wrote:
>> > >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
>> > >> > >>>>> > of a
>> > >> > >>>>> > separate
>> > >> > >>>>> > script that pipes dates (and other relevant
information)
>> > >> > >>>>> > into
>> > >> > >>>>> > the XML
>> > >> > >>>>> > file
>> > >> > >>>>> > used to generate the results.
>> > >> > >>>>> >
>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
>> > >> > >>>>> > Affiliate <
>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>> > >> > >>>>> >
>> > >> > >>>>> > > Thanks Tatiana.  These resources are helpful.
>> > >> > >>>>> > >
>> > >> > >>>>> > > One last question.  Simply typing lines like this
in can
>> > >> > >>>>> > > take
>> > >> > >>>>> > > forever:
>> > >> > >>>>> > >
>> > >> > >>>>> > > <plot_fix>
>> > >> > >>>>> > >     <field equalize="false" name="fcst_init_beg">
>> > >> > >>>>> > >         <set name="fcst_init_beg_0">
>> > >> > >>>>> > >             <val>2019-05-01 12:00:00</val>
>> > >> > >>>>> > >             <val>2019-05-02 06:00:00</val>
>> > >> > >>>>> > >             <val>2019-05-02 12:00:00</val>
>> > >> > >>>>> > >             <val>2019-05-03 06:00:00</val>
>> > >> > >>>>> > >             <val>2019-05-03 12:00:00</val>
>> > >> > >>>>> > >
>> > >> > >>>>> > >
>> > >> > >>>>> > > Is there a better way?   I see date_range,
>> > >> > >>>>> > > date_range_list,
>> > >> > >>>>> > > or
>> > >> > >>>>> > > date_list.
>> > >> > >>>>> > > Can this be used instead of typing individual
values?
>> > >> > >>>>> > > This
>> > >> > >>>>> > > could
>> > >> > >>>>> > > prove
>> > >> > >>>>> > > pretty daunting if one would have to type in 700
entries
>> > >> > >>>>> > > each
>> > >> > >>>>> > > time.
>> > >> > >>>>> > >
>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via
RT
>> > >> > >>>>> > > <met_help at ucar.edu>
>> > >> > >>>>> > > wrote:
>> > >> > >>>>> > >
>> > >> > >>>>> > >> I found a few problems in your XML.
>> > >> > >>>>> > >> The database you are using doesn't have data with
>> > >> > >>>>> > >> vx_mask
>> > >> > >>>>> > >> 'abcREGabc'.
>> > >> > >>>>> > >> That is why METviewer returns an empty dataset
and
>> > >> > >>>>> > >> can't
>> > >> > >>>>> > >> create a
>> > >> > >>>>> > >> plot.
>> > >> > >>>>> > >> The other problem is how you define fcst_lead.
This
>> > >> > >>>>> > >> database
>> > >> > >>>>> > >> has
>> > >> > >>>>> > >> values
>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since
you
>> > >> > >>>>> > >> include
>> > >> > >>>>> > >> all
>> > >> > >>>>> > >> fcst_leads
>> > >> > >>>>> > >> this variable can be omitted.
>> > >> > >>>>> > >> To minimize problems with variables values, I
would
>> > >> > >>>>> > >> suggest
>> > >> > >>>>> > >> to
>> > >> > >>>>> > >> create XML
>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>> > >> > >>>>> > >> I created a plot using you configurations with
GUI
>> > >> > >>>>> > >> (without
>> > >> > >>>>> vx_mack
>> > >> > >>>>> > >> and
>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You can
load it
>> > >> > >>>>> > >> back
>> > >> > >>>>> > >> to
>> > >> > >>>>> the
>> > >> > >>>>> > >> page by
>> > >> > >>>>> > >> using the "History" tab (left bar on GUI). Find
the
>> > >> > >>>>> > >> plot,
>> > >> > >>>>> > >> open XML
>> > >> > >>>>> > >> and
>> > >> > >>>>> > >> click on "Load to page". After that you can
recreate a
>> > >> > >>>>> > >> plot
>> > >> > >>>>> > >> and
>> > >> > >>>>> edit
>> > >> > >>>>> > >> the
>> > >> > >>>>> > >> configurations. You can also download a new XML
and
>> > >> > >>>>> > >> edit it
>> > >> > >>>>> > >> for
>> > >> > >>>>> > >> batch.
>> > >> > >>>>> > >> I attached the example of load XML to this
message.
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> Description of rds parameters:
>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of METviewer
R
>> > >> > >>>>> > >> scripts
>> > >> > >>>>> > >> rds_plots - temporary directory where the plots
will be
>> > >> > >>>>> > >> generated
>> > >> > >>>>> > >> rds_data - temporary directory where the data
retrieved
>> > >> > >>>>> > >> from
>> > >> > >>>>> > >> the
>> > >> > >>>>> > >> database
>> > >> > >>>>> > >> and other intermediate files will be stored
>> > >> > >>>>> > >> rds_scripts - temporary directory where R scripts
for
>> > >> > >>>>> > >> this
>> > >> > >>>>> > >> run
>> > >> > >>>>> will
>> > >> > >>>>> > >> be
>> > >> > >>>>> > >> stored
>> > >> > >>>>> > >> After the execution is done all the files from
>> > >> > >>>>> > >> temporary
>> > >> > >>>>> directories
>> > >> > >>>>> > >> would be removed.
>> > >> > >>>>> > >>
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
>> > >> > >>>>> > >> There
>> > >> > >>>>> > >> are
>> > >> > >>>>> some
>> > >> > >>>>> > >> instructions on how to crate XML config files:
>> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> Tatiana
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
>> > >> > >>>>> > >> wrote:
>> > >> > >>>>> > >> > Hi,
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > I'm attaching the xml below, but am having hard
time
>> > >> > >>>>> > >> > following
>> > >> > >>>>> how
>> > >> > >>>>> > >> > to
>> > >> > >>>>> > >> > go
>> > >> > >>>>> > >> > about the other steps since I have zero
experience on
>> > >> > >>>>> > >> > how
>> > >> > >>>>> > >> > to do
>> > >> > >>>>> > >> > these
>> > >> > >>>>> > >> > things.
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > #2. All parameters in <folders> section should
NOT be
>> > >> > >>>>> > >> > changed
>> > >> > >>>>> and
>> > >> > >>>>> > >> > will
>> > >> > >>>>> > >> > be
>> > >> > >>>>> > >> > replaced by other scripts during the batch
execution.
>> > >> > >>>>> > >> > The
>> > >> > >>>>> > >> > data
>> > >> > >>>>> for
>> > >> > >>>>> > >> > the
>> > >> > >>>>> > >> > batch mode is coming from the database and not
from
>> > >> > >>>>> > >> > files.
>> > >> > >>>>> > >> > The
>> > >> > >>>>> > >> > name of
>> > >> > >>>>> > >> > the
>> > >> > >>>>> > >> > database is in <database> tag.
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > I understand that nothing should be changed in
the
>> > >> > >>>>> > >> > folders
>> > >> > >>>>> > >> > section,
>> > >> > >>>>> > >> > but
>> > >> > >>>>> > >> > there is no indication how and where these
variables
>> > >> > >>>>> > >> > are
>> > >> > >>>>> > >> > being
>> > >> > >>>>> > >> > defined.
>> > >> > >>>>> > >> > Grepping these names of the files available
does not
>> > >> > >>>>> > >> > reveal
>> > >> > >>>>> formal
>> > >> > >>>>> > >> > definitions of these variables.  If I
understood
>> > >> > >>>>> > >> > exactly
>> > >> > >>>>> > >> > how
>> > >> > >>>>> these
>> > >> > >>>>> > >> > were
>> > >> > >>>>> > >> > being defined, then I would be able connect
this
>> > >> > >>>>> > >> > better.
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output files.
They
>> > >> > >>>>> > >> > use
>> > >> > >>>>> METviewer
>> > >> > >>>>> > >> > load
>> > >> > >>>>> > >> > mode to load these files to the specific
database. In
>> > >> > >>>>> > >> > the
>> > >> > >>>>> > >> > config
>> > >> > >>>>> > >> > file
>> > >> > >>>>> > >> > for
>> > >> > >>>>> > >> > this process they also can specify the database
>> > >> > >>>>> > >> > group. For
>> > >> > >>>>> example
>> > >> > >>>>> > >> > database
>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>> > >> > >>>>> > >> > After the data has been loaded to the database
it is
>> > >> > >>>>> > >> > visible and
>> > >> > >>>>> > >> > available
>> > >> > >>>>> > >> > in the GUI and also can be used by the batch
mode.
>> > >> > >>>>> > >> > You can always add more data to the database
using
>> > >> > >>>>> > >> > the
>> > >> > >>>>> > >> > same load
>> > >> > >>>>> > >> > mode.
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > Is there a guide for me to understand the
process
>> > >> > >>>>> > >> > that you
>> > >> > >>>>> > >> > are
>> > >> > >>>>> > >> > referring to?
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek
via RT
>> > >> > >>>>> > >> > <met_help at ucar.edu>
>> > >> > >>>>> > >> > wrote:
>> > >> > >>>>> > >> >
>> > >> > >>>>> > >> > > Hi Edward,
>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA servers
and
>> > >> > >>>>> > >> > > can't
>> > >> > >>>>> > >> > > view your
>> > >> > >>>>> > >> > > XML
>> > >> > >>>>> > >> > > for the
>> > >> > >>>>> > >> > > batch mode. Please, attache it to this issue
and I
>> > >> > >>>>> > >> > > will
>> > >> > >>>>> > >> > > take a
>> > >> > >>>>> > >> > > look.
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
should NOT
>> > >> > >>>>> > >> > > be
>> > >> > >>>>> > >> > > changed
>> > >> > >>>>> > >> > > and
>> > >> > >>>>> > >> > > will be
>> > >> > >>>>> > >> > > replaced by other scripts during the batch
>> > >> > >>>>> > >> > > execution.
>> > >> > >>>>> > >> > > The data
>> > >> > >>>>> > >> > > for
>> > >> > >>>>> > >> > > the
>> > >> > >>>>> > >> > > batch mode is coming from the database and
not from
>> > >> > >>>>> > >> > > files. The
>> > >> > >>>>> > >> > > name
>> > >> > >>>>> > >> > > of the
>> > >> > >>>>> > >> > > database is in <database> tag.
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
>> > >> > >>>>> > >> > > They use
>> > >> > >>>>> > >> > > METviewer
>> > >> > >>>>> > >> > > load
>> > >> > >>>>> > >> > > mode to load these files to the specific
database.
>> > >> > >>>>> > >> > > In
>> > >> > >>>>> > >> > > the
>> > >> > >>>>> config
>> > >> > >>>>> > >> > > file
>> > >> > >>>>> > >> > > for
>> > >> > >>>>> > >> > > this process they also can specify the
database
>> > >> > >>>>> > >> > > group.
>> > >> > >>>>> > >> > > For
>> > >> > >>>>> > >> > > example
>> > >> > >>>>> > >> > > database
>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>> > >> > >>>>> > >> > > After the data has been loaded to the
database it
>> > >> > >>>>> > >> > > is
>> > >> > >>>>> > >> > > visible
>> > >> > >>>>> and
>> > >> > >>>>> > >> > > available
>> > >> > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
>> > >> > >>>>> > >> > > You can always add more data to the database
using
>> > >> > >>>>> > >> > > the
>> > >> > >>>>> > >> > > same
>> > >> > >>>>> load
>> > >> > >>>>> > >> > > mode.
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > > Tatiana
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>> > >> > >>>>> > >> > > wrote:
>> > >> > >>>>> > >> > > > Good morning,
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Over the last two weeks I've made an effort
to
>> > >> > >>>>> > >> > > > understand
>> > >> > >>>>> > >> > > > metviewer,
>> > >> > >>>>> > >> > > > test
>> > >> > >>>>> > >> > > > different cases using the history tab,
changing
>> > >> > >>>>> > >> > > > information
>> > >> > >>>>> to
>> > >> > >>>>> > >> > > > see
>> > >> > >>>>> > >> > > > what
>> > >> > >>>>> > >> > > > works and what does not, and make
connections
>> > >> > >>>>> > >> > > > between
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> GUI
>> > >> > >>>>> > >> > > > settings
>> > >> > >>>>> > >> > > > and
>> > >> > >>>>> > >> > > > the XML file.  Although my comfort level
with
>> > >> > >>>>> > >> > > > metviewer has
>> > >> > >>>>> > >> > > > significantly
>> > >> > >>>>> > >> > > > increased during this time, there are some
>> > >> > >>>>> > >> > > > remaining
>> > >> > >>>>> > >> > > > issues
>> > >> > >>>>> > >> > > > that
>> > >> > >>>>> > >> > > > I'm
>> > >> > >>>>> > >> > > > hoping
>> > >> > >>>>> > >> > > > to clear up.
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > 1.  Problem changing XML file and running
with
>> > >> > >>>>> > >> > > > different
>> > >> > >>>>> > >> > > > statistic
>> > >> > >>>>> > >> > > > and
>> > >> > >>>>> > >> > > > plot
>> > >> > >>>>> > >> > > > type:
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC, provided
me
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> > >> > > > working
>> > >> > >>>>> > >> > > > directory
>> > >> > >>>>> > >> > > > to get
>> > >> > >>>>> > >> > > > started which I copied over:
>> > >> > >>>>> > >> > > >
>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted and
used
>> > >> > >>>>> > >> > > > to
>> > >> > >>>>> > >> > > > run his
>> > >> > >>>>> > >> > > > example.
>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
produced a
>> > >> > >>>>> > >> > > > CSI
>> > >> > >>>>> > >> > > > plot
>> > >> > >>>>> that
>> > >> > >>>>> > >> > > > was
>> > >> > >>>>> > >> > > > stored
>> > >> > >>>>> > >> > > > here:
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >>
>> > >> > >>>>>
>> > >>
>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Running the script produced the expected
result.
>> > >> > >>>>> > >> > > > The
>> > >> > >>>>> > >> > > > next
>> > >> > >>>>> > >> > > > natural
>> > >> > >>>>> > >> > > > step was
>> > >> > >>>>> > >> > > > to copy hochum_xml.template and create a
new xml
>> > >> > >>>>> > >> > > > file
>> > >> > >>>>> > >> > > > called
>> > >> > >>>>> > >> > > > Strobach_xml.template in order to create a
time
>> > >> > >>>>> > >> > > > series
>> > >> > >>>>> > >> > > > where
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg instead
of
>> > >> > >>>>> > >> > > > obs_thresh.
>> > >> > >>>>> > >> > > > Accordingly,
>> > >> > >>>>> > >> > > > I
>> > >> > >>>>> > >> > > > made changes to the fixed fields (removing
>> > >> > >>>>> > >> > > > fcst_valid_beg)
>> > >> > >>>>> and
>> > >> > >>>>> > >> > > > changed
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
>> > >> > >>>>> > >> > > > tried).
>> > >> > >>>>> > >> > > > The
>> > >> > >>>>> error
>> > >> > >>>>> > >> > > > that
>> > >> > >>>>> > >> > > > resulted
>> > >> > >>>>> > >> > > > is as follows:
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-
05-01
>> > >> > >>>>> > >> > > > 13:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 01
>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-05-
01
>> > >> > >>>>> > >> > > > 16:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 01
>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-05-
01
>> > >> > >>>>> > >> > > > 19:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 01
>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-05-
01
>> > >> > >>>>> > >> > > > 22:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 01
>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-05-
02
>> > >> > >>>>> > >> > > > 01:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 02
>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-05-
02
>> > >> > >>>>> > >> > > > 04:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 02
>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-05-
02
>> > >> > >>>>> > >> > > > 07:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 02
>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-05-
02
>> > >> > >>>>> > >> > > > 10:00:00',
>> > >> > >>>>> > >> > > > '2020-05-
>> > >> > >>>>> > >> > > > 02
>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>> > >> > >>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
>> > >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>> > >> > >>>>> > >> > > > No data was returned from database
>> > >> > >>>>> > >> > > > 2020-06-09
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >>
>> > >> > >>>>>
>> > >>
>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable -   **
ERROR:
>> > >> > >>>>> > >> > > > Caught
>> > >> > >>>>> > >> > > > class
>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
in
>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>> > >> > >>>>> > >> > > > res): result set contained no data
>> > >> > >>>>> > >> > > > **  ERROR: Caught class
>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >> > >>>>> > >> > > > in
>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
>> > >> > >>>>> > >> > > > contained no
>> > >> > >>>>> > >> > > > data
>> > >> > >>>>> > >> > > > **  ERROR: Caught class
>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>> > >> > >>>>> > >> > > > in
>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
>> > >> > >>>>> > >> > > > contained no
>> > >> > >>>>> > >> > > > data
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Apparently there was no data during this
time.  I
>> > >> > >>>>> > >> > > > tried
>> > >> > >>>>> > >> > > > changing
>> > >> > >>>>> > >> > > > the
>> > >> > >>>>> > >> > > > range
>> > >> > >>>>> > >> > > > of valid times and changed the valid time
to
>> > >> > >>>>> > >> > > > different
>> > >> > >>>>> > >> > > > days
>> > >> > >>>>> > >> > > > just in
>> > >> > >>>>> > >> > > > case
>> > >> > >>>>> > >> > > > data was missing.  The issue was the same.
To
>> > >> > >>>>> > >> > > > confirm
>> > >> > >>>>> > >> > > > my
>> > >> > >>>>> > >> > > > logic, I
>> > >> > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
>> > >> > >>>>> > >> > > > which I
>> > >> > >>>>> > >> > > > found
>> > >> > >>>>> > >> > > > to
>> > >> > >>>>> > >> > > > work.
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > 2.  Running in batch mode
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > I think I lack an understanding of how to
set up
>> > >> > >>>>> > >> > > > batch
>> > >> > >>>>> > >> > > > mode
>> > >> > >>>>> > >> > > > because
>> > >> > >>>>> > >> > > > I
>> > >> > >>>>> > >> > > > don't
>> > >> > >>>>> > >> > > > see where these definitions are being
explicitly
>> > >> > >>>>> > >> > > > defined:
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > <plot_spec>
>> > >> > >>>>> > >> > > >     <connection>
>> > >> > >>>>> > >> > > >         <host>rds_host:3306</host>
>> > >> > >>>>> > >> > > >
>> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>> > >> > >>>>> > >> > > > <user>rds_user</user>
>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>> > >> > >>>>> > >> > > > </connection>
>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>> > >> > >>>>> > >> > > > <folders>
>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>> > >> > >>>>> > >> > > > <data>rds_data</data>
>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>> > >> > >>>>> > >> > > > </folders>
>> > >> > >>>>> > >> > > > <plot>
>> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > I see the <folders> section, but grepping
all
>> > >> > >>>>> > >> > > > available
>> > >> > >>>>> files
>> > >> > >>>>> > >> > > > does
>> > >> > >>>>> > >> > > > not
>> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well
>> > >> > >>>>> > >> > > > as
>> > >> > >>>>> > >> > > > other
>> > >> > >>>>> > >> > > > similarly
>> > >> > >>>>> > >> > > > defined
>> > >> > >>>>> > >> > > > variables.  I think if I know how these are
being
>> > >> > >>>>> > >> > > > defined,
>> > >> > >>>>> > >> > > > then
>> > >> > >>>>> > >> > > > I'll
>> > >> > >>>>> > >> > > > be
>> > >> > >>>>> > >> > > > able to pipe in my information and process
the
>> > >> > >>>>> > >> > > > data
>> > >> > >>>>> > >> > > > using
>> > >> > >>>>> met
>> > >> > >>>>> > >> > > > and
>> > >> > >>>>> > >> > > > produce
>> > >> > >>>>> > >> > > > results in batch mode.  Am I right on that?
>> > >> > >>>>> > >> > > > Also, if
>> > >> > >>>>> > >> > > > I want
>> > >> > >>>>> > >> > > > to use
>> > >> > >>>>> > >> > > > obs for
>> > >> > >>>>> > >> > > > verification, do I add another line in the
XML
>> > >> > >>>>> > >> > > > file
>> > >> > >>>>> > >> > > > that
>> > >> > >>>>> > >> > > > designates
>> > >> > >>>>> > >> > > > file
>> > >> > >>>>> > >> > > > location of obs?  I assume that other steps
have
>> > >> > >>>>> > >> > > > to be
>> > >> > >>>>> > >> > > > run
>> > >> > >>>>> > >> > > > before
>> > >> > >>>>> > >> > > > all
>> > >> > >>>>> > >> > > > this.  If data is in prepbufr form, for
example,
>> > >> > >>>>> > >> > > > I
>> > >> > >>>>> > >> > > > would
>> > >> > >>>>> have
>> > >> > >>>>> > >> > > > to
>> > >> > >>>>> > >> > > > separately
>> > >> > >>>>> > >> > > > run PB2NC.  If I want grid2obs, then I
would want
>> > >> > >>>>> > >> > > > to
>> > >> > >>>>> > >> > > > do that
>> > >> > >>>>> > >> > > > as
>> > >> > >>>>> > >> > > > well.
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
>> > >> > >>>>> > >> > > > here -
>> > >> > >>>>> > >> > > > see
>> > >> > >>>>> > >> > > > attachment
>> > >> > >>>>> > >> > > >
>> > >> > >>>>> > >> > > > How do I get the results I generate to be
>> > >> > >>>>> > >> > > > uploaded to
>> > >> > >>>>> > >> > > > this
>> > >> > >>>>> > >> > > > location?
>> > >> > >>>>> > >> > > > It's
>> > >> > >>>>> > >> > > > not clear how others are setting this up.
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >> > >
>> > >> > >>>>> > >>
>> > >> > >>>>> > >>
>> > >> > >>>>> > >>
>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>> > >> > >>>>> > >> rds_user
>> > >> > >>>>> rds_pwd
>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
>> > >> > >>>>> rds_scripts
>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>> > >> > >>>>> > >> CMAQAODPRODHIGH
>> > >> > >>>>> 2020-05-12
>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
>> > >> > >>>>> > >> 05-12
>> > >> > >>>>> 20:00:00
>> > >> > >>>>> > >> 2020-05-12
>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
>> > >> > >>>>> > >> 05-13
>> > >> > >>>>> 10:00:00
>> > >> > >>>>> > >> 2020-05-13
>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
>> > >> > >>>>> > >> 05-13
>> > >> > >>>>> 14:00:00
>> > >> > >>>>> > >> 2020-05-13
>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
>> > >> > >>>>> > >> 05-13
>> > >> > >>>>> 18:00:00
>> > >> > >>>>> > >> 2020-05-13
>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
>> > >> > >>>>> > >> 05-13
>> > >> > >>>>> 22:00:00
>> > >> > >>>>> > >> 2020-05-13
>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
>> > >> > >>>>> > >> 05-14
>> > >> > >>>>> 12:00:00
>> > >> > >>>>> > >> true
>> > >> > >>>>> > >> median plot_20200612_173626.data
>> > >> > >>>>> > >> plot_20200612_173626.png
>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
>> > >> > >>>>> > >> false list() list() false false false false false
false
>> > >> > >>>>> > >> true
>> > >> > >>>>> > >> false
>> > >> > >>>>> > >> false
>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
>> > >> > >>>>> > >> c(1,1,0) 1 2
>> > >> > >>>>> > >> 1.4 -2
>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2
0.5 3
>> > >> > >>>>> > >> #cccccc 1
>> > >> > >>>>> > >> listX 1
>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o c(0,
-.25)
>> > >> > >>>>> > >> 3
>> > >> > >>>>> > >> 0.8 1
>> > >> > >>>>> > >> #333333
>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
>> > >> > >>>>> > >> c(TRUE,TRUE)
>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
>> > >> > >>>>> > >> c(1,1)
>> > >> > >>>>> c(1,1)
>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>> > >> > >>>>> > >
>> > >> > >>>>> > >
>> > >> > >>>>> > >
>> > >> > >>>>> > > --
>> > >> > >>>>> > > Edward Strobach
>> > >> > >>>>> > > EMC/NCEP/NWS/
>> > >> > >>>>> > > IMSG Contractor
>> > >> > >>>>> > > Cubicle#: 2029
>> > >> > >>>>> > > 301-683-3717
>> > >> > >>>>> > >
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>
>> > >> > >>>> --
>> > >> > >>>> Edward Strobach
>> > >> > >>>> EMC/NCEP/NWS/
>> > >> > >>>> IMSG Contractor
>> > >> > >>>> Cubicle#: 2029
>> > >> > >>>> 301-683-3717
>> > >> > >>>>
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> --
>> > >> > >>> Edward Strobach
>> > >> > >>> EMC/NCEP/NWS/
>> > >> > >>> IMSG Contractor
>> > >> > >>> Cubicle#: 2029
>> > >> > >>> 301-683-3717
>> > >> > >>>
>> > >> > >>
>> > >> > >>
>> > >> > >> --
>> > >> > >> Edward Strobach
>> > >> > >> EMC/NCEP/NWS/
>> > >> > >> IMSG Contractor
>> > >> > >> Cubicle#: 2029
>> > >> > >> 301-683-3717
>> > >> > >>
>> > >> > >
>> > >> > >
>> > >> > > --
>> > >> > > Edward Strobach
>> > >> > > EMC/NCEP/NWS/
>> > >> > > IMSG Contractor
>> > >> > > Cubicle#: 2029
>> > >> > > 301-683-3717
>> > >> > >
>> > >>
>> > >>
>> > >>
>> > >>
>> > >
>> > > --
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/
>> > > IMSG Contractor
>> > > Cubicle#: 2029
>> > > 301-683-3717
>> > >
>>
>>
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>


--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Jeff.Mcqueen at noaa.gov
Time: Wed Jun 17 11:34:19 2020

Would it be ok to keep perry in the loop

On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> you're right.  I didn't have the storage problem this time.  I'm not
sure
> what changed.  All I know is that I have now created my load xml
which now
> should allow me to move on to the next step of plotting
>
> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> I'm not following you.  How do I now have enough space?  What
changed?
>>
>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
<met_help at ucar.edu>
>> wrote:
>>
>>> Ed, now you should have enough space for the data loading.
>>>
>>> Tatiana
>>>
>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>> > I was basically able to build my xml file but it says I have no
>>> > available
>>> > space.  I'm not sure how that can be since I just started adding
data
>>> > to my
>>> > account
>>> >
>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No space
left
>>> > on
>>> > device
>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No space
left
>>> > on
>>> > device
>>> > ERROR: Command returned with non-zero status (1): scp -r ./*
>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>> > 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
>>> > TABLE_SCHEMA                              USER
SIZE_MB
>>> > mv_met_g2o_gefs                           binbin.zhou
140027.31
>>> > mv_met_system_hrefv2_v3                   binbin.zhou
71986.11
>>> >
>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA Affiliate
<
>>> > edward.strobach at noaa.gov> wrote:
>>> >
>>> > > Thanks Tatiana.  Yes, I've created a database which seems
different
>>> > > from
>>> > > mv_group. How can one create their own group so that I can
store and
>>> > > access
>>> > > the database.  The database exists but cannot be viewed on the
>>> > > metviewer
>>> > > GUI.  The list starts with EMC HREFv3 and ends with vhagerty.
I
>>> > > named my
>>> > > database Ed_Strobach, but that was really intended for the
group
>>> > > name.
>>> > >
>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>> > > <met_help at ucar.edu>
>>> > > wrote:
>>> > >
>>> > >> We don't have database and METviewer running on wcoss.
>>> > >> Instead, users use METviewer-wcoss scripts that help to
connect to
>>> > >> and
>>> > >> use METviewer that is running on AWS. That is why XML
connection and
>>> > >> folder
>>> > >> sections look somewhat different than once from the Docs
page.
>>> > >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>> > >> For data loading :
>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>> > >> And for batch:
>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>> > >>
>>> > >> These scripts copy XML and data files if needed to AWS,
adjust data
>>> > >> path
>>> > >> and users credentials, run METviewer and copy results back to
wcoss.
>>> > >>
>>> > >> Tatiana
>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
>>> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
>>> > >> > -e'create
>>> > >> > database [db_name]
>>> > >> >
>>> > >> > would need to be used to create a database.  I assume that
an sql
>>> > >> > module is
>>> > >> > available on wcoss; I know you don't have access to those
machines
>>> > >> > Tatiana.  If I create my own database with my own name,
then I
>>> > >> > should
>>> > >> > be
>>> > >> > able to proceed with project specific tasks under that tab.
>>> > >> > Something
>>> > >> > like
>>> > >> > mv_EdwardStrobach.
>>> > >> >
>>> > >> > I can then create mv_cmaq_prod_v_para which I add data to
using
>>> > >> > load_spec.
>>> > >> > Once that step is done, then I can run plot_spec.  That's
my
>>> > >> > takeaway
>>> > >> > currently
>>> > >> >
>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate <
>>> > >> > edward.strobach at noaa.gov> wrote:
>>> > >> >
>>> > >> > > Actually, it appears that I need to run the database
loading
>>> > >> > > module
>>> > >> > > first
>>> > >> > > if I want to change/add a data source.  From there I can
run the
>>> > >> > > plotting
>>> > >> > > batch module.  Is that right?   So if I run the database
loading
>>> > >> > > module,
>>> > >> > > then I can specify my dates and other related info
specific to
>>> > >> > > the
>>> > >> > > dataset.  If I create my database with a unique name,
then it
>>> > >> > > should
>>> > >> > > be
>>> > >> > > recognized on the metviewer gui, presumably.
>>> > >> > >
>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
Affiliate
>>> > >> > > <
>>> > >> > > edward.strobach at noaa.gov> wrote:
>>> > >> > >
>>> > >> > >> It seems that those options must be treated separately
outside
>>> > >> > >> of
>>> > >> > >> plot_spec.  Is it acceptable to create both a load_spec
and
>>> > >> > >> plot_spec in
>>> > >> > >> the same xml file?  Would the definitions in load_spec
be able
>>> > >> > >> to
>>> > >> > >> carry
>>> > >> > >> over into plot_spec?  It appears that data loading and
plotting
>>> > >> > >> are
>>> > >> > >> treated
>>> > >> > >> separately.
>>> > >> > >>
>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
>>> > >> > >> Affiliate <
>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>> > >> > >>
>>> > >> > >>> I do plan to use date_range; however, in an example
provided
>>> > >> > >>> by the
>>> > >> > >>> link
>>> > >> > >>> I see this:
>>> > >> > >>>
>>> > >> > >>> <load_spec>
>>> > >> > >>>   <connection>
>>> > >> > >>>     <host>kemosabe:3306</host>
>>> > >> > >>>     <database>metvdb_hwt</database>
>>> > >> > >>>     <user>pgoldenb</user>
>>> > >> > >>>     <password>pgoldenb</password>
>>> > >> > >>>   </connection>
>>> > >> > >>>
>>> > >> > >>> <date_list name="folder_dates">
>>> > >> > >>>   <start>2010051914V</start>
>>> > >> > >>>   <end><date_offset><day_offset>0</day_offset><hour>6
>>> > >> > >>> </hour></date_offset></end>
>>> > >> > >>>   <inc>3600</inc>
>>> > >> > >>>   <format>yyyyMMddHH'V'</format>
>>> > >> > >>> </date_list>
>>> > >> > >>>
>>> > >> > >>> <met_version>V3.0</met_version>
>>> > >> > >>>
>>> > >> > >>> <verbose>false</verbose>
>>> > >> > >>> <insert_size>1</insert_size>
>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>> > >> > >>> <group>Group name</group>
>>> > >> > >>> <load_stat>true</load_stat>
>>> > >> > >>> <load_mode>true</load_mode>
>>> > >> > >>> <load_mtd>true</load_mtd>
>>> > >> > >>> <load_mpr>false</load_mpr>
>>> > >> > >>>
>>> > >> > >>>
>>> <folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>> > >> > >>> <load_val>
>>> > >> > >>>   <field name="model">
>>> > >> > >>>     <val>arw</val>
>>> > >> > >>>     <val>nmm</val>
>>> > >> > >>>   </field>
>>> > >> > >>>
>>> > >> > >>> <field name="valid_time">
>>> > >> > >>>   <date_list name="folder_dates"/>
>>> > >> > >>> </field>
>>> > >> > >>>
>>> > >> > >>> <field name="vx_mask">
>>> > >> > >>>   <val>FULL</val>
>>> > >> > >>>   <val>SWC</val>
>>> > >> > >>> </field>
>>> > >> > >>> </load_val>
>>> > >> > >>> </load_spec>
>>> > >> > >>>
>>> > >> > >>> Are similar settings allowed with <plot_spec>?  I see
>>> > >> > >>> <connection>,
>>> > >> > >>> which seems carried over.  I'm looking to specify my
>>> > >> > >>> folder_tmpl in
>>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>>> > >> > >>> Since my
>>> > >> > >>> directory structure looks like this:
>>> > >> > >>>
>>> > >> > >>>
>>> > >> > >>>
>>> > >>
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>> > >> > >>>
>>> > >> > >>> I would do something like this I would think:
>>> > >> > >>>
>>> > >> > >>> .......
>>> > >> > >>>
>>> > >> > >>> <date_list name="folder_dates">
>>> > >> > >>>   <start>20200612/start>
>>> > >> > >>>   <end><date_offset><day_offset>0</day_offset></end>
>>> > >> > >>>   <inc>86400</inc>
>>> > >> > >>>   <format>yyyyMMdd</format>
>>> > >> > >>> </date_list>
>>> > >> > >>>
>>> > >> > >>> <folder_tmpl>
>>> > >> > >>>
>>> > >>
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>> > >> > >>> </folder_tmpl>
>>> > >> > >>> <load_val>
>>> > >> > >>>   <field name="model">
>>> > >> > >>>     <val>prod</val>
>>> > >> > >>>   </field>
>>> > >> > >>>
>>> > >> > >>> <field name="valid_time">
>>> > >> > >>>   <date_list name="folder_dates"/>
>>> > >> > >>> </field>
>>> > >> > >>>
>>> > >> > >>> ......
>>> > >> > >>>
>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
>>> > >> > >>> Affiliate <
>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>> > >> > >>>
>>> > >> > >>>> Thank you.  This is a very helpful example
>>> > >> > >>>>
>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
>>> > >> > >>>> <met_help at ucar.edu>
>>> > >> > >>>> wrote:
>>> > >> > >>>>
>>> > >> > >>>>> Hi,
>>> > >> > >>>>> Here is some examples how to use <date_range> and
>>> > >> > >>>>> <date_range_list> in
>>> > >> > >>>>> batch XML.
>>> > >> > >>>>> First, you need to defined your range. You can have
more
>>> > >> > >>>>> then one
>>> > >> > >>>>> <date_range>:
>>> > >> > >>>>> <date_range name="may">
>>> > >> > >>>>>         <start>2019-05-01 00:00:00</start>
>>> > >> > >>>>>         <end>2019-05-31 23:59:59</end>
>>> > >> > >>>>> </date_range>
>>> > >> > >>>>> <date_range name="march">
>>> > >> > >>>>>         <start>2019-03-01 00:00:00</start>
>>> > >> > >>>>>         <end>2019-03-31 23:59:59</end>
>>> > >> > >>>>> </date_range>
>>> > >> > >>>>>
>>> > >> > >>>>> and then use one or all date_ranges:
>>> > >> > >>>>> <plot_fix>
>>> > >> > >>>>>     <field name="fcst_init_beg">
>>> > >> > >>>>>         <date_range name="march"/>
>>> > >> > >>>>>         <date_range name="may"/>
>>> > >> > >>>>>    </field>...
>>> > >> > >>>>> This configuration would include all dates for March
and May
>>> > >> > >>>>> of
>>> > >> > >>>>> 2019.
>>> > >> > >>>>>
>>> > >> > >>>>> If you need more flexible configuration you can use
>>> > >> > >>>>> date_range_list.
>>> > >> > >>>>> For example, you want to select first 6 hours every
day
>>> > >> > >>>>> beetween
>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your XML
would
>>> > >> > >>>>> look
>>> > >> > >>>>> like this:
>>> > >> > >>>>> <date_range_list name = 'all'>
>>> > >> > >>>>>        <range_start>2020-01-01 00:00:00</range_start>
>>> > >> > >>>>>         <range_end>2020-01-05 00:00:00</range_end>
>>> > >> > >>>>>         <range_length>360</range_length>
>>> > >> > >>>>>         <inc>1440</inc>
>>> > >> > >>>>> </date_range_list>
>>> > >> > >>>>> where the 1st and 2nd values are the start and end,
>>> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
>>> > >> > >>>>> This example would create following ranges:
>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>>> > >> > >>>>>
>>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
>>> > >> > >>>>> your
>>> > >> > >>>>> data.
>>> > >> > >>>>> For example, if you want to select only dates with
00:00 you
>>> > >> > >>>>> can
>>> > >> > >>>>> specify
>>> > >> > >>>>>  <field name="valid_hour">
>>> > >> > >>>>>         <val>00</val>
>>> > >> > >>>>>  </field>
>>> > >> > >>>>>
>>> > >> > >>>>> Tatiana
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020, edward.strobach at noaa.gov
>>> wrote:
>>> > >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
>>> > >> > >>>>> > of a
>>> > >> > >>>>> > separate
>>> > >> > >>>>> > script that pipes dates (and other relevant
information)
>>> > >> > >>>>> > into
>>> > >> > >>>>> > the XML
>>> > >> > >>>>> > file
>>> > >> > >>>>> > used to generate the results.
>>> > >> > >>>>> >
>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
>>> > >> > >>>>> > Affiliate <
>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>> > >> > >>>>> >
>>> > >> > >>>>> > > Thanks Tatiana.  These resources are helpful.
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > One last question.  Simply typing lines like this
in can
>>> > >> > >>>>> > > take
>>> > >> > >>>>> > > forever:
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > <plot_fix>
>>> > >> > >>>>> > >     <field equalize="false" name="fcst_init_beg">
>>> > >> > >>>>> > >         <set name="fcst_init_beg_0">
>>> > >> > >>>>> > >             <val>2019-05-01 12:00:00</val>
>>> > >> > >>>>> > >             <val>2019-05-02 06:00:00</val>
>>> > >> > >>>>> > >             <val>2019-05-02 12:00:00</val>
>>> > >> > >>>>> > >             <val>2019-05-03 06:00:00</val>
>>> > >> > >>>>> > >             <val>2019-05-03 12:00:00</val>
>>> > >> > >>>>> > >
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > Is there a better way?   I see date_range,
>>> > >> > >>>>> > > date_range_list,
>>> > >> > >>>>> > > or
>>> > >> > >>>>> > > date_list.
>>> > >> > >>>>> > > Can this be used instead of typing individual
values?
>>> > >> > >>>>> > > This
>>> > >> > >>>>> > > could
>>> > >> > >>>>> > > prove
>>> > >> > >>>>> > > pretty daunting if one would have to type in 700
entries
>>> > >> > >>>>> > > each
>>> > >> > >>>>> > > time.
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek via
RT
>>> > >> > >>>>> > > <met_help at ucar.edu>
>>> > >> > >>>>> > > wrote:
>>> > >> > >>>>> > >
>>> > >> > >>>>> > >> I found a few problems in your XML.
>>> > >> > >>>>> > >> The database you are using doesn't have data
with
>>> > >> > >>>>> > >> vx_mask
>>> > >> > >>>>> > >> 'abcREGabc'.
>>> > >> > >>>>> > >> That is why METviewer returns an empty dataset
and
>>> > >> > >>>>> > >> can't
>>> > >> > >>>>> > >> create a
>>> > >> > >>>>> > >> plot.
>>> > >> > >>>>> > >> The other problem is how you define fcst_lead.
This
>>> > >> > >>>>> > >> database
>>> > >> > >>>>> > >> has
>>> > >> > >>>>> > >> values
>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since
you
>>> > >> > >>>>> > >> include
>>> > >> > >>>>> > >> all
>>> > >> > >>>>> > >> fcst_leads
>>> > >> > >>>>> > >> this variable can be omitted.
>>> > >> > >>>>> > >> To minimize problems with variables values, I
would
>>> > >> > >>>>> > >> suggest
>>> > >> > >>>>> > >> to
>>> > >> > >>>>> > >> create XML
>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>>> > >> > >>>>> > >> I created a plot using you configurations with
GUI
>>> > >> > >>>>> > >> (without
>>> > >> > >>>>> vx_mack
>>> > >> > >>>>> > >> and
>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You can
load it
>>> > >> > >>>>> > >> back
>>> > >> > >>>>> > >> to
>>> > >> > >>>>> the
>>> > >> > >>>>> > >> page by
>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI). Find
the
>>> > >> > >>>>> > >> plot,
>>> > >> > >>>>> > >> open XML
>>> > >> > >>>>> > >> and
>>> > >> > >>>>> > >> click on "Load to page". After that you can
recreate a
>>> > >> > >>>>> > >> plot
>>> > >> > >>>>> > >> and
>>> > >> > >>>>> edit
>>> > >> > >>>>> > >> the
>>> > >> > >>>>> > >> configurations. You can also download a new XML
and
>>> > >> > >>>>> > >> edit it
>>> > >> > >>>>> > >> for
>>> > >> > >>>>> > >> batch.
>>> > >> > >>>>> > >> I attached the example of load XML to this
message.
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> Description of rds parameters:
>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
METviewer R
>>> > >> > >>>>> > >> scripts
>>> > >> > >>>>> > >> rds_plots - temporary directory where the plots
will be
>>> > >> > >>>>> > >> generated
>>> > >> > >>>>> > >> rds_data - temporary directory where the data
retrieved
>>> > >> > >>>>> > >> from
>>> > >> > >>>>> > >> the
>>> > >> > >>>>> > >> database
>>> > >> > >>>>> > >> and other intermediate files will be stored
>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
scripts for
>>> > >> > >>>>> > >> this
>>> > >> > >>>>> > >> run
>>> > >> > >>>>> will
>>> > >> > >>>>> > >> be
>>> > >> > >>>>> > >> stored
>>> > >> > >>>>> > >> After the execution is done all the files from
>>> > >> > >>>>> > >> temporary
>>> > >> > >>>>> directories
>>> > >> > >>>>> > >> would be removed.
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
>>> > >> > >>>>> > >> There
>>> > >> > >>>>> > >> are
>>> > >> > >>>>> some
>>> > >> > >>>>> > >> instructions on how to crate XML config files:
>>> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> Tatiana
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
>>> > >> > >>>>> > >> wrote:
>>> > >> > >>>>> > >> > Hi,
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > I'm attaching the xml below, but am having
hard time
>>> > >> > >>>>> > >> > following
>>> > >> > >>>>> how
>>> > >> > >>>>> > >> > to
>>> > >> > >>>>> > >> > go
>>> > >> > >>>>> > >> > about the other steps since I have zero
experience on
>>> > >> > >>>>> > >> > how
>>> > >> > >>>>> > >> > to do
>>> > >> > >>>>> > >> > these
>>> > >> > >>>>> > >> > things.
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > #2. All parameters in <folders> section should
NOT be
>>> > >> > >>>>> > >> > changed
>>> > >> > >>>>> and
>>> > >> > >>>>> > >> > will
>>> > >> > >>>>> > >> > be
>>> > >> > >>>>> > >> > replaced by other scripts during the batch
execution.
>>> > >> > >>>>> > >> > The
>>> > >> > >>>>> > >> > data
>>> > >> > >>>>> for
>>> > >> > >>>>> > >> > the
>>> > >> > >>>>> > >> > batch mode is coming from the database and not
from
>>> > >> > >>>>> > >> > files.
>>> > >> > >>>>> > >> > The
>>> > >> > >>>>> > >> > name of
>>> > >> > >>>>> > >> > the
>>> > >> > >>>>> > >> > database is in <database> tag.
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > I understand that nothing should be changed in
the
>>> > >> > >>>>> > >> > folders
>>> > >> > >>>>> > >> > section,
>>> > >> > >>>>> > >> > but
>>> > >> > >>>>> > >> > there is no indication how and where these
variables
>>> > >> > >>>>> > >> > are
>>> > >> > >>>>> > >> > being
>>> > >> > >>>>> > >> > defined.
>>> > >> > >>>>> > >> > Grepping these names of the files available
does not
>>> > >> > >>>>> > >> > reveal
>>> > >> > >>>>> formal
>>> > >> > >>>>> > >> > definitions of these variables.  If I
understood
>>> > >> > >>>>> > >> > exactly
>>> > >> > >>>>> > >> > how
>>> > >> > >>>>> these
>>> > >> > >>>>> > >> > were
>>> > >> > >>>>> > >> > being defined, then I would be able connect
this
>>> > >> > >>>>> > >> > better.
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
files. They
>>> > >> > >>>>> > >> > use
>>> > >> > >>>>> METviewer
>>> > >> > >>>>> > >> > load
>>> > >> > >>>>> > >> > mode to load these files to the specific
database. In
>>> > >> > >>>>> > >> > the
>>> > >> > >>>>> > >> > config
>>> > >> > >>>>> > >> > file
>>> > >> > >>>>> > >> > for
>>> > >> > >>>>> > >> > this process they also can specify the
database
>>> > >> > >>>>> > >> > group. For
>>> > >> > >>>>> example
>>> > >> > >>>>> > >> > database
>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>> > >> > >>>>> > >> > After the data has been loaded to the database
it is
>>> > >> > >>>>> > >> > visible and
>>> > >> > >>>>> > >> > available
>>> > >> > >>>>> > >> > in the GUI and also can be used by the batch
mode.
>>> > >> > >>>>> > >> > You can always add more data to the database
using
>>> > >> > >>>>> > >> > the
>>> > >> > >>>>> > >> > same load
>>> > >> > >>>>> > >> > mode.
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > Is there a guide for me to understand the
process
>>> > >> > >>>>> > >> > that you
>>> > >> > >>>>> > >> > are
>>> > >> > >>>>> > >> > referring to?
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana Burek
via RT
>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>> > >> > >>>>> > >> > wrote:
>>> > >> > >>>>> > >> >
>>> > >> > >>>>> > >> > > Hi Edward,
>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA servers
and
>>> > >> > >>>>> > >> > > can't
>>> > >> > >>>>> > >> > > view your
>>> > >> > >>>>> > >> > > XML
>>> > >> > >>>>> > >> > > for the
>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this issue
and I
>>> > >> > >>>>> > >> > > will
>>> > >> > >>>>> > >> > > take a
>>> > >> > >>>>> > >> > > look.
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
should NOT
>>> > >> > >>>>> > >> > > be
>>> > >> > >>>>> > >> > > changed
>>> > >> > >>>>> > >> > > and
>>> > >> > >>>>> > >> > > will be
>>> > >> > >>>>> > >> > > replaced by other scripts during the batch
>>> > >> > >>>>> > >> > > execution.
>>> > >> > >>>>> > >> > > The data
>>> > >> > >>>>> > >> > > for
>>> > >> > >>>>> > >> > > the
>>> > >> > >>>>> > >> > > batch mode is coming from the database and
not from
>>> > >> > >>>>> > >> > > files. The
>>> > >> > >>>>> > >> > > name
>>> > >> > >>>>> > >> > > of the
>>> > >> > >>>>> > >> > > database is in <database> tag.
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
>>> > >> > >>>>> > >> > > They use
>>> > >> > >>>>> > >> > > METviewer
>>> > >> > >>>>> > >> > > load
>>> > >> > >>>>> > >> > > mode to load these files to the specific
database.
>>> > >> > >>>>> > >> > > In
>>> > >> > >>>>> > >> > > the
>>> > >> > >>>>> config
>>> > >> > >>>>> > >> > > file
>>> > >> > >>>>> > >> > > for
>>> > >> > >>>>> > >> > > this process they also can specify the
database
>>> > >> > >>>>> > >> > > group.
>>> > >> > >>>>> > >> > > For
>>> > >> > >>>>> > >> > > example
>>> > >> > >>>>> > >> > > database
>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>> > >> > >>>>> > >> > > After the data has been loaded to the
database it
>>> > >> > >>>>> > >> > > is
>>> > >> > >>>>> > >> > > visible
>>> > >> > >>>>> and
>>> > >> > >>>>> > >> > > available
>>> > >> > >>>>> > >> > > in the GUI and also can be used by the batch
mode.
>>> > >> > >>>>> > >> > > You can always add more data to the database
using
>>> > >> > >>>>> > >> > > the
>>> > >> > >>>>> > >> > > same
>>> > >> > >>>>> load
>>> > >> > >>>>> > >> > > mode.
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > > Tatiana
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>> > >> > >>>>> > >> > > wrote:
>>> > >> > >>>>> > >> > > > Good morning,
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
effort to
>>> > >> > >>>>> > >> > > > understand
>>> > >> > >>>>> > >> > > > metviewer,
>>> > >> > >>>>> > >> > > > test
>>> > >> > >>>>> > >> > > > different cases using the history tab,
changing
>>> > >> > >>>>> > >> > > > information
>>> > >> > >>>>> to
>>> > >> > >>>>> > >> > > > see
>>> > >> > >>>>> > >> > > > what
>>> > >> > >>>>> > >> > > > works and what does not, and make
connections
>>> > >> > >>>>> > >> > > > between
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> GUI
>>> > >> > >>>>> > >> > > > settings
>>> > >> > >>>>> > >> > > > and
>>> > >> > >>>>> > >> > > > the XML file.  Although my comfort level
with
>>> > >> > >>>>> > >> > > > metviewer has
>>> > >> > >>>>> > >> > > > significantly
>>> > >> > >>>>> > >> > > > increased during this time, there are some
>>> > >> > >>>>> > >> > > > remaining
>>> > >> > >>>>> > >> > > > issues
>>> > >> > >>>>> > >> > > > that
>>> > >> > >>>>> > >> > > > I'm
>>> > >> > >>>>> > >> > > > hoping
>>> > >> > >>>>> > >> > > > to clear up.
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > 1.  Problem changing XML file and running
with
>>> > >> > >>>>> > >> > > > different
>>> > >> > >>>>> > >> > > > statistic
>>> > >> > >>>>> > >> > > > and
>>> > >> > >>>>> > >> > > > plot
>>> > >> > >>>>> > >> > > > type:
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Ho-Chun Huang, a colleague at EMC,
provided me
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> > >> > > > working
>>> > >> > >>>>> > >> > > > directory
>>> > >> > >>>>> > >> > > > to get
>>> > >> > >>>>> > >> > > > started which I copied over:
>>> > >> > >>>>> > >> > > >
>>> > >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > He has a test.sh script that I adjusted
and used
>>> > >> > >>>>> > >> > > > to
>>> > >> > >>>>> > >> > > > run his
>>> > >> > >>>>> > >> > > > example.
>>> > >> > >>>>> > >> > > > running of the script "bash test.sh"
produced a
>>> > >> > >>>>> > >> > > > CSI
>>> > >> > >>>>> > >> > > > plot
>>> > >> > >>>>> that
>>> > >> > >>>>> > >> > > > was
>>> > >> > >>>>> > >> > > > stored
>>> > >> > >>>>> > >> > > > here:
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >>
>>> > >> > >>>>>
>>> > >>
>>>
/gpfs/dell2/stmp/Edward.Strobach/aws_figure/CONUS_CMAQV5PARA8_AOD_CSI_DAY2_12Z_HIGH_202005.png
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Running the script produced the expected
result.
>>> > >> > >>>>> > >> > > > The
>>> > >> > >>>>> > >> > > > next
>>> > >> > >>>>> > >> > > > natural
>>> > >> > >>>>> > >> > > > step was
>>> > >> > >>>>> > >> > > > to copy hochum_xml.template and create a
new xml
>>> > >> > >>>>> > >> > > > file
>>> > >> > >>>>> > >> > > > called
>>> > >> > >>>>> > >> > > > Strobach_xml.template in order to create a
time
>>> > >> > >>>>> > >> > > > series
>>> > >> > >>>>> > >> > > > where
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> > >> > > > independent time is fcst_valid_beg instead
of
>>> > >> > >>>>> > >> > > > obs_thresh.
>>> > >> > >>>>> > >> > > > Accordingly,
>>> > >> > >>>>> > >> > > > I
>>> > >> > >>>>> > >> > > > made changes to the fixed fields (removing
>>> > >> > >>>>> > >> > > > fcst_valid_beg)
>>> > >> > >>>>> and
>>> > >> > >>>>> > >> > > > changed
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> > >> > > > statistic from CSI to FMEAN (FBAR was also
>>> > >> > >>>>> > >> > > > tried).
>>> > >> > >>>>> > >> > > > The
>>> > >> > >>>>> error
>>> > >> > >>>>> > >> > > > that
>>> > >> > >>>>> > >> > > > resulted
>>> > >> > >>>>> > >> > > > is as follows:
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > AND   BINARY ld.fcst_valid_beg IN ('2020-
05-01
>>> > >> > >>>>> > >> > > > 13:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 01
>>> > >> > >>>>> > >> > > > 14:00:00', '2020-05-01 15:00:00', '2020-
05-01
>>> > >> > >>>>> > >> > > > 16:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 01
>>> > >> > >>>>> > >> > > > 17:00:00', '2020-05-01 18:00:00', '2020-
05-01
>>> > >> > >>>>> > >> > > > 19:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 01
>>> > >> > >>>>> > >> > > > 20:00:00', '2020-05-01 21:00:00', '2020-
05-01
>>> > >> > >>>>> > >> > > > 22:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 01
>>> > >> > >>>>> > >> > > > 23:00:00', '2020-05-02 00:00:00', '2020-
05-02
>>> > >> > >>>>> > >> > > > 01:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 02
>>> > >> > >>>>> > >> > > > 02:00:00', '2020-05-02 03:00:00', '2020-
05-02
>>> > >> > >>>>> > >> > > > 04:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 02
>>> > >> > >>>>> > >> > > > 05:00:00', '2020-05-02 06:00:00', '2020-
05-02
>>> > >> > >>>>> > >> > > > 07:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 02
>>> > >> > >>>>> > >> > > > 08:00:00', '2020-05-02 09:00:00', '2020-
05-02
>>> > >> > >>>>> > >> > > > 10:00:00',
>>> > >> > >>>>> > >> > > > '2020-05-
>>> > >> > >>>>> > >> > > > 02
>>> > >> > >>>>> > >> > > > 11:00:00', '2020-05-02 12:00:00')
>>> > >> > >>>>> > >> > > > AND   BINARY h.fcst_var = 'AOTK'
>>> > >> > >>>>> > >> > > > AND ld.stat_header_id = h.stat_header_id
>>> > >> > >>>>> > >> > > > AND ld.fmean != -9999;
>>> > >> > >>>>> > >> > > > No data was returned from database
>>> > >> > >>>>> > >> > > > 2020-06-09
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >>
>>> > >> > >>>>>
>>> > >>
>>>
edu.ucar.metviewer.db.mysql.MysqlAppDatabaseManager.printFormattedTable(Unknown
>>> > >> > >>>>> > >> > > > Source) -1 printFormattedTable -   **
ERROR:
>>> > >> > >>>>> > >> > > > Caught
>>> > >> > >>>>> > >> > > > class
>>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
in
>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet
>>> > >> > >>>>> > >> > > > res): result set contained no data
>>> > >> > >>>>> > >> > > > **  ERROR: Caught class
>>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>> > >> > >>>>> > >> > > > in
>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
>>> > >> > >>>>> > >> > > > contained no
>>> > >> > >>>>> > >> > > > data
>>> > >> > >>>>> > >> > > > **  ERROR: Caught class
>>> > >> > >>>>> > >> > > > edu.ucar.metviewer.EmptyResultSetException
>>> > >> > >>>>> > >> > > > in
>>> > >> > >>>>> > >> > > > printFormattedTable(ResultSet res): result
set
>>> > >> > >>>>> > >> > > > contained no
>>> > >> > >>>>> > >> > > > data
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Apparently there was no data during this
time.  I
>>> > >> > >>>>> > >> > > > tried
>>> > >> > >>>>> > >> > > > changing
>>> > >> > >>>>> > >> > > > the
>>> > >> > >>>>> > >> > > > range
>>> > >> > >>>>> > >> > > > of valid times and changed the valid time
to
>>> > >> > >>>>> > >> > > > different
>>> > >> > >>>>> > >> > > > days
>>> > >> > >>>>> > >> > > > just in
>>> > >> > >>>>> > >> > > > case
>>> > >> > >>>>> > >> > > > data was missing.  The issue was the same.
To
>>> > >> > >>>>> > >> > > > confirm
>>> > >> > >>>>> > >> > > > my
>>> > >> > >>>>> > >> > > > logic, I
>>> > >> > >>>>> > >> > > > reproduced my attempted approach on
metviewer,
>>> > >> > >>>>> > >> > > > which I
>>> > >> > >>>>> > >> > > > found
>>> > >> > >>>>> > >> > > > to
>>> > >> > >>>>> > >> > > > work.
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > 2.  Running in batch mode
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > I think I lack an understanding of how to
set up
>>> > >> > >>>>> > >> > > > batch
>>> > >> > >>>>> > >> > > > mode
>>> > >> > >>>>> > >> > > > because
>>> > >> > >>>>> > >> > > > I
>>> > >> > >>>>> > >> > > > don't
>>> > >> > >>>>> > >> > > > see where these definitions are being
explicitly
>>> > >> > >>>>> > >> > > > defined:
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > <plot_spec>
>>> > >> > >>>>> > >> > > >     <connection>
>>> > >> > >>>>> > >> > > >         <host>rds_host:3306</host>
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>>
<database>mv_cmaqv5para8_aod_grid2grid_metplus</database>
>>> > >> > >>>>> > >> > > > <user>rds_user</user>
>>> > >> > >>>>> > >> > > > <password>rds_pwd</password>
>>> > >> > >>>>> > >> > > > </connection>
>>> > >> > >>>>> > >> > > > <rscript>Rscript</rscript>
>>> > >> > >>>>> > >> > > > <folders>
>>> > >> > >>>>> > >> > > > <r_tmpl>rds_R_tmpl</r_tmpl>
>>> > >> > >>>>> > >> > > > <r_work>rds_R_work</r_work>
>>> > >> > >>>>> > >> > > > <plots>rds_plots</plots>
>>> > >> > >>>>> > >> > > > <data>rds_data</data>
>>> > >> > >>>>> > >> > > > <scripts>rds_scripts</scripts>
>>> > >> > >>>>> > >> > > > </folders>
>>> > >> > >>>>> > >> > > > <plot>
>>> > >> > >>>>> > >> > > > <template>series_plot.R_tmpl</template>
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > I see the <folders> section, but grepping
all
>>> > >> > >>>>> > >> > > > available
>>> > >> > >>>>> files
>>> > >> > >>>>> > >> > > > does
>>> > >> > >>>>> > >> > > > not
>>> > >> > >>>>> > >> > > > reveal how rds_R_work is being defined, as
well
>>> > >> > >>>>> > >> > > > as
>>> > >> > >>>>> > >> > > > other
>>> > >> > >>>>> > >> > > > similarly
>>> > >> > >>>>> > >> > > > defined
>>> > >> > >>>>> > >> > > > variables.  I think if I know how these
are being
>>> > >> > >>>>> > >> > > > defined,
>>> > >> > >>>>> > >> > > > then
>>> > >> > >>>>> > >> > > > I'll
>>> > >> > >>>>> > >> > > > be
>>> > >> > >>>>> > >> > > > able to pipe in my information and process
the
>>> > >> > >>>>> > >> > > > data
>>> > >> > >>>>> > >> > > > using
>>> > >> > >>>>> met
>>> > >> > >>>>> > >> > > > and
>>> > >> > >>>>> > >> > > > produce
>>> > >> > >>>>> > >> > > > results in batch mode.  Am I right on
that?
>>> > >> > >>>>> > >> > > > Also, if
>>> > >> > >>>>> > >> > > > I want
>>> > >> > >>>>> > >> > > > to use
>>> > >> > >>>>> > >> > > > obs for
>>> > >> > >>>>> > >> > > > verification, do I add another line in the
XML
>>> > >> > >>>>> > >> > > > file
>>> > >> > >>>>> > >> > > > that
>>> > >> > >>>>> > >> > > > designates
>>> > >> > >>>>> > >> > > > file
>>> > >> > >>>>> > >> > > > location of obs?  I assume that other
steps have
>>> > >> > >>>>> > >> > > > to be
>>> > >> > >>>>> > >> > > > run
>>> > >> > >>>>> > >> > > > before
>>> > >> > >>>>> > >> > > > all
>>> > >> > >>>>> > >> > > > this.  If data is in prepbufr form, for
example,
>>> > >> > >>>>> > >> > > > I
>>> > >> > >>>>> > >> > > > would
>>> > >> > >>>>> have
>>> > >> > >>>>> > >> > > > to
>>> > >> > >>>>> > >> > > > separately
>>> > >> > >>>>> > >> > > > run PB2NC.  If I want grid2obs, then I
would want
>>> > >> > >>>>> > >> > > > to
>>> > >> > >>>>> > >> > > > do that
>>> > >> > >>>>> > >> > > > as
>>> > >> > >>>>> > >> > > > well.
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > 3. Communicating results to metviewer
online
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > Lastly, I see a lot of people/projects
listed
>>> > >> > >>>>> > >> > > > here -
>>> > >> > >>>>> > >> > > > see
>>> > >> > >>>>> > >> > > > attachment
>>> > >> > >>>>> > >> > > >
>>> > >> > >>>>> > >> > > > How do I get the results I generate to be
>>> > >> > >>>>> > >> > > > uploaded to
>>> > >> > >>>>> > >> > > > this
>>> > >> > >>>>> > >> > > > location?
>>> > >> > >>>>> > >> > > > It's
>>> > >> > >>>>> > >> > > > not clear how others are setting this up.
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >> > >
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >>
>>> > >> > >>>>> > >> rds_host:3306
mv_cmaqv5para8_aod_grid2grid_metplus
>>> > >> > >>>>> > >> rds_user
>>> > >> > >>>>> rds_pwd
>>> > >> > >>>>> > >> aurora Rscript rds_R_tmpl rds_R_work rds_plots
rds_data
>>> > >> > >>>>> rds_scripts
>>> > >> > >>>>> > >> series_plot.R_tmpl FBAR CMAQAODPARA8HIGH
>>> > >> > >>>>> > >> CMAQAODPRODHIGH
>>> > >> > >>>>> 2020-05-12
>>> > >> > >>>>> > >> 12:00:00 2020-05-13 12:00:00 2020-05-12 19:00:00
2020-
>>> > >> > >>>>> > >> 05-12
>>> > >> > >>>>> 20:00:00
>>> > >> > >>>>> > >> 2020-05-12
>>> > >> > >>>>> > >> 21:00:00 2020-05-12 22:00:00 2020-05-12 23:00:00
2020-
>>> > >> > >>>>> > >> 05-13
>>> > >> > >>>>> 10:00:00
>>> > >> > >>>>> > >> 2020-05-13
>>> > >> > >>>>> > >> 11:00:00 2020-05-13 12:00:00 2020-05-13 13:00:00
2020-
>>> > >> > >>>>> > >> 05-13
>>> > >> > >>>>> 14:00:00
>>> > >> > >>>>> > >> 2020-05-13
>>> > >> > >>>>> > >> 15:00:00 2020-05-13 16:00:00 2020-05-13 17:00:00
2020-
>>> > >> > >>>>> > >> 05-13
>>> > >> > >>>>> 18:00:00
>>> > >> > >>>>> > >> 2020-05-13
>>> > >> > >>>>> > >> 19:00:00 2020-05-13 20:00:00 2020-05-13 21:00:00
2020-
>>> > >> > >>>>> > >> 05-13
>>> > >> > >>>>> 22:00:00
>>> > >> > >>>>> > >> 2020-05-13
>>> > >> > >>>>> > >> 23:00:00 2020-05-14 10:00:00 2020-05-14 11:00:00
2020-
>>> > >> > >>>>> > >> 05-14
>>> > >> > >>>>> 12:00:00
>>> > >> > >>>>> > >> true
>>> > >> > >>>>> > >> median plot_20200612_173626.data
>>> > >> > >>>>> > >> plot_20200612_173626.png
>>> > >> > >>>>> > >> plot_20200612_173626.R test x_label test y_label
>>> > >> > >>>>> > >> false list() list() false false false false
false false
>>> > >> > >>>>> > >> true
>>> > >> > >>>>> > >> false
>>> > >> > >>>>> > >> false
>>> > >> > >>>>> > >> false false false true png16m 8.5 11 72 in
c(8,4,5,4)
>>> > >> > >>>>> > >> c(1,1,0) 1 2
>>> > >> > >>>>> > >> 1.4 -2
>>> > >> > >>>>> > >> 0.5 1 -0.75 0.5 0 1 1 1 2 0.5 1 0.5 0.5 1 1 1 -2
0.5 3
>>> > >> > >>>>> > >> #cccccc 1
>>> > >> > >>>>> > >> listX 1
>>> > >> > >>>>> > >> 1 0.5 0.8 0.8 -0.5 0.5 1 0.5 0.5 1 1 1 0.5 o
c(0, -.25)
>>> > >> > >>>>> > >> 3
>>> > >> > >>>>> > >> 0.8 1
>>> > >> > >>>>> > >> #333333
>>> > >> > >>>>> > >> 0.8 3 0 0.05 c("none","none") c(FALSE,FALSE)
>>> > >> > >>>>> > >> c(TRUE,TRUE)
>>> > >> > >>>>> > >> c("#ff0000FF","#8000ffFF") c(20,20) c("b","b")
c(1,1)
>>> > >> > >>>>> > >> c(1,1)
>>> > >> > >>>>> c(1,1)
>>> > >> > >>>>> > >> c(1,2) c("","") c() c() 0.04 c()
>>> > >> > >>>>> > >
>>> > >> > >>>>> > >
>>> > >> > >>>>> > >
>>> > >> > >>>>> > > --
>>> > >> > >>>>> > > Edward Strobach
>>> > >> > >>>>> > > EMC/NCEP/NWS/
>>> > >> > >>>>> > > IMSG Contractor
>>> > >> > >>>>> > > Cubicle#: 2029
>>> > >> > >>>>> > > 301-683-3717
>>> > >> > >>>>> > >
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>
>>> > >> > >>>> --
>>> > >> > >>>> Edward Strobach
>>> > >> > >>>> EMC/NCEP/NWS/
>>> > >> > >>>> IMSG Contractor
>>> > >> > >>>> Cubicle#: 2029
>>> > >> > >>>> 301-683-3717
>>> > >> > >>>>
>>> > >> > >>>
>>> > >> > >>>
>>> > >> > >>> --
>>> > >> > >>> Edward Strobach
>>> > >> > >>> EMC/NCEP/NWS/
>>> > >> > >>> IMSG Contractor
>>> > >> > >>> Cubicle#: 2029
>>> > >> > >>> 301-683-3717
>>> > >> > >>>
>>> > >> > >>
>>> > >> > >>
>>> > >> > >> --
>>> > >> > >> Edward Strobach
>>> > >> > >> EMC/NCEP/NWS/
>>> > >> > >> IMSG Contractor
>>> > >> > >> Cubicle#: 2029
>>> > >> > >> 301-683-3717
>>> > >> > >>
>>> > >> > >
>>> > >> > >
>>> > >> > > --
>>> > >> > > Edward Strobach
>>> > >> > > EMC/NCEP/NWS/
>>> > >> > > IMSG Contractor
>>> > >> > > Cubicle#: 2029
>>> > >> > > 301-683-3717
>>> > >> > >
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >
>>> > > --
>>> > > Edward Strobach
>>> > > EMC/NCEP/NWS/
>>> > > IMSG Contractor
>>> > > Cubicle#: 2029
>>> > > 301-683-3717
>>> > >
>>>
>>>
>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Jeffery T. McQueen
Meteorologist
National Center for Weather and Climate Prediction
Room 2095
5830 University Research Court
College Park, MD 20741
301-683-3736
Fax: 301-683-3703

------------------------------------------------
Subject: setting up batch mode; communicating results with metviewer online; other remaining issues
From: Edward Strobach - NOAA Affiliate
Time: Wed Jun 17 12:28:11 2020

Just adding more detail to elaborate my concerns...

Not too long ago I was given a test set-up by Ho-Chun to generate a
plot
for aerosol optical depth.  The result was successful and produced a
CSI
plot for AOD, comparing the production run and experimental run.
Although
I created a database, I do not seem to have a group where I can access
specific databases.  I recently created a load_xml file, which should
have
loaded the data based on what I specified into the group.  It seemed
successful based on the log output that was generated.  Additionally,
all
information that I specified in this step was create in the load_*xml.

The next step is to generate plots since the data should be loaded.
Here's
what happens when I run Ho-Chun's - the successful case

HO-CHUN:



















*+ dynamic_figure_name=Test.png+ sed -e 's!abcREGabc!CONUS!'
hochun_xml.template+ bash
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
 ----input file: plot.xmlprocessing 1 jobs:Running 1 plots# # # # # #
# # #
# # # # # # # # # # # # # # # # # # # # # # # # # # #SELECT  model,
ld.fcst_init_beg,  ld.fcst_valid_beg,  ld.fcst_lead,*

This is early on in the script.

ED_STROBACH:





















*+ for i in '"${regs[@]}"'+ dynamic_figure_name=Test_Ed.png+ sed -e
's!abcREGabc!CONUS!' Strobach_Time_Series_xml.template+ bash
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts/mv_batch_on_aws.sh
edward.strobach /gpfs/dell2/stmp/Edward.Strobach/aws_figure
plot.xmlCALLING: scp plot.xml edward.strobach at 205.156.8.85:~----
MVBatch
 ----input file: plot.xmlprocessing 1 jobs:Jun 17, 2020 6:18:20 PM
org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to
create
initial connections of pool.java.sql.SQLException: Communications link
failure with primary. No active connection found for master.        at
org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:538)
      at
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:336)
      at
org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:173)
      at
org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:116)
      at
org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:542)
at
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
      at org.mariadb.jdbc.Driver.connect(Driver.java:92)*

My interpretation of this is that it fails to bridge a connection to
the
group or database; therefore, it's unable to proceed with running the
XML
file to generate the plots based on the specified content.  If I move
past
this step, then I'm golden.

On Wed, Jun 17, 2020 at 1:34 PM Jeff Mcqueen - NOAA Federal <
jeff.mcqueen at noaa.gov> wrote:

> Would it be ok to keep perry in the loop
>
> On Wed, Jun 17, 2020 at 12:18 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> you're right.  I didn't have the storage problem this time.  I'm
not sure
>> what changed.  All I know is that I have now created my load xml
which now
>> should allow me to move on to the next step of plotting
>>
>> On Wed, Jun 17, 2020 at 12:05 PM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>>> I'm not following you.  How do I now have enough space?  What
changed?
>>>
>>> On Wed, Jun 17, 2020 at 12:00 PM Tatiana Burek via RT
<met_help at ucar.edu>
>>> wrote:
>>>
>>>> Ed, now you should have enough space for the data loading.
>>>>
>>>> Tatiana
>>>>
>>>> On Tue Jun 16 18:41:10 2020, edward.strobach at noaa.gov wrote:
>>>> > I was basically able to build my xml file but it says I have no
>>>> > available
>>>> > space.  I'm not sure how that can be since I just started
adding data
>>>> > to my
>>>> > account
>>>> >
>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061322.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/AQ_2020061323.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061300.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061301.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061302.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061303.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061304.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061305.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061306.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061307.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061308.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061309.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061310.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061311.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061312.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061313.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061314.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061315.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061316.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061317.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061318.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061319.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061320.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061321.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061322.stat: No
space left
>>>> > on
>>>> > device
>>>> > scp: //data/mv_data/edward.strobach/PM_2020061323.stat: No
space left
>>>> > on
>>>> > device
>>>> > ERROR: Command returned with non-zero status (1): scp -r ./*
>>>> > edward.strobach at 205.156.8.85://data/mv_data/edward.strobach
>>>> > 4) Check METviewer AWS database list using mv_db_size_on_aws.sh
>>>> > TABLE_SCHEMA                              USER
SIZE_MB
>>>> > mv_met_g2o_gefs                           binbin.zhou
140027.31
>>>> > mv_met_system_hrefv2_v3                   binbin.zhou
71986.11
>>>> >
>>>> > On Tue, Jun 16, 2020 at 5:40 PM Edward Strobach - NOAA
Affiliate <
>>>> > edward.strobach at noaa.gov> wrote:
>>>> >
>>>> > > Thanks Tatiana.  Yes, I've created a database which seems
different
>>>> > > from
>>>> > > mv_group. How can one create their own group so that I can
store and
>>>> > > access
>>>> > > the database.  The database exists but cannot be viewed on
the
>>>> > > metviewer
>>>> > > GUI.  The list starts with EMC HREFv3 and ends with vhagerty.
I
>>>> > > named my
>>>> > > database Ed_Strobach, but that was really intended for the
group
>>>> > > name.
>>>> > >
>>>> > > On Tue, Jun 16, 2020 at 4:45 PM Tatiana Burek via RT
>>>> > > <met_help at ucar.edu>
>>>> > > wrote:
>>>> > >
>>>> > >> We don't have database and METviewer running on wcoss.
>>>> > >> Instead, users use METviewer-wcoss scripts that help to
connect to
>>>> > >> and
>>>> > >> use METviewer that is running on AWS. That is why XML
connection
>>>> and
>>>> > >> folder
>>>> > >> sections look somewhat different than once from the Docs
page.
>>>> > >> To create a database on AWS you need to run
mv_create_db_on_aws.sh
>>>> > >> mv_create_db_on_aws.sh <user_name> <database_name>
>>>> > >> For data loading :
>>>> > >> mv_load_to_aws.sh <user_name> <base_dir> <xml_file>
>>>> > >> And for batch:
>>>> > >> mv_batch_on_aws.sh <user_name> <plots_dir> <xml_file>
>>>> > >>
>>>> > >> These scripts copy XML and data files if needed to AWS,
adjust data
>>>> > >> path
>>>> > >> and users credentials, run METviewer and copy results back
to
>>>> wcoss.
>>>> > >>
>>>> > >> Tatiana
>>>> > >> On Tue Jun 16 08:26:32 2020, edward.strobach at noaa.gov wrote:
>>>> > >> > lastly, it does seem thatmysql -u[db_username]
-p[db_password]
>>>> > >> > -e'create
>>>> > >> > database [db_name]
>>>> > >> >
>>>> > >> > would need to be used to create a database.  I assume that
an sql
>>>> > >> > module is
>>>> > >> > available on wcoss; I know you don't have access to those
>>>> machines
>>>> > >> > Tatiana.  If I create my own database with my own name,
then I
>>>> > >> > should
>>>> > >> > be
>>>> > >> > able to proceed with project specific tasks under that
tab.
>>>> > >> > Something
>>>> > >> > like
>>>> > >> > mv_EdwardStrobach.
>>>> > >> >
>>>> > >> > I can then create mv_cmaq_prod_v_para which I add data to
using
>>>> > >> > load_spec.
>>>> > >> > Once that step is done, then I can run plot_spec.  That's
my
>>>> > >> > takeaway
>>>> > >> > currently
>>>> > >> >
>>>> > >> > On Tue, Jun 16, 2020 at 9:34 AM Edward Strobach - NOAA
Affiliate
>>>> <
>>>> > >> > edward.strobach at noaa.gov> wrote:
>>>> > >> >
>>>> > >> > > Actually, it appears that I need to run the database
loading
>>>> > >> > > module
>>>> > >> > > first
>>>> > >> > > if I want to change/add a data source.  From there I can
run
>>>> the
>>>> > >> > > plotting
>>>> > >> > > batch module.  Is that right?   So if I run the database
>>>> loading
>>>> > >> > > module,
>>>> > >> > > then I can specify my dates and other related info
specific to
>>>> > >> > > the
>>>> > >> > > dataset.  If I create my database with a unique name,
then it
>>>> > >> > > should
>>>> > >> > > be
>>>> > >> > > recognized on the metviewer gui, presumably.
>>>> > >> > >
>>>> > >> > > On Tue, Jun 16, 2020 at 9:12 AM Edward Strobach - NOAA
>>>> Affiliate
>>>> > >> > > <
>>>> > >> > > edward.strobach at noaa.gov> wrote:
>>>> > >> > >
>>>> > >> > >> It seems that those options must be treated separately
outside
>>>> > >> > >> of
>>>> > >> > >> plot_spec.  Is it acceptable to create both a load_spec
and
>>>> > >> > >> plot_spec in
>>>> > >> > >> the same xml file?  Would the definitions in load_spec
be able
>>>> > >> > >> to
>>>> > >> > >> carry
>>>> > >> > >> over into plot_spec?  It appears that data loading and
>>>> plotting
>>>> > >> > >> are
>>>> > >> > >> treated
>>>> > >> > >> separately.
>>>> > >> > >>
>>>> > >> > >> On Mon, Jun 15, 2020 at 5:46 PM Edward Strobach - NOAA
>>>> > >> > >> Affiliate <
>>>> > >> > >> edward.strobach at noaa.gov> wrote:
>>>> > >> > >>
>>>> > >> > >>> I do plan to use date_range; however, in an example
provided
>>>> > >> > >>> by the
>>>> > >> > >>> link
>>>> > >> > >>> I see this:
>>>> > >> > >>>
>>>> > >> > >>> <load_spec>
>>>> > >> > >>>   <connection>
>>>> > >> > >>>     <host>kemosabe:3306</host>
>>>> > >> > >>>     <database>metvdb_hwt</database>
>>>> > >> > >>>     <user>pgoldenb</user>
>>>> > >> > >>>     <password>pgoldenb</password>
>>>> > >> > >>>   </connection>
>>>> > >> > >>>
>>>> > >> > >>> <date_list name="folder_dates">
>>>> > >> > >>>   <start>2010051914V</start>
>>>> > >> > >>>   <end><date_offset><day_offset>0</day_offset><hour>6
>>>> > >> > >>> </hour></date_offset></end>
>>>> > >> > >>>   <inc>3600</inc>
>>>> > >> > >>>   <format>yyyyMMddHH'V'</format>
>>>> > >> > >>> </date_list>
>>>> > >> > >>>
>>>> > >> > >>> <met_version>V3.0</met_version>
>>>> > >> > >>>
>>>> > >> > >>> <verbose>false</verbose>
>>>> > >> > >>> <insert_size>1</insert_size>
>>>> > >> > >>> <mode_header_db_check>true</mode_header_db_check>
>>>> > >> > >>> <drop_indexes>false</drop_indexes>
>>>> > >> > >>> <apply_indexes>true</apply_indexes>
>>>> > >> > >>> <group>Group name</group>
>>>> > >> > >>> <load_stat>true</load_stat>
>>>> > >> > >>> <load_mode>true</load_mode>
>>>> > >> > >>> <load_mtd>true</load_mtd>
>>>> > >> > >>> <load_mpr>false</load_mpr>
>>>> > >> > >>>
>>>> > >> > >>>
>>>>
<folder_tmpl>/d1/data/{model}/{vx_mask}/{valid_time}</folder_tmpl>
>>>> > >> > >>> <load_val>
>>>> > >> > >>>   <field name="model">
>>>> > >> > >>>     <val>arw</val>
>>>> > >> > >>>     <val>nmm</val>
>>>> > >> > >>>   </field>
>>>> > >> > >>>
>>>> > >> > >>> <field name="valid_time">
>>>> > >> > >>>   <date_list name="folder_dates"/>
>>>> > >> > >>> </field>
>>>> > >> > >>>
>>>> > >> > >>> <field name="vx_mask">
>>>> > >> > >>>   <val>FULL</val>
>>>> > >> > >>>   <val>SWC</val>
>>>> > >> > >>> </field>
>>>> > >> > >>> </load_val>
>>>> > >> > >>> </load_spec>
>>>> > >> > >>>
>>>> > >> > >>> Are similar settings allowed with <plot_spec>?  I see
>>>> > >> > >>> <connection>,
>>>> > >> > >>> which seems carried over.  I'm looking to specify my
>>>> > >> > >>> folder_tmpl in
>>>> > >> > >>> <plot_spec>, which according to Tara, should be
possible.
>>>> > >> > >>> Since my
>>>> > >> > >>> directory structure looks like this:
>>>> > >> > >>>
>>>> > >> > >>>
>>>> > >> > >>>
>>>> > >>
>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/prod/stat/aqm/20200612
>>>> > >> > >>>
>>>> > >> > >>> I would do something like this I would think:
>>>> > >> > >>>
>>>> > >> > >>> .......
>>>> > >> > >>>
>>>> > >> > >>> <date_list name="folder_dates">
>>>> > >> > >>>   <start>20200612/start>
>>>> > >> > >>>   <end><date_offset><day_offset>0</day_offset></end>
>>>> > >> > >>>   <inc>86400</inc>
>>>> > >> > >>>   <format>yyyyMMdd</format>
>>>> > >> > >>> </date_list>
>>>> > >> > >>>
>>>> > >> > >>> <folder_tmpl>
>>>> > >> > >>>
>>>> > >>
>>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/{model}/{stat}/aqm/{valid_time}
>>>> > >> > >>> </folder_tmpl>
>>>> > >> > >>> <load_val>
>>>> > >> > >>>   <field name="model">
>>>> > >> > >>>     <val>prod</val>
>>>> > >> > >>>   </field>
>>>> > >> > >>>
>>>> > >> > >>> <field name="valid_time">
>>>> > >> > >>>   <date_list name="folder_dates"/>
>>>> > >> > >>> </field>
>>>> > >> > >>>
>>>> > >> > >>> ......
>>>> > >> > >>>
>>>> > >> > >>> On Mon, Jun 15, 2020 at 4:28 PM Edward Strobach - NOAA
>>>> > >> > >>> Affiliate <
>>>> > >> > >>> edward.strobach at noaa.gov> wrote:
>>>> > >> > >>>
>>>> > >> > >>>> Thank you.  This is a very helpful example
>>>> > >> > >>>>
>>>> > >> > >>>> On Mon, Jun 15, 2020 at 1:56 PM Tatiana Burek via RT
>>>> > >> > >>>> <met_help at ucar.edu>
>>>> > >> > >>>> wrote:
>>>> > >> > >>>>
>>>> > >> > >>>>> Hi,
>>>> > >> > >>>>> Here is some examples how to use <date_range> and
>>>> > >> > >>>>> <date_range_list> in
>>>> > >> > >>>>> batch XML.
>>>> > >> > >>>>> First, you need to defined your range. You can have
more
>>>> > >> > >>>>> then one
>>>> > >> > >>>>> <date_range>:
>>>> > >> > >>>>> <date_range name="may">
>>>> > >> > >>>>>         <start>2019-05-01 00:00:00</start>
>>>> > >> > >>>>>         <end>2019-05-31 23:59:59</end>
>>>> > >> > >>>>> </date_range>
>>>> > >> > >>>>> <date_range name="march">
>>>> > >> > >>>>>         <start>2019-03-01 00:00:00</start>
>>>> > >> > >>>>>         <end>2019-03-31 23:59:59</end>
>>>> > >> > >>>>> </date_range>
>>>> > >> > >>>>>
>>>> > >> > >>>>> and then use one or all date_ranges:
>>>> > >> > >>>>> <plot_fix>
>>>> > >> > >>>>>     <field name="fcst_init_beg">
>>>> > >> > >>>>>         <date_range name="march"/>
>>>> > >> > >>>>>         <date_range name="may"/>
>>>> > >> > >>>>>    </field>...
>>>> > >> > >>>>> This configuration would include all dates for March
and
>>>> May
>>>> > >> > >>>>> of
>>>> > >> > >>>>> 2019.
>>>> > >> > >>>>>
>>>> > >> > >>>>> If you need more flexible configuration you can use
>>>> > >> > >>>>> date_range_list.
>>>> > >> > >>>>> For example, you want to select first 6 hours every
day
>>>> > >> > >>>>> beetween
>>>> > >> > >>>>> 2020-01-01 00:00:00 and 2020-01-05 00:00:00. Your
XML would
>>>> > >> > >>>>> look
>>>> > >> > >>>>> like this:
>>>> > >> > >>>>> <date_range_list name = 'all'>
>>>> > >> > >>>>>        <range_start>2020-01-01
00:00:00</range_start>
>>>> > >> > >>>>>         <range_end>2020-01-05 00:00:00</range_end>
>>>> > >> > >>>>>         <range_length>360</range_length>
>>>> > >> > >>>>>         <inc>1440</inc>
>>>> > >> > >>>>> </date_range_list>
>>>> > >> > >>>>> where the 1st and 2nd values are the start and end,
>>>> > >> > >>>>> range_length - the length of a range in minutes(6
houres)
>>>> > >> > >>>>> inc - lenght beetween ranges in minutes (24 houres)
>>>> > >> > >>>>> This example would create following ranges:
>>>> > >> > >>>>> BETWEEN '2020-01-01 00:00:00' AND '2020-01-01
06:00:00'
>>>> > >> > >>>>> BETWEEN '2020-01-02 00:00:00' AND '2020-01-02
06:00:00'
>>>> > >> > >>>>> BETWEEN '2020-01-03 00:00:00' AND '2020-01-03
06:00:00'
>>>> > >> > >>>>> BETWEEN '2020-01-04 00:00:00' AND '2020-01-04
06:00:00'
>>>> > >> > >>>>> BETWEEN '2020-01-05 00:00:00' AND '2020-01-05
06:00:00'
>>>> > >> > >>>>>
>>>> > >> > >>>>> Also you can add more parameters to <plot_fix> to
filter
>>>> > >> > >>>>> your
>>>> > >> > >>>>> data.
>>>> > >> > >>>>> For example, if you want to select only dates with
00:00
>>>> you
>>>> > >> > >>>>> can
>>>> > >> > >>>>> specify
>>>> > >> > >>>>>  <field name="valid_hour">
>>>> > >> > >>>>>         <val>00</val>
>>>> > >> > >>>>>  </field>
>>>> > >> > >>>>>
>>>> > >> > >>>>> Tatiana
>>>> > >> > >>>>>
>>>> > >> > >>>>>
>>>> > >> > >>>>>
>>>> > >> > >>>>> On Sun Jun 14 09:48:24 2020,
edward.strobach at noaa.gov
>>>> wrote:
>>>> > >> > >>>>> > and if there isn't a better way, I wonder the
whereabouts
>>>> > >> > >>>>> > of a
>>>> > >> > >>>>> > separate
>>>> > >> > >>>>> > script that pipes dates (and other relevant
information)
>>>> > >> > >>>>> > into
>>>> > >> > >>>>> > the XML
>>>> > >> > >>>>> > file
>>>> > >> > >>>>> > used to generate the results.
>>>> > >> > >>>>> >
>>>> > >> > >>>>> > On Sun, Jun 14, 2020 at 11:42 AM Edward Strobach -
NOAA
>>>> > >> > >>>>> > Affiliate <
>>>> > >> > >>>>> > edward.strobach at noaa.gov> wrote:
>>>> > >> > >>>>> >
>>>> > >> > >>>>> > > Thanks Tatiana.  These resources are helpful.
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > > One last question.  Simply typing lines like
this in
>>>> can
>>>> > >> > >>>>> > > take
>>>> > >> > >>>>> > > forever:
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > > <plot_fix>
>>>> > >> > >>>>> > >     <field equalize="false"
name="fcst_init_beg">
>>>> > >> > >>>>> > >         <set name="fcst_init_beg_0">
>>>> > >> > >>>>> > >             <val>2019-05-01 12:00:00</val>
>>>> > >> > >>>>> > >             <val>2019-05-02 06:00:00</val>
>>>> > >> > >>>>> > >             <val>2019-05-02 12:00:00</val>
>>>> > >> > >>>>> > >             <val>2019-05-03 06:00:00</val>
>>>> > >> > >>>>> > >             <val>2019-05-03 12:00:00</val>
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > > Is there a better way?   I see date_range,
>>>> > >> > >>>>> > > date_range_list,
>>>> > >> > >>>>> > > or
>>>> > >> > >>>>> > > date_list.
>>>> > >> > >>>>> > > Can this be used instead of typing individual
values?
>>>> > >> > >>>>> > > This
>>>> > >> > >>>>> > > could
>>>> > >> > >>>>> > > prove
>>>> > >> > >>>>> > > pretty daunting if one would have to type in 700
>>>> entries
>>>> > >> > >>>>> > > each
>>>> > >> > >>>>> > > time.
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > > On Fri, Jun 12, 2020 at 2:23 PM Tatiana Burek
via RT
>>>> > >> > >>>>> > > <met_help at ucar.edu>
>>>> > >> > >>>>> > > wrote:
>>>> > >> > >>>>> > >
>>>> > >> > >>>>> > >> I found a few problems in your XML.
>>>> > >> > >>>>> > >> The database you are using doesn't have data
with
>>>> > >> > >>>>> > >> vx_mask
>>>> > >> > >>>>> > >> 'abcREGabc'.
>>>> > >> > >>>>> > >> That is why METviewer returns an empty dataset
and
>>>> > >> > >>>>> > >> can't
>>>> > >> > >>>>> > >> create a
>>>> > >> > >>>>> > >> plot.
>>>> > >> > >>>>> > >> The other problem is how you define fcst_lead.
This
>>>> > >> > >>>>> > >> database
>>>> > >> > >>>>> > >> has
>>>> > >> > >>>>> > >> values
>>>> > >> > >>>>> > >> like 10000, 20000 and not 010000, 020000. Since
you
>>>> > >> > >>>>> > >> include
>>>> > >> > >>>>> > >> all
>>>> > >> > >>>>> > >> fcst_leads
>>>> > >> > >>>>> > >> this variable can be omitted.
>>>> > >> > >>>>> > >> To minimize problems with variables values, I
would
>>>> > >> > >>>>> > >> suggest
>>>> > >> > >>>>> > >> to
>>>> > >> > >>>>> > >> create XML
>>>> > >> > >>>>> > >> using GUI and then edit it to fit batch
standards.
>>>> > >> > >>>>> > >> I created a plot using you configurations with
GUI
>>>> > >> > >>>>> > >> (without
>>>> > >> > >>>>> vx_mack
>>>> > >> > >>>>> > >> and
>>>> > >> > >>>>> > >> plot and axis titles) #20200612_173626. You can
load
>>>> it
>>>> > >> > >>>>> > >> back
>>>> > >> > >>>>> > >> to
>>>> > >> > >>>>> the
>>>> > >> > >>>>> > >> page by
>>>> > >> > >>>>> > >> using the "History" tab (left bar on GUI). Find
the
>>>> > >> > >>>>> > >> plot,
>>>> > >> > >>>>> > >> open XML
>>>> > >> > >>>>> > >> and
>>>> > >> > >>>>> > >> click on "Load to page". After that you can
recreate a
>>>> > >> > >>>>> > >> plot
>>>> > >> > >>>>> > >> and
>>>> > >> > >>>>> edit
>>>> > >> > >>>>> > >> the
>>>> > >> > >>>>> > >> configurations. You can also download a new XML
and
>>>> > >> > >>>>> > >> edit it
>>>> > >> > >>>>> > >> for
>>>> > >> > >>>>> > >> batch.
>>>> > >> > >>>>> > >> I attached the example of load XML to this
message.
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >> Description of rds parameters:
>>>> > >> > >>>>> > >> rds_R_tmpl and rds_R_work - location of
METviewer R
>>>> > >> > >>>>> > >> scripts
>>>> > >> > >>>>> > >> rds_plots - temporary directory where the plots
will
>>>> be
>>>> > >> > >>>>> > >> generated
>>>> > >> > >>>>> > >> rds_data - temporary directory where the data
>>>> retrieved
>>>> > >> > >>>>> > >> from
>>>> > >> > >>>>> > >> the
>>>> > >> > >>>>> > >> database
>>>> > >> > >>>>> > >> and other intermediate files will be stored
>>>> > >> > >>>>> > >> rds_scripts - temporary directory where R
scripts for
>>>> > >> > >>>>> > >> this
>>>> > >> > >>>>> > >> run
>>>> > >> > >>>>> will
>>>> > >> > >>>>> > >> be
>>>> > >> > >>>>> > >> stored
>>>> > >> > >>>>> > >> After the execution is done all the files from
>>>> > >> > >>>>> > >> temporary
>>>> > >> > >>>>> directories
>>>> > >> > >>>>> > >> would be removed.
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >> We don't have the offitial user guide for
METviewer.
>>>> > >> > >>>>> > >> There
>>>> > >> > >>>>> > >> are
>>>> > >> > >>>>> some
>>>> > >> > >>>>> > >> instructions on how to crate XML config files:
>>>> > >> > >>>>> > >> https://metviewer.nws.noaa.gov/doc/index.html
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >> Tatiana
>>>> > >> > >>>>> > >>
>>>> > >> > >>>>> > >> On Fri Jun 12 11:08:43 2020,
edward.strobach at noaa.gov
>>>> > >> > >>>>> > >> wrote:
>>>> > >> > >>>>> > >> > Hi,
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > I'm attaching the xml below, but am having
hard time
>>>> > >> > >>>>> > >> > following
>>>> > >> > >>>>> how
>>>> > >> > >>>>> > >> > to
>>>> > >> > >>>>> > >> > go
>>>> > >> > >>>>> > >> > about the other steps since I have zero
experience
>>>> on
>>>> > >> > >>>>> > >> > how
>>>> > >> > >>>>> > >> > to do
>>>> > >> > >>>>> > >> > these
>>>> > >> > >>>>> > >> > things.
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > #2. All parameters in <folders> section
should NOT
>>>> be
>>>> > >> > >>>>> > >> > changed
>>>> > >> > >>>>> and
>>>> > >> > >>>>> > >> > will
>>>> > >> > >>>>> > >> > be
>>>> > >> > >>>>> > >> > replaced by other scripts during the batch
>>>> execution.
>>>> > >> > >>>>> > >> > The
>>>> > >> > >>>>> > >> > data
>>>> > >> > >>>>> for
>>>> > >> > >>>>> > >> > the
>>>> > >> > >>>>> > >> > batch mode is coming from the database and
not from
>>>> > >> > >>>>> > >> > files.
>>>> > >> > >>>>> > >> > The
>>>> > >> > >>>>> > >> > name of
>>>> > >> > >>>>> > >> > the
>>>> > >> > >>>>> > >> > database is in <database> tag.
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > I understand that nothing should be changed
in the
>>>> > >> > >>>>> > >> > folders
>>>> > >> > >>>>> > >> > section,
>>>> > >> > >>>>> > >> > but
>>>> > >> > >>>>> > >> > there is no indication how and where these
variables
>>>> > >> > >>>>> > >> > are
>>>> > >> > >>>>> > >> > being
>>>> > >> > >>>>> > >> > defined.
>>>> > >> > >>>>> > >> > Grepping these names of the files available
does not
>>>> > >> > >>>>> > >> > reveal
>>>> > >> > >>>>> formal
>>>> > >> > >>>>> > >> > definitions of these variables.  If I
understood
>>>> > >> > >>>>> > >> > exactly
>>>> > >> > >>>>> > >> > how
>>>> > >> > >>>>> these
>>>> > >> > >>>>> > >> > were
>>>> > >> > >>>>> > >> > being defined, then I would be able connect
this
>>>> > >> > >>>>> > >> > better.
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > #3. Users rum MET and produce MET output
files. They
>>>> > >> > >>>>> > >> > use
>>>> > >> > >>>>> METviewer
>>>> > >> > >>>>> > >> > load
>>>> > >> > >>>>> > >> > mode to load these files to the specific
database.
>>>> In
>>>> > >> > >>>>> > >> > the
>>>> > >> > >>>>> > >> > config
>>>> > >> > >>>>> > >> > file
>>>> > >> > >>>>> > >> > for
>>>> > >> > >>>>> > >> > this process they also can specify the
database
>>>> > >> > >>>>> > >> > group. For
>>>> > >> > >>>>> example
>>>> > >> > >>>>> > >> > database
>>>> > >> > >>>>> > >> > mv_dev is in Test12 group.
>>>> > >> > >>>>> > >> > After the data has been loaded to the
database it is
>>>> > >> > >>>>> > >> > visible and
>>>> > >> > >>>>> > >> > available
>>>> > >> > >>>>> > >> > in the GUI and also can be used by the batch
mode.
>>>> > >> > >>>>> > >> > You can always add more data to the database
using
>>>> > >> > >>>>> > >> > the
>>>> > >> > >>>>> > >> > same load
>>>> > >> > >>>>> > >> > mode.
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > Is there a guide for me to understand the
process
>>>> > >> > >>>>> > >> > that you
>>>> > >> > >>>>> > >> > are
>>>> > >> > >>>>> > >> > referring to?
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > On Fri, Jun 12, 2020 at 12:51 PM Tatiana
Burek via
>>>> RT
>>>> > >> > >>>>> > >> > <met_help at ucar.edu>
>>>> > >> > >>>>> > >> > wrote:
>>>> > >> > >>>>> > >> >
>>>> > >> > >>>>> > >> > > Hi Edward,
>>>> > >> > >>>>> > >> > > #1. I don't have an access to NOAA servers
and
>>>> > >> > >>>>> > >> > > can't
>>>> > >> > >>>>> > >> > > view your
>>>> > >> > >>>>> > >> > > XML
>>>> > >> > >>>>> > >> > > for the
>>>> > >> > >>>>> > >> > > batch mode. Please, attache it to this
issue and I
>>>> > >> > >>>>> > >> > > will
>>>> > >> > >>>>> > >> > > take a
>>>> > >> > >>>>> > >> > > look.
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > > #2. All parameters in <folders> section
should NOT
>>>> > >> > >>>>> > >> > > be
>>>> > >> > >>>>> > >> > > changed
>>>> > >> > >>>>> > >> > > and
>>>> > >> > >>>>> > >> > > will be
>>>> > >> > >>>>> > >> > > replaced by other scripts during the batch
>>>> > >> > >>>>> > >> > > execution.
>>>> > >> > >>>>> > >> > > The data
>>>> > >> > >>>>> > >> > > for
>>>> > >> > >>>>> > >> > > the
>>>> > >> > >>>>> > >> > > batch mode is coming from the database and
not
>>>> from
>>>> > >> > >>>>> > >> > > files. The
>>>> > >> > >>>>> > >> > > name
>>>> > >> > >>>>> > >> > > of the
>>>> > >> > >>>>> > >> > > database is in <database> tag.
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > > #3. Users rum MET and produce MET output
files.
>>>> > >> > >>>>> > >> > > They use
>>>> > >> > >>>>> > >> > > METviewer
>>>> > >> > >>>>> > >> > > load
>>>> > >> > >>>>> > >> > > mode to load these files to the specific
database.
>>>> > >> > >>>>> > >> > > In
>>>> > >> > >>>>> > >> > > the
>>>> > >> > >>>>> config
>>>> > >> > >>>>> > >> > > file
>>>> > >> > >>>>> > >> > > for
>>>> > >> > >>>>> > >> > > this process they also can specify the
database
>>>> > >> > >>>>> > >> > > group.
>>>> > >> > >>>>> > >> > > For
>>>> > >> > >>>>> > >> > > example
>>>> > >> > >>>>> > >> > > database
>>>> > >> > >>>>> > >> > > mv_dev is in Test12 group.
>>>> > >> > >>>>> > >> > > After the data has been loaded to the
database it
>>>> > >> > >>>>> > >> > > is
>>>> > >> > >>>>> > >> > > visible
>>>> > >> > >>>>> and
>>>> > >> > >>>>> > >> > > available
>>>> > >> > >>>>> > >> > > in the GUI and also can be used by the
batch mode.
>>>> > >> > >>>>> > >> > > You can always add more data to the
database using
>>>> > >> > >>>>> > >> > > the
>>>> > >> > >>>>> > >> > > same
>>>> > >> > >>>>> load
>>>> > >> > >>>>> > >> > > mode.
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > > Tatiana
>>>> > >> > >>>>> > >> > >
>>>> > >> > >>>>> > >> > > On Fri Jun 12 09:08:29 2020,
>>>> > >> > >>>>> > >> > > edward.strobach at noaa.gov
>>>> > >> > >>>>> > >> > > wrote:
>>>> > >> > >>>>> > >> > > > Good morning,
>>>> > >> > >>>>> > >> > > >
>>>> > >> > >>>>> > >> > > > Over the last two weeks I've made an
effort to
>>>> > >> > >>>>> > >> > > > understand
>>>> > >> > >>>>> > >> > > > metviewer,
>>>> > >> > >>>>> > >> > > > test
>>>> > >> > >>>>> > >> > > > different cases using the history tab,
changing
>>>> > >> > >>>>> > >> > > > information
>>>> > >> > >>>>> to
>>>> > >> > >>&g