[Met_help] [rt.rap.ucar.edu #95850] History for GridStat Question

George McCabe via RT met_help at ucar.edu
Thu Aug 27 16:45:58 MDT 2020


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

My start date for GridStat is e.g. 2016060118 for FCSTs and OBSs. FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and OBSs at 3-hr intervals. For this time GridStat.conf is configured to take 3rd FCST in the input file [in GridStat.conf: (3,1,*,*) and  6th OBS in the other input file: (6,1,*,*) That only works properly for FCSTs/OBSs starting at 18z with intervals of 24 hours when running over e.g. a month-long period. However, if I wanted to run GridStat to get statistics with data at intervals of 6 hours e.g. SDATE=2016060118 and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*) needs to become (0,1,*,*) etc. Is there a way to pass times of FCSTs/OBSs to GridStat.conf when running master_metplus.py ?

Thanks again,
Mariusz

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: GridStat Question
From: Julie Prestopnik
Time: Wed Jul 08 10:47:03 2020

Hi Mariusz.

I moved your question over to met_help so that others could benefit
from your questions.

I asked a co-worker for some help with this one and he said that if
the data is in NetCDF and in a format that MET can read, you can
specify the time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There is a METplus use case that does this here:

parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf

FCST_GRID_STAT_VAR1_NAME = pr
FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"

He said that likely won't work with this data, but there is likely a
clever way to use the valid time to create the appropriate level value
for each run time in a similar way. If you aren't able to figure it
out, you could provide a sample dataset and he could help figure out
how to configure it.

Thanks,
Julie

On Wed Jul 08 10:39:59 2020, jpresto wrote:
> My start date for GridStat is e.g. 2016060118 for FCSTs and OBSs.
> FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and OBSs at
3-
> hr intervals. For this time GridStat.conf is configured to take 3rd
> FCST in the input file [in GridStat.conf: (3,1,*,*) and  6th OBS in
> the other input file: (6,1,*,*) That only works properly for
> FCSTs/OBSs starting at 18z with intervals of 24 hours when running
> over e.g. a month-long period. However, if I wanted to run GridStat
to
> get statistics with data at intervals of 6 hours e.g.
SDATE=2016060118
> and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*) needs
> to become (0,1,*,*) etc. Is there a way to pass times of FCSTs/OBSs
to
> GridStat.conf when running master_metplus.py ?
>
> Thanks again,
> Mariusz



------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Thu Jul 23 19:06:34 2020

Julie,
It was hanging over my head but only today I tried to input forecast
time
to the conf and did not figure the solution.
valid?fmt string does not produce for me a correct time and GridStat
fails.
If your colleague looked into our problem
we would very much appreciate it, thanks,
Mariusz

models for GridStat processing
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll

other stuff:
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
/scratch1/BMC/gsd-fv3-dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat

On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:

> Hi Mariusz.
>
> I moved your question over to met_help so that others could benefit
from
> your questions.
>
> I asked a co-worker for some help with this one and he said that if
the
> data is in NetCDF and in a format that MET can read, you can specify
the
> time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is a
METplus
> use case that does this here:
>
>
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
>
> FCST_GRID_STAT_VAR1_NAME = pr
> FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
>
> He said that likely won't work with this data, but there is likely a
> clever way to use the valid time to create the appropriate level
value for
> each run time in a similar way. If you aren't able to figure it out,
you
> could provide a sample dataset and he could help figure out how to
> configure it.
>
> Thanks,
> Julie
>
> On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > My start date for GridStat is e.g. 2016060118 for FCSTs and OBSs.
> > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and OBSs at
3-
> > hr intervals. For this time GridStat.conf is configured to take
3rd
> > FCST in the input file [in GridStat.conf: (3,1,*,*) and  6th OBS
in
> > the other input file: (6,1,*,*) That only works properly for
> > FCSTs/OBSs starting at 18z with intervals of 24 hours when running
> > over e.g. a month-long period. However, if I wanted to run
GridStat to
> > get statistics with data at intervals of 6 hours e.g.
SDATE=2016060118
> > and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*)
needs
> > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs to
> > GridStat.conf when running master_metplus.py ?
> >
> > Thanks again,
> > Mariusz
>
>
>
>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Mon Jul 27 11:13:14 2020

Hi Mariusz,

To specify the time in the NetCDF structure, you need to use the
format
YYYYMMDD_HHMMSS, so your FTIME variable should look like this:

{valid?fmt=%Y%m%d_%H%M%S}

If the NetCDF time info is formatted in the way the MET tools expect,
the
code is smart enough to find the correct data from the time info.

If that doesn't work, you may need to do something clever to
manipulate the
hour value to the corresponding number of the NetCDF dimension. It
looks
like your data has a time dimension of size 4, so passing in the hour
(18)
is outside of the valid range. I would recommend trying to format the
valid
time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and I try
to
help you figure out a way to get the correct number for the time you
need.

Thanks,
George

On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> Julie,
> It was hanging over my head but only today I tried to input forecast
time
> to the conf and did not figure the solution.
> valid?fmt string does not produce for me a correct time and GridStat
fails.
> If your colleague looked into our problem
> we would very much appreciate it, thanks,
> Mariusz
>
> models for GridStat processing
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
>
> other stuff:
> /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> /scratch1/BMC/gsd-fv3-dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
>
> On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> > Hi Mariusz.
> >
> > I moved your question over to met_help so that others could
benefit from
> > your questions.
> >
> > I asked a co-worker for some help with this one and he said that
if the
> > data is in NetCDF and in a format that MET can read, you can
specify the
> > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is a
METplus
> > use case that does this here:
> >
> >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> >
> > FCST_GRID_STAT_VAR1_NAME = pr
> > FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
> >
> > He said that likely won't work with this data, but there is likely
a
> > clever way to use the valid time to create the appropriate level
value
> for
> > each run time in a similar way. If you aren't able to figure it
out, you
> > could provide a sample dataset and he could help figure out how to
> > configure it.
> >
> > Thanks,
> > Julie
> >
> > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
> > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and OBSs
at 3-
> > > hr intervals. For this time GridStat.conf is configured to take
3rd
> > > FCST in the input file [in GridStat.conf: (3,1,*,*) and  6th OBS
in
> > > the other input file: (6,1,*,*) That only works properly for
> > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
> > > over e.g. a month-long period. However, if I wanted to run
GridStat to
> > > get statistics with data at intervals of 6 hours e.g.
SDATE=2016060118
> > > and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*)
needs
> > > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs to
> > > GridStat.conf when running master_metplus.py ?
> > >
> > > Thanks again,
> > > Mariusz
> >
> >
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Mon Jul 27 22:14:31 2020

Thanks, George, works as promised,
Mariusz

On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Mariusz,
>
> To specify the time in the NetCDF structure, you need to use the
format
> YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
>
> {valid?fmt=%Y%m%d_%H%M%S}
>
> If the NetCDF time info is formatted in the way the MET tools
expect, the
> code is smart enough to find the correct data from the time info.
>
> If that doesn't work, you may need to do something clever to
manipulate the
> hour value to the corresponding number of the NetCDF dimension. It
looks
> like your data has a time dimension of size 4, so passing in the
hour (18)
> is outside of the valid range. I would recommend trying to format
the valid
> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and I
try to
> help you figure out a way to get the correct number for the time you
need.
>
> Thanks,
> George
>
> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > Julie,
> > It was hanging over my head but only today I tried to input
forecast time
> > to the conf and did not figure the solution.
> > valid?fmt string does not produce for me a correct time and
GridStat
> fails.
> > If your colleague looked into our problem
> > we would very much appreciate it, thanks,
> > Mariusz
> >
> > models for GridStat processing
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> >
> > other stuff:
> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> >
> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > > Hi Mariusz.
> > >
> > > I moved your question over to met_help so that others could
benefit
> from
> > > your questions.
> > >
> > > I asked a co-worker for some help with this one and he said that
if the
> > > data is in NetCDF and in a format that MET can read, you can
specify
> the
> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is a
> METplus
> > > use case that does this here:
> > >
> > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > >
> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
> > >
> > > He said that likely won't work with this data, but there is
likely a
> > > clever way to use the valid time to create the appropriate level
value
> > for
> > > each run time in a similar way. If you aren't able to figure it
out,
> you
> > > could provide a sample dataset and he could help figure out how
to
> > > configure it.
> > >
> > > Thanks,
> > > Julie
> > >
> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and
OBSs at 3-
> > > > hr intervals. For this time GridStat.conf is configured to
take 3rd
> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and  6th
OBS in
> > > > the other input file: (6,1,*,*) That only works properly for
> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
> > > > over e.g. a month-long period. However, if I wanted to run
GridStat
> to
> > > > get statistics with data at intervals of 6 hours e.g.
> SDATE=2016060118
> > > > and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*)
needs
> > > > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs
> to
> > > > GridStat.conf when running master_metplus.py ?
> > > >
> > > > Thanks again,
> > > > Mariusz
> > >
> > >
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Tue Jul 28 08:19:32 2020

Good to hear! I am going to close this ticket, but feel free to open a
new
ticket via met_help at ucar.edu if you have any other issues.

Thanks,
George

On Mon, Jul 27, 2020 at 10:15 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> Thanks, George, works as promised,
> Mariusz
>
> On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Mariusz,
> >
> > To specify the time in the NetCDF structure, you need to use the
format
> > YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
> >
> > {valid?fmt=%Y%m%d_%H%M%S}
> >
> > If the NetCDF time info is formatted in the way the MET tools
expect, the
> > code is smart enough to find the correct data from the time info.
> >
> > If that doesn't work, you may need to do something clever to
manipulate
> the
> > hour value to the corresponding number of the NetCDF dimension. It
looks
> > like your data has a time dimension of size 4, so passing in the
hour
> (18)
> > is outside of the valid range. I would recommend trying to format
the
> valid
> > time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and I
try to
> > help you figure out a way to get the correct number for the time
you
> need.
> >
> > Thanks,
> > George
> >
> > On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > Julie,
> > > It was hanging over my head but only today I tried to input
forecast
> time
> > > to the conf and did not figure the solution.
> > > valid?fmt string does not produce for me a correct time and
GridStat
> > fails.
> > > If your colleague looked into our problem
> > > we would very much appreciate it, thanks,
> > > Mariusz
> > >
> > > models for GridStat processing
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > >
> > > other stuff:
> > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > >
> > > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > > Hi Mariusz.
> > > >
> > > > I moved your question over to met_help so that others could
benefit
> > from
> > > > your questions.
> > > >
> > > > I asked a co-worker for some help with this one and he said
that if
> the
> > > > data is in NetCDF and in a format that MET can read, you can
specify
> > the
> > > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is
a
> > METplus
> > > > use case that does this here:
> > > >
> > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > >
> > > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
> > > >
> > > > He said that likely won't work with this data, but there is
likely a
> > > > clever way to use the valid time to create the appropriate
level
> value
> > > for
> > > > each run time in a similar way. If you aren't able to figure
it out,
> > you
> > > > could provide a sample dataset and he could help figure out
how to
> > > > configure it.
> > > >
> > > > Thanks,
> > > > Julie
> > > >
> > > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
> > > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and
OBSs at
> 3-
> > > > > hr intervals. For this time GridStat.conf is configured to
take 3rd
> > > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and  6th
OBS in
> > > > > the other input file: (6,1,*,*) That only works properly for
> > > > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
> > > > > over e.g. a month-long period. However, if I wanted to run
GridStat
> > to
> > > > > get statistics with data at intervals of 6 hours e.g.
> > SDATE=2016060118
> > > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> needs
> > > > > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs
> > to
> > > > > GridStat.conf when running master_metplus.py ?
> > > > >
> > > > > Thanks again,
> > > > > Mariusz
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Tue Aug 04 21:00:55 2020

George,
actually, there is a problem with the next step: StatAnalysis.
GridStat outputs time levels in the form 20160601_060000,0,*,*
but StatAnalysis does not accept
{valid?fmt=%Y%m%d_%H%M%S}
so ends up either producing an error or empty files depending how the
config is set up.
Is there a solution to this problem?
Thanks,
Mariusz

on Orion:
script
/home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
...
    | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
    | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
    > ./StatAnalysis.conf

KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:  Fatal error
occurred

Outputs in
/work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats



On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA Affiliate <
mariusz.pagowski at noaa.gov> wrote:

> Thanks, George, works as promised,
> Mariusz
>
> On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
>> Hi Mariusz,
>>
>> To specify the time in the NetCDF structure, you need to use the
format
>> YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
>>
>> {valid?fmt=%Y%m%d_%H%M%S}
>>
>> If the NetCDF time info is formatted in the way the MET tools
expect, the
>> code is smart enough to find the correct data from the time info.
>>
>> If that doesn't work, you may need to do something clever to
manipulate
>> the
>> hour value to the corresponding number of the NetCDF dimension. It
looks
>> like your data has a time dimension of size 4, so passing in the
hour (18)
>> is outside of the valid range. I would recommend trying to format
the
>> valid
>> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and I
try to
>> help you figure out a way to get the correct number for the time
you need.
>>
>> Thanks,
>> George
>>
>> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>> >
>> > Julie,
>> > It was hanging over my head but only today I tried to input
forecast
>> time
>> > to the conf and did not figure the solution.
>> > valid?fmt string does not produce for me a correct time and
GridStat
>> fails.
>> > If your colleague looked into our problem
>> > we would very much appreciate it, thanks,
>> > Mariusz
>> >
>> > models for GridStat processing
>> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
>> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
>> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
>> >
>> > other stuff:
>> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
>> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
>> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
>> >
>> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
>> met_help at ucar.edu
>> > >
>> > wrote:
>> >
>> > > Hi Mariusz.
>> > >
>> > > I moved your question over to met_help so that others could
benefit
>> from
>> > > your questions.
>> > >
>> > > I asked a co-worker for some help with this one and he said
that if
>> the
>> > > data is in NetCDF and in a format that MET can read, you can
specify
>> the
>> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is
a
>> METplus
>> > > use case that does this here:
>> > >
>> > >
>> > >
>> >
>>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
>> > >
>> > > FCST_GRID_STAT_VAR1_NAME = pr
>> > > FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
>> > >
>> > > He said that likely won't work with this data, but there is
likely a
>> > > clever way to use the valid time to create the appropriate
level value
>> > for
>> > > each run time in a similar way. If you aren't able to figure it
out,
>> you
>> > > could provide a sample dataset and he could help figure out how
to
>> > > configure it.
>> > >
>> > > Thanks,
>> > > Julie
>> > >
>> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
>> > > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
>> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and
OBSs at
>> 3-
>> > > > hr intervals. For this time GridStat.conf is configured to
take 3rd
>> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and  6th
OBS in
>> > > > the other input file: (6,1,*,*) That only works properly for
>> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
>> > > > over e.g. a month-long period. However, if I wanted to run
GridStat
>> to
>> > > > get statistics with data at intervals of 6 hours e.g.
>> SDATE=2016060118
>> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*) needs
>> > > > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs
>> to
>> > > > GridStat.conf when running master_metplus.py ?
>> > > >
>> > > > Thanks again,
>> > > > Mariusz
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Wed Aug 05 08:04:08 2020

Hi Mariusz,

You are correct that StatAnalysis cannot use that syntax to specify
the
levels in this case. Fortunately there is a recently added feature
that
will allow you to get around this issue. You can specify values to
override
attributes in GridStat. This will allow you to change the level value
20160601_060000,0,*,* to a value that is more easily specified in the
StatAnalysis config file. GitHub issue #1020 outlines these changes
and
this link takes you directly to a comment that lists the new
configurations:

https://github.com/NCAR/MET/issues/1020#issuecomment-653724572

You will want to set set_attr_level in GridStat, which can be added
via
METplus with:

FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";

I don't have access to Orion yet so I can't look at your configuration
to
know what exactly "something" should be. If you need help figuring
that
out, you could attach your METplus config file to this ticket and I
could
take a look.

These changes are available since MET 9.1-beta2 and will be in MET
9.1,
which is close to being released. It looks like the beta release is
not
installed on Orion but 9.1 is planned to be installed there.

Let me know if you have any other questions.

Thanks,
George

On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> George,
> actually, there is a problem with the next step: StatAnalysis.
> GridStat outputs time levels in the form 20160601_060000,0,*,*
> but StatAnalysis does not accept
> {valid?fmt=%Y%m%d_%H%M%S}
> so ends up either producing an error or empty files depending how
the
> config is set up.
> Is there a solution to this problem?
> Thanks,
> Mariusz
>
> on Orion:
> script
> /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> ...
>     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>     > ./StatAnalysis.conf
>
> KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:  Fatal
error
> occurred
>
> Outputs in
> /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
>
>
>
> On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA Affiliate <
> mariusz.pagowski at noaa.gov> wrote:
>
> > Thanks, George, works as promised,
> > Mariusz
> >
> > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> >> Hi Mariusz,
> >>
> >> To specify the time in the NetCDF structure, you need to use the
format
> >> YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
> >>
> >> {valid?fmt=%Y%m%d_%H%M%S}
> >>
> >> If the NetCDF time info is formatted in the way the MET tools
expect,
> the
> >> code is smart enough to find the correct data from the time info.
> >>
> >> If that doesn't work, you may need to do something clever to
manipulate
> >> the
> >> hour value to the corresponding number of the NetCDF dimension.
It looks
> >> like your data has a time dimension of size 4, so passing in the
hour
> (18)
> >> is outside of the valid range. I would recommend trying to format
the
> >> valid
> >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and
I try
> to
> >> help you figure out a way to get the correct number for the time
you
> need.
> >>
> >> Thanks,
> >> George
> >>
> >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >> >
> >> > Julie,
> >> > It was hanging over my head but only today I tried to input
forecast
> >> time
> >> > to the conf and did not figure the solution.
> >> > valid?fmt string does not produce for me a correct time and
GridStat
> >> fails.
> >> > If your colleague looked into our problem
> >> > we would very much appreciate it, thanks,
> >> > Mariusz
> >> >
> >> > models for GridStat processing
> >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> >> >
> >> > other stuff:
> >> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> >> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> >> >
> >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> >> met_help at ucar.edu
> >> > >
> >> > wrote:
> >> >
> >> > > Hi Mariusz.
> >> > >
> >> > > I moved your question over to met_help so that others could
benefit
> >> from
> >> > > your questions.
> >> > >
> >> > > I asked a co-worker for some help with this one and he said
that if
> >> the
> >> > > data is in NetCDF and in a format that MET can read, you can
specify
> >> the
> >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There
is a
> >> METplus
> >> > > use case that does this here:
> >> > >
> >> > >
> >> > >
> >> >
> >>
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> >> > >
> >> > > FCST_GRID_STAT_VAR1_NAME = pr
> >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> >> > >
> >> > > He said that likely won't work with this data, but there is
likely a
> >> > > clever way to use the valid time to create the appropriate
level
> value
> >> > for
> >> > > each run time in a similar way. If you aren't able to figure
it out,
> >> you
> >> > > could provide a sample dataset and he could help figure out
how to
> >> > > configure it.
> >> > >
> >> > > Thanks,
> >> > > Julie
> >> > >
> >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> >> > > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
> >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and
OBSs at
> >> 3-
> >> > > > hr intervals. For this time GridStat.conf is configured to
take
> 3rd
> >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and
6th OBS
> in
> >> > > > the other input file: (6,1,*,*) That only works properly
for
> >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
> >> > > > over e.g. a month-long period. However, if I wanted to run
> GridStat
> >> to
> >> > > > get statistics with data at intervals of 6 hours e.g.
> >> SDATE=2016060118
> >> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> needs
> >> > > > to become (0,1,*,*) etc. Is there a way to pass times of
> FCSTs/OBSs
> >> to
> >> > > > GridStat.conf when running master_metplus.py ?
> >> > > >
> >> > > > Thanks again,
> >> > > > Mariusz
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >> --
> >> George McCabe - Software Engineer III
> >> National Center for Atmospheric Research
> >> Research Applications Laboratory
> >> 303-497-2768
> >> ---
> >> My working day may not be your working day. Please do not feel
obliged
> to
> >> reply to this email outside of your normal working hours.
> >>
> >>
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Wed Aug 05 09:53:28 2020

Thanks, George,
What is a timeline for releasing 9.1?
Mariusz


On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Mariusz,
>
> You are correct that StatAnalysis cannot use that syntax to specify
the
> levels in this case. Fortunately there is a recently added feature
that
> will allow you to get around this issue. You can specify values to
override
> attributes in GridStat. This will allow you to change the level
value
> 20160601_060000,0,*,* to a value that is more easily specified in
the
> StatAnalysis config file. GitHub issue #1020 outlines these changes
and
> this link takes you directly to a comment that lists the new
> configurations:
>
> https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
>
> You will want to set set_attr_level in GridStat, which can be added
via
> METplus with:
>
> FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
>
> I don't have access to Orion yet so I can't look at your
configuration to
> know what exactly "something" should be. If you need help figuring
that
> out, you could attach your METplus config file to this ticket and I
could
> take a look.
>
> These changes are available since MET 9.1-beta2 and will be in MET
9.1,
> which is close to being released. It looks like the beta release is
not
> installed on Orion but 9.1 is planned to be installed there.
>
> Let me know if you have any other questions.
>
> Thanks,
> George
>
> On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > George,
> > actually, there is a problem with the next step: StatAnalysis.
> > GridStat outputs time levels in the form 20160601_060000,0,*,*
> > but StatAnalysis does not accept
> > {valid?fmt=%Y%m%d_%H%M%S}
> > so ends up either producing an error or empty files depending how
the
> > config is set up.
> > Is there a solution to this problem?
> > Thanks,
> > Mariusz
> >
> > on Orion:
> > script
> > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > ...
> >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> >     > ./StatAnalysis.conf
> >
> > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:  Fatal
error
> > occurred
> >
> > Outputs in
> > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> >
> >
> >
> > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA Affiliate
<
> > mariusz.pagowski at noaa.gov> wrote:
> >
> > > Thanks, George, works as promised,
> > > Mariusz
> > >
> > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > >> Hi Mariusz,
> > >>
> > >> To specify the time in the NetCDF structure, you need to use
the
> format
> > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
> > >>
> > >> {valid?fmt=%Y%m%d_%H%M%S}
> > >>
> > >> If the NetCDF time info is formatted in the way the MET tools
expect,
> > the
> > >> code is smart enough to find the correct data from the time
info.
> > >>
> > >> If that doesn't work, you may need to do something clever to
> manipulate
> > >> the
> > >> hour value to the corresponding number of the NetCDF dimension.
It
> looks
> > >> like your data has a time dimension of size 4, so passing in
the hour
> > (18)
> > >> is outside of the valid range. I would recommend trying to
format the
> > >> valid
> > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know
and I try
> > to
> > >> help you figure out a way to get the correct number for the
time you
> > need.
> > >>
> > >> Thanks,
> > >> George
> > >>
> > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > >> met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > >> >
> > >> > Julie,
> > >> > It was hanging over my head but only today I tried to input
forecast
> > >> time
> > >> > to the conf and did not figure the solution.
> > >> > valid?fmt string does not produce for me a correct time and
GridStat
> > >> fails.
> > >> > If your colleague looked into our problem
> > >> > we would very much appreciate it, thanks,
> > >> > Mariusz
> > >> >
> > >> > models for GridStat processing
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > >> >
> > >> > other stuff:
> > >> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > >> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > >> >
> > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Hi Mariusz.
> > >> > >
> > >> > > I moved your question over to met_help so that others could
> benefit
> > >> from
> > >> > > your questions.
> > >> > >
> > >> > > I asked a co-worker for some help with this one and he said
that
> if
> > >> the
> > >> > > data is in NetCDF and in a format that MET can read, you
can
> specify
> > >> the
> > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There
is a
> > >> METplus
> > >> > > use case that does this here:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > >> > >
> > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> > >> > >
> > >> > > He said that likely won't work with this data, but there is
> likely a
> > >> > > clever way to use the valid time to create the appropriate
level
> > value
> > >> > for
> > >> > > each run time in a similar way. If you aren't able to
figure it
> out,
> > >> you
> > >> > > could provide a sample dataset and he could help figure out
how to
> > >> > > configure it.
> > >> > >
> > >> > > Thanks,
> > >> > > Julie
> > >> > >
> > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > >> > > > My start date for GridStat is e.g. 2016060118 for FCSTs
and
> OBSs.
> > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z)
and OBSs
> at
> > >> 3-
> > >> > > > hr intervals. For this time GridStat.conf is configured
to take
> > 3rd
> > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and
6th OBS
> > in
> > >> > > > the other input file: (6,1,*,*) That only works properly
for
> > >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours
when
> running
> > >> > > > over e.g. a month-long period. However, if I wanted to
run
> > GridStat
> > >> to
> > >> > > > get statistics with data at intervals of 6 hours e.g.
> > >> SDATE=2016060118
> > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> > needs
> > >> > > > to become (0,1,*,*) etc. Is there a way to pass times of
> > FCSTs/OBSs
> > >> to
> > >> > > > GridStat.conf when running master_metplus.py ?
> > >> > > >
> > >> > > > Thanks again,
> > >> > > > Mariusz
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >> --
> > >> George McCabe - Software Engineer III
> > >> National Center for Atmospheric Research
> > >> Research Applications Laboratory
> > >> 303-497-2768
> > >> ---
> > >> My working day may not be your working day. Please do not feel
obliged
> > to
> > >> reply to this email outside of your normal working hours.
> > >>
> > >>
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Wed Aug 05 10:12:03 2020

Hi Mariusz,

We are aiming to release during this week, however the engineer who
installs on Orion is out for the rest of the week. If all goes
according to
plan, it should be available on Orion early next week. If you'd like,
I can
reply to this ticket when that has been completed to let you know that
it
is available.

Thanks,
George

On Wed, Aug 5, 2020 at 9:54 AM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> Thanks, George,
> What is a timeline for releasing 9.1?
> Mariusz
>
>
> On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Mariusz,
> >
> > You are correct that StatAnalysis cannot use that syntax to
specify the
> > levels in this case. Fortunately there is a recently added feature
that
> > will allow you to get around this issue. You can specify values to
> override
> > attributes in GridStat. This will allow you to change the level
value
> > 20160601_060000,0,*,* to a value that is more easily specified in
the
> > StatAnalysis config file. GitHub issue #1020 outlines these
changes and
> > this link takes you directly to a comment that lists the new
> > configurations:
> >
> > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> >
> > You will want to set set_attr_level in GridStat, which can be
added via
> > METplus with:
> >
> > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> >
> > I don't have access to Orion yet so I can't look at your
configuration to
> > know what exactly "something" should be. If you need help figuring
that
> > out, you could attach your METplus config file to this ticket and
I could
> > take a look.
> >
> > These changes are available since MET 9.1-beta2 and will be in MET
9.1,
> > which is close to being released. It looks like the beta release
is not
> > installed on Orion but 9.1 is planned to be installed there.
> >
> > Let me know if you have any other questions.
> >
> > Thanks,
> > George
> >
> > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > George,
> > > actually, there is a problem with the next step: StatAnalysis.
> > > GridStat outputs time levels in the form 20160601_060000,0,*,*
> > > but StatAnalysis does not accept
> > > {valid?fmt=%Y%m%d_%H%M%S}
> > > so ends up either producing an error or empty files depending
how the
> > > config is set up.
> > > Is there a solution to this problem?
> > > Thanks,
> > > Mariusz
> > >
> > > on Orion:
> > > script
> > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > ...
> > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > >     > ./StatAnalysis.conf
> > >
> > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:  Fatal
error
> > > occurred
> > >
> > > Outputs in
> > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > >
> > >
> > >
> > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate <
> > > mariusz.pagowski at noaa.gov> wrote:
> > >
> > > > Thanks, George, works as promised,
> > > > Mariusz
> > > >
> > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > >> Hi Mariusz,
> > > >>
> > > >> To specify the time in the NetCDF structure, you need to use
the
> > format
> > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
> > > >>
> > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > >>
> > > >> If the NetCDF time info is formatted in the way the MET tools
> expect,
> > > the
> > > >> code is smart enough to find the correct data from the time
info.
> > > >>
> > > >> If that doesn't work, you may need to do something clever to
> > manipulate
> > > >> the
> > > >> hour value to the corresponding number of the NetCDF
dimension. It
> > looks
> > > >> like your data has a time dimension of size 4, so passing in
the
> hour
> > > (18)
> > > >> is outside of the valid range. I would recommend trying to
format
> the
> > > >> valid
> > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know
and I
> try
> > > to
> > > >> help you figure out a way to get the correct number for the
time you
> > > need.
> > > >>
> > > >> Thanks,
> > > >> George
> > > >>
> > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > > >> met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > >> >
> > > >> > Julie,
> > > >> > It was hanging over my head but only today I tried to input
> forecast
> > > >> time
> > > >> > to the conf and did not figure the solution.
> > > >> > valid?fmt string does not produce for me a correct time and
> GridStat
> > > >> fails.
> > > >> > If your colleague looked into our problem
> > > >> > we would very much appreciate it, thanks,
> > > >> > Mariusz
> > > >> >
> > > >> > models for GridStat processing
> > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > >> >
> > > >> > other stuff:
> > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > >> >
> /scratch1/BMC/gsd-fv3-dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > >> >
> > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Mariusz.
> > > >> > >
> > > >> > > I moved your question over to met_help so that others
could
> > benefit
> > > >> from
> > > >> > > your questions.
> > > >> > >
> > > >> > > I asked a co-worker for some help with this one and he
said that
> > if
> > > >> the
> > > >> > > data is in NetCDF and in a format that MET can read, you
can
> > specify
> > > >> the
> > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There is a
> > > >> METplus
> > > >> > > use case that does this here:
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > >> > >
> > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> > > >> > >
> > > >> > > He said that likely won't work with this data, but there
is
> > likely a
> > > >> > > clever way to use the valid time to create the
appropriate level
> > > value
> > > >> > for
> > > >> > > each run time in a similar way. If you aren't able to
figure it
> > out,
> > > >> you
> > > >> > > could provide a sample dataset and he could help figure
out how
> to
> > > >> > > configure it.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Julie
> > > >> > >
> > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > >> > > > My start date for GridStat is e.g. 2016060118 for FCSTs
and
> > OBSs.
> > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z)
and
> OBSs
> > at
> > > >> 3-
> > > >> > > > hr intervals. For this time GridStat.conf is configured
to
> take
> > > 3rd
> > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and
6th
> OBS
> > > in
> > > >> > > > the other input file: (6,1,*,*) That only works
properly for
> > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours
when
> > running
> > > >> > > > over e.g. a month-long period. However, if I wanted to
run
> > > GridStat
> > > >> to
> > > >> > > > get statistics with data at intervals of 6 hours e.g.
> > > >> SDATE=2016060118
> > > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> > > needs
> > > >> > > > to become (0,1,*,*) etc. Is there a way to pass times
of
> > > FCSTs/OBSs
> > > >> to
> > > >> > > > GridStat.conf when running master_metplus.py ?
> > > >> > > >
> > > >> > > > Thanks again,
> > > >> > > > Mariusz
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >>
> > > >> --
> > > >> George McCabe - Software Engineer III
> > > >> National Center for Atmospheric Research
> > > >> Research Applications Laboratory
> > > >> 303-497-2768
> > > >> ---
> > > >> My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > >> reply to this email outside of your normal working hours.
> > > >>
> > > >>
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Wed Aug 05 10:15:16 2020

Thanks again, George, that's quick. And yes, pls let me know when
the installation on Orion is complete/available,
Mariusz

On Wed, Aug 5, 2020 at 10:12 AM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Mariusz,
>
> We are aiming to release during this week, however the engineer who
> installs on Orion is out for the rest of the week. If all goes
according to
> plan, it should be available on Orion early next week. If you'd
like, I can
> reply to this ticket when that has been completed to let you know
that it
> is available.
>
> Thanks,
> George
>
> On Wed, Aug 5, 2020 at 9:54 AM Mariusz Pagowski via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > Thanks, George,
> > What is a timeline for releasing 9.1?
> > Mariusz
> >
> >
> > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Mariusz,
> > >
> > > You are correct that StatAnalysis cannot use that syntax to
specify the
> > > levels in this case. Fortunately there is a recently added
feature that
> > > will allow you to get around this issue. You can specify values
to
> > override
> > > attributes in GridStat. This will allow you to change the level
value
> > > 20160601_060000,0,*,* to a value that is more easily specified
in the
> > > StatAnalysis config file. GitHub issue #1020 outlines these
changes and
> > > this link takes you directly to a comment that lists the new
> > > configurations:
> > >
> > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> > >
> > > You will want to set set_attr_level in GridStat, which can be
added via
> > > METplus with:
> > >
> > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> > >
> > > I don't have access to Orion yet so I can't look at your
configuration
> to
> > > know what exactly "something" should be. If you need help
figuring that
> > > out, you could attach your METplus config file to this ticket
and I
> could
> > > take a look.
> > >
> > > These changes are available since MET 9.1-beta2 and will be in
MET 9.1,
> > > which is close to being released. It looks like the beta release
is not
> > > installed on Orion but 9.1 is planned to be installed there.
> > >
> > > Let me know if you have any other questions.
> > >
> > > Thanks,
> > > George
> > >
> > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > > >
> > > > George,
> > > > actually, there is a problem with the next step: StatAnalysis.
> > > > GridStat outputs time levels in the form 20160601_060000,0,*,*
> > > > but StatAnalysis does not accept
> > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > so ends up either producing an error or empty files depending
how the
> > > > config is set up.
> > > > Is there a solution to this problem?
> > > > Thanks,
> > > > Mariusz
> > > >
> > > > on Orion:
> > > > script
> > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > ...
> > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > >     > ./StatAnalysis.conf
> > > >
> > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal error
> > > > occurred
> > > >
> > > > Outputs in
> > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > >
> > > >
> > > >
> > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate <
> > > > mariusz.pagowski at noaa.gov> wrote:
> > > >
> > > > > Thanks, George, works as promised,
> > > > > Mariusz
> > > > >
> > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Mariusz,
> > > > >>
> > > > >> To specify the time in the NetCDF structure, you need to
use the
> > > format
> > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
> > > > >>
> > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > >>
> > > > >> If the NetCDF time info is formatted in the way the MET
tools
> > expect,
> > > > the
> > > > >> code is smart enough to find the correct data from the time
info.
> > > > >>
> > > > >> If that doesn't work, you may need to do something clever
to
> > > manipulate
> > > > >> the
> > > > >> hour value to the corresponding number of the NetCDF
dimension. It
> > > looks
> > > > >> like your data has a time dimension of size 4, so passing
in the
> > hour
> > > > (18)
> > > > >> is outside of the valid range. I would recommend trying to
format
> > the
> > > > >> valid
> > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me
know and I
> > try
> > > > to
> > > > >> help you figure out a way to get the correct number for the
time
> you
> > > > need.
> > > > >>
> > > > >> Thanks,
> > > > >> George
> > > > >>
> > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > > > >> met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >> >
> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > >> >
> > > > >> > Julie,
> > > > >> > It was hanging over my head but only today I tried to
input
> > forecast
> > > > >> time
> > > > >> > to the conf and did not figure the solution.
> > > > >> > valid?fmt string does not produce for me a correct time
and
> > GridStat
> > > > >> fails.
> > > > >> > If your colleague looked into our problem
> > > > >> > we would very much appreciate it, thanks,
> > > > >> > Mariusz
> > > > >> >
> > > > >> > models for GridStat processing
> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > >> >
> > > > >> > other stuff:
> > > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > >> >
> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > >> >
> > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > > > >> met_help at ucar.edu
> > > > >> > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Mariusz.
> > > > >> > >
> > > > >> > > I moved your question over to met_help so that others
could
> > > benefit
> > > > >> from
> > > > >> > > your questions.
> > > > >> > >
> > > > >> > > I asked a co-worker for some help with this one and he
said
> that
> > > if
> > > > >> the
> > > > >> > > data is in NetCDF and in a format that MET can read,
you can
> > > specify
> > > > >> the
> > > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There
> is a
> > > > >> METplus
> > > > >> > > use case that does this here:
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > >> > >
> > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> > > > >> > >
> > > > >> > > He said that likely won't work with this data, but
there is
> > > likely a
> > > > >> > > clever way to use the valid time to create the
appropriate
> level
> > > > value
> > > > >> > for
> > > > >> > > each run time in a similar way. If you aren't able to
figure
> it
> > > out,
> > > > >> you
> > > > >> > > could provide a sample dataset and he could help figure
out
> how
> > to
> > > > >> > > configure it.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Julie
> > > > >> > >
> > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > >> > > > My start date for GridStat is e.g. 2016060118 for
FCSTs and
> > > OBSs.
> > > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z,
18Z) and
> > OBSs
> > > at
> > > > >> 3-
> > > > >> > > > hr intervals. For this time GridStat.conf is
configured to
> > take
> > > > 3rd
> > > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*)
and  6th
> > OBS
> > > > in
> > > > >> > > > the other input file: (6,1,*,*) That only works
properly for
> > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours
when
> > > running
> > > > >> > > > over e.g. a month-long period. However, if I wanted
to run
> > > > GridStat
> > > > >> to
> > > > >> > > > get statistics with data at intervals of 6 hours e.g.
> > > > >> SDATE=2016060118
> > > > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
> (3,1,*,*)
> > > > needs
> > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass times
of
> > > > FCSTs/OBSs
> > > > >> to
> > > > >> > > > GridStat.conf when running master_metplus.py ?
> > > > >> > > >
> > > > >> > > > Thanks again,
> > > > >> > > > Mariusz
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >> --
> > > > >> George McCabe - Software Engineer III
> > > > >> National Center for Atmospheric Research
> > > > >> Research Applications Laboratory
> > > > >> 303-497-2768
> > > > >> ---
> > > > >> My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > >> reply to this email outside of your normal working hours.
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: GridStat Question
From: Julie Prestopnik
Time: Thu Aug 13 16:17:50 2020

HI Mariusz.

Unfortunately, I do not have permissions to install in the proper area
on
Orion.  I am working on getting that changed.  I will follow up once
the
install is ready on Orion.

Julie

On Wed, Aug 5, 2020 at 10:15 AM Mariusz Pagowski - NOAA Affiliate <
mariusz.pagowski at noaa.gov> wrote:

> Thanks again, George, that's quick. And yes, pls let me know when
> the installation on Orion is complete/available,
> Mariusz
>
> On Wed, Aug 5, 2020 at 10:12 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
>> Hi Mariusz,
>>
>> We are aiming to release during this week, however the engineer who
>> installs on Orion is out for the rest of the week. If all goes
according
>> to
>> plan, it should be available on Orion early next week. If you'd
like, I
>> can
>> reply to this ticket when that has been completed to let you know
that it
>> is available.
>>
>> Thanks,
>> George
>>
>> On Wed, Aug 5, 2020 at 9:54 AM Mariusz Pagowski via RT
<met_help at ucar.edu
>> >
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>> >
>> > Thanks, George,
>> > What is a timeline for releasing 9.1?
>> > Mariusz
>> >
>> >
>> > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
>> > wrote:
>> >
>> > > Hi Mariusz,
>> > >
>> > > You are correct that StatAnalysis cannot use that syntax to
specify
>> the
>> > > levels in this case. Fortunately there is a recently added
feature
>> that
>> > > will allow you to get around this issue. You can specify values
to
>> > override
>> > > attributes in GridStat. This will allow you to change the level
value
>> > > 20160601_060000,0,*,* to a value that is more easily specified
in the
>> > > StatAnalysis config file. GitHub issue #1020 outlines these
changes
>> and
>> > > this link takes you directly to a comment that lists the new
>> > > configurations:
>> > >
>> > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
>> > >
>> > > You will want to set set_attr_level in GridStat, which can be
added
>> via
>> > > METplus with:
>> > >
>> > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
>> > >
>> > > I don't have access to Orion yet so I can't look at your
>> configuration to
>> > > know what exactly "something" should be. If you need help
figuring
>> that
>> > > out, you could attach your METplus config file to this ticket
and I
>> could
>> > > take a look.
>> > >
>> > > These changes are available since MET 9.1-beta2 and will be in
MET
>> 9.1,
>> > > which is close to being released. It looks like the beta
release is
>> not
>> > > installed on Orion but 9.1 is planned to be installed there.
>> > >
>> > > Let me know if you have any other questions.
>> > >
>> > > Thanks,
>> > > George
>> > >
>> > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
>> > met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
>> > > >
>> > > > George,
>> > > > actually, there is a problem with the next step:
StatAnalysis.
>> > > > GridStat outputs time levels in the form
20160601_060000,0,*,*
>> > > > but StatAnalysis does not accept
>> > > > {valid?fmt=%Y%m%d_%H%M%S}
>> > > > so ends up either producing an error or empty files depending
how
>> the
>> > > > config is set up.
>> > > > Is there a solution to this problem?
>> > > > Thanks,
>> > > > Mariusz
>> > > >
>> > > > on Orion:
>> > > > script
>> > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
>> > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
>> > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
>> > > > ...
>> > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>> > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>> > > >     > ./StatAnalysis.conf
>> > > >
>> > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
>> > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal
>> error
>> > > > occurred
>> > > >
>> > > > Outputs in
>> > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate <
>> > > > mariusz.pagowski at noaa.gov> wrote:
>> > > >
>> > > > > Thanks, George, works as promised,
>> > > > > Mariusz
>> > > > >
>> > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
>> > > met_help at ucar.edu
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > >> Hi Mariusz,
>> > > > >>
>> > > > >> To specify the time in the NetCDF structure, you need to
use the
>> > > format
>> > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
>> > > > >>
>> > > > >> {valid?fmt=%Y%m%d_%H%M%S}
>> > > > >>
>> > > > >> If the NetCDF time info is formatted in the way the MET
tools
>> > expect,
>> > > > the
>> > > > >> code is smart enough to find the correct data from the
time info.
>> > > > >>
>> > > > >> If that doesn't work, you may need to do something clever
to
>> > > manipulate
>> > > > >> the
>> > > > >> hour value to the corresponding number of the NetCDF
dimension.
>> It
>> > > looks
>> > > > >> like your data has a time dimension of size 4, so passing
in the
>> > hour
>> > > > (18)
>> > > > >> is outside of the valid range. I would recommend trying to
format
>> > the
>> > > > >> valid
>> > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me
know and
>> I
>> > try
>> > > > to
>> > > > >> help you figure out a way to get the correct number for
the time
>> you
>> > > > need.
>> > > > >>
>> > > > >> Thanks,
>> > > > >> George
>> > > > >>
>> > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
>> > > > >> met_help at ucar.edu>
>> > > > >> wrote:
>> > > > >>
>> > > > >> >
>> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>> >
>> > > > >> >
>> > > > >> > Julie,
>> > > > >> > It was hanging over my head but only today I tried to
input
>> > forecast
>> > > > >> time
>> > > > >> > to the conf and did not figure the solution.
>> > > > >> > valid?fmt string does not produce for me a correct time
and
>> > GridStat
>> > > > >> fails.
>> > > > >> > If your colleague looked into our problem
>> > > > >> > we would very much appreciate it, thanks,
>> > > > >> > Mariusz
>> > > > >> >
>> > > > >> > models for GridStat processing
>> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
>> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
>> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
>> > > > >> >
>> > > > >> > other stuff:
>> > > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
>> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
>> > > > >> >
>> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
>> > > > >> >
>> > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT
<
>> > > > >> met_help at ucar.edu
>> > > > >> > >
>> > > > >> > wrote:
>> > > > >> >
>> > > > >> > > Hi Mariusz.
>> > > > >> > >
>> > > > >> > > I moved your question over to met_help so that others
could
>> > > benefit
>> > > > >> from
>> > > > >> > > your questions.
>> > > > >> > >
>> > > > >> > > I asked a co-worker for some help with this one and he
said
>> that
>> > > if
>> > > > >> the
>> > > > >> > > data is in NetCDF and in a format that MET can read,
you can
>> > > specify
>> > > > >> the
>> > > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There
>> is a
>> > > > >> METplus
>> > > > >> > > use case that does this here:
>> > > > >> > >
>> > > > >> > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
>> > > > >> > >
>> > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
>> > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
>> "({valid?fmt=%Y%m01_000000},*,*)"
>> > > > >> > >
>> > > > >> > > He said that likely won't work with this data, but
there is
>> > > likely a
>> > > > >> > > clever way to use the valid time to create the
appropriate
>> level
>> > > > value
>> > > > >> > for
>> > > > >> > > each run time in a similar way. If you aren't able to
figure
>> it
>> > > out,
>> > > > >> you
>> > > > >> > > could provide a sample dataset and he could help
figure out
>> how
>> > to
>> > > > >> > > configure it.
>> > > > >> > >
>> > > > >> > > Thanks,
>> > > > >> > > Julie
>> > > > >> > >
>> > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
>> > > > >> > > > My start date for GridStat is e.g. 2016060118 for
FCSTs and
>> > > OBSs.
>> > > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z,
18Z) and
>> > OBSs
>> > > at
>> > > > >> 3-
>> > > > >> > > > hr intervals. For this time GridStat.conf is
configured to
>> > take
>> > > > 3rd
>> > > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*)
and
>> 6th
>> > OBS
>> > > > in
>> > > > >> > > > the other input file: (6,1,*,*) That only works
properly
>> for
>> > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24
hours when
>> > > running
>> > > > >> > > > over e.g. a month-long period. However, if I wanted
to run
>> > > > GridStat
>> > > > >> to
>> > > > >> > > > get statistics with data at intervals of 6 hours
e.g.
>> > > > >> SDATE=2016060118
>> > > > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
>> (3,1,*,*)
>> > > > needs
>> > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass
times of
>> > > > FCSTs/OBSs
>> > > > >> to
>> > > > >> > > > GridStat.conf when running master_metplus.py ?
>> > > > >> > > >
>> > > > >> > > > Thanks again,
>> > > > >> > > > Mariusz
>> > > > >> > >
>> > > > >> > >
>> > > > >> > >
>> > > > >> > >
>> > > > >> >
>> > > > >> >
>> > > > >>
>> > > > >> --
>> > > > >> George McCabe - Software Engineer III
>> > > > >> National Center for Atmospheric Research
>> > > > >> Research Applications Laboratory
>> > > > >> 303-497-2768
>> > > > >> ---
>> > > > >> My working day may not be your working day. Please do not
feel
>> > obliged
>> > > > to
>> > > > >> reply to this email outside of your normal working hours.
>> > > > >>
>> > > > >>
>> > > >
>> > > >
>> > >
>> > > --
>> > > George McCabe - Software Engineer III
>> > > National Center for Atmospheric Research
>> > > Research Applications Laboratory
>> > > 303-497-2768
>> > > ---
>> > > My working day may not be your working day. Please do not feel
>> obliged to
>> > > reply to this email outside of your normal working hours.
>> > >
>> > >
>> >
>> >
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>>

--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Mon Aug 17 16:20:00 2020

Julie,
it is not for lack of interest in MET 9.1 that my response to your e-
mail
is delayed. Very much looking forward
to the new version, thanks,
Mariusz

On Thu, Aug 13, 2020 at 4:17 PM Julie Prestopnik <jpresto at ucar.edu>
wrote:

> HI Mariusz.
>
> Unfortunately, I do not have permissions to install in the proper
area on
> Orion.  I am working on getting that changed.  I will follow up once
the
> install is ready on Orion.
>
> Julie
>
> On Wed, Aug 5, 2020 at 10:15 AM Mariusz Pagowski - NOAA Affiliate <
> mariusz.pagowski at noaa.gov> wrote:
>
>> Thanks again, George, that's quick. And yes, pls let me know when
>> the installation on Orion is complete/available,
>> Mariusz
>>
>> On Wed, Aug 5, 2020 at 10:12 AM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>>> Hi Mariusz,
>>>
>>> We are aiming to release during this week, however the engineer
who
>>> installs on Orion is out for the rest of the week. If all goes
according
>>> to
>>> plan, it should be available on Orion early next week. If you'd
like, I
>>> can
>>> reply to this ticket when that has been completed to let you know
that it
>>> is available.
>>>
>>> Thanks,
>>> George
>>>
>>> On Wed, Aug 5, 2020 at 9:54 AM Mariusz Pagowski via RT <
>>> met_help at ucar.edu>
>>> wrote:
>>>
>>> >
>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>>> >
>>> > Thanks, George,
>>> > What is a timeline for releasing 9.1?
>>> > Mariusz
>>> >
>>> >
>>> > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu
>>> >
>>> > wrote:
>>> >
>>> > > Hi Mariusz,
>>> > >
>>> > > You are correct that StatAnalysis cannot use that syntax to
specify
>>> the
>>> > > levels in this case. Fortunately there is a recently added
feature
>>> that
>>> > > will allow you to get around this issue. You can specify
values to
>>> > override
>>> > > attributes in GridStat. This will allow you to change the
level value
>>> > > 20160601_060000,0,*,* to a value that is more easily specified
in the
>>> > > StatAnalysis config file. GitHub issue #1020 outlines these
changes
>>> and
>>> > > this link takes you directly to a comment that lists the new
>>> > > configurations:
>>> > >
>>> > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
>>> > >
>>> > > You will want to set set_attr_level in GridStat, which can be
added
>>> via
>>> > > METplus with:
>>> > >
>>> > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
>>> > >
>>> > > I don't have access to Orion yet so I can't look at your
>>> configuration to
>>> > > know what exactly "something" should be. If you need help
figuring
>>> that
>>> > > out, you could attach your METplus config file to this ticket
and I
>>> could
>>> > > take a look.
>>> > >
>>> > > These changes are available since MET 9.1-beta2 and will be in
MET
>>> 9.1,
>>> > > which is close to being released. It looks like the beta
release is
>>> not
>>> > > installed on Orion but 9.1 is planned to be installed there.
>>> > >
>>> > > Let me know if you have any other questions.
>>> > >
>>> > > Thanks,
>>> > > George
>>> > >
>>> > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
>>> > met_help at ucar.edu>
>>> > > wrote:
>>> > >
>>> > > >
>>> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>>> > > >
>>> > > > George,
>>> > > > actually, there is a problem with the next step:
StatAnalysis.
>>> > > > GridStat outputs time levels in the form
20160601_060000,0,*,*
>>> > > > but StatAnalysis does not accept
>>> > > > {valid?fmt=%Y%m%d_%H%M%S}
>>> > > > so ends up either producing an error or empty files
depending how
>>> the
>>> > > > config is set up.
>>> > > > Is there a solution to this problem?
>>> > > > Thanks,
>>> > > > Mariusz
>>> > > >
>>> > > > on Orion:
>>> > > > script
>>> > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
>>> > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
>>> > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
>>> > > > ...
>>> > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>>> > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>>> > > >     > ./StatAnalysis.conf
>>> > > >
>>> > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
>>> > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal
>>> error
>>> > > > occurred
>>> > > >
>>> > > > Outputs in
>>> > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate
>>> <
>>> > > > mariusz.pagowski at noaa.gov> wrote:
>>> > > >
>>> > > > > Thanks, George, works as promised,
>>> > > > > Mariusz
>>> > > > >
>>> > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
>>> > > met_help at ucar.edu
>>> > > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > >> Hi Mariusz,
>>> > > > >>
>>> > > > >> To specify the time in the NetCDF structure, you need to
use the
>>> > > format
>>> > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
>>> > > > >>
>>> > > > >> {valid?fmt=%Y%m%d_%H%M%S}
>>> > > > >>
>>> > > > >> If the NetCDF time info is formatted in the way the MET
tools
>>> > expect,
>>> > > > the
>>> > > > >> code is smart enough to find the correct data from the
time
>>> info.
>>> > > > >>
>>> > > > >> If that doesn't work, you may need to do something clever
to
>>> > > manipulate
>>> > > > >> the
>>> > > > >> hour value to the corresponding number of the NetCDF
dimension.
>>> It
>>> > > looks
>>> > > > >> like your data has a time dimension of size 4, so passing
in the
>>> > hour
>>> > > > (18)
>>> > > > >> is outside of the valid range. I would recommend trying
to
>>> format
>>> > the
>>> > > > >> valid
>>> > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me
know
>>> and I
>>> > try
>>> > > > to
>>> > > > >> help you figure out a way to get the correct number for
the
>>> time you
>>> > > > need.
>>> > > > >>
>>> > > > >> Thanks,
>>> > > > >> George
>>> > > > >>
>>> > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
>>> > > > >> met_help at ucar.edu>
>>> > > > >> wrote:
>>> > > > >>
>>> > > > >> >
>>> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>>> >
>>> > > > >> >
>>> > > > >> > Julie,
>>> > > > >> > It was hanging over my head but only today I tried to
input
>>> > forecast
>>> > > > >> time
>>> > > > >> > to the conf and did not figure the solution.
>>> > > > >> > valid?fmt string does not produce for me a correct time
and
>>> > GridStat
>>> > > > >> fails.
>>> > > > >> > If your colleague looked into our problem
>>> > > > >> > we would very much appreciate it, thanks,
>>> > > > >> > Mariusz
>>> > > > >> >
>>> > > > >> > models for GridStat processing
>>> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
>>> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
>>> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
>>> > > > >> >
>>> > > > >> > other stuff:
>>> > > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
>>> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
>>> > > > >> >
>>> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
>>> > > > >> >
>>> > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT
<
>>> > > > >> met_help at ucar.edu
>>> > > > >> > >
>>> > > > >> > wrote:
>>> > > > >> >
>>> > > > >> > > Hi Mariusz.
>>> > > > >> > >
>>> > > > >> > > I moved your question over to met_help so that others
could
>>> > > benefit
>>> > > > >> from
>>> > > > >> > > your questions.
>>> > > > >> > >
>>> > > > >> > > I asked a co-worker for some help with this one and
he said
>>> that
>>> > > if
>>> > > > >> the
>>> > > > >> > > data is in NetCDF and in a format that MET can read,
you can
>>> > > specify
>>> > > > >> the
>>> > > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There
>>> is a
>>> > > > >> METplus
>>> > > > >> > > use case that does this here:
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> >
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
>>> > > > >> > >
>>> > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
>>> > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
>>> "({valid?fmt=%Y%m01_000000},*,*)"
>>> > > > >> > >
>>> > > > >> > > He said that likely won't work with this data, but
there is
>>> > > likely a
>>> > > > >> > > clever way to use the valid time to create the
appropriate
>>> level
>>> > > > value
>>> > > > >> > for
>>> > > > >> > > each run time in a similar way. If you aren't able to
>>> figure it
>>> > > out,
>>> > > > >> you
>>> > > > >> > > could provide a sample dataset and he could help
figure out
>>> how
>>> > to
>>> > > > >> > > configure it.
>>> > > > >> > >
>>> > > > >> > > Thanks,
>>> > > > >> > > Julie
>>> > > > >> > >
>>> > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
>>> > > > >> > > > My start date for GridStat is e.g. 2016060118 for
FCSTs
>>> and
>>> > > OBSs.
>>> > > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z,
18Z) and
>>> > OBSs
>>> > > at
>>> > > > >> 3-
>>> > > > >> > > > hr intervals. For this time GridStat.conf is
configured to
>>> > take
>>> > > > 3rd
>>> > > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*)
and
>>> 6th
>>> > OBS
>>> > > > in
>>> > > > >> > > > the other input file: (6,1,*,*) That only works
properly
>>> for
>>> > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24
hours when
>>> > > running
>>> > > > >> > > > over e.g. a month-long period. However, if I wanted
to run
>>> > > > GridStat
>>> > > > >> to
>>> > > > >> > > > get statistics with data at intervals of 6 hours
e.g.
>>> > > > >> SDATE=2016060118
>>> > > > >> > > > and EDATE=2016063118 that is incorrect since for
00Z
>>> (3,1,*,*)
>>> > > > needs
>>> > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass
times of
>>> > > > FCSTs/OBSs
>>> > > > >> to
>>> > > > >> > > > GridStat.conf when running master_metplus.py ?
>>> > > > >> > > >
>>> > > > >> > > > Thanks again,
>>> > > > >> > > > Mariusz
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> >
>>> > > > >> >
>>> > > > >>
>>> > > > >> --
>>> > > > >> George McCabe - Software Engineer III
>>> > > > >> National Center for Atmospheric Research
>>> > > > >> Research Applications Laboratory
>>> > > > >> 303-497-2768
>>> > > > >> ---
>>> > > > >> My working day may not be your working day. Please do not
feel
>>> > obliged
>>> > > > to
>>> > > > >> reply to this email outside of your normal working hours.
>>> > > > >>
>>> > > > >>
>>> > > >
>>> > > >
>>> > >
>>> > > --
>>> > > George McCabe - Software Engineer III
>>> > > National Center for Atmospheric Research
>>> > > Research Applications Laboratory
>>> > > 303-497-2768
>>> > > ---
>>> > > My working day may not be your working day. Please do not feel
>>> obliged to
>>> > > reply to this email outside of your normal working hours.
>>> > >
>>> > >
>>> >
>>> >
>>>
>>> --
>>> George McCabe - Software Engineer III
>>> National Center for Atmospheric Research
>>> Research Applications Laboratory
>>> 303-497-2768
>>> ---
>>> My working day may not be your working day. Please do not feel
obliged to
>>> reply to this email outside of your normal working hours.
>>>
>>>
>
> --
> Julie Prestopnik (she/her/hers)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day.  Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Wed Aug 26 12:51:17 2020

George,
I am not able to figure out how to set up GridStat to have a correct
attribute for StatAnalysis to handle it.
Attached are the confs I am using. Where should I add the option you
mentioned and in what form?
I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
What is  "something" ?
Thanks,
Mariusz



On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Mariusz,
>
> You are correct that StatAnalysis cannot use that syntax to specify
the
> levels in this case. Fortunately there is a recently added feature
that
> will allow you to get around this issue. You can specify values to
override
> attributes in GridStat. This will allow you to change the level
value
> 20160601_060000,0,*,* to a value that is more easily specified in
the
> StatAnalysis config file. GitHub issue #1020 outlines these changes
and
> this link takes you directly to a comment that lists the new
> configurations:
>
> https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
>
> You will want to set set_attr_level in GridStat, which can be added
via
> METplus with:
>
> FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
>
> I don't have access to Orion yet so I can't look at your
configuration to
> know what exactly "something" should be. If you need help figuring
that
> out, you could attach your METplus config file to this ticket and I
could
> take a look.
>
> These changes are available since MET 9.1-beta2 and will be in MET
9.1,
> which is close to being released. It looks like the beta release is
not
> installed on Orion but 9.1 is planned to be installed there.
>
> Let me know if you have any other questions.
>
> Thanks,
> George
>
> On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > George,
> > actually, there is a problem with the next step: StatAnalysis.
> > GridStat outputs time levels in the form 20160601_060000,0,*,*
> > but StatAnalysis does not accept
> > {valid?fmt=%Y%m%d_%H%M%S}
> > so ends up either producing an error or empty files depending how
the
> > config is set up.
> > Is there a solution to this problem?
> > Thanks,
> > Mariusz
> >
> > on Orion:
> > script
> > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > ...
> >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> >     > ./StatAnalysis.conf
> >
> > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:  Fatal
error
> > occurred
> >
> > Outputs in
> > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> >
> >
> >
> > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA Affiliate
<
> > mariusz.pagowski at noaa.gov> wrote:
> >
> > > Thanks, George, works as promised,
> > > Mariusz
> > >
> > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > >> Hi Mariusz,
> > >>
> > >> To specify the time in the NetCDF structure, you need to use
the
> format
> > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
> > >>
> > >> {valid?fmt=%Y%m%d_%H%M%S}
> > >>
> > >> If the NetCDF time info is formatted in the way the MET tools
expect,
> > the
> > >> code is smart enough to find the correct data from the time
info.
> > >>
> > >> If that doesn't work, you may need to do something clever to
> manipulate
> > >> the
> > >> hour value to the corresponding number of the NetCDF dimension.
It
> looks
> > >> like your data has a time dimension of size 4, so passing in
the hour
> > (18)
> > >> is outside of the valid range. I would recommend trying to
format the
> > >> valid
> > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know
and I try
> > to
> > >> help you figure out a way to get the correct number for the
time you
> > need.
> > >>
> > >> Thanks,
> > >> George
> > >>
> > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > >> met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > >> >
> > >> > Julie,
> > >> > It was hanging over my head but only today I tried to input
forecast
> > >> time
> > >> > to the conf and did not figure the solution.
> > >> > valid?fmt string does not produce for me a correct time and
GridStat
> > >> fails.
> > >> > If your colleague looked into our problem
> > >> > we would very much appreciate it, thanks,
> > >> > Mariusz
> > >> >
> > >> > models for GridStat processing
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > >> >
> > >> > other stuff:
> > >> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > >> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > >> >
> > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Hi Mariusz.
> > >> > >
> > >> > > I moved your question over to met_help so that others could
> benefit
> > >> from
> > >> > > your questions.
> > >> > >
> > >> > > I asked a co-worker for some help with this one and he said
that
> if
> > >> the
> > >> > > data is in NetCDF and in a format that MET can read, you
can
> specify
> > >> the
> > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There
is a
> > >> METplus
> > >> > > use case that does this here:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > >> > >
> > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> > >> > >
> > >> > > He said that likely won't work with this data, but there is
> likely a
> > >> > > clever way to use the valid time to create the appropriate
level
> > value
> > >> > for
> > >> > > each run time in a similar way. If you aren't able to
figure it
> out,
> > >> you
> > >> > > could provide a sample dataset and he could help figure out
how to
> > >> > > configure it.
> > >> > >
> > >> > > Thanks,
> > >> > > Julie
> > >> > >
> > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > >> > > > My start date for GridStat is e.g. 2016060118 for FCSTs
and
> OBSs.
> > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z)
and OBSs
> at
> > >> 3-
> > >> > > > hr intervals. For this time GridStat.conf is configured
to take
> > 3rd
> > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and
6th OBS
> > in
> > >> > > > the other input file: (6,1,*,*) That only works properly
for
> > >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours
when
> running
> > >> > > > over e.g. a month-long period. However, if I wanted to
run
> > GridStat
> > >> to
> > >> > > > get statistics with data at intervals of 6 hours e.g.
> > >> SDATE=2016060118
> > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> > needs
> > >> > > > to become (0,1,*,*) etc. Is there a way to pass times of
> > FCSTs/OBSs
> > >> to
> > >> > > > GridStat.conf when running master_metplus.py ?
> > >> > > >
> > >> > > > Thanks again,
> > >> > > > Mariusz
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >> --
> > >> George McCabe - Software Engineer III
> > >> National Center for Atmospheric Research
> > >> Research Applications Laboratory
> > >> 303-497-2768
> > >> ---
> > >> My working day may not be your working day. Please do not feel
obliged
> > to
> > >> reply to this email outside of your normal working hours.
> > >>
> > >>
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Wed Aug 26 16:17:02 2020

Hi Mariusz,

That is a good question. It looks like you have multiple levels
specified
for *_VAR1_LEVELS.

FCST_VAR1_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"

If you were to add FCST_VAR1_OPTIONS, it would apply to all of those
fields, so all of those fields would contain the same value in the
FCST_LEV
column of the GridStat output. If you need to specify a different
FCST_LEV
value for each of those levels, you would have to split them up into
separate VAR<n> items, like this:

FCST_VAR1_NAME = DUSTTOTAL
FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"

FCST_VAR2_NAME = {FCST_VAR1_NAME}
FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"

FCST_VAR3_NAME =  {FCST_VAR1_NAME}
FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"

etc. and do the same for the OBS_VAR<n>_* variables.

I'm not sure what each of the levels contain, so it is hard to know
what to
name them. You really just need to use a unique identifier so that
they can
be grouped via StatAnalysis. You could set them to the number
corresponding
to each NetCDF item:

FCST_VAR1_NAME = DUSTTOTAL
FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
FCST_VAR1_OPTIONS = set_attr_level = "0";

FCST_VAR2_NAME = {FCST_VAR1_NAME}
FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
FCST_VAR2_OPTIONS = set_attr_level = "1";

FCST_VAR3_NAME =  {FCST_VAR1_NAME}
FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
FCST_VAR3_OPTIONS = set_attr_level = "2";

Or you could use a string that identifies each level. If they are
pressure
levels, you could do:

FCST_VAR1_OPTIONS = set_attr_level = "P750";
FCST_VAR2_OPTIONS = set_attr_level = "P500";
FCST_VAR3_OPTIONS = set_attr_level = "P250";

Whatever you decide to use, you should use the same values when
configuring
StatAnalysis:

FCST_LEVEL_LIST = 0, 1, 2
or
FCST_LEVEL_LIST = P750, P500, P250

Let me know if you have any questions or if that solution doesn't work
for
you.

Thanks,
George


On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> George,
> I am not able to figure out how to set up GridStat to have a correct
> attribute for StatAnalysis to handle it.
> Attached are the confs I am using. Where should I add the option you
> mentioned and in what form?
> I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> What is  "something" ?
> Thanks,
> Mariusz
>
>
>
> On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Mariusz,
> >
> > You are correct that StatAnalysis cannot use that syntax to
specify the
> > levels in this case. Fortunately there is a recently added feature
that
> > will allow you to get around this issue. You can specify values to
> override
> > attributes in GridStat. This will allow you to change the level
value
> > 20160601_060000,0,*,* to a value that is more easily specified in
the
> > StatAnalysis config file. GitHub issue #1020 outlines these
changes and
> > this link takes you directly to a comment that lists the new
> > configurations:
> >
> > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> >
> > You will want to set set_attr_level in GridStat, which can be
added via
> > METplus with:
> >
> > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> >
> > I don't have access to Orion yet so I can't look at your
configuration to
> > know what exactly "something" should be. If you need help figuring
that
> > out, you could attach your METplus config file to this ticket and
I could
> > take a look.
> >
> > These changes are available since MET 9.1-beta2 and will be in MET
9.1,
> > which is close to being released. It looks like the beta release
is not
> > installed on Orion but 9.1 is planned to be installed there.
> >
> > Let me know if you have any other questions.
> >
> > Thanks,
> > George
> >
> > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > George,
> > > actually, there is a problem with the next step: StatAnalysis.
> > > GridStat outputs time levels in the form 20160601_060000,0,*,*
> > > but StatAnalysis does not accept
> > > {valid?fmt=%Y%m%d_%H%M%S}
> > > so ends up either producing an error or empty files depending
how the
> > > config is set up.
> > > Is there a solution to this problem?
> > > Thanks,
> > > Mariusz
> > >
> > > on Orion:
> > > script
> > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > ...
> > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > >     > ./StatAnalysis.conf
> > >
> > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:  Fatal
error
> > > occurred
> > >
> > > Outputs in
> > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > >
> > >
> > >
> > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate <
> > > mariusz.pagowski at noaa.gov> wrote:
> > >
> > > > Thanks, George, works as promised,
> > > > Mariusz
> > > >
> > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > >> Hi Mariusz,
> > > >>
> > > >> To specify the time in the NetCDF structure, you need to use
the
> > format
> > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
> > > >>
> > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > >>
> > > >> If the NetCDF time info is formatted in the way the MET tools
> expect,
> > > the
> > > >> code is smart enough to find the correct data from the time
info.
> > > >>
> > > >> If that doesn't work, you may need to do something clever to
> > manipulate
> > > >> the
> > > >> hour value to the corresponding number of the NetCDF
dimension. It
> > looks
> > > >> like your data has a time dimension of size 4, so passing in
the
> hour
> > > (18)
> > > >> is outside of the valid range. I would recommend trying to
format
> the
> > > >> valid
> > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know
and I
> try
> > > to
> > > >> help you figure out a way to get the correct number for the
time you
> > > need.
> > > >>
> > > >> Thanks,
> > > >> George
> > > >>
> > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > > >> met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > >> >
> > > >> > Julie,
> > > >> > It was hanging over my head but only today I tried to input
> forecast
> > > >> time
> > > >> > to the conf and did not figure the solution.
> > > >> > valid?fmt string does not produce for me a correct time and
> GridStat
> > > >> fails.
> > > >> > If your colleague looked into our problem
> > > >> > we would very much appreciate it, thanks,
> > > >> > Mariusz
> > > >> >
> > > >> > models for GridStat processing
> > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > >> >
> > > >> > other stuff:
> > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > >> >
> /scratch1/BMC/gsd-fv3-dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > >> >
> > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Mariusz.
> > > >> > >
> > > >> > > I moved your question over to met_help so that others
could
> > benefit
> > > >> from
> > > >> > > your questions.
> > > >> > >
> > > >> > > I asked a co-worker for some help with this one and he
said that
> > if
> > > >> the
> > > >> > > data is in NetCDF and in a format that MET can read, you
can
> > specify
> > > >> the
> > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There is a
> > > >> METplus
> > > >> > > use case that does this here:
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > >> > >
> > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> > > >> > >
> > > >> > > He said that likely won't work with this data, but there
is
> > likely a
> > > >> > > clever way to use the valid time to create the
appropriate level
> > > value
> > > >> > for
> > > >> > > each run time in a similar way. If you aren't able to
figure it
> > out,
> > > >> you
> > > >> > > could provide a sample dataset and he could help figure
out how
> to
> > > >> > > configure it.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Julie
> > > >> > >
> > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > >> > > > My start date for GridStat is e.g. 2016060118 for FCSTs
and
> > OBSs.
> > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z)
and
> OBSs
> > at
> > > >> 3-
> > > >> > > > hr intervals. For this time GridStat.conf is configured
to
> take
> > > 3rd
> > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and
6th
> OBS
> > > in
> > > >> > > > the other input file: (6,1,*,*) That only works
properly for
> > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours
when
> > running
> > > >> > > > over e.g. a month-long period. However, if I wanted to
run
> > > GridStat
> > > >> to
> > > >> > > > get statistics with data at intervals of 6 hours e.g.
> > > >> SDATE=2016060118
> > > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> > > needs
> > > >> > > > to become (0,1,*,*) etc. Is there a way to pass times
of
> > > FCSTs/OBSs
> > > >> to
> > > >> > > > GridStat.conf when running master_metplus.py ?
> > > >> > > >
> > > >> > > > Thanks again,
> > > >> > > > Mariusz
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >>
> > > >> --
> > > >> George McCabe - Software Engineer III
> > > >> National Center for Atmospheric Research
> > > >> Research Applications Laboratory
> > > >> 303-497-2768
> > > >> ---
> > > >> My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > >> reply to this email outside of your normal working hours.
> > > >>
> > > >>
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Wed Aug 26 17:24:02 2020

George,
I configured it the way you suggested but GridStat produced the same
output
as before i.e. in SL1L2 a string with time in format %Y%m%d_%H%M%S and
after that StatAnalysis with empty files.
The issue is that I have two sorts of files - one with a single
forecast at
a time and another one with four fcsts daily.
Maybe I misunderstood something. If I put files on Hera - would you be
able
to look at them and the scripts?
Thanks,
Mariusz







On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Mariusz,
>
> That is a good question. It looks like you have multiple levels
specified
> for *_VAR1_LEVELS.
>
> FCST_VAR1_LEVELS =
>
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
>
> If you were to add FCST_VAR1_OPTIONS, it would apply to all of those
> fields, so all of those fields would contain the same value in the
FCST_LEV
> column of the GridStat output. If you need to specify a different
FCST_LEV
> value for each of those levels, you would have to split them up into
> separate VAR<n> items, like this:
>
> FCST_VAR1_NAME = DUSTTOTAL
> FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
>
> FCST_VAR2_NAME = {FCST_VAR1_NAME}
> FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
>
> FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
>
> etc. and do the same for the OBS_VAR<n>_* variables.
>
> I'm not sure what each of the levels contain, so it is hard to know
what to
> name them. You really just need to use a unique identifier so that
they can
> be grouped via StatAnalysis. You could set them to the number
corresponding
> to each NetCDF item:
>
> FCST_VAR1_NAME = DUSTTOTAL
> FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> FCST_VAR1_OPTIONS = set_attr_level = "0";
>
> FCST_VAR2_NAME = {FCST_VAR1_NAME}
> FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> FCST_VAR2_OPTIONS = set_attr_level = "1";
>
> FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> FCST_VAR3_OPTIONS = set_attr_level = "2";
>
> Or you could use a string that identifies each level. If they are
pressure
> levels, you could do:
>
> FCST_VAR1_OPTIONS = set_attr_level = "P750";
> FCST_VAR2_OPTIONS = set_attr_level = "P500";
> FCST_VAR3_OPTIONS = set_attr_level = "P250";
>
> Whatever you decide to use, you should use the same values when
configuring
> StatAnalysis:
>
> FCST_LEVEL_LIST = 0, 1, 2
> or
> FCST_LEVEL_LIST = P750, P500, P250
>
> Let me know if you have any questions or if that solution doesn't
work for
> you.
>
> Thanks,
> George
>
>
> On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > George,
> > I am not able to figure out how to set up GridStat to have a
correct
> > attribute for StatAnalysis to handle it.
> > Attached are the confs I am using. Where should I add the option
you
> > mentioned and in what form?
> > I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> > What is  "something" ?
> > Thanks,
> > Mariusz
> >
> >
> >
> > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Mariusz,
> > >
> > > You are correct that StatAnalysis cannot use that syntax to
specify the
> > > levels in this case. Fortunately there is a recently added
feature that
> > > will allow you to get around this issue. You can specify values
to
> > override
> > > attributes in GridStat. This will allow you to change the level
value
> > > 20160601_060000,0,*,* to a value that is more easily specified
in the
> > > StatAnalysis config file. GitHub issue #1020 outlines these
changes and
> > > this link takes you directly to a comment that lists the new
> > > configurations:
> > >
> > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> > >
> > > You will want to set set_attr_level in GridStat, which can be
added via
> > > METplus with:
> > >
> > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> > >
> > > I don't have access to Orion yet so I can't look at your
configuration
> to
> > > know what exactly "something" should be. If you need help
figuring that
> > > out, you could attach your METplus config file to this ticket
and I
> could
> > > take a look.
> > >
> > > These changes are available since MET 9.1-beta2 and will be in
MET 9.1,
> > > which is close to being released. It looks like the beta release
is not
> > > installed on Orion but 9.1 is planned to be installed there.
> > >
> > > Let me know if you have any other questions.
> > >
> > > Thanks,
> > > George
> > >
> > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > > >
> > > > George,
> > > > actually, there is a problem with the next step: StatAnalysis.
> > > > GridStat outputs time levels in the form 20160601_060000,0,*,*
> > > > but StatAnalysis does not accept
> > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > so ends up either producing an error or empty files depending
how the
> > > > config is set up.
> > > > Is there a solution to this problem?
> > > > Thanks,
> > > > Mariusz
> > > >
> > > > on Orion:
> > > > script
> > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > ...
> > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > >     > ./StatAnalysis.conf
> > > >
> > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal error
> > > > occurred
> > > >
> > > > Outputs in
> > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > >
> > > >
> > > >
> > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate <
> > > > mariusz.pagowski at noaa.gov> wrote:
> > > >
> > > > > Thanks, George, works as promised,
> > > > > Mariusz
> > > > >
> > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Mariusz,
> > > > >>
> > > > >> To specify the time in the NetCDF structure, you need to
use the
> > > format
> > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
> > > > >>
> > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > >>
> > > > >> If the NetCDF time info is formatted in the way the MET
tools
> > expect,
> > > > the
> > > > >> code is smart enough to find the correct data from the time
info.
> > > > >>
> > > > >> If that doesn't work, you may need to do something clever
to
> > > manipulate
> > > > >> the
> > > > >> hour value to the corresponding number of the NetCDF
dimension. It
> > > looks
> > > > >> like your data has a time dimension of size 4, so passing
in the
> > hour
> > > > (18)
> > > > >> is outside of the valid range. I would recommend trying to
format
> > the
> > > > >> valid
> > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me
know and I
> > try
> > > > to
> > > > >> help you figure out a way to get the correct number for the
time
> you
> > > > need.
> > > > >>
> > > > >> Thanks,
> > > > >> George
> > > > >>
> > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > > > >> met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >> >
> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > >> >
> > > > >> > Julie,
> > > > >> > It was hanging over my head but only today I tried to
input
> > forecast
> > > > >> time
> > > > >> > to the conf and did not figure the solution.
> > > > >> > valid?fmt string does not produce for me a correct time
and
> > GridStat
> > > > >> fails.
> > > > >> > If your colleague looked into our problem
> > > > >> > we would very much appreciate it, thanks,
> > > > >> > Mariusz
> > > > >> >
> > > > >> > models for GridStat processing
> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > >> >
> > > > >> > other stuff:
> > > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > >> >
> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > >> >
> > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > > > >> met_help at ucar.edu
> > > > >> > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Mariusz.
> > > > >> > >
> > > > >> > > I moved your question over to met_help so that others
could
> > > benefit
> > > > >> from
> > > > >> > > your questions.
> > > > >> > >
> > > > >> > > I asked a co-worker for some help with this one and he
said
> that
> > > if
> > > > >> the
> > > > >> > > data is in NetCDF and in a format that MET can read,
you can
> > > specify
> > > > >> the
> > > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There
> is a
> > > > >> METplus
> > > > >> > > use case that does this here:
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > >> > >
> > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> > > > >> > >
> > > > >> > > He said that likely won't work with this data, but
there is
> > > likely a
> > > > >> > > clever way to use the valid time to create the
appropriate
> level
> > > > value
> > > > >> > for
> > > > >> > > each run time in a similar way. If you aren't able to
figure
> it
> > > out,
> > > > >> you
> > > > >> > > could provide a sample dataset and he could help figure
out
> how
> > to
> > > > >> > > configure it.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Julie
> > > > >> > >
> > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > >> > > > My start date for GridStat is e.g. 2016060118 for
FCSTs and
> > > OBSs.
> > > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z,
18Z) and
> > OBSs
> > > at
> > > > >> 3-
> > > > >> > > > hr intervals. For this time GridStat.conf is
configured to
> > take
> > > > 3rd
> > > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*)
and  6th
> > OBS
> > > > in
> > > > >> > > > the other input file: (6,1,*,*) That only works
properly for
> > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours
when
> > > running
> > > > >> > > > over e.g. a month-long period. However, if I wanted
to run
> > > > GridStat
> > > > >> to
> > > > >> > > > get statistics with data at intervals of 6 hours e.g.
> > > > >> SDATE=2016060118
> > > > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
> (3,1,*,*)
> > > > needs
> > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass times
of
> > > > FCSTs/OBSs
> > > > >> to
> > > > >> > > > GridStat.conf when running master_metplus.py ?
> > > > >> > > >
> > > > >> > > > Thanks again,
> > > > >> > > > Mariusz
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >> --
> > > > >> George McCabe - Software Engineer III
> > > > >> National Center for Atmospheric Research
> > > > >> Research Applications Laboratory
> > > > >> 303-497-2768
> > > > >> ---
> > > > >> My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > >> reply to this email outside of your normal working hours.
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Wed Aug 26 17:36:48 2020

Yes, if you put them on hera and send me the path, I can take a look.

Thanks,
George

On Wed, Aug 26, 2020 at 5:24 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> George,
> I configured it the way you suggested but GridStat produced the same
output
> as before i.e. in SL1L2 a string with time in format %Y%m%d_%H%M%S
and
> after that StatAnalysis with empty files.
> The issue is that I have two sorts of files - one with a single
forecast at
> a time and another one with four fcsts daily.
> Maybe I misunderstood something. If I put files on Hera - would you
be able
> to look at them and the scripts?
> Thanks,
> Mariusz
>
>
>
>
>
>
>
> On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Mariusz,
> >
> > That is a good question. It looks like you have multiple levels
specified
> > for *_VAR1_LEVELS.
> >
> > FCST_VAR1_LEVELS =
> >
> >
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
> >
> > If you were to add FCST_VAR1_OPTIONS, it would apply to all of
those
> > fields, so all of those fields would contain the same value in the
> FCST_LEV
> > column of the GridStat output. If you need to specify a different
> FCST_LEV
> > value for each of those levels, you would have to split them up
into
> > separate VAR<n> items, like this:
> >
> > FCST_VAR1_NAME = DUSTTOTAL
> > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> >
> > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> >
> > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> >
> > etc. and do the same for the OBS_VAR<n>_* variables.
> >
> > I'm not sure what each of the levels contain, so it is hard to
know what
> to
> > name them. You really just need to use a unique identifier so that
they
> can
> > be grouped via StatAnalysis. You could set them to the number
> corresponding
> > to each NetCDF item:
> >
> > FCST_VAR1_NAME = DUSTTOTAL
> > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > FCST_VAR1_OPTIONS = set_attr_level = "0";
> >
> > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > FCST_VAR2_OPTIONS = set_attr_level = "1";
> >
> > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > FCST_VAR3_OPTIONS = set_attr_level = "2";
> >
> > Or you could use a string that identifies each level. If they are
> pressure
> > levels, you could do:
> >
> > FCST_VAR1_OPTIONS = set_attr_level = "P750";
> > FCST_VAR2_OPTIONS = set_attr_level = "P500";
> > FCST_VAR3_OPTIONS = set_attr_level = "P250";
> >
> > Whatever you decide to use, you should use the same values when
> configuring
> > StatAnalysis:
> >
> > FCST_LEVEL_LIST = 0, 1, 2
> > or
> > FCST_LEVEL_LIST = P750, P500, P250
> >
> > Let me know if you have any questions or if that solution doesn't
work
> for
> > you.
> >
> > Thanks,
> > George
> >
> >
> > On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > George,
> > > I am not able to figure out how to set up GridStat to have a
correct
> > > attribute for StatAnalysis to handle it.
> > > Attached are the confs I am using. Where should I add the option
you
> > > mentioned and in what form?
> > > I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> > > What is  "something" ?
> > > Thanks,
> > > Mariusz
> > >
> > >
> > >
> > > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Mariusz,
> > > >
> > > > You are correct that StatAnalysis cannot use that syntax to
specify
> the
> > > > levels in this case. Fortunately there is a recently added
feature
> that
> > > > will allow you to get around this issue. You can specify
values to
> > > override
> > > > attributes in GridStat. This will allow you to change the
level value
> > > > 20160601_060000,0,*,* to a value that is more easily specified
in the
> > > > StatAnalysis config file. GitHub issue #1020 outlines these
changes
> and
> > > > this link takes you directly to a comment that lists the new
> > > > configurations:
> > > >
> > > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> > > >
> > > > You will want to set set_attr_level in GridStat, which can be
added
> via
> > > > METplus with:
> > > >
> > > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> > > >
> > > > I don't have access to Orion yet so I can't look at your
> configuration
> > to
> > > > know what exactly "something" should be. If you need help
figuring
> that
> > > > out, you could attach your METplus config file to this ticket
and I
> > could
> > > > take a look.
> > > >
> > > > These changes are available since MET 9.1-beta2 and will be in
MET
> 9.1,
> > > > which is close to being released. It looks like the beta
release is
> not
> > > > installed on Orion but 9.1 is planned to be installed there.
> > > >
> > > > Let me know if you have any other questions.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > >
> > > > > George,
> > > > > actually, there is a problem with the next step:
StatAnalysis.
> > > > > GridStat outputs time levels in the form
20160601_060000,0,*,*
> > > > > but StatAnalysis does not accept
> > > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > > so ends up either producing an error or empty files
depending how
> the
> > > > > config is set up.
> > > > > Is there a solution to this problem?
> > > > > Thanks,
> > > > > Mariusz
> > > > >
> > > > > on Orion:
> > > > > script
> > > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > > ...
> > > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > >     > ./StatAnalysis.conf
> > > > >
> > > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal
> error
> > > > > occurred
> > > > >
> > > > > Outputs in
> > > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate
> <
> > > > > mariusz.pagowski at noaa.gov> wrote:
> > > > >
> > > > > > Thanks, George, works as promised,
> > > > > > Mariusz
> > > > > >
> > > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Mariusz,
> > > > > >>
> > > > > >> To specify the time in the NetCDF structure, you need to
use the
> > > > format
> > > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
> > > > > >>
> > > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > > >>
> > > > > >> If the NetCDF time info is formatted in the way the MET
tools
> > > expect,
> > > > > the
> > > > > >> code is smart enough to find the correct data from the
time
> info.
> > > > > >>
> > > > > >> If that doesn't work, you may need to do something clever
to
> > > > manipulate
> > > > > >> the
> > > > > >> hour value to the corresponding number of the NetCDF
dimension.
> It
> > > > looks
> > > > > >> like your data has a time dimension of size 4, so passing
in the
> > > hour
> > > > > (18)
> > > > > >> is outside of the valid range. I would recommend trying
to
> format
> > > the
> > > > > >> valid
> > > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me
know
> and I
> > > try
> > > > > to
> > > > > >> help you figure out a way to get the correct number for
the time
> > you
> > > > > need.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> George
> > > > > >>
> > > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > > > > >> met_help at ucar.edu>
> > > > > >> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> >
> > > > > >> >
> > > > > >> > Julie,
> > > > > >> > It was hanging over my head but only today I tried to
input
> > > forecast
> > > > > >> time
> > > > > >> > to the conf and did not figure the solution.
> > > > > >> > valid?fmt string does not produce for me a correct time
and
> > > GridStat
> > > > > >> fails.
> > > > > >> > If your colleague looked into our problem
> > > > > >> > we would very much appreciate it, thanks,
> > > > > >> > Mariusz
> > > > > >> >
> > > > > >> > models for GridStat processing
> > > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
> > > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > >> >
> > > > > >> > other stuff:
> > > > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > > >> >
> > > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > > >> >
> > > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT
<
> > > > > >> met_help at ucar.edu
> > > > > >> > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi Mariusz.
> > > > > >> > >
> > > > > >> > > I moved your question over to met_help so that others
could
> > > > benefit
> > > > > >> from
> > > > > >> > > your questions.
> > > > > >> > >
> > > > > >> > > I asked a co-worker for some help with this one and
he said
> > that
> > > > if
> > > > > >> the
> > > > > >> > > data is in NetCDF and in a format that MET can read,
you can
> > > > specify
> > > > > >> the
> > > > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There
> > is a
> > > > > >> METplus
> > > > > >> > > use case that does this here:
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > > >> > >
> > > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
> "({valid?fmt=%Y%m01_000000},*,*)"
> > > > > >> > >
> > > > > >> > > He said that likely won't work with this data, but
there is
> > > > likely a
> > > > > >> > > clever way to use the valid time to create the
appropriate
> > level
> > > > > value
> > > > > >> > for
> > > > > >> > > each run time in a similar way. If you aren't able to
figure
> > it
> > > > out,
> > > > > >> you
> > > > > >> > > could provide a sample dataset and he could help
figure out
> > how
> > > to
> > > > > >> > > configure it.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Julie
> > > > > >> > >
> > > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > > >> > > > My start date for GridStat is e.g. 2016060118 for
FCSTs
> and
> > > > OBSs.
> > > > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z,
18Z) and
> > > OBSs
> > > > at
> > > > > >> 3-
> > > > > >> > > > hr intervals. For this time GridStat.conf is
configured to
> > > take
> > > > > 3rd
> > > > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*)
and
> 6th
> > > OBS
> > > > > in
> > > > > >> > > > the other input file: (6,1,*,*) That only works
properly
> for
> > > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24
hours when
> > > > running
> > > > > >> > > > over e.g. a month-long period. However, if I wanted
to run
> > > > > GridStat
> > > > > >> to
> > > > > >> > > > get statistics with data at intervals of 6 hours
e.g.
> > > > > >> SDATE=2016060118
> > > > > >> > > > and EDATE=2016063118 that is incorrect since for
00Z
> > (3,1,*,*)
> > > > > needs
> > > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass
times of
> > > > > FCSTs/OBSs
> > > > > >> to
> > > > > >> > > > GridStat.conf when running master_metplus.py ?
> > > > > >> > > >
> > > > > >> > > > Thanks again,
> > > > > >> > > > Mariusz
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >> --
> > > > > >> George McCabe - Software Engineer III
> > > > > >> National Center for Atmospheric Research
> > > > > >> Research Applications Laboratory
> > > > > >> 303-497-2768
> > > > > >> ---
> > > > > >> My working day may not be your working day. Please do not
feel
> > > obliged
> > > > > to
> > > > > >> reply to this email outside of your normal working hours.
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Wed Aug 26 17:54:47 2020

Thanks, the scripts are
/home/Mariusz.Pagowski/mapp_2018/scripts/met/
met_grid_stat.sh
met_stat_anal.sh
with locations of config directories in the same location.

Models are in
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll

Mariusz




On Wed, Aug 26, 2020 at 5:36 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Yes, if you put them on hera and send me the path, I can take a
look.
>
> Thanks,
> George
>
> On Wed, Aug 26, 2020 at 5:24 PM Mariusz Pagowski via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > George,
> > I configured it the way you suggested but GridStat produced the
same
> output
> > as before i.e. in SL1L2 a string with time in format %Y%m%d_%H%M%S
and
> > after that StatAnalysis with empty files.
> > The issue is that I have two sorts of files - one with a single
forecast
> at
> > a time and another one with four fcsts daily.
> > Maybe I misunderstood something. If I put files on Hera - would
you be
> able
> > to look at them and the scripts?
> > Thanks,
> > Mariusz
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Mariusz,
> > >
> > > That is a good question. It looks like you have multiple levels
> specified
> > > for *_VAR1_LEVELS.
> > >
> > > FCST_VAR1_LEVELS =
> > >
> > >
> >
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
> > >
> > > If you were to add FCST_VAR1_OPTIONS, it would apply to all of
those
> > > fields, so all of those fields would contain the same value in
the
> > FCST_LEV
> > > column of the GridStat output. If you need to specify a
different
> > FCST_LEV
> > > value for each of those levels, you would have to split them up
into
> > > separate VAR<n> items, like this:
> > >
> > > FCST_VAR1_NAME = DUSTTOTAL
> > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > >
> > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > >
> > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > >
> > > etc. and do the same for the OBS_VAR<n>_* variables.
> > >
> > > I'm not sure what each of the levels contain, so it is hard to
know
> what
> > to
> > > name them. You really just need to use a unique identifier so
that they
> > can
> > > be grouped via StatAnalysis. You could set them to the number
> > corresponding
> > > to each NetCDF item:
> > >
> > > FCST_VAR1_NAME = DUSTTOTAL
> > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > FCST_VAR1_OPTIONS = set_attr_level = "0";
> > >
> > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > FCST_VAR2_OPTIONS = set_attr_level = "1";
> > >
> > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > FCST_VAR3_OPTIONS = set_attr_level = "2";
> > >
> > > Or you could use a string that identifies each level. If they
are
> > pressure
> > > levels, you could do:
> > >
> > > FCST_VAR1_OPTIONS = set_attr_level = "P750";
> > > FCST_VAR2_OPTIONS = set_attr_level = "P500";
> > > FCST_VAR3_OPTIONS = set_attr_level = "P250";
> > >
> > > Whatever you decide to use, you should use the same values when
> > configuring
> > > StatAnalysis:
> > >
> > > FCST_LEVEL_LIST = 0, 1, 2
> > > or
> > > FCST_LEVEL_LIST = P750, P500, P250
> > >
> > > Let me know if you have any questions or if that solution
doesn't work
> > for
> > > you.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > > On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > > >
> > > > George,
> > > > I am not able to figure out how to set up GridStat to have a
correct
> > > > attribute for StatAnalysis to handle it.
> > > > Attached are the confs I am using. Where should I add the
option you
> > > > mentioned and in what form?
> > > > I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
"something";
> > > > What is  "something" ?
> > > > Thanks,
> > > > Mariusz
> > > >
> > > >
> > > >
> > > > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > Hi Mariusz,
> > > > >
> > > > > You are correct that StatAnalysis cannot use that syntax to
specify
> > the
> > > > > levels in this case. Fortunately there is a recently added
feature
> > that
> > > > > will allow you to get around this issue. You can specify
values to
> > > > override
> > > > > attributes in GridStat. This will allow you to change the
level
> value
> > > > > 20160601_060000,0,*,* to a value that is more easily
specified in
> the
> > > > > StatAnalysis config file. GitHub issue #1020 outlines these
changes
> > and
> > > > > this link takes you directly to a comment that lists the new
> > > > > configurations:
> > > > >
> > > > > https://github.com/NCAR/MET/issues/1020#issuecomment-
653724572
> > > > >
> > > > > You will want to set set_attr_level in GridStat, which can
be added
> > via
> > > > > METplus with:
> > > > >
> > > > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> > > > >
> > > > > I don't have access to Orion yet so I can't look at your
> > configuration
> > > to
> > > > > know what exactly "something" should be. If you need help
figuring
> > that
> > > > > out, you could attach your METplus config file to this
ticket and I
> > > could
> > > > > take a look.
> > > > >
> > > > > These changes are available since MET 9.1-beta2 and will be
in MET
> > 9.1,
> > > > > which is close to being released. It looks like the beta
release is
> > not
> > > > > installed on Orion but 9.1 is planned to be installed there.
> > > > >
> > > > > Let me know if you have any other questions.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > > >
> > > > > > George,
> > > > > > actually, there is a problem with the next step:
StatAnalysis.
> > > > > > GridStat outputs time levels in the form
20160601_060000,0,*,*
> > > > > > but StatAnalysis does not accept
> > > > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > so ends up either producing an error or empty files
depending how
> > the
> > > > > > config is set up.
> > > > > > Is there a solution to this problem?
> > > > > > Thanks,
> > > > > > Mariusz
> > > > > >
> > > > > > on Orion:
> > > > > > script
> > > > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > > > ...
> > > > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > >     > ./StatAnalysis.conf
> > > > > >
> > > > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal
> > error
> > > > > > occurred
> > > > > >
> > > > > > Outputs in
> > > > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
> Affiliate
> > <
> > > > > > mariusz.pagowski at noaa.gov> wrote:
> > > > > >
> > > > > > > Thanks, George, works as promised,
> > > > > > > Mariusz
> > > > > > >
> > > > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Mariusz,
> > > > > > >>
> > > > > > >> To specify the time in the NetCDF structure, you need
to use
> the
> > > > > format
> > > > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look
like this:
> > > > > > >>
> > > > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > >>
> > > > > > >> If the NetCDF time info is formatted in the way the MET
tools
> > > > expect,
> > > > > > the
> > > > > > >> code is smart enough to find the correct data from the
time
> > info.
> > > > > > >>
> > > > > > >> If that doesn't work, you may need to do something
clever to
> > > > > manipulate
> > > > > > >> the
> > > > > > >> hour value to the corresponding number of the NetCDF
> dimension.
> > It
> > > > > looks
> > > > > > >> like your data has a time dimension of size 4, so
passing in
> the
> > > > hour
> > > > > > (18)
> > > > > > >> is outside of the valid range. I would recommend trying
to
> > format
> > > > the
> > > > > > >> valid
> > > > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me
know
> > and I
> > > > try
> > > > > > to
> > > > > > >> help you figure out a way to get the correct number for
the
> time
> > > you
> > > > > > need.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> George
> > > > > > >>
> > > > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT
<
> > > > > > >> met_help at ucar.edu>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> >
> > > > > > >> > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > >
> > > > > > >> >
> > > > > > >> > Julie,
> > > > > > >> > It was hanging over my head but only today I tried to
input
> > > > forecast
> > > > > > >> time
> > > > > > >> > to the conf and did not figure the solution.
> > > > > > >> > valid?fmt string does not produce for me a correct
time and
> > > > GridStat
> > > > > > >> fails.
> > > > > > >> > If your colleague looked into our problem
> > > > > > >> > we would very much appreciate it, thanks,
> > > > > > >> > Mariusz
> > > > > > >> >
> > > > > > >> > models for GridStat processing
> > > > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > >> >
> > > > > > >> > other stuff:
> > > > > > >> >
> /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > > > >> >
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > > > >> >
> > > > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > > > >> >
> > > > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via
RT <
> > > > > > >> met_help at ucar.edu
> > > > > > >> > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Mariusz.
> > > > > > >> > >
> > > > > > >> > > I moved your question over to met_help so that
others
> could
> > > > > benefit
> > > > > > >> from
> > > > > > >> > > your questions.
> > > > > > >> > >
> > > > > > >> > > I asked a co-worker for some help with this one and
he
> said
> > > that
> > > > > if
> > > > > > >> the
> > > > > > >> > > data is in NetCDF and in a format that MET can
read, you
> can
> > > > > specify
> > > > > > >> the
> > > > > > >> > > time to use in the level, i.e.
(YYYYMMDD_HHMMSS,*,*).
> There
> > > is a
> > > > > > >> METplus
> > > > > > >> > > use case that does this here:
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > > > >> > >
> > > > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
> > "({valid?fmt=%Y%m01_000000},*,*)"
> > > > > > >> > >
> > > > > > >> > > He said that likely won't work with this data, but
there
> is
> > > > > likely a
> > > > > > >> > > clever way to use the valid time to create the
appropriate
> > > level
> > > > > > value
> > > > > > >> > for
> > > > > > >> > > each run time in a similar way. If you aren't able
to
> figure
> > > it
> > > > > out,
> > > > > > >> you
> > > > > > >> > > could provide a sample dataset and he could help
figure
> out
> > > how
> > > > to
> > > > > > >> > > configure it.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Julie
> > > > > > >> > >
> > > > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > > > >> > > > My start date for GridStat is e.g. 2016060118 for
FCSTs
> > and
> > > > > OBSs.
> > > > > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z,
18Z)
> and
> > > > OBSs
> > > > > at
> > > > > > >> 3-
> > > > > > >> > > > hr intervals. For this time GridStat.conf is
configured
> to
> > > > take
> > > > > > 3rd
> > > > > > >> > > > FCST in the input file [in GridStat.conf:
(3,1,*,*) and
> > 6th
> > > > OBS
> > > > > > in
> > > > > > >> > > > the other input file: (6,1,*,*) That only works
properly
> > for
> > > > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24
hours
> when
> > > > > running
> > > > > > >> > > > over e.g. a month-long period. However, if I
wanted to
> run
> > > > > > GridStat
> > > > > > >> to
> > > > > > >> > > > get statistics with data at intervals of 6 hours
e.g.
> > > > > > >> SDATE=2016060118
> > > > > > >> > > > and EDATE=2016063118 that is incorrect since for
00Z
> > > (3,1,*,*)
> > > > > > needs
> > > > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass
times of
> > > > > > FCSTs/OBSs
> > > > > > >> to
> > > > > > >> > > > GridStat.conf when running master_metplus.py ?
> > > > > > >> > > >
> > > > > > >> > > > Thanks again,
> > > > > > >> > > > Mariusz
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >> --
> > > > > > >> George McCabe - Software Engineer III
> > > > > > >> National Center for Atmospheric Research
> > > > > > >> Research Applications Laboratory
> > > > > > >> 303-497-2768
> > > > > > >> ---
> > > > > > >> My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > > to
> > > > > > >> reply to this email outside of your normal working
hours.
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Wed Aug 26 18:10:44 2020

Hi Mariusz,

Which GridStat conf file did you modify? I don't see any files that
have
the FCST_VAR<n>_OPTIONS set.

- George

On Wed, Aug 26, 2020 at 5:54 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> Thanks, the scripts are
> /home/Mariusz.Pagowski/mapp_2018/scripts/met/
> met_grid_stat.sh
> met_stat_anal.sh
> with locations of config directories in the same location.
>
> Models are in
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll
>
> Mariusz
>
>
>
>
> On Wed, Aug 26, 2020 at 5:36 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Yes, if you put them on hera and send me the path, I can take a
look.
> >
> > Thanks,
> > George
> >
> > On Wed, Aug 26, 2020 at 5:24 PM Mariusz Pagowski via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > George,
> > > I configured it the way you suggested but GridStat produced the
same
> > output
> > > as before i.e. in SL1L2 a string with time in format
%Y%m%d_%H%M%S and
> > > after that StatAnalysis with empty files.
> > > The issue is that I have two sorts of files - one with a single
> forecast
> > at
> > > a time and another one with four fcsts daily.
> > > Maybe I misunderstood something. If I put files on Hera - would
you be
> > able
> > > to look at them and the scripts?
> > > Thanks,
> > > Mariusz
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Mariusz,
> > > >
> > > > That is a good question. It looks like you have multiple
levels
> > specified
> > > > for *_VAR1_LEVELS.
> > > >
> > > > FCST_VAR1_LEVELS =
> > > >
> > > >
> > >
> >
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
> > > >
> > > > If you were to add FCST_VAR1_OPTIONS, it would apply to all of
those
> > > > fields, so all of those fields would contain the same value in
the
> > > FCST_LEV
> > > > column of the GridStat output. If you need to specify a
different
> > > FCST_LEV
> > > > value for each of those levels, you would have to split them
up into
> > > > separate VAR<n> items, like this:
> > > >
> > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > >
> > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > >
> > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > >
> > > > etc. and do the same for the OBS_VAR<n>_* variables.
> > > >
> > > > I'm not sure what each of the levels contain, so it is hard to
know
> > what
> > > to
> > > > name them. You really just need to use a unique identifier so
that
> they
> > > can
> > > > be grouped via StatAnalysis. You could set them to the number
> > > corresponding
> > > > to each NetCDF item:
> > > >
> > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > FCST_VAR1_OPTIONS = set_attr_level = "0";
> > > >
> > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > FCST_VAR2_OPTIONS = set_attr_level = "1";
> > > >
> > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > FCST_VAR3_OPTIONS = set_attr_level = "2";
> > > >
> > > > Or you could use a string that identifies each level. If they
are
> > > pressure
> > > > levels, you could do:
> > > >
> > > > FCST_VAR1_OPTIONS = set_attr_level = "P750";
> > > > FCST_VAR2_OPTIONS = set_attr_level = "P500";
> > > > FCST_VAR3_OPTIONS = set_attr_level = "P250";
> > > >
> > > > Whatever you decide to use, you should use the same values
when
> > > configuring
> > > > StatAnalysis:
> > > >
> > > > FCST_LEVEL_LIST = 0, 1, 2
> > > > or
> > > > FCST_LEVEL_LIST = P750, P500, P250
> > > >
> > > > Let me know if you have any questions or if that solution
doesn't
> work
> > > for
> > > > you.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > > On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > >
> > > > > George,
> > > > > I am not able to figure out how to set up GridStat to have a
> correct
> > > > > attribute for StatAnalysis to handle it.
> > > > > Attached are the confs I am using. Where should I add the
option
> you
> > > > > mentioned and in what form?
> > > > > I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
"something";
> > > > > What is  "something" ?
> > > > > Thanks,
> > > > > Mariusz
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Mariusz,
> > > > > >
> > > > > > You are correct that StatAnalysis cannot use that syntax
to
> specify
> > > the
> > > > > > levels in this case. Fortunately there is a recently added
> feature
> > > that
> > > > > > will allow you to get around this issue. You can specify
values
> to
> > > > > override
> > > > > > attributes in GridStat. This will allow you to change the
level
> > value
> > > > > > 20160601_060000,0,*,* to a value that is more easily
specified in
> > the
> > > > > > StatAnalysis config file. GitHub issue #1020 outlines
these
> changes
> > > and
> > > > > > this link takes you directly to a comment that lists the
new
> > > > > > configurations:
> > > > > >
> > > > > > https://github.com/NCAR/MET/issues/1020#issuecomment-
653724572
> > > > > >
> > > > > > You will want to set set_attr_level in GridStat, which can
be
> added
> > > via
> > > > > > METplus with:
> > > > > >
> > > > > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
"something";
> > > > > >
> > > > > > I don't have access to Orion yet so I can't look at your
> > > configuration
> > > > to
> > > > > > know what exactly "something" should be. If you need help
> figuring
> > > that
> > > > > > out, you could attach your METplus config file to this
ticket
> and I
> > > > could
> > > > > > take a look.
> > > > > >
> > > > > > These changes are available since MET 9.1-beta2 and will
be in
> MET
> > > 9.1,
> > > > > > which is close to being released. It looks like the beta
release
> is
> > > not
> > > > > > installed on Orion but 9.1 is planned to be installed
there.
> > > > > >
> > > > > > Let me know if you have any other questions.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> >
> > > > > > >
> > > > > > > George,
> > > > > > > actually, there is a problem with the next step:
StatAnalysis.
> > > > > > > GridStat outputs time levels in the form
20160601_060000,0,*,*
> > > > > > > but StatAnalysis does not accept
> > > > > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > so ends up either producing an error or empty files
depending
> how
> > > the
> > > > > > > config is set up.
> > > > > > > Is there a solution to this problem?
> > > > > > > Thanks,
> > > > > > > Mariusz
> > > > > > >
> > > > > > > on Orion:
> > > > > > > script
> > > > > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > > > > ...
> > > > > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > >     > ./StatAnalysis.conf
> > > > > > >
> > > > > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal
> > > error
> > > > > > > occurred
> > > > > > >
> > > > > > > Outputs in
> > > > > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
> > Affiliate
> > > <
> > > > > > > mariusz.pagowski at noaa.gov> wrote:
> > > > > > >
> > > > > > > > Thanks, George, works as promised,
> > > > > > > > Mariusz
> > > > > > > >
> > > > > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Mariusz,
> > > > > > > >>
> > > > > > > >> To specify the time in the NetCDF structure, you need
to use
> > the
> > > > > > format
> > > > > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look
like
> this:
> > > > > > > >>
> > > > > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > >>
> > > > > > > >> If the NetCDF time info is formatted in the way the
MET
> tools
> > > > > expect,
> > > > > > > the
> > > > > > > >> code is smart enough to find the correct data from
the time
> > > info.
> > > > > > > >>
> > > > > > > >> If that doesn't work, you may need to do something
clever to
> > > > > > manipulate
> > > > > > > >> the
> > > > > > > >> hour value to the corresponding number of the NetCDF
> > dimension.
> > > It
> > > > > > looks
> > > > > > > >> like your data has a time dimension of size 4, so
passing in
> > the
> > > > > hour
> > > > > > > (18)
> > > > > > > >> is outside of the valid range. I would recommend
trying to
> > > format
> > > > > the
> > > > > > > >> valid
> > > > > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let
me know
> > > and I
> > > > > try
> > > > > > > to
> > > > > > > >> help you figure out a way to get the correct number
for the
> > time
> > > > you
> > > > > > > need.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> George
> > > > > > > >>
> > > > > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via
RT <
> > > > > > > >> met_help at ucar.edu>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > >
> > > > > > > >> >
> > > > > > > >> > Julie,
> > > > > > > >> > It was hanging over my head but only today I tried
to
> input
> > > > > forecast
> > > > > > > >> time
> > > > > > > >> > to the conf and did not figure the solution.
> > > > > > > >> > valid?fmt string does not produce for me a correct
time
> and
> > > > > GridStat
> > > > > > > >> fails.
> > > > > > > >> > If your colleague looked into our problem
> > > > > > > >> > we would very much appreciate it, thanks,
> > > > > > > >> > Mariusz
> > > > > > > >> >
> > > > > > > >> > models for GridStat processing
> > > > > > > >> >
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > > > > >> >
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > > > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > > >> >
> > > > > > > >> > other stuff:
> > > > > > > >> >
> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > > > > >> >
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > > > > >> >
> > > > >
> /scratch1/BMC/gsd-fv3-dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > > > > >> >
> > > > > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik
via RT <
> > > > > > > >> met_help at ucar.edu
> > > > > > > >> > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Mariusz.
> > > > > > > >> > >
> > > > > > > >> > > I moved your question over to met_help so that
others
> > could
> > > > > > benefit
> > > > > > > >> from
> > > > > > > >> > > your questions.
> > > > > > > >> > >
> > > > > > > >> > > I asked a co-worker for some help with this one
and he
> > said
> > > > that
> > > > > > if
> > > > > > > >> the
> > > > > > > >> > > data is in NetCDF and in a format that MET can
read, you
> > can
> > > > > > specify
> > > > > > > >> the
> > > > > > > >> > > time to use in the level, i.e.
(YYYYMMDD_HHMMSS,*,*).
> > There
> > > > is a
> > > > > > > >> METplus
> > > > > > > >> > > use case that does this here:
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > > > > >> > >
> > > > > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
> > > "({valid?fmt=%Y%m01_000000},*,*)"
> > > > > > > >> > >
> > > > > > > >> > > He said that likely won't work with this data,
but there
> > is
> > > > > > likely a
> > > > > > > >> > > clever way to use the valid time to create the
> appropriate
> > > > level
> > > > > > > value
> > > > > > > >> > for
> > > > > > > >> > > each run time in a similar way. If you aren't
able to
> > figure
> > > > it
> > > > > > out,
> > > > > > > >> you
> > > > > > > >> > > could provide a sample dataset and he could help
figure
> > out
> > > > how
> > > > > to
> > > > > > > >> > > configure it.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Julie
> > > > > > > >> > >
> > > > > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > > > > >> > > > My start date for GridStat is e.g. 2016060118
for
> FCSTs
> > > and
> > > > > > OBSs.
> > > > > > > >> > > > FCSTs are output at 6-hr intervals (00Z,
06Z,12Z, 18Z)
> > and
> > > > > OBSs
> > > > > > at
> > > > > > > >> 3-
> > > > > > > >> > > > hr intervals. For this time GridStat.conf is
> configured
> > to
> > > > > take
> > > > > > > 3rd
> > > > > > > >> > > > FCST in the input file [in GridStat.conf:
(3,1,*,*)
> and
> > > 6th
> > > > > OBS
> > > > > > > in
> > > > > > > >> > > > the other input file: (6,1,*,*) That only works
> properly
> > > for
> > > > > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24
hours
> > when
> > > > > > running
> > > > > > > >> > > > over e.g. a month-long period. However, if I
wanted to
> > run
> > > > > > > GridStat
> > > > > > > >> to
> > > > > > > >> > > > get statistics with data at intervals of 6
hours e.g.
> > > > > > > >> SDATE=2016060118
> > > > > > > >> > > > and EDATE=2016063118 that is incorrect since
for 00Z
> > > > (3,1,*,*)
> > > > > > > needs
> > > > > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass
times
> of
> > > > > > > FCSTs/OBSs
> > > > > > > >> to
> > > > > > > >> > > > GridStat.conf when running master_metplus.py ?
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks again,
> > > > > > > >> > > > Mariusz
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> George McCabe - Software Engineer III
> > > > > > > >> National Center for Atmospheric Research
> > > > > > > >> Research Applications Laboratory
> > > > > > > >> 303-497-2768
> > > > > > > >> ---
> > > > > > > >> My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > > to
> > > > > > > >> reply to this email outside of your normal working
hours.
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Wed Aug 26 18:17:14 2020

Sorry, that was on Orion, now modified  similarly on Hera in
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
Mariusz


On Wed, Aug 26, 2020 at 6:10 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Mariusz,
>
> Which GridStat conf file did you modify? I don't see any files that
have
> the FCST_VAR<n>_OPTIONS set.
>
> - George
>
> On Wed, Aug 26, 2020 at 5:54 PM Mariusz Pagowski via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > Thanks, the scripts are
> > /home/Mariusz.Pagowski/mapp_2018/scripts/met/
> > met_grid_stat.sh
> > met_stat_anal.sh
> > with locations of config directories in the same location.
> >
> > Models are in
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll
> >
> > Mariusz
> >
> >
> >
> >
> > On Wed, Aug 26, 2020 at 5:36 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Yes, if you put them on hera and send me the path, I can take a
look.
> > >
> > > Thanks,
> > > George
> > >
> > > On Wed, Aug 26, 2020 at 5:24 PM Mariusz Pagowski via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > > >
> > > > George,
> > > > I configured it the way you suggested but GridStat produced
the same
> > > output
> > > > as before i.e. in SL1L2 a string with time in format
%Y%m%d_%H%M%S
> and
> > > > after that StatAnalysis with empty files.
> > > > The issue is that I have two sorts of files - one with a
single
> > forecast
> > > at
> > > > a time and another one with four fcsts daily.
> > > > Maybe I misunderstood something. If I put files on Hera -
would you
> be
> > > able
> > > > to look at them and the scripts?
> > > > Thanks,
> > > > Mariusz
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Mariusz,
> > > > >
> > > > > That is a good question. It looks like you have multiple
levels
> > > specified
> > > > > for *_VAR1_LEVELS.
> > > > >
> > > > > FCST_VAR1_LEVELS =
> > > > >
> > > > >
> > > >
> > >
> >
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
> > > > >
> > > > > If you were to add FCST_VAR1_OPTIONS, it would apply to all
of
> those
> > > > > fields, so all of those fields would contain the same value
in the
> > > > FCST_LEV
> > > > > column of the GridStat output. If you need to specify a
different
> > > > FCST_LEV
> > > > > value for each of those levels, you would have to split them
up
> into
> > > > > separate VAR<n> items, like this:
> > > > >
> > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > >
> > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > >
> > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > >
> > > > > etc. and do the same for the OBS_VAR<n>_* variables.
> > > > >
> > > > > I'm not sure what each of the levels contain, so it is hard
to know
> > > what
> > > > to
> > > > > name them. You really just need to use a unique identifier
so that
> > they
> > > > can
> > > > > be grouped via StatAnalysis. You could set them to the
number
> > > > corresponding
> > > > > to each NetCDF item:
> > > > >
> > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > FCST_VAR1_OPTIONS = set_attr_level = "0";
> > > > >
> > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > FCST_VAR2_OPTIONS = set_attr_level = "1";
> > > > >
> > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > FCST_VAR3_OPTIONS = set_attr_level = "2";
> > > > >
> > > > > Or you could use a string that identifies each level. If
they are
> > > > pressure
> > > > > levels, you could do:
> > > > >
> > > > > FCST_VAR1_OPTIONS = set_attr_level = "P750";
> > > > > FCST_VAR2_OPTIONS = set_attr_level = "P500";
> > > > > FCST_VAR3_OPTIONS = set_attr_level = "P250";
> > > > >
> > > > > Whatever you decide to use, you should use the same values
when
> > > > configuring
> > > > > StatAnalysis:
> > > > >
> > > > > FCST_LEVEL_LIST = 0, 1, 2
> > > > > or
> > > > > FCST_LEVEL_LIST = P750, P500, P250
> > > > >
> > > > > Let me know if you have any questions or if that solution
doesn't
> > work
> > > > for
> > > > > you.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > >
> > > > > On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > > >
> > > > > > George,
> > > > > > I am not able to figure out how to set up GridStat to have
a
> > correct
> > > > > > attribute for StatAnalysis to handle it.
> > > > > > Attached are the confs I am using. Where should I add the
option
> > you
> > > > > > mentioned and in what form?
> > > > > > I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
"something";
> > > > > > What is  "something" ?
> > > > > > Thanks,
> > > > > > Mariusz
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Mariusz,
> > > > > > >
> > > > > > > You are correct that StatAnalysis cannot use that syntax
to
> > specify
> > > > the
> > > > > > > levels in this case. Fortunately there is a recently
added
> > feature
> > > > that
> > > > > > > will allow you to get around this issue. You can specify
values
> > to
> > > > > > override
> > > > > > > attributes in GridStat. This will allow you to change
the level
> > > value
> > > > > > > 20160601_060000,0,*,* to a value that is more easily
specified
> in
> > > the
> > > > > > > StatAnalysis config file. GitHub issue #1020 outlines
these
> > changes
> > > > and
> > > > > > > this link takes you directly to a comment that lists the
new
> > > > > > > configurations:
> > > > > > >
> > > > > > > https://github.com/NCAR/MET/issues/1020#issuecomment-
653724572
> > > > > > >
> > > > > > > You will want to set set_attr_level in GridStat, which
can be
> > added
> > > > via
> > > > > > > METplus with:
> > > > > > >
> > > > > > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
"something";
> > > > > > >
> > > > > > > I don't have access to Orion yet so I can't look at your
> > > > configuration
> > > > > to
> > > > > > > know what exactly "something" should be. If you need
help
> > figuring
> > > > that
> > > > > > > out, you could attach your METplus config file to this
ticket
> > and I
> > > > > could
> > > > > > > take a look.
> > > > > > >
> > > > > > > These changes are available since MET 9.1-beta2 and will
be in
> > MET
> > > > 9.1,
> > > > > > > which is close to being released. It looks like the beta
> release
> > is
> > > > not
> > > > > > > installed on Orion but 9.1 is planned to be installed
there.
> > > > > > >
> > > > > > > Let me know if you have any other questions.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > >
> > > > > > > >
> > > > > > > > George,
> > > > > > > > actually, there is a problem with the next step:
> StatAnalysis.
> > > > > > > > GridStat outputs time levels in the form
> 20160601_060000,0,*,*
> > > > > > > > but StatAnalysis does not accept
> > > > > > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > so ends up either producing an error or empty files
depending
> > how
> > > > the
> > > > > > > > config is set up.
> > > > > > > > Is there a solution to this problem?
> > > > > > > > Thanks,
> > > > > > > > Mariusz
> > > > > > > >
> > > > > > > > on Orion:
> > > > > > > > script
> > > > > > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > > > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > > > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > > > > > ...
> > > > > > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > >     > ./StatAnalysis.conf
> > > > > > > >
> > > > > > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > > > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis:
ERROR:
> Fatal
> > > > error
> > > > > > > > occurred
> > > > > > > >
> > > > > > > > Outputs in
> > > > > > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski -
NOAA
> > > Affiliate
> > > > <
> > > > > > > > mariusz.pagowski at noaa.gov> wrote:
> > > > > > > >
> > > > > > > > > Thanks, George, works as promised,
> > > > > > > > > Mariusz
> > > > > > > > >
> > > > > > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi Mariusz,
> > > > > > > > >>
> > > > > > > > >> To specify the time in the NetCDF structure, you
need to
> use
> > > the
> > > > > > > format
> > > > > > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look
like
> > this:
> > > > > > > > >>
> > > > > > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > >>
> > > > > > > > >> If the NetCDF time info is formatted in the way the
MET
> > tools
> > > > > > expect,
> > > > > > > > the
> > > > > > > > >> code is smart enough to find the correct data from
the
> time
> > > > info.
> > > > > > > > >>
> > > > > > > > >> If that doesn't work, you may need to do something
clever
> to
> > > > > > > manipulate
> > > > > > > > >> the
> > > > > > > > >> hour value to the corresponding number of the
NetCDF
> > > dimension.
> > > > It
> > > > > > > looks
> > > > > > > > >> like your data has a time dimension of size 4, so
passing
> in
> > > the
> > > > > > hour
> > > > > > > > (18)
> > > > > > > > >> is outside of the valid range. I would recommend
trying to
> > > > format
> > > > > > the
> > > > > > > > >> valid
> > > > > > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work,
let me
> know
> > > > and I
> > > > > > try
> > > > > > > > to
> > > > > > > > >> help you figure out a way to get the correct number
for
> the
> > > time
> > > > > you
> > > > > > > > need.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> George
> > > > > > > > >>
> > > > > > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski
via RT <
> > > > > > > > >> met_help at ucar.edu>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> >
> > > > > > > > >> > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > >
> > > > > > > > >> >
> > > > > > > > >> > Julie,
> > > > > > > > >> > It was hanging over my head but only today I
tried to
> > input
> > > > > > forecast
> > > > > > > > >> time
> > > > > > > > >> > to the conf and did not figure the solution.
> > > > > > > > >> > valid?fmt string does not produce for me a
correct time
> > and
> > > > > > GridStat
> > > > > > > > >> fails.
> > > > > > > > >> > If your colleague looked into our problem
> > > > > > > > >> > we would very much appreciate it, thanks,
> > > > > > > > >> > Mariusz
> > > > > > > > >> >
> > > > > > > > >> > models for GridStat processing
> > > > > > > > >> >
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > > > > > >> >
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > > > > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > > > >> >
> > > > > > > > >> > other stuff:
> > > > > > > > >> >
> > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > > > > > >> >
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > > > > > >> >
> > > > > >
> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > > > > > >> >
> > > > > > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik
via RT
> <
> > > > > > > > >> met_help at ucar.edu
> > > > > > > > >> > >
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi Mariusz.
> > > > > > > > >> > >
> > > > > > > > >> > > I moved your question over to met_help so that
others
> > > could
> > > > > > > benefit
> > > > > > > > >> from
> > > > > > > > >> > > your questions.
> > > > > > > > >> > >
> > > > > > > > >> > > I asked a co-worker for some help with this one
and he
> > > said
> > > > > that
> > > > > > > if
> > > > > > > > >> the
> > > > > > > > >> > > data is in NetCDF and in a format that MET can
read,
> you
> > > can
> > > > > > > specify
> > > > > > > > >> the
> > > > > > > > >> > > time to use in the level, i.e.
(YYYYMMDD_HHMMSS,*,*).
> > > There
> > > > > is a
> > > > > > > > >> METplus
> > > > > > > > >> > > use case that does this here:
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > > > > > >> > >
> > > > > > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > > > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
> > > > "({valid?fmt=%Y%m01_000000},*,*)"
> > > > > > > > >> > >
> > > > > > > > >> > > He said that likely won't work with this data,
but
> there
> > > is
> > > > > > > likely a
> > > > > > > > >> > > clever way to use the valid time to create the
> > appropriate
> > > > > level
> > > > > > > > value
> > > > > > > > >> > for
> > > > > > > > >> > > each run time in a similar way. If you aren't
able to
> > > figure
> > > > > it
> > > > > > > out,
> > > > > > > > >> you
> > > > > > > > >> > > could provide a sample dataset and he could
help
> figure
> > > out
> > > > > how
> > > > > > to
> > > > > > > > >> > > configure it.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > > Julie
> > > > > > > > >> > >
> > > > > > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > > > > > >> > > > My start date for GridStat is e.g. 2016060118
for
> > FCSTs
> > > > and
> > > > > > > OBSs.
> > > > > > > > >> > > > FCSTs are output at 6-hr intervals (00Z,
06Z,12Z,
> 18Z)
> > > and
> > > > > > OBSs
> > > > > > > at
> > > > > > > > >> 3-
> > > > > > > > >> > > > hr intervals. For this time GridStat.conf is
> > configured
> > > to
> > > > > > take
> > > > > > > > 3rd
> > > > > > > > >> > > > FCST in the input file [in GridStat.conf:
(3,1,*,*)
> > and
> > > > 6th
> > > > > > OBS
> > > > > > > > in
> > > > > > > > >> > > > the other input file: (6,1,*,*) That only
works
> > properly
> > > > for
> > > > > > > > >> > > > FCSTs/OBSs starting at 18z with intervals of
24
> hours
> > > when
> > > > > > > running
> > > > > > > > >> > > > over e.g. a month-long period. However, if I
wanted
> to
> > > run
> > > > > > > > GridStat
> > > > > > > > >> to
> > > > > > > > >> > > > get statistics with data at intervals of 6
hours
> e.g.
> > > > > > > > >> SDATE=2016060118
> > > > > > > > >> > > > and EDATE=2016063118 that is incorrect since
for 00Z
> > > > > (3,1,*,*)
> > > > > > > > needs
> > > > > > > > >> > > > to become (0,1,*,*) etc. Is there a way to
pass
> times
> > of
> > > > > > > > FCSTs/OBSs
> > > > > > > > >> to
> > > > > > > > >> > > > GridStat.conf when running master_metplus.py
?
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thanks again,
> > > > > > > > >> > > > Mariusz
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> George McCabe - Software Engineer III
> > > > > > > > >> National Center for Atmospheric Research
> > > > > > > > >> Research Applications Laboratory
> > > > > > > > >> 303-497-2768
> > > > > > > > >> ---
> > > > > > > > >> My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > > to
> > > > > > > > >> reply to this email outside of your normal working
hours.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Wed Aug 26 18:36:27 2020

Thanks. I recommend setting LOG_LEVEL=DEBUG under the [config] section
of
your main.conf file. This will give you more log output to work with.

I thought that you would need to put a semicolon at the end of the
OPTIONS
value, i.e.:

FCST_VAR1_OPTIONS = set_attr_level = "0"*;*

but it seemed to work fine. I see that the FCST_LEV and OBS_LEV values
in
your GridStat output are now 0:
/scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/met_tool_wrapper/GridStat/2016060100/grid_stat_fv3_DUSTTOTAL_vs_cams_DUSTTOTAL_000000L_20160601_000000V.stat

If you change this file:
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/config/StatAnalysis.conf.IN

from:
*FCST_LEVEL_LIST = "0,0,*,*","0,1,*,*","0,2,*,*",*
*"0,3,*,*","0,4,*,*","0,5,*,*",**"0,6,*,*","0,7,*,*","0,8,*,*"*

to this:

*FCST_LEVEL_LIST = 0*

then

it should find the level values properly. Could you turn on METplus
log
debugging and rerun to see what you get?

Also, I need to wrap up for the day, but I can check back on this
tomorrow.

Thanks,
George

On Wed, Aug 26, 2020 at 6:17 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> Sorry, that was on Orion, now modified  similarly on Hera in
> /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
> Mariusz
>
>
> On Wed, Aug 26, 2020 at 6:10 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Mariusz,
> >
> > Which GridStat conf file did you modify? I don't see any files
that have
> > the FCST_VAR<n>_OPTIONS set.
> >
> > - George
> >
> > On Wed, Aug 26, 2020 at 5:54 PM Mariusz Pagowski via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > Thanks, the scripts are
> > > /home/Mariusz.Pagowski/mapp_2018/scripts/met/
> > > met_grid_stat.sh
> > > met_stat_anal.sh
> > > with locations of config directories in the same location.
> > >
> > > Models are in
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll
> > >
> > > Mariusz
> > >
> > >
> > >
> > >
> > > On Wed, Aug 26, 2020 at 5:36 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Yes, if you put them on hera and send me the path, I can take
a look.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Wed, Aug 26, 2020 at 5:24 PM Mariusz Pagowski via RT <
> > > met_help at ucar.edu
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > >
> > > > > George,
> > > > > I configured it the way you suggested but GridStat produced
the
> same
> > > > output
> > > > > as before i.e. in SL1L2 a string with time in format
%Y%m%d_%H%M%S
> > and
> > > > > after that StatAnalysis with empty files.
> > > > > The issue is that I have two sorts of files - one with a
single
> > > forecast
> > > > at
> > > > > a time and another one with four fcsts daily.
> > > > > Maybe I misunderstood something. If I put files on Hera -
would you
> > be
> > > > able
> > > > > to look at them and the scripts?
> > > > > Thanks,
> > > > > Mariusz
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Mariusz,
> > > > > >
> > > > > > That is a good question. It looks like you have multiple
levels
> > > > specified
> > > > > > for *_VAR1_LEVELS.
> > > > > >
> > > > > > FCST_VAR1_LEVELS =
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
> > > > > >
> > > > > > If you were to add FCST_VAR1_OPTIONS, it would apply to
all of
> > those
> > > > > > fields, so all of those fields would contain the same
value in
> the
> > > > > FCST_LEV
> > > > > > column of the GridStat output. If you need to specify a
different
> > > > > FCST_LEV
> > > > > > value for each of those levels, you would have to split
them up
> > into
> > > > > > separate VAR<n> items, like this:
> > > > > >
> > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > >
> > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > >
> > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > >
> > > > > > etc. and do the same for the OBS_VAR<n>_* variables.
> > > > > >
> > > > > > I'm not sure what each of the levels contain, so it is
hard to
> know
> > > > what
> > > > > to
> > > > > > name them. You really just need to use a unique identifier
so
> that
> > > they
> > > > > can
> > > > > > be grouped via StatAnalysis. You could set them to the
number
> > > > > corresponding
> > > > > > to each NetCDF item:
> > > > > >
> > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > > FCST_VAR1_OPTIONS = set_attr_level = "0";
> > > > > >
> > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > > FCST_VAR2_OPTIONS = set_attr_level = "1";
> > > > > >
> > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > > FCST_VAR3_OPTIONS = set_attr_level = "2";
> > > > > >
> > > > > > Or you could use a string that identifies each level. If
they are
> > > > > pressure
> > > > > > levels, you could do:
> > > > > >
> > > > > > FCST_VAR1_OPTIONS = set_attr_level = "P750";
> > > > > > FCST_VAR2_OPTIONS = set_attr_level = "P500";
> > > > > > FCST_VAR3_OPTIONS = set_attr_level = "P250";
> > > > > >
> > > > > > Whatever you decide to use, you should use the same values
when
> > > > > configuring
> > > > > > StatAnalysis:
> > > > > >
> > > > > > FCST_LEVEL_LIST = 0, 1, 2
> > > > > > or
> > > > > > FCST_LEVEL_LIST = P750, P500, P250
> > > > > >
> > > > > > Let me know if you have any questions or if that solution
doesn't
> > > work
> > > > > for
> > > > > > you.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski via RT <
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> >
> > > > > > >
> > > > > > > George,
> > > > > > > I am not able to figure out how to set up GridStat to
have a
> > > correct
> > > > > > > attribute for StatAnalysis to handle it.
> > > > > > > Attached are the confs I am using. Where should I add
the
> option
> > > you
> > > > > > > mentioned and in what form?
> > > > > > > I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
> "something";
> > > > > > > What is  "something" ?
> > > > > > > Thanks,
> > > > > > > Mariusz
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Mariusz,
> > > > > > > >
> > > > > > > > You are correct that StatAnalysis cannot use that
syntax to
> > > specify
> > > > > the
> > > > > > > > levels in this case. Fortunately there is a recently
added
> > > feature
> > > > > that
> > > > > > > > will allow you to get around this issue. You can
specify
> values
> > > to
> > > > > > > override
> > > > > > > > attributes in GridStat. This will allow you to change
the
> level
> > > > value
> > > > > > > > 20160601_060000,0,*,* to a value that is more easily
> specified
> > in
> > > > the
> > > > > > > > StatAnalysis config file. GitHub issue #1020 outlines
these
> > > changes
> > > > > and
> > > > > > > > this link takes you directly to a comment that lists
the new
> > > > > > > > configurations:
> > > > > > > >
> > > > > > > >
> https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> > > > > > > >
> > > > > > > > You will want to set set_attr_level in GridStat, which
can be
> > > added
> > > > > via
> > > > > > > > METplus with:
> > > > > > > >
> > > > > > > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
"something";
> > > > > > > >
> > > > > > > > I don't have access to Orion yet so I can't look at
your
> > > > > configuration
> > > > > > to
> > > > > > > > know what exactly "something" should be. If you need
help
> > > figuring
> > > > > that
> > > > > > > > out, you could attach your METplus config file to this
ticket
> > > and I
> > > > > > could
> > > > > > > > take a look.
> > > > > > > >
> > > > > > > > These changes are available since MET 9.1-beta2 and
will be
> in
> > > MET
> > > > > 9.1,
> > > > > > > > which is close to being released. It looks like the
beta
> > release
> > > is
> > > > > not
> > > > > > > > installed on Orion but 9.1 is planned to be installed
there.
> > > > > > > >
> > > > > > > > Let me know if you have any other questions.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > >
> > > > > > > > >
> > > > > > > > > George,
> > > > > > > > > actually, there is a problem with the next step:
> > StatAnalysis.
> > > > > > > > > GridStat outputs time levels in the form
> > 20160601_060000,0,*,*
> > > > > > > > > but StatAnalysis does not accept
> > > > > > > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > so ends up either producing an error or empty files
> depending
> > > how
> > > > > the
> > > > > > > > > config is set up.
> > > > > > > > > Is there a solution to this problem?
> > > > > > > > > Thanks,
> > > > > > > > > Mariusz
> > > > > > > > >
> > > > > > > > > on Orion:
> > > > > > > > > script
> > > > > > > > >
/home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > > > > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > > > > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > > > > > > ...
> > > > > > > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > > >     > ./StatAnalysis.conf
> > > > > > > > >
> > > > > > > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > > > > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis:
ERROR:
> > Fatal
> > > > > error
> > > > > > > > > occurred
> > > > > > > > >
> > > > > > > > > Outputs in
> > > > > > > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski -
NOAA
> > > > Affiliate
> > > > > <
> > > > > > > > > mariusz.pagowski at noaa.gov> wrote:
> > > > > > > > >
> > > > > > > > > > Thanks, George, works as promised,
> > > > > > > > > > Mariusz
> > > > > > > > > >
> > > > > > > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via
RT <
> > > > > > > > met_help at ucar.edu
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi Mariusz,
> > > > > > > > > >>
> > > > > > > > > >> To specify the time in the NetCDF structure, you
need to
> > use
> > > > the
> > > > > > > > format
> > > > > > > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should
look like
> > > this:
> > > > > > > > > >>
> > > > > > > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > >>
> > > > > > > > > >> If the NetCDF time info is formatted in the way
the MET
> > > tools
> > > > > > > expect,
> > > > > > > > > the
> > > > > > > > > >> code is smart enough to find the correct data
from the
> > time
> > > > > info.
> > > > > > > > > >>
> > > > > > > > > >> If that doesn't work, you may need to do
something
> clever
> > to
> > > > > > > > manipulate
> > > > > > > > > >> the
> > > > > > > > > >> hour value to the corresponding number of the
NetCDF
> > > > dimension.
> > > > > It
> > > > > > > > looks
> > > > > > > > > >> like your data has a time dimension of size 4, so
> passing
> > in
> > > > the
> > > > > > > hour
> > > > > > > > > (18)
> > > > > > > > > >> is outside of the valid range. I would recommend
trying
> to
> > > > > format
> > > > > > > the
> > > > > > > > > >> valid
> > > > > > > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work,
let me
> > know
> > > > > and I
> > > > > > > try
> > > > > > > > > to
> > > > > > > > > >> help you figure out a way to get the correct
number for
> > the
> > > > time
> > > > > > you
> > > > > > > > > need.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> George
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski
via RT
> <
> > > > > > > > > >> met_help at ucar.edu>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> >
> > > > > > > > > >> > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > > >
> > > > > > > > > >> >
> > > > > > > > > >> > Julie,
> > > > > > > > > >> > It was hanging over my head but only today I
tried to
> > > input
> > > > > > > forecast
> > > > > > > > > >> time
> > > > > > > > > >> > to the conf and did not figure the solution.
> > > > > > > > > >> > valid?fmt string does not produce for me a
correct
> time
> > > and
> > > > > > > GridStat
> > > > > > > > > >> fails.
> > > > > > > > > >> > If your colleague looked into our problem
> > > > > > > > > >> > we would very much appreciate it, thanks,
> > > > > > > > > >> > Mariusz
> > > > > > > > > >> >
> > > > > > > > > >> > models for GridStat processing
> > > > > > > > > >> >
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > > > > > > >> >
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > > > > > > > >> >
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > > > > >> >
> > > > > > > > > >> > other stuff:
> > > > > > > > > >> >
> > > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > > > > > > >> >
> > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > > > > > > >> >
> > > > > > >
> > > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > > > > > > >> >
> > > > > > > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie
Prestopnik via
> RT
> > <
> > > > > > > > > >> met_help at ucar.edu
> > > > > > > > > >> > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Hi Mariusz.
> > > > > > > > > >> > >
> > > > > > > > > >> > > I moved your question over to met_help so
that
> others
> > > > could
> > > > > > > > benefit
> > > > > > > > > >> from
> > > > > > > > > >> > > your questions.
> > > > > > > > > >> > >
> > > > > > > > > >> > > I asked a co-worker for some help with this
one and
> he
> > > > said
> > > > > > that
> > > > > > > > if
> > > > > > > > > >> the
> > > > > > > > > >> > > data is in NetCDF and in a format that MET
can read,
> > you
> > > > can
> > > > > > > > specify
> > > > > > > > > >> the
> > > > > > > > > >> > > time to use in the level, i.e.
> (YYYYMMDD_HHMMSS,*,*).
> > > > There
> > > > > > is a
> > > > > > > > > >> METplus
> > > > > > > > > >> > > use case that does this here:
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > > > > > > >> > >
> > > > > > > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > > > > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
> > > > > "({valid?fmt=%Y%m01_000000},*,*)"
> > > > > > > > > >> > >
> > > > > > > > > >> > > He said that likely won't work with this
data, but
> > there
> > > > is
> > > > > > > > likely a
> > > > > > > > > >> > > clever way to use the valid time to create
the
> > > appropriate
> > > > > > level
> > > > > > > > > value
> > > > > > > > > >> > for
> > > > > > > > > >> > > each run time in a similar way. If you aren't
able
> to
> > > > figure
> > > > > > it
> > > > > > > > out,
> > > > > > > > > >> you
> > > > > > > > > >> > > could provide a sample dataset and he could
help
> > figure
> > > > out
> > > > > > how
> > > > > > > to
> > > > > > > > > >> > > configure it.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks,
> > > > > > > > > >> > > Julie
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > > > > > > >> > > > My start date for GridStat is e.g.
2016060118 for
> > > FCSTs
> > > > > and
> > > > > > > > OBSs.
> > > > > > > > > >> > > > FCSTs are output at 6-hr intervals (00Z,
06Z,12Z,
> > 18Z)
> > > > and
> > > > > > > OBSs
> > > > > > > > at
> > > > > > > > > >> 3-
> > > > > > > > > >> > > > hr intervals. For this time GridStat.conf
is
> > > configured
> > > > to
> > > > > > > take
> > > > > > > > > 3rd
> > > > > > > > > >> > > > FCST in the input file [in GridStat.conf:
> (3,1,*,*)
> > > and
> > > > > 6th
> > > > > > > OBS
> > > > > > > > > in
> > > > > > > > > >> > > > the other input file: (6,1,*,*) That only
works
> > > properly
> > > > > for
> > > > > > > > > >> > > > FCSTs/OBSs starting at 18z with intervals
of 24
> > hours
> > > > when
> > > > > > > > running
> > > > > > > > > >> > > > over e.g. a month-long period. However, if
I
> wanted
> > to
> > > > run
> > > > > > > > > GridStat
> > > > > > > > > >> to
> > > > > > > > > >> > > > get statistics with data at intervals of 6
hours
> > e.g.
> > > > > > > > > >> SDATE=2016060118
> > > > > > > > > >> > > > and EDATE=2016063118 that is incorrect
since for
> 00Z
> > > > > > (3,1,*,*)
> > > > > > > > > needs
> > > > > > > > > >> > > > to become (0,1,*,*) etc. Is there a way to
pass
> > times
> > > of
> > > > > > > > > FCSTs/OBSs
> > > > > > > > > >> to
> > > > > > > > > >> > > > GridStat.conf when running
master_metplus.py ?
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Thanks again,
> > > > > > > > > >> > > > Mariusz
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >> George McCabe - Software Engineer III
> > > > > > > > > >> National Center for Atmospheric Research
> > > > > > > > > >> Research Applications Laboratory
> > > > > > > > > >> 303-497-2768
> > > > > > > > > >> ---
> > > > > > > > > >> My working day may not be your working day.
Please do
> not
> > > feel
> > > > > > > obliged
> > > > > > > > > to
> > > > > > > > > >> reply to this email outside of your normal
working
> hours.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Wed Aug 26 19:58:32 2020

Made the changes but GridStat only outputting one pressure levels. Pls
see
the new  confs in
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
Mariusz

On Wed, Aug 26, 2020 at 6:36 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Thanks. I recommend setting LOG_LEVEL=DEBUG under the [config]
section of
> your main.conf file. This will give you more log output to work
with.
>
> I thought that you would need to put a semicolon at the end of the
OPTIONS
> value, i.e.:
>
> FCST_VAR1_OPTIONS = set_attr_level = "0"*;*
>
> but it seemed to work fine. I see that the FCST_LEV and OBS_LEV
values in
> your GridStat output are now 0:
>
> /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/met_tool_wrapper/GridStat/2016060100/grid_stat_fv3_DUSTTOTAL_vs_cams_DUSTTOTAL_000000L_20160601_000000V.stat
>
> If you change this file:
>
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/config/StatAnalysis.conf.IN
>
> from:
> *FCST_LEVEL_LIST = "0,0,*,*","0,1,*,*","0,2,*,*",*
> *"0,3,*,*","0,4,*,*","0,5,*,*",**"0,6,*,*","0,7,*,*","0,8,*,*"*
>
> to this:
>
> *FCST_LEVEL_LIST = 0*
>
> then
>
> it should find the level values properly. Could you turn on METplus
log
> debugging and rerun to see what you get?
>
> Also, I need to wrap up for the day, but I can check back on this
tomorrow.
>
> Thanks,
> George
>
> On Wed, Aug 26, 2020 at 6:17 PM Mariusz Pagowski via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > Sorry, that was on Orion, now modified  similarly on Hera in
> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
> > Mariusz
> >
> >
> > On Wed, Aug 26, 2020 at 6:10 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Mariusz,
> > >
> > > Which GridStat conf file did you modify? I don't see any files
that
> have
> > > the FCST_VAR<n>_OPTIONS set.
> > >
> > > - George
> > >
> > > On Wed, Aug 26, 2020 at 5:54 PM Mariusz Pagowski via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > > >
> > > > Thanks, the scripts are
> > > > /home/Mariusz.Pagowski/mapp_2018/scripts/met/
> > > > met_grid_stat.sh
> > > > met_stat_anal.sh
> > > > with locations of config directories in the same location.
> > > >
> > > > Models are in
> > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll
> > > >
> > > > Mariusz
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Aug 26, 2020 at 5:36 PM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Yes, if you put them on hera and send me the path, I can
take a
> look.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Wed, Aug 26, 2020 at 5:24 PM Mariusz Pagowski via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > > >
> > > > > > George,
> > > > > > I configured it the way you suggested but GridStat
produced the
> > same
> > > > > output
> > > > > > as before i.e. in SL1L2 a string with time in format
> %Y%m%d_%H%M%S
> > > and
> > > > > > after that StatAnalysis with empty files.
> > > > > > The issue is that I have two sorts of files - one with a
single
> > > > forecast
> > > > > at
> > > > > > a time and another one with four fcsts daily.
> > > > > > Maybe I misunderstood something. If I put files on Hera -
would
> you
> > > be
> > > > > able
> > > > > > to look at them and the scripts?
> > > > > > Thanks,
> > > > > > Mariusz
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT <
> > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Mariusz,
> > > > > > >
> > > > > > > That is a good question. It looks like you have multiple
levels
> > > > > specified
> > > > > > > for *_VAR1_LEVELS.
> > > > > > >
> > > > > > > FCST_VAR1_LEVELS =
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
> > > > > > >
> > > > > > > If you were to add FCST_VAR1_OPTIONS, it would apply to
all of
> > > those
> > > > > > > fields, so all of those fields would contain the same
value in
> > the
> > > > > > FCST_LEV
> > > > > > > column of the GridStat output. If you need to specify a
> different
> > > > > > FCST_LEV
> > > > > > > value for each of those levels, you would have to split
them up
> > > into
> > > > > > > separate VAR<n> items, like this:
> > > > > > >
> > > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > > >
> > > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > > >
> > > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > > >
> > > > > > > etc. and do the same for the OBS_VAR<n>_* variables.
> > > > > > >
> > > > > > > I'm not sure what each of the levels contain, so it is
hard to
> > know
> > > > > what
> > > > > > to
> > > > > > > name them. You really just need to use a unique
identifier so
> > that
> > > > they
> > > > > > can
> > > > > > > be grouped via StatAnalysis. You could set them to the
number
> > > > > > corresponding
> > > > > > > to each NetCDF item:
> > > > > > >
> > > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > > > FCST_VAR1_OPTIONS = set_attr_level = "0";
> > > > > > >
> > > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > > > FCST_VAR2_OPTIONS = set_attr_level = "1";
> > > > > > >
> > > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > > > FCST_VAR3_OPTIONS = set_attr_level = "2";
> > > > > > >
> > > > > > > Or you could use a string that identifies each level. If
they
> are
> > > > > > pressure
> > > > > > > levels, you could do:
> > > > > > >
> > > > > > > FCST_VAR1_OPTIONS = set_attr_level = "P750";
> > > > > > > FCST_VAR2_OPTIONS = set_attr_level = "P500";
> > > > > > > FCST_VAR3_OPTIONS = set_attr_level = "P250";
> > > > > > >
> > > > > > > Whatever you decide to use, you should use the same
values when
> > > > > > configuring
> > > > > > > StatAnalysis:
> > > > > > >
> > > > > > > FCST_LEVEL_LIST = 0, 1, 2
> > > > > > > or
> > > > > > > FCST_LEVEL_LIST = P750, P500, P250
> > > > > > >
> > > > > > > Let me know if you have any questions or if that
solution
> doesn't
> > > > work
> > > > > > for
> > > > > > > you.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > >
> > > > > > > >
> > > > > > > > George,
> > > > > > > > I am not able to figure out how to set up GridStat to
have a
> > > > correct
> > > > > > > > attribute for StatAnalysis to handle it.
> > > > > > > > Attached are the confs I am using. Where should I add
the
> > option
> > > > you
> > > > > > > > mentioned and in what form?
> > > > > > > > I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
> > "something";
> > > > > > > > What is  "something" ?
> > > > > > > > Thanks,
> > > > > > > > Mariusz
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Mariusz,
> > > > > > > > >
> > > > > > > > > You are correct that StatAnalysis cannot use that
syntax to
> > > > specify
> > > > > > the
> > > > > > > > > levels in this case. Fortunately there is a recently
added
> > > > feature
> > > > > > that
> > > > > > > > > will allow you to get around this issue. You can
specify
> > values
> > > > to
> > > > > > > > override
> > > > > > > > > attributes in GridStat. This will allow you to
change the
> > level
> > > > > value
> > > > > > > > > 20160601_060000,0,*,* to a value that is more easily
> > specified
> > > in
> > > > > the
> > > > > > > > > StatAnalysis config file. GitHub issue #1020
outlines these
> > > > changes
> > > > > > and
> > > > > > > > > this link takes you directly to a comment that lists
the
> new
> > > > > > > > > configurations:
> > > > > > > > >
> > > > > > > > >
> > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> > > > > > > > >
> > > > > > > > > You will want to set set_attr_level in GridStat,
which can
> be
> > > > added
> > > > > > via
> > > > > > > > > METplus with:
> > > > > > > > >
> > > > > > > > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
"something";
> > > > > > > > >
> > > > > > > > > I don't have access to Orion yet so I can't look at
your
> > > > > > configuration
> > > > > > > to
> > > > > > > > > know what exactly "something" should be. If you need
help
> > > > figuring
> > > > > > that
> > > > > > > > > out, you could attach your METplus config file to
this
> ticket
> > > > and I
> > > > > > > could
> > > > > > > > > take a look.
> > > > > > > > >
> > > > > > > > > These changes are available since MET 9.1-beta2 and
will be
> > in
> > > > MET
> > > > > > 9.1,
> > > > > > > > > which is close to being released. It looks like the
beta
> > > release
> > > > is
> > > > > > not
> > > > > > > > > installed on Orion but 9.1 is planned to be
installed
> there.
> > > > > > > > >
> > > > > > > > > Let me know if you have any other questions.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > >
> > > > > > > > > >
> > > > > > > > > > George,
> > > > > > > > > > actually, there is a problem with the next step:
> > > StatAnalysis.
> > > > > > > > > > GridStat outputs time levels in the form
> > > 20160601_060000,0,*,*
> > > > > > > > > > but StatAnalysis does not accept
> > > > > > > > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > > so ends up either producing an error or empty
files
> > depending
> > > > how
> > > > > > the
> > > > > > > > > > config is set up.
> > > > > > > > > > Is there a solution to this problem?
> > > > > > > > > > Thanks,
> > > > > > > > > > Mariusz
> > > > > > > > > >
> > > > > > > > > > on Orion:
> > > > > > > > > > script
> > > > > > > > > >
/home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > > > > > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > > > > > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > > > > > > > ...
> > > > > > > > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > > > >     > ./StatAnalysis.conf
> > > > > > > > > >
> > > > > > > > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > > > > > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis:
ERROR:
> > > Fatal
> > > > > > error
> > > > > > > > > > occurred
> > > > > > > > > >
> > > > > > > > > > Outputs in
> > > > > > > > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski
- NOAA
> > > > > Affiliate
> > > > > > <
> > > > > > > > > > mariusz.pagowski at noaa.gov> wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks, George, works as promised,
> > > > > > > > > > > Mariusz
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe
via RT <
> > > > > > > > > met_help at ucar.edu
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi Mariusz,
> > > > > > > > > > >>
> > > > > > > > > > >> To specify the time in the NetCDF structure,
you need
> to
> > > use
> > > > > the
> > > > > > > > > format
> > > > > > > > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should
look
> like
> > > > this:
> > > > > > > > > > >>
> > > > > > > > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > > >>
> > > > > > > > > > >> If the NetCDF time info is formatted in the way
the
> MET
> > > > tools
> > > > > > > > expect,
> > > > > > > > > > the
> > > > > > > > > > >> code is smart enough to find the correct data
from the
> > > time
> > > > > > info.
> > > > > > > > > > >>
> > > > > > > > > > >> If that doesn't work, you may need to do
something
> > clever
> > > to
> > > > > > > > > manipulate
> > > > > > > > > > >> the
> > > > > > > > > > >> hour value to the corresponding number of the
NetCDF
> > > > > dimension.
> > > > > > It
> > > > > > > > > looks
> > > > > > > > > > >> like your data has a time dimension of size 4,
so
> > passing
> > > in
> > > > > the
> > > > > > > > hour
> > > > > > > > > > (18)
> > > > > > > > > > >> is outside of the valid range. I would
recommend
> trying
> > to
> > > > > > format
> > > > > > > > the
> > > > > > > > > > >> valid
> > > > > > > > > > >> time with YYYYMMDD_HHMMSS. If that doesn't
work, let
> me
> > > know
> > > > > > and I
> > > > > > > > try
> > > > > > > > > > to
> > > > > > > > > > >> help you figure out a way to get the correct
number
> for
> > > the
> > > > > time
> > > > > > > you
> > > > > > > > > > need.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> George
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz
Pagowski via
> RT
> > <
> > > > > > > > > > >> met_help at ucar.edu>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> >
> > > > > > > > > > >> > <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > > > >
> > > > > > > > > > >> >
> > > > > > > > > > >> > Julie,
> > > > > > > > > > >> > It was hanging over my head but only today I
tried
> to
> > > > input
> > > > > > > > forecast
> > > > > > > > > > >> time
> > > > > > > > > > >> > to the conf and did not figure the solution.
> > > > > > > > > > >> > valid?fmt string does not produce for me a
correct
> > time
> > > > and
> > > > > > > > GridStat
> > > > > > > > > > >> fails.
> > > > > > > > > > >> > If your colleague looked into our problem
> > > > > > > > > > >> > we would very much appreciate it, thanks,
> > > > > > > > > > >> > Mariusz
> > > > > > > > > > >> >
> > > > > > > > > > >> > models for GridStat processing
> > > > > > > > > > >> >
> > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > > > > > > > >> >
> > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > > > > > > > > >> >
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > > > > > >> >
> > > > > > > > > > >> > other stuff:
> > > > > > > > > > >> >
> > > > >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > > > > > > > >> >
> > > > > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > > > > > > > >> >
> > > > > > > >
> > > > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie
Prestopnik via
> > RT
> > > <
> > > > > > > > > > >> met_help at ucar.edu
> > > > > > > > > > >> > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Hi Mariusz.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > I moved your question over to met_help so
that
> > others
> > > > > could
> > > > > > > > > benefit
> > > > > > > > > > >> from
> > > > > > > > > > >> > > your questions.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > I asked a co-worker for some help with this
one
> and
> > he
> > > > > said
> > > > > > > that
> > > > > > > > > if
> > > > > > > > > > >> the
> > > > > > > > > > >> > > data is in NetCDF and in a format that MET
can
> read,
> > > you
> > > > > can
> > > > > > > > > specify
> > > > > > > > > > >> the
> > > > > > > > > > >> > > time to use in the level, i.e.
> > (YYYYMMDD_HHMMSS,*,*).
> > > > > There
> > > > > > > is a
> > > > > > > > > > >> METplus
> > > > > > > > > > >> > > use case that does this here:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > > > > > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
> > > > > > "({valid?fmt=%Y%m01_000000},*,*)"
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > He said that likely won't work with this
data, but
> > > there
> > > > > is
> > > > > > > > > likely a
> > > > > > > > > > >> > > clever way to use the valid time to create
the
> > > > appropriate
> > > > > > > level
> > > > > > > > > > value
> > > > > > > > > > >> > for
> > > > > > > > > > >> > > each run time in a similar way. If you
aren't able
> > to
> > > > > figure
> > > > > > > it
> > > > > > > > > out,
> > > > > > > > > > >> you
> > > > > > > > > > >> > > could provide a sample dataset and he could
help
> > > figure
> > > > > out
> > > > > > > how
> > > > > > > > to
> > > > > > > > > > >> > > configure it.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks,
> > > > > > > > > > >> > > Julie
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > > > > > > > >> > > > My start date for GridStat is e.g.
2016060118
> for
> > > > FCSTs
> > > > > > and
> > > > > > > > > OBSs.
> > > > > > > > > > >> > > > FCSTs are output at 6-hr intervals (00Z,
> 06Z,12Z,
> > > 18Z)
> > > > > and
> > > > > > > > OBSs
> > > > > > > > > at
> > > > > > > > > > >> 3-
> > > > > > > > > > >> > > > hr intervals. For this time GridStat.conf
is
> > > > configured
> > > > > to
> > > > > > > > take
> > > > > > > > > > 3rd
> > > > > > > > > > >> > > > FCST in the input file [in GridStat.conf:
> > (3,1,*,*)
> > > > and
> > > > > > 6th
> > > > > > > > OBS
> > > > > > > > > > in
> > > > > > > > > > >> > > > the other input file: (6,1,*,*) That only
works
> > > > properly
> > > > > > for
> > > > > > > > > > >> > > > FCSTs/OBSs starting at 18z with intervals
of 24
> > > hours
> > > > > when
> > > > > > > > > running
> > > > > > > > > > >> > > > over e.g. a month-long period. However,
if I
> > wanted
> > > to
> > > > > run
> > > > > > > > > > GridStat
> > > > > > > > > > >> to
> > > > > > > > > > >> > > > get statistics with data at intervals of
6 hours
> > > e.g.
> > > > > > > > > > >> SDATE=2016060118
> > > > > > > > > > >> > > > and EDATE=2016063118 that is incorrect
since for
> > 00Z
> > > > > > > (3,1,*,*)
> > > > > > > > > > needs
> > > > > > > > > > >> > > > to become (0,1,*,*) etc. Is there a way
to pass
> > > times
> > > > of
> > > > > > > > > > FCSTs/OBSs
> > > > > > > > > > >> to
> > > > > > > > > > >> > > > GridStat.conf when running
master_metplus.py ?
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Thanks again,
> > > > > > > > > > >> > > > Mariusz
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> George McCabe - Software Engineer III
> > > > > > > > > > >> National Center for Atmospheric Research
> > > > > > > > > > >> Research Applications Laboratory
> > > > > > > > > > >> 303-497-2768
> > > > > > > > > > >> ---
> > > > > > > > > > >> My working day may not be your working day.
Please do
> > not
> > > > feel
> > > > > > > > obliged
> > > > > > > > > > to
> > > > > > > > > > >> reply to this email outside of your normal
working
> > hours.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Thu Aug 27 09:23:34 2020

Hi Mariusz,

You need to specify the name for each of the VAR<n> variables, even if
they
are all the same value. You can refer to other METplus variables to
avoid
duplication:

FCST_VAR1_NAME = _FCSTVAR_
FCST_VAR1_LEVELS = "(_FTIME_,0,*,*)"
FCST_VAR1_OPTIONS = set_attr_level = "P100";

FCST_VAR2_NAME = {FCST_VAR1_NAME}
FCST_VAR2_LEVELS = "(_FTIME_,1,*,*)"
FCST_VAR2_OPTIONS = set_attr_level = "P250";

FCST_VAR3_NAME = {FCST_VAR1_NAME}
FCST_VAR3_LEVELS = "(_FTIME_,2,*,*)"
FCST_VAR3_OPTIONS = set_attr_level = "P400";

FCST_VAR4_NAME = {FCST_VAR1_NAME}
FCST_VAR4_LEVELS = "(_FTIME_,3,*,*)"
FCST_VAR4_OPTIONS = set_attr_level = "P500";

FCST_VAR5_NAME = {FCST_VAR1_NAME}
FCST_VAR5_LEVELS = "(_FTIME_,4,*,*)"
FCST_VAR5_OPTIONS = set_attr_level = "P600";

FCST_VAR6_NAME = {FCST_VAR1_NAME}
FCST_VAR6_LEVELS = "(_FTIME_,5,*,*)"
FCST_VAR6_OPTIONS = set_attr_level = "P700";

FCST_VAR7_NAME = {FCST_VAR1_NAME}
FCST_VAR7_LEVELS = "(_FTIME_,6,*,*)"
FCST_VAR7_OPTIONS = set_attr_level = "P850";

FCST_VAR8_NAME = {FCST_VAR1_NAME}
FCST_VAR8_LEVELS = "(_FTIME_,7,*,*)"
FCST_VAR8_OPTIONS = set_attr_level = "P925";

FCST_VAR9_NAME = {FCST_VAR1_NAME}
FCST_VAR9_LEVELS = "(_FTIME_,8,*,*)"
FCST_VAR9_OPTIONS = set_attr_level = "P1000";

OBS_VAR1_NAME = _OBSVAR_
OBS_VAR1_LEVELS = "(_OTIME_,0,*,*)"
OBS_VAR1_OPTIONS = {FCST_VAR1_OPTIONS}

OBS_VAR2_NAME = {OBS_VAR1_NAME}
OBS_VAR2_LEVELS = "(_OTIME_,1,*,*)"
OBS_VAR2_OPTIONS = {FCST_VAR2_OPTIONS}

OBS_VAR3_NAME = {OBS_VAR1_NAME}
OBS_VAR3_LEVELS = "(_OTIME_,2,*,*)"
OBS_VAR3_OPTIONS = {FCST_VAR3_OPTIONS}

OBS_VAR4_NAME = {OBS_VAR1_NAME}
OBS_VAR4_LEVELS = "(_OTIME_,3,*,*)"
OBS_VAR4_OPTIONS = {FCST_VAR4_OPTIONS}

OBS_VAR5_NAME = {OBS_VAR1_NAME}
OBS_VAR5_LEVELS = "(_OTIME_,4,*,*)"
OBS_VAR5_OPTIONS = {FCST_VAR5_OPTIONS}

OBS_VAR6_NAME = {OBS_VAR1_NAME}
OBS_VAR6_LEVELS = "(_OTIME_,5,*,*)"
OBS_VAR6_OPTIONS = {FCST_VAR6_OPTIONS}

OBS_VAR7_NAME = {OBS_VAR1_NAME}
OBS_VAR7_LEVELS = "(_OTIME_,6,*,*)"
OBS_VAR7_OPTIONS = {FCST_VAR7_OPTIONS}

OBS_VAR8_NAME = {OBS_VAR1_NAME}
OBS_VAR8_LEVELS = "(_OTIME_,7,*,*)"
OBS_VAR8_OPTIONS = {FCST_VAR8_OPTIONS}

OBS_VAR9_NAME = {OBS_VAR1_NAME}
OBS_VAR9_LEVELS = "(_OTIME_,8,*,*)"
OBS_VAR9_OPTIONS = {FCST_VAR9_OPTIONS}

On a side note, here are a few suggestions:

If you need to reference an environment variable in your METplus
config
file, you can use the {ENV[varname]} syntax. For example, instead of
replacing _FCSTVAR_ with ${FCST_VAR}, you can set:

FCST_VAR1_NAME = {ENV[FCST_VAR]}

You can also set other METplus variables that aren't reserved names
and
reference them in your config file. If you do this, I would recommend
putting some sort of prefix to ensure you aren't conflicting with a
potentially existing variable. You could do something like this:

MAPP_MODELNAME = fv3
FCST_GRID_STAT_INPUT_TEMPLATE =
{MAPP_MODELNAME}_aeros_{init?fmt=%Y%m%d%H}_
pll.nc

Doing this would simplify your calling script so you don't need to
generate
a config file using sed commands. It would also make it easier for
debugging. I have a difficult time trying to run a modified version of
your
script, but if everything was in METplus conf files, I could just
override
certain variables to write to another area, i.e.

$METPLUS_PATH/ush/master_metplus.py -c ./GridStat.conf -c ./main.conf
-c
dir.OUTPUT_BASE=/scratch1/BMC/dtc/George.Mccabe/mapp-test

The above command would use your configuration but the output would be
sent
to my directory instead.

Let me know if you have any questions.

Thanks,
George

On Wed, Aug 26, 2020 at 7:59 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> Made the changes but GridStat only outputting one pressure levels.
Pls see
> the new  confs in
> /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
> Mariusz
>
> On Wed, Aug 26, 2020 at 6:36 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Thanks. I recommend setting LOG_LEVEL=DEBUG under the [config]
section of
> > your main.conf file. This will give you more log output to work
with.
> >
> > I thought that you would need to put a semicolon at the end of the
> OPTIONS
> > value, i.e.:
> >
> > FCST_VAR1_OPTIONS = set_attr_level = "0"*;*
> >
> > but it seemed to work fine. I see that the FCST_LEV and OBS_LEV
values in
> > your GridStat output are now 0:
> >
> >
> /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/met_tool_wrapper/GridStat/2016060100/grid_stat_fv3_DUSTTOTAL_vs_cams_DUSTTOTAL_000000L_20160601_000000V.stat
> >
> > If you change this file:
> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/config/StatAnalysis.conf.IN
> >
> > from:
> > *FCST_LEVEL_LIST = "0,0,*,*","0,1,*,*","0,2,*,*",*
> > *"0,3,*,*","0,4,*,*","0,5,*,*",**"0,6,*,*","0,7,*,*","0,8,*,*"*
> >
> > to this:
> >
> > *FCST_LEVEL_LIST = 0*
> >
> > then
> >
> > it should find the level values properly. Could you turn on
METplus log
> > debugging and rerun to see what you get?
> >
> > Also, I need to wrap up for the day, but I can check back on this
> tomorrow.
> >
> > Thanks,
> > George
> >
> > On Wed, Aug 26, 2020 at 6:17 PM Mariusz Pagowski via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > Sorry, that was on Orion, now modified  similarly on Hera in
> > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
> > > Mariusz
> > >
> > >
> > > On Wed, Aug 26, 2020 at 6:10 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Mariusz,
> > > >
> > > > Which GridStat conf file did you modify? I don't see any files
that
> > have
> > > > the FCST_VAR<n>_OPTIONS set.
> > > >
> > > > - George
> > > >
> > > > On Wed, Aug 26, 2020 at 5:54 PM Mariusz Pagowski via RT <
> > > met_help at ucar.edu
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > >
> > > > > Thanks, the scripts are
> > > > > /home/Mariusz.Pagowski/mapp_2018/scripts/met/
> > > > > met_grid_stat.sh
> > > > > met_stat_anal.sh
> > > > > with locations of config directories in the same location.
> > > > >
> > > > > Models are in
> > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll
> > > > >
> > > > > Mariusz
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 26, 2020 at 5:36 PM George McCabe via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Yes, if you put them on hera and send me the path, I can
take a
> > look.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Wed, Aug 26, 2020 at 5:24 PM Mariusz Pagowski via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> >
> > > > > > >
> > > > > > > George,
> > > > > > > I configured it the way you suggested but GridStat
produced the
> > > same
> > > > > > output
> > > > > > > as before i.e. in SL1L2 a string with time in format
> > %Y%m%d_%H%M%S
> > > > and
> > > > > > > after that StatAnalysis with empty files.
> > > > > > > The issue is that I have two sorts of files - one with a
single
> > > > > forecast
> > > > > > at
> > > > > > > a time and another one with four fcsts daily.
> > > > > > > Maybe I misunderstood something. If I put files on Hera
- would
> > you
> > > > be
> > > > > > able
> > > > > > > to look at them and the scripts?
> > > > > > > Thanks,
> > > > > > > Mariusz
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT <
> > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Mariusz,
> > > > > > > >
> > > > > > > > That is a good question. It looks like you have
multiple
> levels
> > > > > > specified
> > > > > > > > for *_VAR1_LEVELS.
> > > > > > > >
> > > > > > > > FCST_VAR1_LEVELS =
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
> > > > > > > >
> > > > > > > > If you were to add FCST_VAR1_OPTIONS, it would apply
to all
> of
> > > > those
> > > > > > > > fields, so all of those fields would contain the same
value
> in
> > > the
> > > > > > > FCST_LEV
> > > > > > > > column of the GridStat output. If you need to specify
a
> > different
> > > > > > > FCST_LEV
> > > > > > > > value for each of those levels, you would have to
split them
> up
> > > > into
> > > > > > > > separate VAR<n> items, like this:
> > > > > > > >
> > > > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > > > >
> > > > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > > > >
> > > > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > > > >
> > > > > > > > etc. and do the same for the OBS_VAR<n>_* variables.
> > > > > > > >
> > > > > > > > I'm not sure what each of the levels contain, so it is
hard
> to
> > > know
> > > > > > what
> > > > > > > to
> > > > > > > > name them. You really just need to use a unique
identifier so
> > > that
> > > > > they
> > > > > > > can
> > > > > > > > be grouped via StatAnalysis. You could set them to the
number
> > > > > > > corresponding
> > > > > > > > to each NetCDF item:
> > > > > > > >
> > > > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > > > FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > > > > FCST_VAR1_OPTIONS = set_attr_level = "0";
> > > > > > > >
> > > > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > > > FCST_VAR2_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > > > > FCST_VAR2_OPTIONS = set_attr_level = "1";
> > > > > > > >
> > > > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > > > FCST_VAR3_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > > > > FCST_VAR3_OPTIONS = set_attr_level = "2";
> > > > > > > >
> > > > > > > > Or you could use a string that identifies each level.
If they
> > are
> > > > > > > pressure
> > > > > > > > levels, you could do:
> > > > > > > >
> > > > > > > > FCST_VAR1_OPTIONS = set_attr_level = "P750";
> > > > > > > > FCST_VAR2_OPTIONS = set_attr_level = "P500";
> > > > > > > > FCST_VAR3_OPTIONS = set_attr_level = "P250";
> > > > > > > >
> > > > > > > > Whatever you decide to use, you should use the same
values
> when
> > > > > > > configuring
> > > > > > > > StatAnalysis:
> > > > > > > >
> > > > > > > > FCST_LEVEL_LIST = 0, 1, 2
> > > > > > > > or
> > > > > > > > FCST_LEVEL_LIST = P750, P500, P250
> > > > > > > >
> > > > > > > > Let me know if you have any questions or if that
solution
> > doesn't
> > > > > work
> > > > > > > for
> > > > > > > > you.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > >
> > > > > > > > >
> > > > > > > > > George,
> > > > > > > > > I am not able to figure out how to set up GridStat
to have
> a
> > > > > correct
> > > > > > > > > attribute for StatAnalysis to handle it.
> > > > > > > > > Attached are the confs I am using. Where should I
add the
> > > option
> > > > > you
> > > > > > > > > mentioned and in what form?
> > > > > > > > > I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
> > > "something";
> > > > > > > > > What is  "something" ?
> > > > > > > > > Thanks,
> > > > > > > > > Mariusz
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Mariusz,
> > > > > > > > > >
> > > > > > > > > > You are correct that StatAnalysis cannot use that
syntax
> to
> > > > > specify
> > > > > > > the
> > > > > > > > > > levels in this case. Fortunately there is a
recently
> added
> > > > > feature
> > > > > > > that
> > > > > > > > > > will allow you to get around this issue. You can
specify
> > > values
> > > > > to
> > > > > > > > > override
> > > > > > > > > > attributes in GridStat. This will allow you to
change the
> > > level
> > > > > > value
> > > > > > > > > > 20160601_060000,0,*,* to a value that is more
easily
> > > specified
> > > > in
> > > > > > the
> > > > > > > > > > StatAnalysis config file. GitHub issue #1020
outlines
> these
> > > > > changes
> > > > > > > and
> > > > > > > > > > this link takes you directly to a comment that
lists the
> > new
> > > > > > > > > > configurations:
> > > > > > > > > >
> > > > > > > > > >
> > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> > > > > > > > > >
> > > > > > > > > > You will want to set set_attr_level in GridStat,
which
> can
> > be
> > > > > added
> > > > > > > via
> > > > > > > > > > METplus with:
> > > > > > > > > >
> > > > > > > > > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
> "something";
> > > > > > > > > >
> > > > > > > > > > I don't have access to Orion yet so I can't look
at your
> > > > > > > configuration
> > > > > > > > to
> > > > > > > > > > know what exactly "something" should be. If you
need help
> > > > > figuring
> > > > > > > that
> > > > > > > > > > out, you could attach your METplus config file to
this
> > ticket
> > > > > and I
> > > > > > > > could
> > > > > > > > > > take a look.
> > > > > > > > > >
> > > > > > > > > > These changes are available since MET 9.1-beta2
and will
> be
> > > in
> > > > > MET
> > > > > > > 9.1,
> > > > > > > > > > which is close to being released. It looks like
the beta
> > > > release
> > > > > is
> > > > > > > not
> > > > > > > > > > installed on Orion but 9.1 is planned to be
installed
> > there.
> > > > > > > > > >
> > > > > > > > > > Let me know if you have any other questions.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski
via RT <
> > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > George,
> > > > > > > > > > > actually, there is a problem with the next step:
> > > > StatAnalysis.
> > > > > > > > > > > GridStat outputs time levels in the form
> > > > 20160601_060000,0,*,*
> > > > > > > > > > > but StatAnalysis does not accept
> > > > > > > > > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > > > so ends up either producing an error or empty
files
> > > depending
> > > > > how
> > > > > > > the
> > > > > > > > > > > config is set up.
> > > > > > > > > > > Is there a solution to this problem?
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Mariusz
> > > > > > > > > > >
> > > > > > > > > > > on Orion:
> > > > > > > > > > > script
> > > > > > > > > > >
/home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > > > > > > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > > > > > > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > > > > > > > > ...
> > > > > > > > > > >     | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g
\
> > > > > > > > > > >     | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g
\
> > > > > > > > > > >     > ./StatAnalysis.conf
> > > > > > > > > > >
> > > > > > > > > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > > > > > > > > 08/04 21:48:57Z run-METplus-
metplus.StatAnalysis:
> ERROR:
> > > > Fatal
> > > > > > > error
> > > > > > > > > > > occurred
> > > > > > > > > > >
> > > > > > > > > > > Outputs in
> > > > > > > > > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz
Pagowski -
> NOAA
> > > > > > Affiliate
> > > > > > > <
> > > > > > > > > > > mariusz.pagowski at noaa.gov> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks, George, works as promised,
> > > > > > > > > > > > Mariusz
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe
via
> RT <
> > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi Mariusz,
> > > > > > > > > > > >>
> > > > > > > > > > > >> To specify the time in the NetCDF structure,
you
> need
> > to
> > > > use
> > > > > > the
> > > > > > > > > > format
> > > > > > > > > > > >> YYYYMMDD_HHMMSS, so your FTIME variable
should look
> > like
> > > > > this:
> > > > > > > > > > > >>
> > > > > > > > > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > > > >>
> > > > > > > > > > > >> If the NetCDF time info is formatted in the
way the
> > MET
> > > > > tools
> > > > > > > > > expect,
> > > > > > > > > > > the
> > > > > > > > > > > >> code is smart enough to find the correct data
from
> the
> > > > time
> > > > > > > info.
> > > > > > > > > > > >>
> > > > > > > > > > > >> If that doesn't work, you may need to do
something
> > > clever
> > > > to
> > > > > > > > > > manipulate
> > > > > > > > > > > >> the
> > > > > > > > > > > >> hour value to the corresponding number of the
NetCDF
> > > > > > dimension.
> > > > > > > It
> > > > > > > > > > looks
> > > > > > > > > > > >> like your data has a time dimension of size
4, so
> > > passing
> > > > in
> > > > > > the
> > > > > > > > > hour
> > > > > > > > > > > (18)
> > > > > > > > > > > >> is outside of the valid range. I would
recommend
> > trying
> > > to
> > > > > > > format
> > > > > > > > > the
> > > > > > > > > > > >> valid
> > > > > > > > > > > >> time with YYYYMMDD_HHMMSS. If that doesn't
work, let
> > me
> > > > know
> > > > > > > and I
> > > > > > > > > try
> > > > > > > > > > > to
> > > > > > > > > > > >> help you figure out a way to get the correct
number
> > for
> > > > the
> > > > > > time
> > > > > > > > you
> > > > > > > > > > > need.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks,
> > > > > > > > > > > >> George
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz
Pagowski via
> > RT
> > > <
> > > > > > > > > > > >> met_help at ucar.edu>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > > > > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Julie,
> > > > > > > > > > > >> > It was hanging over my head but only today
I tried
> > to
> > > > > input
> > > > > > > > > forecast
> > > > > > > > > > > >> time
> > > > > > > > > > > >> > to the conf and did not figure the
solution.
> > > > > > > > > > > >> > valid?fmt string does not produce for me a
correct
> > > time
> > > > > and
> > > > > > > > > GridStat
> > > > > > > > > > > >> fails.
> > > > > > > > > > > >> > If your colleague looked into our problem
> > > > > > > > > > > >> > we would very much appreciate it, thanks,
> > > > > > > > > > > >> > Mariusz
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > models for GridStat processing
> > > > > > > > > > > >> >
> > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > > > > > > > > >> >
> > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > > > > > > > > > >> >
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > other stuff:
> > > > > > > > > > > >> >
> > > > > >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > > > > > > > > >> >
> > > > > > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > > > > > > > > >> >
> > > > > > > > >
> > > > >
> /scratch1/BMC/gsd-fv3-dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie
Prestopnik
> via
> > > RT
> > > > <
> > > > > > > > > > > >> met_help at ucar.edu
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > Hi Mariusz.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > I moved your question over to met_help so
that
> > > others
> > > > > > could
> > > > > > > > > > benefit
> > > > > > > > > > > >> from
> > > > > > > > > > > >> > > your questions.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > I asked a co-worker for some help with
this one
> > and
> > > he
> > > > > > said
> > > > > > > > that
> > > > > > > > > > if
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > data is in NetCDF and in a format that
MET can
> > read,
> > > > you
> > > > > > can
> > > > > > > > > > specify
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > time to use in the level, i.e.
> > > (YYYYMMDD_HHMMSS,*,*).
> > > > > > There
> > > > > > > > is a
> > > > > > > > > > > >> METplus
> > > > > > > > > > > >> > > use case that does this here:
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > > > > > > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
> > > > > > > "({valid?fmt=%Y%m01_000000},*,*)"
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > He said that likely won't work with this
data,
> but
> > > > there
> > > > > > is
> > > > > > > > > > likely a
> > > > > > > > > > > >> > > clever way to use the valid time to
create the
> > > > > appropriate
> > > > > > > > level
> > > > > > > > > > > value
> > > > > > > > > > > >> > for
> > > > > > > > > > > >> > > each run time in a similar way. If you
aren't
> able
> > > to
> > > > > > figure
> > > > > > > > it
> > > > > > > > > > out,
> > > > > > > > > > > >> you
> > > > > > > > > > > >> > > could provide a sample dataset and he
could help
> > > > figure
> > > > > > out
> > > > > > > > how
> > > > > > > > > to
> > > > > > > > > > > >> > > configure it.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Thanks,
> > > > > > > > > > > >> > > Julie
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto
wrote:
> > > > > > > > > > > >> > > > My start date for GridStat is e.g.
2016060118
> > for
> > > > > FCSTs
> > > > > > > and
> > > > > > > > > > OBSs.
> > > > > > > > > > > >> > > > FCSTs are output at 6-hr intervals
(00Z,
> > 06Z,12Z,
> > > > 18Z)
> > > > > > and
> > > > > > > > > OBSs
> > > > > > > > > > at
> > > > > > > > > > > >> 3-
> > > > > > > > > > > >> > > > hr intervals. For this time
GridStat.conf is
> > > > > configured
> > > > > > to
> > > > > > > > > take
> > > > > > > > > > > 3rd
> > > > > > > > > > > >> > > > FCST in the input file [in
GridStat.conf:
> > > (3,1,*,*)
> > > > > and
> > > > > > > 6th
> > > > > > > > > OBS
> > > > > > > > > > > in
> > > > > > > > > > > >> > > > the other input file: (6,1,*,*) That
only
> works
> > > > > properly
> > > > > > > for
> > > > > > > > > > > >> > > > FCSTs/OBSs starting at 18z with
intervals of
> 24
> > > > hours
> > > > > > when
> > > > > > > > > > running
> > > > > > > > > > > >> > > > over e.g. a month-long period. However,
if I
> > > wanted
> > > > to
> > > > > > run
> > > > > > > > > > > GridStat
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > > > get statistics with data at intervals
of 6
> hours
> > > > e.g.
> > > > > > > > > > > >> SDATE=2016060118
> > > > > > > > > > > >> > > > and EDATE=2016063118 that is incorrect
since
> for
> > > 00Z
> > > > > > > > (3,1,*,*)
> > > > > > > > > > > needs
> > > > > > > > > > > >> > > > to become (0,1,*,*) etc. Is there a way
to
> pass
> > > > times
> > > > > of
> > > > > > > > > > > FCSTs/OBSs
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > > > GridStat.conf when running
master_metplus.py ?
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Thanks again,
> > > > > > > > > > > >> > > > Mariusz
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >> --
> > > > > > > > > > > >> George McCabe - Software Engineer III
> > > > > > > > > > > >> National Center for Atmospheric Research
> > > > > > > > > > > >> Research Applications Laboratory
> > > > > > > > > > > >> 303-497-2768
> > > > > > > > > > > >> ---
> > > > > > > > > > > >> My working day may not be your working day.
Please
> do
> > > not
> > > > > feel
> > > > > > > > > obliged
> > > > > > > > > > > to
> > > > > > > > > > > >> reply to this email outside of your normal
working
> > > hours.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > 303-497-2768
> > > > > > > > > > ---
> > > > > > > > > > My working day may not be your working day. Please
do not
> > > feel
> > > > > > > obliged
> > > > > > > > to
> > > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Thu Aug 27 14:55:16 2020

George,
Thanks for helping, I don't have all the bells and whistles that you
suggested but METPlus appears to be working correctly.
Pls have a look one last time, thanks,
Mariusz

/home/Mariusz.Pagowski/mapp_2018/scripts/met

/scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/met_tool_wrapper/
GridStat/
StatAnalysis/


On Thu, Aug 27, 2020 at 9:23 AM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Mariusz,
>
> You need to specify the name for each of the VAR<n> variables, even
if they
> are all the same value. You can refer to other METplus variables to
avoid
> duplication:
>
> FCST_VAR1_NAME = _FCSTVAR_
> FCST_VAR1_LEVELS = "(_FTIME_,0,*,*)"
> FCST_VAR1_OPTIONS = set_attr_level = "P100";
>
> FCST_VAR2_NAME = {FCST_VAR1_NAME}
> FCST_VAR2_LEVELS = "(_FTIME_,1,*,*)"
> FCST_VAR2_OPTIONS = set_attr_level = "P250";
>
> FCST_VAR3_NAME = {FCST_VAR1_NAME}
> FCST_VAR3_LEVELS = "(_FTIME_,2,*,*)"
> FCST_VAR3_OPTIONS = set_attr_level = "P400";
>
> FCST_VAR4_NAME = {FCST_VAR1_NAME}
> FCST_VAR4_LEVELS = "(_FTIME_,3,*,*)"
> FCST_VAR4_OPTIONS = set_attr_level = "P500";
>
> FCST_VAR5_NAME = {FCST_VAR1_NAME}
> FCST_VAR5_LEVELS = "(_FTIME_,4,*,*)"
> FCST_VAR5_OPTIONS = set_attr_level = "P600";
>
> FCST_VAR6_NAME = {FCST_VAR1_NAME}
> FCST_VAR6_LEVELS = "(_FTIME_,5,*,*)"
> FCST_VAR6_OPTIONS = set_attr_level = "P700";
>
> FCST_VAR7_NAME = {FCST_VAR1_NAME}
> FCST_VAR7_LEVELS = "(_FTIME_,6,*,*)"
> FCST_VAR7_OPTIONS = set_attr_level = "P850";
>
> FCST_VAR8_NAME = {FCST_VAR1_NAME}
> FCST_VAR8_LEVELS = "(_FTIME_,7,*,*)"
> FCST_VAR8_OPTIONS = set_attr_level = "P925";
>
> FCST_VAR9_NAME = {FCST_VAR1_NAME}
> FCST_VAR9_LEVELS = "(_FTIME_,8,*,*)"
> FCST_VAR9_OPTIONS = set_attr_level = "P1000";
>
> OBS_VAR1_NAME = _OBSVAR_
> OBS_VAR1_LEVELS = "(_OTIME_,0,*,*)"
> OBS_VAR1_OPTIONS = {FCST_VAR1_OPTIONS}
>
> OBS_VAR2_NAME = {OBS_VAR1_NAME}
> OBS_VAR2_LEVELS = "(_OTIME_,1,*,*)"
> OBS_VAR2_OPTIONS = {FCST_VAR2_OPTIONS}
>
> OBS_VAR3_NAME = {OBS_VAR1_NAME}
> OBS_VAR3_LEVELS = "(_OTIME_,2,*,*)"
> OBS_VAR3_OPTIONS = {FCST_VAR3_OPTIONS}
>
> OBS_VAR4_NAME = {OBS_VAR1_NAME}
> OBS_VAR4_LEVELS = "(_OTIME_,3,*,*)"
> OBS_VAR4_OPTIONS = {FCST_VAR4_OPTIONS}
>
> OBS_VAR5_NAME = {OBS_VAR1_NAME}
> OBS_VAR5_LEVELS = "(_OTIME_,4,*,*)"
> OBS_VAR5_OPTIONS = {FCST_VAR5_OPTIONS}
>
> OBS_VAR6_NAME = {OBS_VAR1_NAME}
> OBS_VAR6_LEVELS = "(_OTIME_,5,*,*)"
> OBS_VAR6_OPTIONS = {FCST_VAR6_OPTIONS}
>
> OBS_VAR7_NAME = {OBS_VAR1_NAME}
> OBS_VAR7_LEVELS = "(_OTIME_,6,*,*)"
> OBS_VAR7_OPTIONS = {FCST_VAR7_OPTIONS}
>
> OBS_VAR8_NAME = {OBS_VAR1_NAME}
> OBS_VAR8_LEVELS = "(_OTIME_,7,*,*)"
> OBS_VAR8_OPTIONS = {FCST_VAR8_OPTIONS}
>
> OBS_VAR9_NAME = {OBS_VAR1_NAME}
> OBS_VAR9_LEVELS = "(_OTIME_,8,*,*)"
> OBS_VAR9_OPTIONS = {FCST_VAR9_OPTIONS}
>
> On a side note, here are a few suggestions:
>
> If you need to reference an environment variable in your METplus
config
> file, you can use the {ENV[varname]} syntax. For example, instead of
> replacing _FCSTVAR_ with ${FCST_VAR}, you can set:
>
> FCST_VAR1_NAME = {ENV[FCST_VAR]}
>
> You can also set other METplus variables that aren't reserved names
and
> reference them in your config file. If you do this, I would
recommend
> putting some sort of prefix to ensure you aren't conflicting with a
> potentially existing variable. You could do something like this:
>
> MAPP_MODELNAME = fv3
> FCST_GRID_STAT_INPUT_TEMPLATE =
{MAPP_MODELNAME}_aeros_{init?fmt=%Y%m%d%H}_
> pll.nc
>
> Doing this would simplify your calling script so you don't need to
generate
> a config file using sed commands. It would also make it easier for
> debugging. I have a difficult time trying to run a modified version
of your
> script, but if everything was in METplus conf files, I could just
override
> certain variables to write to another area, i.e.
>
> $METPLUS_PATH/ush/master_metplus.py -c ./GridStat.conf -c
./main.conf -c
> dir.OUTPUT_BASE=/scratch1/BMC/dtc/George.Mccabe/mapp-test
>
> The above command would use your configuration but the output would
be sent
> to my directory instead.
>
> Let me know if you have any questions.
>
> Thanks,
> George
>
> On Wed, Aug 26, 2020 at 7:59 PM Mariusz Pagowski via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > Made the changes but GridStat only outputting one pressure levels.
Pls
> see
> > the new  confs in
> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
> > Mariusz
> >
> > On Wed, Aug 26, 2020 at 6:36 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Thanks. I recommend setting LOG_LEVEL=DEBUG under the [config]
section
> of
> > > your main.conf file. This will give you more log output to work
with.
> > >
> > > I thought that you would need to put a semicolon at the end of
the
> > OPTIONS
> > > value, i.e.:
> > >
> > > FCST_VAR1_OPTIONS = set_attr_level = "0"*;*
> > >
> > > but it seemed to work fine. I see that the FCST_LEV and OBS_LEV
values
> in
> > > your GridStat output are now 0:
> > >
> > >
> >
> /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/met_tool_wrapper/GridStat/2016060100/grid_stat_fv3_DUSTTOTAL_vs_cams_DUSTTOTAL_000000L_20160601_000000V.stat
> > >
> > > If you change this file:
> > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config/
> StatAnalysis.conf.IN
> > >
> > > from:
> > > *FCST_LEVEL_LIST = "0,0,*,*","0,1,*,*","0,2,*,*",*
> > > *"0,3,*,*","0,4,*,*","0,5,*,*",**"0,6,*,*","0,7,*,*","0,8,*,*"*
> > >
> > > to this:
> > >
> > > *FCST_LEVEL_LIST = 0*
> > >
> > > then
> > >
> > > it should find the level values properly. Could you turn on
METplus log
> > > debugging and rerun to see what you get?
> > >
> > > Also, I need to wrap up for the day, but I can check back on
this
> > tomorrow.
> > >
> > > Thanks,
> > > George
> > >
> > > On Wed, Aug 26, 2020 at 6:17 PM Mariusz Pagowski via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > > >
> > > > Sorry, that was on Orion, now modified  similarly on Hera in
> > > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
> > > > Mariusz
> > > >
> > > >
> > > > On Wed, Aug 26, 2020 at 6:10 PM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Mariusz,
> > > > >
> > > > > Which GridStat conf file did you modify? I don't see any
files that
> > > have
> > > > > the FCST_VAR<n>_OPTIONS set.
> > > > >
> > > > > - George
> > > > >
> > > > > On Wed, Aug 26, 2020 at 5:54 PM Mariusz Pagowski via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > > >
> > > > > > Thanks, the scripts are
> > > > > > /home/Mariusz.Pagowski/mapp_2018/scripts/met/
> > > > > > met_grid_stat.sh
> > > > > > met_stat_anal.sh
> > > > > > with locations of config directories in the same location.
> > > > > >
> > > > > > Models are in
> > > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll
> > > > > >
> > > > > > Mariusz
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 26, 2020 at 5:36 PM George McCabe via RT <
> > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Yes, if you put them on hera and send me the path, I can
take a
> > > look.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > > On Wed, Aug 26, 2020 at 5:24 PM Mariusz Pagowski via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > >
> > > > > > > >
> > > > > > > > George,
> > > > > > > > I configured it the way you suggested but GridStat
produced
> the
> > > > same
> > > > > > > output
> > > > > > > > as before i.e. in SL1L2 a string with time in format
> > > %Y%m%d_%H%M%S
> > > > > and
> > > > > > > > after that StatAnalysis with empty files.
> > > > > > > > The issue is that I have two sorts of files - one with
a
> single
> > > > > > forecast
> > > > > > > at
> > > > > > > > a time and another one with four fcsts daily.
> > > > > > > > Maybe I misunderstood something. If I put files on
Hera -
> would
> > > you
> > > > > be
> > > > > > > able
> > > > > > > > to look at them and the scripts?
> > > > > > > > Thanks,
> > > > > > > > Mariusz
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Mariusz,
> > > > > > > > >
> > > > > > > > > That is a good question. It looks like you have
multiple
> > levels
> > > > > > > specified
> > > > > > > > > for *_VAR1_LEVELS.
> > > > > > > > >
> > > > > > > > > FCST_VAR1_LEVELS =
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
> > > > > > > > >
> > > > > > > > > If you were to add FCST_VAR1_OPTIONS, it would apply
to all
> > of
> > > > > those
> > > > > > > > > fields, so all of those fields would contain the
same value
> > in
> > > > the
> > > > > > > > FCST_LEV
> > > > > > > > > column of the GridStat output. If you need to
specify a
> > > different
> > > > > > > > FCST_LEV
> > > > > > > > > value for each of those levels, you would have to
split
> them
> > up
> > > > > into
> > > > > > > > > separate VAR<n> items, like this:
> > > > > > > > >
> > > > > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > > > > FCST_VAR1_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > > > > >
> > > > > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > > > > FCST_VAR2_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > > > > >
> > > > > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > > > > FCST_VAR3_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > > > > >
> > > > > > > > > etc. and do the same for the OBS_VAR<n>_* variables.
> > > > > > > > >
> > > > > > > > > I'm not sure what each of the levels contain, so it
is hard
> > to
> > > > know
> > > > > > > what
> > > > > > > > to
> > > > > > > > > name them. You really just need to use a unique
identifier
> so
> > > > that
> > > > > > they
> > > > > > > > can
> > > > > > > > > be grouped via StatAnalysis. You could set them to
the
> number
> > > > > > > > corresponding
> > > > > > > > > to each NetCDF item:
> > > > > > > > >
> > > > > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > > > > FCST_VAR1_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > > > > > FCST_VAR1_OPTIONS = set_attr_level = "0";
> > > > > > > > >
> > > > > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > > > > FCST_VAR2_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > > > > > FCST_VAR2_OPTIONS = set_attr_level = "1";
> > > > > > > > >
> > > > > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > > > > FCST_VAR3_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > > > > > FCST_VAR3_OPTIONS = set_attr_level = "2";
> > > > > > > > >
> > > > > > > > > Or you could use a string that identifies each
level. If
> they
> > > are
> > > > > > > > pressure
> > > > > > > > > levels, you could do:
> > > > > > > > >
> > > > > > > > > FCST_VAR1_OPTIONS = set_attr_level = "P750";
> > > > > > > > > FCST_VAR2_OPTIONS = set_attr_level = "P500";
> > > > > > > > > FCST_VAR3_OPTIONS = set_attr_level = "P250";
> > > > > > > > >
> > > > > > > > > Whatever you decide to use, you should use the same
values
> > when
> > > > > > > > configuring
> > > > > > > > > StatAnalysis:
> > > > > > > > >
> > > > > > > > > FCST_LEVEL_LIST = 0, 1, 2
> > > > > > > > > or
> > > > > > > > > FCST_LEVEL_LIST = P750, P500, P250
> > > > > > > > >
> > > > > > > > > Let me know if you have any questions or if that
solution
> > > doesn't
> > > > > > work
> > > > > > > > for
> > > > > > > > > you.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski
via RT <
> > > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > >
> > > > > > > > > >
> > > > > > > > > > George,
> > > > > > > > > > I am not able to figure out how to set up GridStat
to
> have
> > a
> > > > > > correct
> > > > > > > > > > attribute for StatAnalysis to handle it.
> > > > > > > > > > Attached are the confs I am using. Where should I
add the
> > > > option
> > > > > > you
> > > > > > > > > > mentioned and in what form?
> > > > > > > > > > I.e. FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level
=
> > > > "something";
> > > > > > > > > > What is  "something" ?
> > > > > > > > > > Thanks,
> > > > > > > > > > Mariusz
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Mariusz,
> > > > > > > > > > >
> > > > > > > > > > > You are correct that StatAnalysis cannot use
that
> syntax
> > to
> > > > > > specify
> > > > > > > > the
> > > > > > > > > > > levels in this case. Fortunately there is a
recently
> > added
> > > > > > feature
> > > > > > > > that
> > > > > > > > > > > will allow you to get around this issue. You can
> specify
> > > > values
> > > > > > to
> > > > > > > > > > override
> > > > > > > > > > > attributes in GridStat. This will allow you to
change
> the
> > > > level
> > > > > > > value
> > > > > > > > > > > 20160601_060000,0,*,* to a value that is more
easily
> > > > specified
> > > > > in
> > > > > > > the
> > > > > > > > > > > StatAnalysis config file. GitHub issue #1020
outlines
> > these
> > > > > > changes
> > > > > > > > and
> > > > > > > > > > > this link takes you directly to a comment that
lists
> the
> > > new
> > > > > > > > > > > configurations:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> > > > > > > > > > >
> > > > > > > > > > > You will want to set set_attr_level in GridStat,
which
> > can
> > > be
> > > > > > added
> > > > > > > > via
> > > > > > > > > > > METplus with:
> > > > > > > > > > >
> > > > > > > > > > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
> > "something";
> > > > > > > > > > >
> > > > > > > > > > > I don't have access to Orion yet so I can't look
at
> your
> > > > > > > > configuration
> > > > > > > > > to
> > > > > > > > > > > know what exactly "something" should be. If you
need
> help
> > > > > > figuring
> > > > > > > > that
> > > > > > > > > > > out, you could attach your METplus config file
to this
> > > ticket
> > > > > > and I
> > > > > > > > > could
> > > > > > > > > > > take a look.
> > > > > > > > > > >
> > > > > > > > > > > These changes are available since MET 9.1-beta2
and
> will
> > be
> > > > in
> > > > > > MET
> > > > > > > > 9.1,
> > > > > > > > > > > which is close to being released. It looks like
the
> beta
> > > > > release
> > > > > > is
> > > > > > > > not
> > > > > > > > > > > installed on Orion but 9.1 is planned to be
installed
> > > there.
> > > > > > > > > > >
> > > > > > > > > > > Let me know if you have any other questions.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > George
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski
via RT
> <
> > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > George,
> > > > > > > > > > > > actually, there is a problem with the next
step:
> > > > > StatAnalysis.
> > > > > > > > > > > > GridStat outputs time levels in the form
> > > > > 20160601_060000,0,*,*
> > > > > > > > > > > > but StatAnalysis does not accept
> > > > > > > > > > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > > > > so ends up either producing an error or empty
files
> > > > depending
> > > > > > how
> > > > > > > > the
> > > > > > > > > > > > config is set up.
> > > > > > > > > > > > Is there a solution to this problem?
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Mariusz
> > > > > > > > > > > >
> > > > > > > > > > > > on Orion:
> > > > > > > > > > > > script
> > > > > > > > > > > >
/home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > > > > > > > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > > > > > > > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > > > > > > > > > ...
> > > > > > > > > > > >     | sed
s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > > > > > >     | sed
s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > > > > > >     > ./StatAnalysis.conf
> > > > > > > > > > > >
> > > > > > > > > > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > > > > > > > > > 08/04 21:48:57Z run-METplus-
metplus.StatAnalysis:
> > ERROR:
> > > > > Fatal
> > > > > > > > error
> > > > > > > > > > > > occurred
> > > > > > > > > > > >
> > > > > > > > > > > > Outputs in
> > > > > > > > > > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz
Pagowski -
> > NOAA
> > > > > > > Affiliate
> > > > > > > > <
> > > > > > > > > > > > mariusz.pagowski at noaa.gov> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks, George, works as promised,
> > > > > > > > > > > > > Mariusz
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jul 27, 2020 at 11:13 AM George
McCabe via
> > RT <
> > > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi Mariusz,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> To specify the time in the NetCDF
structure, you
> > need
> > > to
> > > > > use
> > > > > > > the
> > > > > > > > > > > format
> > > > > > > > > > > > >> YYYYMMDD_HHMMSS, so your FTIME variable
should
> look
> > > like
> > > > > > this:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> If the NetCDF time info is formatted in the
way
> the
> > > MET
> > > > > > tools
> > > > > > > > > > expect,
> > > > > > > > > > > > the
> > > > > > > > > > > > >> code is smart enough to find the correct
data from
> > the
> > > > > time
> > > > > > > > info.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> If that doesn't work, you may need to do
something
> > > > clever
> > > > > to
> > > > > > > > > > > manipulate
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> hour value to the corresponding number of
the
> NetCDF
> > > > > > > dimension.
> > > > > > > > It
> > > > > > > > > > > looks
> > > > > > > > > > > > >> like your data has a time dimension of size
4, so
> > > > passing
> > > > > in
> > > > > > > the
> > > > > > > > > > hour
> > > > > > > > > > > > (18)
> > > > > > > > > > > > >> is outside of the valid range. I would
recommend
> > > trying
> > > > to
> > > > > > > > format
> > > > > > > > > > the
> > > > > > > > > > > > >> valid
> > > > > > > > > > > > >> time with YYYYMMDD_HHMMSS. If that doesn't
work,
> let
> > > me
> > > > > know
> > > > > > > > and I
> > > > > > > > > > try
> > > > > > > > > > > > to
> > > > > > > > > > > > >> help you figure out a way to get the
correct
> number
> > > for
> > > > > the
> > > > > > > time
> > > > > > > > > you
> > > > > > > > > > > > need.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >> George
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz
Pagowski
> via
> > > RT
> > > > <
> > > > > > > > > > > > >> met_help at ucar.edu>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > > > > > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Julie,
> > > > > > > > > > > > >> > It was hanging over my head but only
today I
> tried
> > > to
> > > > > > input
> > > > > > > > > > forecast
> > > > > > > > > > > > >> time
> > > > > > > > > > > > >> > to the conf and did not figure the
solution.
> > > > > > > > > > > > >> > valid?fmt string does not produce for me
a
> correct
> > > > time
> > > > > > and
> > > > > > > > > > GridStat
> > > > > > > > > > > > >> fails.
> > > > > > > > > > > > >> > If your colleague looked into our problem
> > > > > > > > > > > > >> > we would very much appreciate it, thanks,
> > > > > > > > > > > > >> > Mariusz
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > models for GridStat processing
> > > > > > > > > > > > >> >
> > > > > > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > > > > > > > > > >> >
> > > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
> > > > > > > > > > > > >> >
> > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > other stuff:
> > > > > > > > > > > > >> >
> > > > > > >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > > > > > > > > > >> >
> > > > > > > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > > > > > > > > > >> >
> > > > > > > > > >
> > > > > >
> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie
Prestopnik
> > via
> > > > RT
> > > > > <
> > > > > > > > > > > > >> met_help at ucar.edu
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > Hi Mariusz.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > I moved your question over to met_help
so that
> > > > others
> > > > > > > could
> > > > > > > > > > > benefit
> > > > > > > > > > > > >> from
> > > > > > > > > > > > >> > > your questions.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > I asked a co-worker for some help with
this
> one
> > > and
> > > > he
> > > > > > > said
> > > > > > > > > that
> > > > > > > > > > > if
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > > data is in NetCDF and in a format that
MET can
> > > read,
> > > > > you
> > > > > > > can
> > > > > > > > > > > specify
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > > time to use in the level, i.e.
> > > > (YYYYMMDD_HHMMSS,*,*).
> > > > > > > There
> > > > > > > > > is a
> > > > > > > > > > > > >> METplus
> > > > > > > > > > > > >> > > use case that does this here:
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > > > > > > > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
> > > > > > > > "({valid?fmt=%Y%m01_000000},*,*)"
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > He said that likely won't work with
this data,
> > but
> > > > > there
> > > > > > > is
> > > > > > > > > > > likely a
> > > > > > > > > > > > >> > > clever way to use the valid time to
create the
> > > > > > appropriate
> > > > > > > > > level
> > > > > > > > > > > > value
> > > > > > > > > > > > >> > for
> > > > > > > > > > > > >> > > each run time in a similar way. If you
aren't
> > able
> > > > to
> > > > > > > figure
> > > > > > > > > it
> > > > > > > > > > > out,
> > > > > > > > > > > > >> you
> > > > > > > > > > > > >> > > could provide a sample dataset and he
could
> help
> > > > > figure
> > > > > > > out
> > > > > > > > > how
> > > > > > > > > > to
> > > > > > > > > > > > >> > > configure it.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Thanks,
> > > > > > > > > > > > >> > > Julie
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto
wrote:
> > > > > > > > > > > > >> > > > My start date for GridStat is e.g.
> 2016060118
> > > for
> > > > > > FCSTs
> > > > > > > > and
> > > > > > > > > > > OBSs.
> > > > > > > > > > > > >> > > > FCSTs are output at 6-hr intervals
(00Z,
> > > 06Z,12Z,
> > > > > 18Z)
> > > > > > > and
> > > > > > > > > > OBSs
> > > > > > > > > > > at
> > > > > > > > > > > > >> 3-
> > > > > > > > > > > > >> > > > hr intervals. For this time
GridStat.conf is
> > > > > > configured
> > > > > > > to
> > > > > > > > > > take
> > > > > > > > > > > > 3rd
> > > > > > > > > > > > >> > > > FCST in the input file [in
GridStat.conf:
> > > > (3,1,*,*)
> > > > > > and
> > > > > > > > 6th
> > > > > > > > > > OBS
> > > > > > > > > > > > in
> > > > > > > > > > > > >> > > > the other input file: (6,1,*,*) That
only
> > works
> > > > > > properly
> > > > > > > > for
> > > > > > > > > > > > >> > > > FCSTs/OBSs starting at 18z with
intervals of
> > 24
> > > > > hours
> > > > > > > when
> > > > > > > > > > > running
> > > > > > > > > > > > >> > > > over e.g. a month-long period.
However, if I
> > > > wanted
> > > > > to
> > > > > > > run
> > > > > > > > > > > > GridStat
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> > > > get statistics with data at intervals
of 6
> > hours
> > > > > e.g.
> > > > > > > > > > > > >> SDATE=2016060118
> > > > > > > > > > > > >> > > > and EDATE=2016063118 that is
incorrect since
> > for
> > > > 00Z
> > > > > > > > > (3,1,*,*)
> > > > > > > > > > > > needs
> > > > > > > > > > > > >> > > > to become (0,1,*,*) etc. Is there a
way to
> > pass
> > > > > times
> > > > > > of
> > > > > > > > > > > > FCSTs/OBSs
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> > > > GridStat.conf when running
> master_metplus.py ?
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Thanks again,
> > > > > > > > > > > > >> > > > Mariusz
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> --
> > > > > > > > > > > > >> George McCabe - Software Engineer III
> > > > > > > > > > > > >> National Center for Atmospheric Research
> > > > > > > > > > > > >> Research Applications Laboratory
> > > > > > > > > > > > >> 303-497-2768
> > > > > > > > > > > > >> ---
> > > > > > > > > > > > >> My working day may not be your working day.
Please
> > do
> > > > not
> > > > > > feel
> > > > > > > > > > obliged
> > > > > > > > > > > > to
> > > > > > > > > > > > >> reply to this email outside of your normal
working
> > > > hours.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > 303-497-2768
> > > > > > > > > > > ---
> > > > > > > > > > > My working day may not be your working day.
Please do
> not
> > > > feel
> > > > > > > > obliged
> > > > > > > > > to
> > > > > > > > > > > reply to this email outside of your normal
working
> hours.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Thu Aug 27 16:45:57 2020

Looks good to me! Glad to hear it.

Thanks,
George

On Thu, Aug 27, 2020 at 2:55 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> George,
> Thanks for helping, I don't have all the bells and whistles that you
> suggested but METPlus appears to be working correctly.
> Pls have a look one last time, thanks,
> Mariusz
>
> /home/Mariusz.Pagowski/mapp_2018/scripts/met
>
> /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/met_tool_wrapper/
> GridStat/
> StatAnalysis/
>
>
> On Thu, Aug 27, 2020 at 9:23 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Mariusz,
> >
> > You need to specify the name for each of the VAR<n> variables,
even if
> they
> > are all the same value. You can refer to other METplus variables
to avoid
> > duplication:
> >
> > FCST_VAR1_NAME = _FCSTVAR_
> > FCST_VAR1_LEVELS = "(_FTIME_,0,*,*)"
> > FCST_VAR1_OPTIONS = set_attr_level = "P100";
> >
> > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > FCST_VAR2_LEVELS = "(_FTIME_,1,*,*)"
> > FCST_VAR2_OPTIONS = set_attr_level = "P250";
> >
> > FCST_VAR3_NAME = {FCST_VAR1_NAME}
> > FCST_VAR3_LEVELS = "(_FTIME_,2,*,*)"
> > FCST_VAR3_OPTIONS = set_attr_level = "P400";
> >
> > FCST_VAR4_NAME = {FCST_VAR1_NAME}
> > FCST_VAR4_LEVELS = "(_FTIME_,3,*,*)"
> > FCST_VAR4_OPTIONS = set_attr_level = "P500";
> >
> > FCST_VAR5_NAME = {FCST_VAR1_NAME}
> > FCST_VAR5_LEVELS = "(_FTIME_,4,*,*)"
> > FCST_VAR5_OPTIONS = set_attr_level = "P600";
> >
> > FCST_VAR6_NAME = {FCST_VAR1_NAME}
> > FCST_VAR6_LEVELS = "(_FTIME_,5,*,*)"
> > FCST_VAR6_OPTIONS = set_attr_level = "P700";
> >
> > FCST_VAR7_NAME = {FCST_VAR1_NAME}
> > FCST_VAR7_LEVELS = "(_FTIME_,6,*,*)"
> > FCST_VAR7_OPTIONS = set_attr_level = "P850";
> >
> > FCST_VAR8_NAME = {FCST_VAR1_NAME}
> > FCST_VAR8_LEVELS = "(_FTIME_,7,*,*)"
> > FCST_VAR8_OPTIONS = set_attr_level = "P925";
> >
> > FCST_VAR9_NAME = {FCST_VAR1_NAME}
> > FCST_VAR9_LEVELS = "(_FTIME_,8,*,*)"
> > FCST_VAR9_OPTIONS = set_attr_level = "P1000";
> >
> > OBS_VAR1_NAME = _OBSVAR_
> > OBS_VAR1_LEVELS = "(_OTIME_,0,*,*)"
> > OBS_VAR1_OPTIONS = {FCST_VAR1_OPTIONS}
> >
> > OBS_VAR2_NAME = {OBS_VAR1_NAME}
> > OBS_VAR2_LEVELS = "(_OTIME_,1,*,*)"
> > OBS_VAR2_OPTIONS = {FCST_VAR2_OPTIONS}
> >
> > OBS_VAR3_NAME = {OBS_VAR1_NAME}
> > OBS_VAR3_LEVELS = "(_OTIME_,2,*,*)"
> > OBS_VAR3_OPTIONS = {FCST_VAR3_OPTIONS}
> >
> > OBS_VAR4_NAME = {OBS_VAR1_NAME}
> > OBS_VAR4_LEVELS = "(_OTIME_,3,*,*)"
> > OBS_VAR4_OPTIONS = {FCST_VAR4_OPTIONS}
> >
> > OBS_VAR5_NAME = {OBS_VAR1_NAME}
> > OBS_VAR5_LEVELS = "(_OTIME_,4,*,*)"
> > OBS_VAR5_OPTIONS = {FCST_VAR5_OPTIONS}
> >
> > OBS_VAR6_NAME = {OBS_VAR1_NAME}
> > OBS_VAR6_LEVELS = "(_OTIME_,5,*,*)"
> > OBS_VAR6_OPTIONS = {FCST_VAR6_OPTIONS}
> >
> > OBS_VAR7_NAME = {OBS_VAR1_NAME}
> > OBS_VAR7_LEVELS = "(_OTIME_,6,*,*)"
> > OBS_VAR7_OPTIONS = {FCST_VAR7_OPTIONS}
> >
> > OBS_VAR8_NAME = {OBS_VAR1_NAME}
> > OBS_VAR8_LEVELS = "(_OTIME_,7,*,*)"
> > OBS_VAR8_OPTIONS = {FCST_VAR8_OPTIONS}
> >
> > OBS_VAR9_NAME = {OBS_VAR1_NAME}
> > OBS_VAR9_LEVELS = "(_OTIME_,8,*,*)"
> > OBS_VAR9_OPTIONS = {FCST_VAR9_OPTIONS}
> >
> > On a side note, here are a few suggestions:
> >
> > If you need to reference an environment variable in your METplus
config
> > file, you can use the {ENV[varname]} syntax. For example, instead
of
> > replacing _FCSTVAR_ with ${FCST_VAR}, you can set:
> >
> > FCST_VAR1_NAME = {ENV[FCST_VAR]}
> >
> > You can also set other METplus variables that aren't reserved
names and
> > reference them in your config file. If you do this, I would
recommend
> > putting some sort of prefix to ensure you aren't conflicting with
a
> > potentially existing variable. You could do something like this:
> >
> > MAPP_MODELNAME = fv3
> > FCST_GRID_STAT_INPUT_TEMPLATE =
> {MAPP_MODELNAME}_aeros_{init?fmt=%Y%m%d%H}_
> > pll.nc
> >
> > Doing this would simplify your calling script so you don't need to
> generate
> > a config file using sed commands. It would also make it easier for
> > debugging. I have a difficult time trying to run a modified
version of
> your
> > script, but if everything was in METplus conf files, I could just
> override
> > certain variables to write to another area, i.e.
> >
> > $METPLUS_PATH/ush/master_metplus.py -c ./GridStat.conf -c
./main.conf -c
> > dir.OUTPUT_BASE=/scratch1/BMC/dtc/George.Mccabe/mapp-test
> >
> > The above command would use your configuration but the output
would be
> sent
> > to my directory instead.
> >
> > Let me know if you have any questions.
> >
> > Thanks,
> > George
> >
> > On Wed, Aug 26, 2020 at 7:59 PM Mariusz Pagowski via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > Made the changes but GridStat only outputting one pressure
levels. Pls
> > see
> > > the new  confs in
> > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
> > > Mariusz
> > >
> > > On Wed, Aug 26, 2020 at 6:36 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Thanks. I recommend setting LOG_LEVEL=DEBUG under the [config]
> section
> > of
> > > > your main.conf file. This will give you more log output to
work with.
> > > >
> > > > I thought that you would need to put a semicolon at the end of
the
> > > OPTIONS
> > > > value, i.e.:
> > > >
> > > > FCST_VAR1_OPTIONS = set_attr_level = "0"*;*
> > > >
> > > > but it seemed to work fine. I see that the FCST_LEV and
OBS_LEV
> values
> > in
> > > > your GridStat output are now 0:
> > > >
> > > >
> > >
> >
> /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/met_tool_wrapper/GridStat/2016060100/grid_stat_fv3_DUSTTOTAL_vs_cams_DUSTTOTAL_000000L_20160601_000000V.stat
> > > >
> > > > If you change this file:
> > > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config/
> > StatAnalysis.conf.IN
> > > >
> > > > from:
> > > > *FCST_LEVEL_LIST = "0,0,*,*","0,1,*,*","0,2,*,*",*
> > > >
*"0,3,*,*","0,4,*,*","0,5,*,*",**"0,6,*,*","0,7,*,*","0,8,*,*"*
> > > >
> > > > to this:
> > > >
> > > > *FCST_LEVEL_LIST = 0*
> > > >
> > > > then
> > > >
> > > > it should find the level values properly. Could you turn on
METplus
> log
> > > > debugging and rerun to see what you get?
> > > >
> > > > Also, I need to wrap up for the day, but I can check back on
this
> > > tomorrow.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Wed, Aug 26, 2020 at 6:17 PM Mariusz Pagowski via RT <
> > > met_help at ucar.edu
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > >
> > > > > Sorry, that was on Orion, now modified  similarly on Hera in
> > > > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/config
> > > > > Mariusz
> > > > >
> > > > >
> > > > > On Wed, Aug 26, 2020 at 6:10 PM George McCabe via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Mariusz,
> > > > > >
> > > > > > Which GridStat conf file did you modify? I don't see any
files
> that
> > > > have
> > > > > > the FCST_VAR<n>_OPTIONS set.
> > > > > >
> > > > > > - George
> > > > > >
> > > > > > On Wed, Aug 26, 2020 at 5:54 PM Mariusz Pagowski via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> >
> > > > > > >
> > > > > > > Thanks, the scripts are
> > > > > > > /home/Mariusz.Pagowski/mapp_2018/scripts/met/
> > > > > > > met_grid_stat.sh
> > > > > > > met_stat_anal.sh
> > > > > > > with locations of config directories in the same
location.
> > > > > > >
> > > > > > > Models are in
> > > > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll
> > > > > > >
> > > > > > > Mariusz
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 26, 2020 at 5:36 PM George McCabe via RT <
> > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yes, if you put them on hera and send me the path, I
can
> take a
> > > > look.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > > On Wed, Aug 26, 2020 at 5:24 PM Mariusz Pagowski via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > >
> > > > > > > > >
> > > > > > > > > George,
> > > > > > > > > I configured it the way you suggested but GridStat
produced
> > the
> > > > > same
> > > > > > > > output
> > > > > > > > > as before i.e. in SL1L2 a string with time in format
> > > > %Y%m%d_%H%M%S
> > > > > > and
> > > > > > > > > after that StatAnalysis with empty files.
> > > > > > > > > The issue is that I have two sorts of files - one
with a
> > single
> > > > > > > forecast
> > > > > > > > at
> > > > > > > > > a time and another one with four fcsts daily.
> > > > > > > > > Maybe I misunderstood something. If I put files on
Hera -
> > would
> > > > you
> > > > > > be
> > > > > > > > able
> > > > > > > > > to look at them and the scripts?
> > > > > > > > > Thanks,
> > > > > > > > > Mariusz
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Aug 26, 2020 at 4:17 PM George McCabe via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Mariusz,
> > > > > > > > > >
> > > > > > > > > > That is a good question. It looks like you have
multiple
> > > levels
> > > > > > > > specified
> > > > > > > > > > for *_VAR1_LEVELS.
> > > > > > > > > >
> > > > > > > > > > FCST_VAR1_LEVELS =
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)","({valid?fmt=%Y%m%d_%H%M%S},1,*,*)","({valid?fmt=%Y%m%d_%H%M%S},2,*,*)","({valid?fmt=%Y%m%d_%H%M%S},3,*,*)","({valid?fmt=%Y%m%d_%H%M%S},4,*,*)","({valid?fmt=%Y%m%d_%H%M%S},5,*,*)","({valid?fmt=%Y%m%d_%H%M%S},6,*,*)","({valid?fmt=%Y%m%d_%H%M%S},7,*,*)","({valid?fmt=%Y%m%d_%H%M%S},8,*,*)"
> > > > > > > > > >
> > > > > > > > > > If you were to add FCST_VAR1_OPTIONS, it would
apply to
> all
> > > of
> > > > > > those
> > > > > > > > > > fields, so all of those fields would contain the
same
> value
> > > in
> > > > > the
> > > > > > > > > FCST_LEV
> > > > > > > > > > column of the GridStat output. If you need to
specify a
> > > > different
> > > > > > > > > FCST_LEV
> > > > > > > > > > value for each of those levels, you would have to
split
> > them
> > > up
> > > > > > into
> > > > > > > > > > separate VAR<n> items, like this:
> > > > > > > > > >
> > > > > > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > > > > > FCST_VAR1_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > > > > > >
> > > > > > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > > > > > FCST_VAR2_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > > > > > >
> > > > > > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > > > > > FCST_VAR3_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > > > > > >
> > > > > > > > > > etc. and do the same for the OBS_VAR<n>_*
variables.
> > > > > > > > > >
> > > > > > > > > > I'm not sure what each of the levels contain, so
it is
> hard
> > > to
> > > > > know
> > > > > > > > what
> > > > > > > > > to
> > > > > > > > > > name them. You really just need to use a unique
> identifier
> > so
> > > > > that
> > > > > > > they
> > > > > > > > > can
> > > > > > > > > > be grouped via StatAnalysis. You could set them to
the
> > number
> > > > > > > > > corresponding
> > > > > > > > > > to each NetCDF item:
> > > > > > > > > >
> > > > > > > > > > FCST_VAR1_NAME = DUSTTOTAL
> > > > > > > > > > FCST_VAR1_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},0,*,*)"
> > > > > > > > > > FCST_VAR1_OPTIONS = set_attr_level = "0";
> > > > > > > > > >
> > > > > > > > > > FCST_VAR2_NAME = {FCST_VAR1_NAME}
> > > > > > > > > > FCST_VAR2_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},1,*,*)"
> > > > > > > > > > FCST_VAR2_OPTIONS = set_attr_level = "1";
> > > > > > > > > >
> > > > > > > > > > FCST_VAR3_NAME =  {FCST_VAR1_NAME}
> > > > > > > > > > FCST_VAR3_LEVELS =
"({valid?fmt=%Y%m%d_%H%M%S},2,*,*)"
> > > > > > > > > > FCST_VAR3_OPTIONS = set_attr_level = "2";
> > > > > > > > > >
> > > > > > > > > > Or you could use a string that identifies each
level. If
> > they
> > > > are
> > > > > > > > > pressure
> > > > > > > > > > levels, you could do:
> > > > > > > > > >
> > > > > > > > > > FCST_VAR1_OPTIONS = set_attr_level = "P750";
> > > > > > > > > > FCST_VAR2_OPTIONS = set_attr_level = "P500";
> > > > > > > > > > FCST_VAR3_OPTIONS = set_attr_level = "P250";
> > > > > > > > > >
> > > > > > > > > > Whatever you decide to use, you should use the
same
> values
> > > when
> > > > > > > > > configuring
> > > > > > > > > > StatAnalysis:
> > > > > > > > > >
> > > > > > > > > > FCST_LEVEL_LIST = 0, 1, 2
> > > > > > > > > > or
> > > > > > > > > > FCST_LEVEL_LIST = P750, P500, P250
> > > > > > > > > >
> > > > > > > > > > Let me know if you have any questions or if that
solution
> > > > doesn't
> > > > > > > work
> > > > > > > > > for
> > > > > > > > > > you.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Aug 26, 2020 at 12:51 PM Mariusz Pagowski
via RT
> <
> > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > George,
> > > > > > > > > > > I am not able to figure out how to set up
GridStat to
> > have
> > > a
> > > > > > > correct
> > > > > > > > > > > attribute for StatAnalysis to handle it.
> > > > > > > > > > > Attached are the confs I am using. Where should
I add
> the
> > > > > option
> > > > > > > you
> > > > > > > > > > > mentioned and in what form?
> > > > > > > > > > > I.e. FCST_GRID_STAT_VAR1_OPTIONS =
set_attr_level =
> > > > > "something";
> > > > > > > > > > > What is  "something" ?
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Mariusz
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via
RT <
> > > > > > > > met_help at ucar.edu
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Mariusz,
> > > > > > > > > > > >
> > > > > > > > > > > > You are correct that StatAnalysis cannot use
that
> > syntax
> > > to
> > > > > > > specify
> > > > > > > > > the
> > > > > > > > > > > > levels in this case. Fortunately there is a
recently
> > > added
> > > > > > > feature
> > > > > > > > > that
> > > > > > > > > > > > will allow you to get around this issue. You
can
> > specify
> > > > > values
> > > > > > > to
> > > > > > > > > > > override
> > > > > > > > > > > > attributes in GridStat. This will allow you to
change
> > the
> > > > > level
> > > > > > > > value
> > > > > > > > > > > > 20160601_060000,0,*,* to a value that is more
easily
> > > > > specified
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > StatAnalysis config file. GitHub issue #1020
outlines
> > > these
> > > > > > > changes
> > > > > > > > > and
> > > > > > > > > > > > this link takes you directly to a comment that
lists
> > the
> > > > new
> > > > > > > > > > > > configurations:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > https://github.com/NCAR/MET/issues/1020#issuecomment-
653724572
> > > > > > > > > > > >
> > > > > > > > > > > > You will want to set set_attr_level in
GridStat,
> which
> > > can
> > > > be
> > > > > > > added
> > > > > > > > > via
> > > > > > > > > > > > METplus with:
> > > > > > > > > > > >
> > > > > > > > > > > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level =
> > > "something";
> > > > > > > > > > > >
> > > > > > > > > > > > I don't have access to Orion yet so I can't
look at
> > your
> > > > > > > > > configuration
> > > > > > > > > > to
> > > > > > > > > > > > know what exactly "something" should be. If
you need
> > help
> > > > > > > figuring
> > > > > > > > > that
> > > > > > > > > > > > out, you could attach your METplus config file
to
> this
> > > > ticket
> > > > > > > and I
> > > > > > > > > > could
> > > > > > > > > > > > take a look.
> > > > > > > > > > > >
> > > > > > > > > > > > These changes are available since MET 9.1-
beta2 and
> > will
> > > be
> > > > > in
> > > > > > > MET
> > > > > > > > > 9.1,
> > > > > > > > > > > > which is close to being released. It looks
like the
> > beta
> > > > > > release
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > > > installed on Orion but 9.1 is planned to be
installed
> > > > there.
> > > > > > > > > > > >
> > > > > > > > > > > > Let me know if you have any other questions.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > George
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz
Pagowski via
> RT
> > <
> > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > George,
> > > > > > > > > > > > > actually, there is a problem with the next
step:
> > > > > > StatAnalysis.
> > > > > > > > > > > > > GridStat outputs time levels in the form
> > > > > > 20160601_060000,0,*,*
> > > > > > > > > > > > > but StatAnalysis does not accept
> > > > > > > > > > > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > > > > > so ends up either producing an error or
empty files
> > > > > depending
> > > > > > > how
> > > > > > > > > the
> > > > > > > > > > > > > config is set up.
> > > > > > > > > > > > > Is there a solution to this problem?
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Mariusz
> > > > > > > > > > > > >
> > > > > > > > > > > > > on Orion:
> > > > > > > > > > > > > script
> > > > > > > > > > > > >
> /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > > > > > > > > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > > > > > > > > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > > > > > > > > > > ...
> > > > > > > > > > > > >     | sed
s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > > > > > > >     | sed
s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > > > > > > > > >     > ./StatAnalysis.conf
> > > > > > > > > > > > >
> > > > > > > > > > > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > > > > > > > > > > 08/04 21:48:57Z run-METplus-
metplus.StatAnalysis:
> > > ERROR:
> > > > > > Fatal
> > > > > > > > > error
> > > > > > > > > > > > > occurred
> > > > > > > > > > > > >
> > > > > > > > > > > > > Outputs in
> > > > > > > > > > > > > /work/noaa/gsd-fv3-
dev/pagowski/MODEL/metstats
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz
Pagowski -
> > > NOAA
> > > > > > > > Affiliate
> > > > > > > > > <
> > > > > > > > > > > > > mariusz.pagowski at noaa.gov> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks, George, works as promised,
> > > > > > > > > > > > > > Mariusz
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Jul 27, 2020 at 11:13 AM George
McCabe
> via
> > > RT <
> > > > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Hi Mariusz,
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> To specify the time in the NetCDF
structure, you
> > > need
> > > > to
> > > > > > use
> > > > > > > > the
> > > > > > > > > > > > format
> > > > > > > > > > > > > >> YYYYMMDD_HHMMSS, so your FTIME variable
should
> > look
> > > > like
> > > > > > > this:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> If the NetCDF time info is formatted in
the way
> > the
> > > > MET
> > > > > > > tools
> > > > > > > > > > > expect,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> code is smart enough to find the correct
data
> from
> > > the
> > > > > > time
> > > > > > > > > info.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> If that doesn't work, you may need to do
> something
> > > > > clever
> > > > > > to
> > > > > > > > > > > > manipulate
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> hour value to the corresponding number of
the
> > NetCDF
> > > > > > > > dimension.
> > > > > > > > > It
> > > > > > > > > > > > looks
> > > > > > > > > > > > > >> like your data has a time dimension of
size 4,
> so
> > > > > passing
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > hour
> > > > > > > > > > > > > (18)
> > > > > > > > > > > > > >> is outside of the valid range. I would
recommend
> > > > trying
> > > > > to
> > > > > > > > > format
> > > > > > > > > > > the
> > > > > > > > > > > > > >> valid
> > > > > > > > > > > > > >> time with YYYYMMDD_HHMMSS. If that
doesn't work,
> > let
> > > > me
> > > > > > know
> > > > > > > > > and I
> > > > > > > > > > > try
> > > > > > > > > > > > > to
> > > > > > > > > > > > > >> help you figure out a way to get the
correct
> > number
> > > > for
> > > > > > the
> > > > > > > > time
> > > > > > > > > > you
> > > > > > > > > > > > > need.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > >> George
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz
Pagowski
> > via
> > > > RT
> > > > > <
> > > > > > > > > > > > > >> met_help at ucar.edu>
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
> > > > > > > > > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Julie,
> > > > > > > > > > > > > >> > It was hanging over my head but only
today I
> > tried
> > > > to
> > > > > > > input
> > > > > > > > > > > forecast
> > > > > > > > > > > > > >> time
> > > > > > > > > > > > > >> > to the conf and did not figure the
solution.
> > > > > > > > > > > > > >> > valid?fmt string does not produce for
me a
> > correct
> > > > > time
> > > > > > > and
> > > > > > > > > > > GridStat
> > > > > > > > > > > > > >> fails.
> > > > > > > > > > > > > >> > If your colleague looked into our
problem
> > > > > > > > > > > > > >> > we would very much appreciate it,
thanks,
> > > > > > > > > > > > > >> > Mariusz
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > models for GridStat processing
> > > > > > > > > > > > > >> >
> > > > > > > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > > > > > > > > > > >> >
> > > > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
> > > > > > > > > > > > > >> >
> > > > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > other stuff:
> > > > > > > > > > > > > >> >
> > > > > > > >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > > > > > > > > > > >> >
> > > > > > > > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > > > > > > > > > > >> >
> > > > > > > > > > >
> > > > > > >
> > > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie
> Prestopnik
> > > via
> > > > > RT
> > > > > > <
> > > > > > > > > > > > > >> met_help at ucar.edu
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > > Hi Mariusz.
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > I moved your question over to
met_help so
> that
> > > > > others
> > > > > > > > could
> > > > > > > > > > > > benefit
> > > > > > > > > > > > > >> from
> > > > > > > > > > > > > >> > > your questions.
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > I asked a co-worker for some help
with this
> > one
> > > > and
> > > > > he
> > > > > > > > said
> > > > > > > > > > that
> > > > > > > > > > > > if
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> > > data is in NetCDF and in a format
that MET
> can
> > > > read,
> > > > > > you
> > > > > > > > can
> > > > > > > > > > > > specify
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> > > time to use in the level, i.e.
> > > > > (YYYYMMDD_HHMMSS,*,*).
> > > > > > > > There
> > > > > > > > > > is a
> > > > > > > > > > > > > >> METplus
> > > > > > > > > > > > > >> > > use case that does this here:
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > > > > > > > > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
> > > > > > > > > "({valid?fmt=%Y%m01_000000},*,*)"
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > He said that likely won't work with
this
> data,
> > > but
> > > > > > there
> > > > > > > > is
> > > > > > > > > > > > likely a
> > > > > > > > > > > > > >> > > clever way to use the valid time to
create
> the
> > > > > > > appropriate
> > > > > > > > > > level
> > > > > > > > > > > > > value
> > > > > > > > > > > > > >> > for
> > > > > > > > > > > > > >> > > each run time in a similar way. If
you
> aren't
> > > able
> > > > > to
> > > > > > > > figure
> > > > > > > > > > it
> > > > > > > > > > > > out,
> > > > > > > > > > > > > >> you
> > > > > > > > > > > > > >> > > could provide a sample dataset and he
could
> > help
> > > > > > figure
> > > > > > > > out
> > > > > > > > > > how
> > > > > > > > > > > to
> > > > > > > > > > > > > >> > > configure it.
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > Thanks,
> > > > > > > > > > > > > >> > > Julie
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto
wrote:
> > > > > > > > > > > > > >> > > > My start date for GridStat is e.g.
> > 2016060118
> > > > for
> > > > > > > FCSTs
> > > > > > > > > and
> > > > > > > > > > > > OBSs.
> > > > > > > > > > > > > >> > > > FCSTs are output at 6-hr intervals
(00Z,
> > > > 06Z,12Z,
> > > > > > 18Z)
> > > > > > > > and
> > > > > > > > > > > OBSs
> > > > > > > > > > > > at
> > > > > > > > > > > > > >> 3-
> > > > > > > > > > > > > >> > > > hr intervals. For this time
GridStat.conf
> is
> > > > > > > configured
> > > > > > > > to
> > > > > > > > > > > take
> > > > > > > > > > > > > 3rd
> > > > > > > > > > > > > >> > > > FCST in the input file [in
GridStat.conf:
> > > > > (3,1,*,*)
> > > > > > > and
> > > > > > > > > 6th
> > > > > > > > > > > OBS
> > > > > > > > > > > > > in
> > > > > > > > > > > > > >> > > > the other input file: (6,1,*,*)
That only
> > > works
> > > > > > > properly
> > > > > > > > > for
> > > > > > > > > > > > > >> > > > FCSTs/OBSs starting at 18z with
intervals
> of
> > > 24
> > > > > > hours
> > > > > > > > when
> > > > > > > > > > > > running
> > > > > > > > > > > > > >> > > > over e.g. a month-long period.
However,
> if I
> > > > > wanted
> > > > > > to
> > > > > > > > run
> > > > > > > > > > > > > GridStat
> > > > > > > > > > > > > >> to
> > > > > > > > > > > > > >> > > > get statistics with data at
intervals of 6
> > > hours
> > > > > > e.g.
> > > > > > > > > > > > > >> SDATE=2016060118
> > > > > > > > > > > > > >> > > > and EDATE=2016063118 that is
incorrect
> since
> > > for
> > > > > 00Z
> > > > > > > > > > (3,1,*,*)
> > > > > > > > > > > > > needs
> > > > > > > > > > > > > >> > > > to become (0,1,*,*) etc. Is there a
way to
> > > pass
> > > > > > times
> > > > > > > of
> > > > > > > > > > > > > FCSTs/OBSs
> > > > > > > > > > > > > >> to
> > > > > > > > > > > > > >> > > > GridStat.conf when running
> > master_metplus.py ?
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > Thanks again,
> > > > > > > > > > > > > >> > > > Mariusz
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> --
> > > > > > > > > > > > > >> George McCabe - Software Engineer III
> > > > > > > > > > > > > >> National Center for Atmospheric Research
> > > > > > > > > > > > > >> Research Applications Laboratory
> > > > > > > > > > > > > >> 303-497-2768
> > > > > > > > > > > > > >> ---
> > > > > > > > > > > > > >> My working day may not be your working
day.
> Please
> > > do
> > > > > not
> > > > > > > feel
> > > > > > > > > > > obliged
> > > > > > > > > > > > > to
> > > > > > > > > > > > > >> reply to this email outside of your
normal
> working
> > > > > hours.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > > 303-497-2768
> > > > > > > > > > > > ---
> > > > > > > > > > > > My working day may not be your working day.
Please do
> > not
> > > > > feel
> > > > > > > > > obliged
> > > > > > > > > > to
> > > > > > > > > > > > reply to this email outside of your normal
working
> > hours.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > 303-497-2768
> > > > > > > > > > ---
> > > > > > > > > > My working day may not be your working day. Please
do not
> > > feel
> > > > > > > obliged
> > > > > > > > to
> > > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------


More information about the Met_help mailing list