[Met_help] [rt.rap.ucar.edu #95850] History for GridStat Question
George McCabe via RT
met_help at ucar.edu
Wed Aug 19 15:10:24 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
My start date for GridStat is e.g. 2016060118 for FCSTs and OBSs. FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and OBSs at 3-hr intervals. For this time GridStat.conf is configured to take 3rd FCST in the input file [in GridStat.conf: (3,1,*,*) and 6th OBS in the other input file: (6,1,*,*) That only works properly for FCSTs/OBSs starting at 18z with intervals of 24 hours when running over e.g. a month-long period. However, if I wanted to run GridStat to get statistics with data at intervals of 6 hours e.g. SDATE=2016060118 and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*) needs to become (0,1,*,*) etc. Is there a way to pass times of FCSTs/OBSs to GridStat.conf when running master_metplus.py ?
Thanks again,
Mariusz
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: GridStat Question
From: Julie Prestopnik
Time: Wed Jul 08 10:47:03 2020
Hi Mariusz.
I moved your question over to met_help so that others could benefit
from your questions.
I asked a co-worker for some help with this one and he said that if
the data is in NetCDF and in a format that MET can read, you can
specify the time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There is a METplus use case that does this here:
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
FCST_GRID_STAT_VAR1_NAME = pr
FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
He said that likely won't work with this data, but there is likely a
clever way to use the valid time to create the appropriate level value
for each run time in a similar way. If you aren't able to figure it
out, you could provide a sample dataset and he could help figure out
how to configure it.
Thanks,
Julie
On Wed Jul 08 10:39:59 2020, jpresto wrote:
> My start date for GridStat is e.g. 2016060118 for FCSTs and OBSs.
> FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and OBSs at
3-
> hr intervals. For this time GridStat.conf is configured to take 3rd
> FCST in the input file [in GridStat.conf: (3,1,*,*) and 6th OBS in
> the other input file: (6,1,*,*) That only works properly for
> FCSTs/OBSs starting at 18z with intervals of 24 hours when running
> over e.g. a month-long period. However, if I wanted to run GridStat
to
> get statistics with data at intervals of 6 hours e.g.
SDATE=2016060118
> and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*) needs
> to become (0,1,*,*) etc. Is there a way to pass times of FCSTs/OBSs
to
> GridStat.conf when running master_metplus.py ?
>
> Thanks again,
> Mariusz
------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Thu Jul 23 19:06:34 2020
Julie,
It was hanging over my head but only today I tried to input forecast
time
to the conf and did not figure the solution.
valid?fmt string does not produce for me a correct time and GridStat
fails.
If your colleague looked into our problem
we would very much appreciate it, thanks,
Mariusz
models for GridStat processing
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
other stuff:
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
/scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
/scratch1/BMC/gsd-fv3-dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Mariusz.
>
> I moved your question over to met_help so that others could benefit
from
> your questions.
>
> I asked a co-worker for some help with this one and he said that if
the
> data is in NetCDF and in a format that MET can read, you can specify
the
> time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is a
METplus
> use case that does this here:
>
>
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
>
> FCST_GRID_STAT_VAR1_NAME = pr
> FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
>
> He said that likely won't work with this data, but there is likely a
> clever way to use the valid time to create the appropriate level
value for
> each run time in a similar way. If you aren't able to figure it out,
you
> could provide a sample dataset and he could help figure out how to
> configure it.
>
> Thanks,
> Julie
>
> On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > My start date for GridStat is e.g. 2016060118 for FCSTs and OBSs.
> > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and OBSs at
3-
> > hr intervals. For this time GridStat.conf is configured to take
3rd
> > FCST in the input file [in GridStat.conf: (3,1,*,*) and 6th OBS
in
> > the other input file: (6,1,*,*) That only works properly for
> > FCSTs/OBSs starting at 18z with intervals of 24 hours when running
> > over e.g. a month-long period. However, if I wanted to run
GridStat to
> > get statistics with data at intervals of 6 hours e.g.
SDATE=2016060118
> > and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*)
needs
> > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs to
> > GridStat.conf when running master_metplus.py ?
> >
> > Thanks again,
> > Mariusz
>
>
>
>
------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Mon Jul 27 11:13:14 2020
Hi Mariusz,
To specify the time in the NetCDF structure, you need to use the
format
YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
{valid?fmt=%Y%m%d_%H%M%S}
If the NetCDF time info is formatted in the way the MET tools expect,
the
code is smart enough to find the correct data from the time info.
If that doesn't work, you may need to do something clever to
manipulate the
hour value to the corresponding number of the NetCDF dimension. It
looks
like your data has a time dimension of size 4, so passing in the hour
(18)
is outside of the valid range. I would recommend trying to format the
valid
time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and I try
to
help you figure out a way to get the correct number for the time you
need.
Thanks,
George
On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> Julie,
> It was hanging over my head but only today I tried to input forecast
time
> to the conf and did not figure the solution.
> valid?fmt string does not produce for me a correct time and GridStat
fails.
> If your colleague looked into our problem
> we would very much appreciate it, thanks,
> Mariusz
>
> models for GridStat processing
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
>
> other stuff:
> /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> /scratch1/BMC/gsd-fv3-dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
>
> On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> > Hi Mariusz.
> >
> > I moved your question over to met_help so that others could
benefit from
> > your questions.
> >
> > I asked a co-worker for some help with this one and he said that
if the
> > data is in NetCDF and in a format that MET can read, you can
specify the
> > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is a
METplus
> > use case that does this here:
> >
> >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> >
> > FCST_GRID_STAT_VAR1_NAME = pr
> > FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
> >
> > He said that likely won't work with this data, but there is likely
a
> > clever way to use the valid time to create the appropriate level
value
> for
> > each run time in a similar way. If you aren't able to figure it
out, you
> > could provide a sample dataset and he could help figure out how to
> > configure it.
> >
> > Thanks,
> > Julie
> >
> > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
> > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and OBSs
at 3-
> > > hr intervals. For this time GridStat.conf is configured to take
3rd
> > > FCST in the input file [in GridStat.conf: (3,1,*,*) and 6th OBS
in
> > > the other input file: (6,1,*,*) That only works properly for
> > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
> > > over e.g. a month-long period. However, if I wanted to run
GridStat to
> > > get statistics with data at intervals of 6 hours e.g.
SDATE=2016060118
> > > and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*)
needs
> > > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs to
> > > GridStat.conf when running master_metplus.py ?
> > >
> > > Thanks again,
> > > Mariusz
> >
> >
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Mon Jul 27 22:14:31 2020
Thanks, George, works as promised,
Mariusz
On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Mariusz,
>
> To specify the time in the NetCDF structure, you need to use the
format
> YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
>
> {valid?fmt=%Y%m%d_%H%M%S}
>
> If the NetCDF time info is formatted in the way the MET tools
expect, the
> code is smart enough to find the correct data from the time info.
>
> If that doesn't work, you may need to do something clever to
manipulate the
> hour value to the corresponding number of the NetCDF dimension. It
looks
> like your data has a time dimension of size 4, so passing in the
hour (18)
> is outside of the valid range. I would recommend trying to format
the valid
> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and I
try to
> help you figure out a way to get the correct number for the time you
need.
>
> Thanks,
> George
>
> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > Julie,
> > It was hanging over my head but only today I tried to input
forecast time
> > to the conf and did not figure the solution.
> > valid?fmt string does not produce for me a correct time and
GridStat
> fails.
> > If your colleague looked into our problem
> > we would very much appreciate it, thanks,
> > Mariusz
> >
> > models for GridStat processing
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> >
> > other stuff:
> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> >
> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > > Hi Mariusz.
> > >
> > > I moved your question over to met_help so that others could
benefit
> from
> > > your questions.
> > >
> > > I asked a co-worker for some help with this one and he said that
if the
> > > data is in NetCDF and in a format that MET can read, you can
specify
> the
> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is a
> METplus
> > > use case that does this here:
> > >
> > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > >
> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
> > >
> > > He said that likely won't work with this data, but there is
likely a
> > > clever way to use the valid time to create the appropriate level
value
> > for
> > > each run time in a similar way. If you aren't able to figure it
out,
> you
> > > could provide a sample dataset and he could help figure out how
to
> > > configure it.
> > >
> > > Thanks,
> > > Julie
> > >
> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and
OBSs at 3-
> > > > hr intervals. For this time GridStat.conf is configured to
take 3rd
> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and 6th
OBS in
> > > > the other input file: (6,1,*,*) That only works properly for
> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
> > > > over e.g. a month-long period. However, if I wanted to run
GridStat
> to
> > > > get statistics with data at intervals of 6 hours e.g.
> SDATE=2016060118
> > > > and EDATE=2016063118 that is incorrect since for 00Z (3,1,*,*)
needs
> > > > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs
> to
> > > > GridStat.conf when running master_metplus.py ?
> > > >
> > > > Thanks again,
> > > > Mariusz
> > >
> > >
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Tue Jul 28 08:19:32 2020
Good to hear! I am going to close this ticket, but feel free to open a
new
ticket via met_help at ucar.edu if you have any other issues.
Thanks,
George
On Mon, Jul 27, 2020 at 10:15 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> Thanks, George, works as promised,
> Mariusz
>
> On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Mariusz,
> >
> > To specify the time in the NetCDF structure, you need to use the
format
> > YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
> >
> > {valid?fmt=%Y%m%d_%H%M%S}
> >
> > If the NetCDF time info is formatted in the way the MET tools
expect, the
> > code is smart enough to find the correct data from the time info.
> >
> > If that doesn't work, you may need to do something clever to
manipulate
> the
> > hour value to the corresponding number of the NetCDF dimension. It
looks
> > like your data has a time dimension of size 4, so passing in the
hour
> (18)
> > is outside of the valid range. I would recommend trying to format
the
> valid
> > time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and I
try to
> > help you figure out a way to get the correct number for the time
you
> need.
> >
> > Thanks,
> > George
> >
> > On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > Julie,
> > > It was hanging over my head but only today I tried to input
forecast
> time
> > > to the conf and did not figure the solution.
> > > valid?fmt string does not produce for me a correct time and
GridStat
> > fails.
> > > If your colleague looked into our problem
> > > we would very much appreciate it, thanks,
> > > Mariusz
> > >
> > > models for GridStat processing
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > >
> > > other stuff:
> > > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > >
> > > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > > Hi Mariusz.
> > > >
> > > > I moved your question over to met_help so that others could
benefit
> > from
> > > > your questions.
> > > >
> > > > I asked a co-worker for some help with this one and he said
that if
> the
> > > > data is in NetCDF and in a format that MET can read, you can
specify
> > the
> > > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is
a
> > METplus
> > > > use case that does this here:
> > > >
> > > >
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > >
> > > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
> > > >
> > > > He said that likely won't work with this data, but there is
likely a
> > > > clever way to use the valid time to create the appropriate
level
> value
> > > for
> > > > each run time in a similar way. If you aren't able to figure
it out,
> > you
> > > > could provide a sample dataset and he could help figure out
how to
> > > > configure it.
> > > >
> > > > Thanks,
> > > > Julie
> > > >
> > > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
> > > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and
OBSs at
> 3-
> > > > > hr intervals. For this time GridStat.conf is configured to
take 3rd
> > > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and 6th
OBS in
> > > > > the other input file: (6,1,*,*) That only works properly for
> > > > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
> > > > > over e.g. a month-long period. However, if I wanted to run
GridStat
> > to
> > > > > get statistics with data at intervals of 6 hours e.g.
> > SDATE=2016060118
> > > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> needs
> > > > > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs
> > to
> > > > > GridStat.conf when running master_metplus.py ?
> > > > >
> > > > > Thanks again,
> > > > > Mariusz
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Tue Aug 04 21:00:55 2020
George,
actually, there is a problem with the next step: StatAnalysis.
GridStat outputs time levels in the form 20160601_060000,0,*,*
but StatAnalysis does not accept
{valid?fmt=%Y%m%d_%H%M%S}
so ends up either producing an error or empty files depending how the
config is set up.
Is there a solution to this problem?
Thanks,
Mariusz
on Orion:
script
/home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
...
| sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
| sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> ./StatAnalysis.conf
KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR: Fatal error
occurred
Outputs in
/work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA Affiliate <
mariusz.pagowski at noaa.gov> wrote:
> Thanks, George, works as promised,
> Mariusz
>
> On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
>> Hi Mariusz,
>>
>> To specify the time in the NetCDF structure, you need to use the
format
>> YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
>>
>> {valid?fmt=%Y%m%d_%H%M%S}
>>
>> If the NetCDF time info is formatted in the way the MET tools
expect, the
>> code is smart enough to find the correct data from the time info.
>>
>> If that doesn't work, you may need to do something clever to
manipulate
>> the
>> hour value to the corresponding number of the NetCDF dimension. It
looks
>> like your data has a time dimension of size 4, so passing in the
hour (18)
>> is outside of the valid range. I would recommend trying to format
the
>> valid
>> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and I
try to
>> help you figure out a way to get the correct number for the time
you need.
>>
>> Thanks,
>> George
>>
>> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>> >
>> > Julie,
>> > It was hanging over my head but only today I tried to input
forecast
>> time
>> > to the conf and did not figure the solution.
>> > valid?fmt string does not produce for me a correct time and
GridStat
>> fails.
>> > If your colleague looked into our problem
>> > we would very much appreciate it, thanks,
>> > Mariusz
>> >
>> > models for GridStat processing
>> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
>> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
>> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
>> >
>> > other stuff:
>> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
>> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
>> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
>> >
>> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
>> met_help at ucar.edu
>> > >
>> > wrote:
>> >
>> > > Hi Mariusz.
>> > >
>> > > I moved your question over to met_help so that others could
benefit
>> from
>> > > your questions.
>> > >
>> > > I asked a co-worker for some help with this one and he said
that if
>> the
>> > > data is in NetCDF and in a format that MET can read, you can
specify
>> the
>> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There is
a
>> METplus
>> > > use case that does this here:
>> > >
>> > >
>> > >
>> >
>>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
>> > >
>> > > FCST_GRID_STAT_VAR1_NAME = pr
>> > > FCST_GRID_STAT_VAR1_LEVELS = "({valid?fmt=%Y%m01_000000},*,*)"
>> > >
>> > > He said that likely won't work with this data, but there is
likely a
>> > > clever way to use the valid time to create the appropriate
level value
>> > for
>> > > each run time in a similar way. If you aren't able to figure it
out,
>> you
>> > > could provide a sample dataset and he could help figure out how
to
>> > > configure it.
>> > >
>> > > Thanks,
>> > > Julie
>> > >
>> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
>> > > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
>> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and
OBSs at
>> 3-
>> > > > hr intervals. For this time GridStat.conf is configured to
take 3rd
>> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and 6th
OBS in
>> > > > the other input file: (6,1,*,*) That only works properly for
>> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
>> > > > over e.g. a month-long period. However, if I wanted to run
GridStat
>> to
>> > > > get statistics with data at intervals of 6 hours e.g.
>> SDATE=2016060118
>> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*) needs
>> > > > to become (0,1,*,*) etc. Is there a way to pass times of
FCSTs/OBSs
>> to
>> > > > GridStat.conf when running master_metplus.py ?
>> > > >
>> > > > Thanks again,
>> > > > Mariusz
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>>
------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Wed Aug 05 08:04:08 2020
Hi Mariusz,
You are correct that StatAnalysis cannot use that syntax to specify
the
levels in this case. Fortunately there is a recently added feature
that
will allow you to get around this issue. You can specify values to
override
attributes in GridStat. This will allow you to change the level value
20160601_060000,0,*,* to a value that is more easily specified in the
StatAnalysis config file. GitHub issue #1020 outlines these changes
and
this link takes you directly to a comment that lists the new
configurations:
https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
You will want to set set_attr_level in GridStat, which can be added
via
METplus with:
FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
I don't have access to Orion yet so I can't look at your configuration
to
know what exactly "something" should be. If you need help figuring
that
out, you could attach your METplus config file to this ticket and I
could
take a look.
These changes are available since MET 9.1-beta2 and will be in MET
9.1,
which is close to being released. It looks like the beta release is
not
installed on Orion but 9.1 is planned to be installed there.
Let me know if you have any other questions.
Thanks,
George
On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> George,
> actually, there is a problem with the next step: StatAnalysis.
> GridStat outputs time levels in the form 20160601_060000,0,*,*
> but StatAnalysis does not accept
> {valid?fmt=%Y%m%d_%H%M%S}
> so ends up either producing an error or empty files depending how
the
> config is set up.
> Is there a solution to this problem?
> Thanks,
> Mariusz
>
> on Orion:
> script
> /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> ...
> | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > ./StatAnalysis.conf
>
> KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR: Fatal
error
> occurred
>
> Outputs in
> /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
>
>
>
> On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA Affiliate <
> mariusz.pagowski at noaa.gov> wrote:
>
> > Thanks, George, works as promised,
> > Mariusz
> >
> > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> >> Hi Mariusz,
> >>
> >> To specify the time in the NetCDF structure, you need to use the
format
> >> YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
> >>
> >> {valid?fmt=%Y%m%d_%H%M%S}
> >>
> >> If the NetCDF time info is formatted in the way the MET tools
expect,
> the
> >> code is smart enough to find the correct data from the time info.
> >>
> >> If that doesn't work, you may need to do something clever to
manipulate
> >> the
> >> hour value to the corresponding number of the NetCDF dimension.
It looks
> >> like your data has a time dimension of size 4, so passing in the
hour
> (18)
> >> is outside of the valid range. I would recommend trying to format
the
> >> valid
> >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know and
I try
> to
> >> help you figure out a way to get the correct number for the time
you
> need.
> >>
> >> Thanks,
> >> George
> >>
> >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >> >
> >> > Julie,
> >> > It was hanging over my head but only today I tried to input
forecast
> >> time
> >> > to the conf and did not figure the solution.
> >> > valid?fmt string does not produce for me a correct time and
GridStat
> >> fails.
> >> > If your colleague looked into our problem
> >> > we would very much appreciate it, thanks,
> >> > Mariusz
> >> >
> >> > models for GridStat processing
> >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> >> >
> >> > other stuff:
> >> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> >> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> >> >
> >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> >> met_help at ucar.edu
> >> > >
> >> > wrote:
> >> >
> >> > > Hi Mariusz.
> >> > >
> >> > > I moved your question over to met_help so that others could
benefit
> >> from
> >> > > your questions.
> >> > >
> >> > > I asked a co-worker for some help with this one and he said
that if
> >> the
> >> > > data is in NetCDF and in a format that MET can read, you can
specify
> >> the
> >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There
is a
> >> METplus
> >> > > use case that does this here:
> >> > >
> >> > >
> >> > >
> >> >
> >>
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> >> > >
> >> > > FCST_GRID_STAT_VAR1_NAME = pr
> >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> >> > >
> >> > > He said that likely won't work with this data, but there is
likely a
> >> > > clever way to use the valid time to create the appropriate
level
> value
> >> > for
> >> > > each run time in a similar way. If you aren't able to figure
it out,
> >> you
> >> > > could provide a sample dataset and he could help figure out
how to
> >> > > configure it.
> >> > >
> >> > > Thanks,
> >> > > Julie
> >> > >
> >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> >> > > > My start date for GridStat is e.g. 2016060118 for FCSTs and
OBSs.
> >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z) and
OBSs at
> >> 3-
> >> > > > hr intervals. For this time GridStat.conf is configured to
take
> 3rd
> >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and
6th OBS
> in
> >> > > > the other input file: (6,1,*,*) That only works properly
for
> >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours when
running
> >> > > > over e.g. a month-long period. However, if I wanted to run
> GridStat
> >> to
> >> > > > get statistics with data at intervals of 6 hours e.g.
> >> SDATE=2016060118
> >> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> needs
> >> > > > to become (0,1,*,*) etc. Is there a way to pass times of
> FCSTs/OBSs
> >> to
> >> > > > GridStat.conf when running master_metplus.py ?
> >> > > >
> >> > > > Thanks again,
> >> > > > Mariusz
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >> --
> >> George McCabe - Software Engineer III
> >> National Center for Atmospheric Research
> >> Research Applications Laboratory
> >> 303-497-2768
> >> ---
> >> My working day may not be your working day. Please do not feel
obliged
> to
> >> reply to this email outside of your normal working hours.
> >>
> >>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Wed Aug 05 09:53:28 2020
Thanks, George,
What is a timeline for releasing 9.1?
Mariusz
On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Mariusz,
>
> You are correct that StatAnalysis cannot use that syntax to specify
the
> levels in this case. Fortunately there is a recently added feature
that
> will allow you to get around this issue. You can specify values to
override
> attributes in GridStat. This will allow you to change the level
value
> 20160601_060000,0,*,* to a value that is more easily specified in
the
> StatAnalysis config file. GitHub issue #1020 outlines these changes
and
> this link takes you directly to a comment that lists the new
> configurations:
>
> https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
>
> You will want to set set_attr_level in GridStat, which can be added
via
> METplus with:
>
> FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
>
> I don't have access to Orion yet so I can't look at your
configuration to
> know what exactly "something" should be. If you need help figuring
that
> out, you could attach your METplus config file to this ticket and I
could
> take a look.
>
> These changes are available since MET 9.1-beta2 and will be in MET
9.1,
> which is close to being released. It looks like the beta release is
not
> installed on Orion but 9.1 is planned to be installed there.
>
> Let me know if you have any other questions.
>
> Thanks,
> George
>
> On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > George,
> > actually, there is a problem with the next step: StatAnalysis.
> > GridStat outputs time levels in the form 20160601_060000,0,*,*
> > but StatAnalysis does not accept
> > {valid?fmt=%Y%m%d_%H%M%S}
> > so ends up either producing an error or empty files depending how
the
> > config is set up.
> > Is there a solution to this problem?
> > Thanks,
> > Mariusz
> >
> > on Orion:
> > script
> > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > ...
> > | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > ./StatAnalysis.conf
> >
> > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR: Fatal
error
> > occurred
> >
> > Outputs in
> > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> >
> >
> >
> > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA Affiliate
<
> > mariusz.pagowski at noaa.gov> wrote:
> >
> > > Thanks, George, works as promised,
> > > Mariusz
> > >
> > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > >> Hi Mariusz,
> > >>
> > >> To specify the time in the NetCDF structure, you need to use
the
> format
> > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like this:
> > >>
> > >> {valid?fmt=%Y%m%d_%H%M%S}
> > >>
> > >> If the NetCDF time info is formatted in the way the MET tools
expect,
> > the
> > >> code is smart enough to find the correct data from the time
info.
> > >>
> > >> If that doesn't work, you may need to do something clever to
> manipulate
> > >> the
> > >> hour value to the corresponding number of the NetCDF dimension.
It
> looks
> > >> like your data has a time dimension of size 4, so passing in
the hour
> > (18)
> > >> is outside of the valid range. I would recommend trying to
format the
> > >> valid
> > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know
and I try
> > to
> > >> help you figure out a way to get the correct number for the
time you
> > need.
> > >>
> > >> Thanks,
> > >> George
> > >>
> > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > >> met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > >> >
> > >> > Julie,
> > >> > It was hanging over my head but only today I tried to input
forecast
> > >> time
> > >> > to the conf and did not figure the solution.
> > >> > valid?fmt string does not produce for me a correct time and
GridStat
> > >> fails.
> > >> > If your colleague looked into our problem
> > >> > we would very much appreciate it, thanks,
> > >> > Mariusz
> > >> >
> > >> > models for GridStat processing
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > >> >
> > >> > other stuff:
> > >> > /home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > >> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > >> >
> > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Hi Mariusz.
> > >> > >
> > >> > > I moved your question over to met_help so that others could
> benefit
> > >> from
> > >> > > your questions.
> > >> > >
> > >> > > I asked a co-worker for some help with this one and he said
that
> if
> > >> the
> > >> > > data is in NetCDF and in a format that MET can read, you
can
> specify
> > >> the
> > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*). There
is a
> > >> METplus
> > >> > > use case that does this here:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > >> > >
> > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> > >> > >
> > >> > > He said that likely won't work with this data, but there is
> likely a
> > >> > > clever way to use the valid time to create the appropriate
level
> > value
> > >> > for
> > >> > > each run time in a similar way. If you aren't able to
figure it
> out,
> > >> you
> > >> > > could provide a sample dataset and he could help figure out
how to
> > >> > > configure it.
> > >> > >
> > >> > > Thanks,
> > >> > > Julie
> > >> > >
> > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > >> > > > My start date for GridStat is e.g. 2016060118 for FCSTs
and
> OBSs.
> > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z)
and OBSs
> at
> > >> 3-
> > >> > > > hr intervals. For this time GridStat.conf is configured
to take
> > 3rd
> > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and
6th OBS
> > in
> > >> > > > the other input file: (6,1,*,*) That only works properly
for
> > >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours
when
> running
> > >> > > > over e.g. a month-long period. However, if I wanted to
run
> > GridStat
> > >> to
> > >> > > > get statistics with data at intervals of 6 hours e.g.
> > >> SDATE=2016060118
> > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> > needs
> > >> > > > to become (0,1,*,*) etc. Is there a way to pass times of
> > FCSTs/OBSs
> > >> to
> > >> > > > GridStat.conf when running master_metplus.py ?
> > >> > > >
> > >> > > > Thanks again,
> > >> > > > Mariusz
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >> --
> > >> George McCabe - Software Engineer III
> > >> National Center for Atmospheric Research
> > >> Research Applications Laboratory
> > >> 303-497-2768
> > >> ---
> > >> My working day may not be your working day. Please do not feel
obliged
> > to
> > >> reply to this email outside of your normal working hours.
> > >>
> > >>
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: GridStat Question
From: George McCabe
Time: Wed Aug 05 10:12:03 2020
Hi Mariusz,
We are aiming to release during this week, however the engineer who
installs on Orion is out for the rest of the week. If all goes
according to
plan, it should be available on Orion early next week. If you'd like,
I can
reply to this ticket when that has been completed to let you know that
it
is available.
Thanks,
George
On Wed, Aug 5, 2020 at 9:54 AM Mariusz Pagowski via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>
> Thanks, George,
> What is a timeline for releasing 9.1?
> Mariusz
>
>
> On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Mariusz,
> >
> > You are correct that StatAnalysis cannot use that syntax to
specify the
> > levels in this case. Fortunately there is a recently added feature
that
> > will allow you to get around this issue. You can specify values to
> override
> > attributes in GridStat. This will allow you to change the level
value
> > 20160601_060000,0,*,* to a value that is more easily specified in
the
> > StatAnalysis config file. GitHub issue #1020 outlines these
changes and
> > this link takes you directly to a comment that lists the new
> > configurations:
> >
> > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> >
> > You will want to set set_attr_level in GridStat, which can be
added via
> > METplus with:
> >
> > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> >
> > I don't have access to Orion yet so I can't look at your
configuration to
> > know what exactly "something" should be. If you need help figuring
that
> > out, you could attach your METplus config file to this ticket and
I could
> > take a look.
> >
> > These changes are available since MET 9.1-beta2 and will be in MET
9.1,
> > which is close to being released. It looks like the beta release
is not
> > installed on Orion but 9.1 is planned to be installed there.
> >
> > Let me know if you have any other questions.
> >
> > Thanks,
> > George
> >
> > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > >
> > > George,
> > > actually, there is a problem with the next step: StatAnalysis.
> > > GridStat outputs time levels in the form 20160601_060000,0,*,*
> > > but StatAnalysis does not accept
> > > {valid?fmt=%Y%m%d_%H%M%S}
> > > so ends up either producing an error or empty files depending
how the
> > > config is set up.
> > > Is there a solution to this problem?
> > > Thanks,
> > > Mariusz
> > >
> > > on Orion:
> > > script
> > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > ...
> > > | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > ./StatAnalysis.conf
> > >
> > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR: Fatal
error
> > > occurred
> > >
> > > Outputs in
> > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > >
> > >
> > >
> > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate <
> > > mariusz.pagowski at noaa.gov> wrote:
> > >
> > > > Thanks, George, works as promised,
> > > > Mariusz
> > > >
> > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > >> Hi Mariusz,
> > > >>
> > > >> To specify the time in the NetCDF structure, you need to use
the
> > format
> > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
> > > >>
> > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > >>
> > > >> If the NetCDF time info is formatted in the way the MET tools
> expect,
> > > the
> > > >> code is smart enough to find the correct data from the time
info.
> > > >>
> > > >> If that doesn't work, you may need to do something clever to
> > manipulate
> > > >> the
> > > >> hour value to the corresponding number of the NetCDF
dimension. It
> > looks
> > > >> like your data has a time dimension of size 4, so passing in
the
> hour
> > > (18)
> > > >> is outside of the valid range. I would recommend trying to
format
> the
> > > >> valid
> > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me know
and I
> try
> > > to
> > > >> help you figure out a way to get the correct number for the
time you
> > > need.
> > > >>
> > > >> Thanks,
> > > >> George
> > > >>
> > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > > >> met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > >> >
> > > >> > Julie,
> > > >> > It was hanging over my head but only today I tried to input
> forecast
> > > >> time
> > > >> > to the conf and did not figure the solution.
> > > >> > valid?fmt string does not produce for me a correct time and
> GridStat
> > > >> fails.
> > > >> > If your colleague looked into our problem
> > > >> > we would very much appreciate it, thanks,
> > > >> > Mariusz
> > > >> >
> > > >> > models for GridStat processing
> > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-2/pll
> > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > >> >
> > > >> > other stuff:
> > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > >> >
> /scratch1/BMC/gsd-fv3-dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > >> >
> > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Mariusz.
> > > >> > >
> > > >> > > I moved your question over to met_help so that others
could
> > benefit
> > > >> from
> > > >> > > your questions.
> > > >> > >
> > > >> > > I asked a co-worker for some help with this one and he
said that
> > if
> > > >> the
> > > >> > > data is in NetCDF and in a format that MET can read, you
can
> > specify
> > > >> the
> > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There is a
> > > >> METplus
> > > >> > > use case that does this here:
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > >> > >
> > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> > > >> > >
> > > >> > > He said that likely won't work with this data, but there
is
> > likely a
> > > >> > > clever way to use the valid time to create the
appropriate level
> > > value
> > > >> > for
> > > >> > > each run time in a similar way. If you aren't able to
figure it
> > out,
> > > >> you
> > > >> > > could provide a sample dataset and he could help figure
out how
> to
> > > >> > > configure it.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Julie
> > > >> > >
> > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > >> > > > My start date for GridStat is e.g. 2016060118 for FCSTs
and
> > OBSs.
> > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z, 18Z)
and
> OBSs
> > at
> > > >> 3-
> > > >> > > > hr intervals. For this time GridStat.conf is configured
to
> take
> > > 3rd
> > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*) and
6th
> OBS
> > > in
> > > >> > > > the other input file: (6,1,*,*) That only works
properly for
> > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours
when
> > running
> > > >> > > > over e.g. a month-long period. However, if I wanted to
run
> > > GridStat
> > > >> to
> > > >> > > > get statistics with data at intervals of 6 hours e.g.
> > > >> SDATE=2016060118
> > > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
(3,1,*,*)
> > > needs
> > > >> > > > to become (0,1,*,*) etc. Is there a way to pass times
of
> > > FCSTs/OBSs
> > > >> to
> > > >> > > > GridStat.conf when running master_metplus.py ?
> > > >> > > >
> > > >> > > > Thanks again,
> > > >> > > > Mariusz
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >>
> > > >> --
> > > >> George McCabe - Software Engineer III
> > > >> National Center for Atmospheric Research
> > > >> Research Applications Laboratory
> > > >> 303-497-2768
> > > >> ---
> > > >> My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > >> reply to this email outside of your normal working hours.
> > > >>
> > > >>
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Wed Aug 05 10:15:16 2020
Thanks again, George, that's quick. And yes, pls let me know when
the installation on Orion is complete/available,
Mariusz
On Wed, Aug 5, 2020 at 10:12 AM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Mariusz,
>
> We are aiming to release during this week, however the engineer who
> installs on Orion is out for the rest of the week. If all goes
according to
> plan, it should be available on Orion early next week. If you'd
like, I can
> reply to this ticket when that has been completed to let you know
that it
> is available.
>
> Thanks,
> George
>
> On Wed, Aug 5, 2020 at 9:54 AM Mariusz Pagowski via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> >
> > Thanks, George,
> > What is a timeline for releasing 9.1?
> > Mariusz
> >
> >
> > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Mariusz,
> > >
> > > You are correct that StatAnalysis cannot use that syntax to
specify the
> > > levels in this case. Fortunately there is a recently added
feature that
> > > will allow you to get around this issue. You can specify values
to
> > override
> > > attributes in GridStat. This will allow you to change the level
value
> > > 20160601_060000,0,*,* to a value that is more easily specified
in the
> > > StatAnalysis config file. GitHub issue #1020 outlines these
changes and
> > > this link takes you directly to a comment that lists the new
> > > configurations:
> > >
> > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
> > >
> > > You will want to set set_attr_level in GridStat, which can be
added via
> > > METplus with:
> > >
> > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
> > >
> > > I don't have access to Orion yet so I can't look at your
configuration
> to
> > > know what exactly "something" should be. If you need help
figuring that
> > > out, you could attach your METplus config file to this ticket
and I
> could
> > > take a look.
> > >
> > > These changes are available since MET 9.1-beta2 and will be in
MET 9.1,
> > > which is close to being released. It looks like the beta release
is not
> > > installed on Orion but 9.1 is planned to be installed there.
> > >
> > > Let me know if you have any other questions.
> > >
> > > Thanks,
> > > George
> > >
> > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
> > > >
> > > > George,
> > > > actually, there is a problem with the next step: StatAnalysis.
> > > > GridStat outputs time levels in the form 20160601_060000,0,*,*
> > > > but StatAnalysis does not accept
> > > > {valid?fmt=%Y%m%d_%H%M%S}
> > > > so ends up either producing an error or empty files depending
how the
> > > > config is set up.
> > > > Is there a solution to this problem?
> > > > Thanks,
> > > > Mariusz
> > > >
> > > > on Orion:
> > > > script
> > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
> > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
> > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
> > > > ...
> > > > | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
> > > > > ./StatAnalysis.conf
> > > >
> > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
> > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal error
> > > > occurred
> > > >
> > > > Outputs in
> > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
> > > >
> > > >
> > > >
> > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate <
> > > > mariusz.pagowski at noaa.gov> wrote:
> > > >
> > > > > Thanks, George, works as promised,
> > > > > Mariusz
> > > > >
> > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Mariusz,
> > > > >>
> > > > >> To specify the time in the NetCDF structure, you need to
use the
> > > format
> > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
> > > > >>
> > > > >> {valid?fmt=%Y%m%d_%H%M%S}
> > > > >>
> > > > >> If the NetCDF time info is formatted in the way the MET
tools
> > expect,
> > > > the
> > > > >> code is smart enough to find the correct data from the time
info.
> > > > >>
> > > > >> If that doesn't work, you may need to do something clever
to
> > > manipulate
> > > > >> the
> > > > >> hour value to the corresponding number of the NetCDF
dimension. It
> > > looks
> > > > >> like your data has a time dimension of size 4, so passing
in the
> > hour
> > > > (18)
> > > > >> is outside of the valid range. I would recommend trying to
format
> > the
> > > > >> valid
> > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me
know and I
> > try
> > > > to
> > > > >> help you figure out a way to get the correct number for the
time
> you
> > > > need.
> > > > >>
> > > > >> Thanks,
> > > > >> George
> > > > >>
> > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
> > > > >> met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >> >
> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
> > > > >> >
> > > > >> > Julie,
> > > > >> > It was hanging over my head but only today I tried to
input
> > forecast
> > > > >> time
> > > > >> > to the conf and did not figure the solution.
> > > > >> > valid?fmt string does not produce for me a correct time
and
> > GridStat
> > > > >> fails.
> > > > >> > If your colleague looked into our problem
> > > > >> > we would very much appreciate it, thanks,
> > > > >> > Mariusz
> > > > >> >
> > > > >> > models for GridStat processing
> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
> > > > >> >
> > > > >> > other stuff:
> > > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
> > > > >> >
> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
> > > > >> >
> > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT <
> > > > >> met_help at ucar.edu
> > > > >> > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Mariusz.
> > > > >> > >
> > > > >> > > I moved your question over to met_help so that others
could
> > > benefit
> > > > >> from
> > > > >> > > your questions.
> > > > >> > >
> > > > >> > > I asked a co-worker for some help with this one and he
said
> that
> > > if
> > > > >> the
> > > > >> > > data is in NetCDF and in a format that MET can read,
you can
> > > specify
> > > > >> the
> > > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There
> is a
> > > > >> METplus
> > > > >> > > use case that does this here:
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
> > > > >> > >
> > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
> > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
"({valid?fmt=%Y%m01_000000},*,*)"
> > > > >> > >
> > > > >> > > He said that likely won't work with this data, but
there is
> > > likely a
> > > > >> > > clever way to use the valid time to create the
appropriate
> level
> > > > value
> > > > >> > for
> > > > >> > > each run time in a similar way. If you aren't able to
figure
> it
> > > out,
> > > > >> you
> > > > >> > > could provide a sample dataset and he could help figure
out
> how
> > to
> > > > >> > > configure it.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Julie
> > > > >> > >
> > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
> > > > >> > > > My start date for GridStat is e.g. 2016060118 for
FCSTs and
> > > OBSs.
> > > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z,
18Z) and
> > OBSs
> > > at
> > > > >> 3-
> > > > >> > > > hr intervals. For this time GridStat.conf is
configured to
> > take
> > > > 3rd
> > > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*)
and 6th
> > OBS
> > > > in
> > > > >> > > > the other input file: (6,1,*,*) That only works
properly for
> > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24 hours
when
> > > running
> > > > >> > > > over e.g. a month-long period. However, if I wanted
to run
> > > > GridStat
> > > > >> to
> > > > >> > > > get statistics with data at intervals of 6 hours e.g.
> > > > >> SDATE=2016060118
> > > > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
> (3,1,*,*)
> > > > needs
> > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass times
of
> > > > FCSTs/OBSs
> > > > >> to
> > > > >> > > > GridStat.conf when running master_metplus.py ?
> > > > >> > > >
> > > > >> > > > Thanks again,
> > > > >> > > > Mariusz
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >> --
> > > > >> George McCabe - Software Engineer III
> > > > >> National Center for Atmospheric Research
> > > > >> Research Applications Laboratory
> > > > >> 303-497-2768
> > > > >> ---
> > > > >> My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > >> reply to this email outside of your normal working hours.
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: GridStat Question
From: Julie Prestopnik
Time: Thu Aug 13 16:17:50 2020
HI Mariusz.
Unfortunately, I do not have permissions to install in the proper area
on
Orion. I am working on getting that changed. I will follow up once
the
install is ready on Orion.
Julie
On Wed, Aug 5, 2020 at 10:15 AM Mariusz Pagowski - NOAA Affiliate <
mariusz.pagowski at noaa.gov> wrote:
> Thanks again, George, that's quick. And yes, pls let me know when
> the installation on Orion is complete/available,
> Mariusz
>
> On Wed, Aug 5, 2020 at 10:12 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
>> Hi Mariusz,
>>
>> We are aiming to release during this week, however the engineer who
>> installs on Orion is out for the rest of the week. If all goes
according
>> to
>> plan, it should be available on Orion early next week. If you'd
like, I
>> can
>> reply to this ticket when that has been completed to let you know
that it
>> is available.
>>
>> Thanks,
>> George
>>
>> On Wed, Aug 5, 2020 at 9:54 AM Mariusz Pagowski via RT
<met_help at ucar.edu
>> >
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>> >
>> > Thanks, George,
>> > What is a timeline for releasing 9.1?
>> > Mariusz
>> >
>> >
>> > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu>
>> > wrote:
>> >
>> > > Hi Mariusz,
>> > >
>> > > You are correct that StatAnalysis cannot use that syntax to
specify
>> the
>> > > levels in this case. Fortunately there is a recently added
feature
>> that
>> > > will allow you to get around this issue. You can specify values
to
>> > override
>> > > attributes in GridStat. This will allow you to change the level
value
>> > > 20160601_060000,0,*,* to a value that is more easily specified
in the
>> > > StatAnalysis config file. GitHub issue #1020 outlines these
changes
>> and
>> > > this link takes you directly to a comment that lists the new
>> > > configurations:
>> > >
>> > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
>> > >
>> > > You will want to set set_attr_level in GridStat, which can be
added
>> via
>> > > METplus with:
>> > >
>> > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
>> > >
>> > > I don't have access to Orion yet so I can't look at your
>> configuration to
>> > > know what exactly "something" should be. If you need help
figuring
>> that
>> > > out, you could attach your METplus config file to this ticket
and I
>> could
>> > > take a look.
>> > >
>> > > These changes are available since MET 9.1-beta2 and will be in
MET
>> 9.1,
>> > > which is close to being released. It looks like the beta
release is
>> not
>> > > installed on Orion but 9.1 is planned to be installed there.
>> > >
>> > > Let me know if you have any other questions.
>> > >
>> > > Thanks,
>> > > George
>> > >
>> > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
>> > met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>
>> > > >
>> > > > George,
>> > > > actually, there is a problem with the next step:
StatAnalysis.
>> > > > GridStat outputs time levels in the form
20160601_060000,0,*,*
>> > > > but StatAnalysis does not accept
>> > > > {valid?fmt=%Y%m%d_%H%M%S}
>> > > > so ends up either producing an error or empty files depending
how
>> the
>> > > > config is set up.
>> > > > Is there a solution to this problem?
>> > > > Thanks,
>> > > > Mariusz
>> > > >
>> > > > on Orion:
>> > > > script
>> > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
>> > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
>> > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
>> > > > ...
>> > > > | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>> > > > | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>> > > > > ./StatAnalysis.conf
>> > > >
>> > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
>> > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal
>> error
>> > > > occurred
>> > > >
>> > > > Outputs in
>> > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate <
>> > > > mariusz.pagowski at noaa.gov> wrote:
>> > > >
>> > > > > Thanks, George, works as promised,
>> > > > > Mariusz
>> > > > >
>> > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
>> > > met_help at ucar.edu
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > >> Hi Mariusz,
>> > > > >>
>> > > > >> To specify the time in the NetCDF structure, you need to
use the
>> > > format
>> > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
>> > > > >>
>> > > > >> {valid?fmt=%Y%m%d_%H%M%S}
>> > > > >>
>> > > > >> If the NetCDF time info is formatted in the way the MET
tools
>> > expect,
>> > > > the
>> > > > >> code is smart enough to find the correct data from the
time info.
>> > > > >>
>> > > > >> If that doesn't work, you may need to do something clever
to
>> > > manipulate
>> > > > >> the
>> > > > >> hour value to the corresponding number of the NetCDF
dimension.
>> It
>> > > looks
>> > > > >> like your data has a time dimension of size 4, so passing
in the
>> > hour
>> > > > (18)
>> > > > >> is outside of the valid range. I would recommend trying to
format
>> > the
>> > > > >> valid
>> > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me
know and
>> I
>> > try
>> > > > to
>> > > > >> help you figure out a way to get the correct number for
the time
>> you
>> > > > need.
>> > > > >>
>> > > > >> Thanks,
>> > > > >> George
>> > > > >>
>> > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
>> > > > >> met_help at ucar.edu>
>> > > > >> wrote:
>> > > > >>
>> > > > >> >
>> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>> >
>> > > > >> >
>> > > > >> > Julie,
>> > > > >> > It was hanging over my head but only today I tried to
input
>> > forecast
>> > > > >> time
>> > > > >> > to the conf and did not figure the solution.
>> > > > >> > valid?fmt string does not produce for me a correct time
and
>> > GridStat
>> > > > >> fails.
>> > > > >> > If your colleague looked into our problem
>> > > > >> > we would very much appreciate it, thanks,
>> > > > >> > Mariusz
>> > > > >> >
>> > > > >> > models for GridStat processing
>> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
>> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
>> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
>> > > > >> >
>> > > > >> > other stuff:
>> > > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
>> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
>> > > > >> >
>> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
>> > > > >> >
>> > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT
<
>> > > > >> met_help at ucar.edu
>> > > > >> > >
>> > > > >> > wrote:
>> > > > >> >
>> > > > >> > > Hi Mariusz.
>> > > > >> > >
>> > > > >> > > I moved your question over to met_help so that others
could
>> > > benefit
>> > > > >> from
>> > > > >> > > your questions.
>> > > > >> > >
>> > > > >> > > I asked a co-worker for some help with this one and he
said
>> that
>> > > if
>> > > > >> the
>> > > > >> > > data is in NetCDF and in a format that MET can read,
you can
>> > > specify
>> > > > >> the
>> > > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There
>> is a
>> > > > >> METplus
>> > > > >> > > use case that does this here:
>> > > > >> > >
>> > > > >> > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
>> > > > >> > >
>> > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
>> > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
>> "({valid?fmt=%Y%m01_000000},*,*)"
>> > > > >> > >
>> > > > >> > > He said that likely won't work with this data, but
there is
>> > > likely a
>> > > > >> > > clever way to use the valid time to create the
appropriate
>> level
>> > > > value
>> > > > >> > for
>> > > > >> > > each run time in a similar way. If you aren't able to
figure
>> it
>> > > out,
>> > > > >> you
>> > > > >> > > could provide a sample dataset and he could help
figure out
>> how
>> > to
>> > > > >> > > configure it.
>> > > > >> > >
>> > > > >> > > Thanks,
>> > > > >> > > Julie
>> > > > >> > >
>> > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
>> > > > >> > > > My start date for GridStat is e.g. 2016060118 for
FCSTs and
>> > > OBSs.
>> > > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z,
18Z) and
>> > OBSs
>> > > at
>> > > > >> 3-
>> > > > >> > > > hr intervals. For this time GridStat.conf is
configured to
>> > take
>> > > > 3rd
>> > > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*)
and
>> 6th
>> > OBS
>> > > > in
>> > > > >> > > > the other input file: (6,1,*,*) That only works
properly
>> for
>> > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24
hours when
>> > > running
>> > > > >> > > > over e.g. a month-long period. However, if I wanted
to run
>> > > > GridStat
>> > > > >> to
>> > > > >> > > > get statistics with data at intervals of 6 hours
e.g.
>> > > > >> SDATE=2016060118
>> > > > >> > > > and EDATE=2016063118 that is incorrect since for 00Z
>> (3,1,*,*)
>> > > > needs
>> > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass
times of
>> > > > FCSTs/OBSs
>> > > > >> to
>> > > > >> > > > GridStat.conf when running master_metplus.py ?
>> > > > >> > > >
>> > > > >> > > > Thanks again,
>> > > > >> > > > Mariusz
>> > > > >> > >
>> > > > >> > >
>> > > > >> > >
>> > > > >> > >
>> > > > >> >
>> > > > >> >
>> > > > >>
>> > > > >> --
>> > > > >> George McCabe - Software Engineer III
>> > > > >> National Center for Atmospheric Research
>> > > > >> Research Applications Laboratory
>> > > > >> 303-497-2768
>> > > > >> ---
>> > > > >> My working day may not be your working day. Please do not
feel
>> > obliged
>> > > > to
>> > > > >> reply to this email outside of your normal working hours.
>> > > > >>
>> > > > >>
>> > > >
>> > > >
>> > >
>> > > --
>> > > George McCabe - Software Engineer III
>> > > National Center for Atmospheric Research
>> > > Research Applications Laboratory
>> > > 303-497-2768
>> > > ---
>> > > My working day may not be your working day. Please do not feel
>> obliged to
>> > > reply to this email outside of your normal working hours.
>> > >
>> > >
>> >
>> >
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>>
--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: GridStat Question
From: Mariusz Pagowski
Time: Mon Aug 17 16:20:00 2020
Julie,
it is not for lack of interest in MET 9.1 that my response to your e-
mail
is delayed. Very much looking forward
to the new version, thanks,
Mariusz
On Thu, Aug 13, 2020 at 4:17 PM Julie Prestopnik <jpresto at ucar.edu>
wrote:
> HI Mariusz.
>
> Unfortunately, I do not have permissions to install in the proper
area on
> Orion. I am working on getting that changed. I will follow up once
the
> install is ready on Orion.
>
> Julie
>
> On Wed, Aug 5, 2020 at 10:15 AM Mariusz Pagowski - NOAA Affiliate <
> mariusz.pagowski at noaa.gov> wrote:
>
>> Thanks again, George, that's quick. And yes, pls let me know when
>> the installation on Orion is complete/available,
>> Mariusz
>>
>> On Wed, Aug 5, 2020 at 10:12 AM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>>> Hi Mariusz,
>>>
>>> We are aiming to release during this week, however the engineer
who
>>> installs on Orion is out for the rest of the week. If all goes
according
>>> to
>>> plan, it should be available on Orion early next week. If you'd
like, I
>>> can
>>> reply to this ticket when that has been completed to let you know
that it
>>> is available.
>>>
>>> Thanks,
>>> George
>>>
>>> On Wed, Aug 5, 2020 at 9:54 AM Mariusz Pagowski via RT <
>>> met_help at ucar.edu>
>>> wrote:
>>>
>>> >
>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>>> >
>>> > Thanks, George,
>>> > What is a timeline for releasing 9.1?
>>> > Mariusz
>>> >
>>> >
>>> > On Wed, Aug 5, 2020 at 8:04 AM George McCabe via RT
<met_help at ucar.edu
>>> >
>>> > wrote:
>>> >
>>> > > Hi Mariusz,
>>> > >
>>> > > You are correct that StatAnalysis cannot use that syntax to
specify
>>> the
>>> > > levels in this case. Fortunately there is a recently added
feature
>>> that
>>> > > will allow you to get around this issue. You can specify
values to
>>> > override
>>> > > attributes in GridStat. This will allow you to change the
level value
>>> > > 20160601_060000,0,*,* to a value that is more easily specified
in the
>>> > > StatAnalysis config file. GitHub issue #1020 outlines these
changes
>>> and
>>> > > this link takes you directly to a comment that lists the new
>>> > > configurations:
>>> > >
>>> > > https://github.com/NCAR/MET/issues/1020#issuecomment-653724572
>>> > >
>>> > > You will want to set set_attr_level in GridStat, which can be
added
>>> via
>>> > > METplus with:
>>> > >
>>> > > FCST_GRID_STAT_VAR1_OPTIONS = set_attr_level = "something";
>>> > >
>>> > > I don't have access to Orion yet so I can't look at your
>>> configuration to
>>> > > know what exactly "something" should be. If you need help
figuring
>>> that
>>> > > out, you could attach your METplus config file to this ticket
and I
>>> could
>>> > > take a look.
>>> > >
>>> > > These changes are available since MET 9.1-beta2 and will be in
MET
>>> 9.1,
>>> > > which is close to being released. It looks like the beta
release is
>>> not
>>> > > installed on Orion but 9.1 is planned to be installed there.
>>> > >
>>> > > Let me know if you have any other questions.
>>> > >
>>> > > Thanks,
>>> > > George
>>> > >
>>> > > On Tue, Aug 4, 2020 at 9:00 PM Mariusz Pagowski via RT <
>>> > met_help at ucar.edu>
>>> > > wrote:
>>> > >
>>> > > >
>>> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850 >
>>> > > >
>>> > > > George,
>>> > > > actually, there is a problem with the next step:
StatAnalysis.
>>> > > > GridStat outputs time levels in the form
20160601_060000,0,*,*
>>> > > > but StatAnalysis does not accept
>>> > > > {valid?fmt=%Y%m%d_%H%M%S}
>>> > > > so ends up either producing an error or empty files
depending how
>>> the
>>> > > > config is set up.
>>> > > > Is there a solution to this problem?
>>> > > > Thanks,
>>> > > > Mariusz
>>> > > >
>>> > > > on Orion:
>>> > > > script
>>> > > > /home/mpagowsk/mapp_2018/scripts/met/met_stat_anal.sh
>>> > > > INCONFIG=${CONFIG_DIR}/StatAnalysis.conf.IN
>>> > > > cat $INCONFIG | sed s:_SDATE_:${SDATE}:g \
>>> > > > ...
>>> > > > | sed s:_FTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>>> > > > | sed s:_OTIME_:{valid?fmt=%Y%m%d_%H%M%S}:g \
>>> > > > > ./StatAnalysis.conf
>>> > > >
>>> > > > KeyError: 'valid?fmt=%Y%m%d_%H%M%S'
>>> > > > 08/04 21:48:57Z run-METplus-metplus.StatAnalysis: ERROR:
Fatal
>>> error
>>> > > > occurred
>>> > > >
>>> > > > Outputs in
>>> > > > /work/noaa/gsd-fv3-dev/pagowski/MODEL/metstats
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Mon, Jul 27, 2020 at 10:14 PM Mariusz Pagowski - NOAA
Affiliate
>>> <
>>> > > > mariusz.pagowski at noaa.gov> wrote:
>>> > > >
>>> > > > > Thanks, George, works as promised,
>>> > > > > Mariusz
>>> > > > >
>>> > > > > On Mon, Jul 27, 2020 at 11:13 AM George McCabe via RT <
>>> > > met_help at ucar.edu
>>> > > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > >> Hi Mariusz,
>>> > > > >>
>>> > > > >> To specify the time in the NetCDF structure, you need to
use the
>>> > > format
>>> > > > >> YYYYMMDD_HHMMSS, so your FTIME variable should look like
this:
>>> > > > >>
>>> > > > >> {valid?fmt=%Y%m%d_%H%M%S}
>>> > > > >>
>>> > > > >> If the NetCDF time info is formatted in the way the MET
tools
>>> > expect,
>>> > > > the
>>> > > > >> code is smart enough to find the correct data from the
time
>>> info.
>>> > > > >>
>>> > > > >> If that doesn't work, you may need to do something clever
to
>>> > > manipulate
>>> > > > >> the
>>> > > > >> hour value to the corresponding number of the NetCDF
dimension.
>>> It
>>> > > looks
>>> > > > >> like your data has a time dimension of size 4, so passing
in the
>>> > hour
>>> > > > (18)
>>> > > > >> is outside of the valid range. I would recommend trying
to
>>> format
>>> > the
>>> > > > >> valid
>>> > > > >> time with YYYYMMDD_HHMMSS. If that doesn't work, let me
know
>>> and I
>>> > try
>>> > > > to
>>> > > > >> help you figure out a way to get the correct number for
the
>>> time you
>>> > > > need.
>>> > > > >>
>>> > > > >> Thanks,
>>> > > > >> George
>>> > > > >>
>>> > > > >> On Thu, Jul 23, 2020 at 7:06 PM Mariusz Pagowski via RT <
>>> > > > >> met_help at ucar.edu>
>>> > > > >> wrote:
>>> > > > >>
>>> > > > >> >
>>> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95850
>>> >
>>> > > > >> >
>>> > > > >> > Julie,
>>> > > > >> > It was hanging over my head but only today I tried to
input
>>> > forecast
>>> > > > >> time
>>> > > > >> > to the conf and did not figure the solution.
>>> > > > >> > valid?fmt string does not produce for me a correct time
and
>>> > GridStat
>>> > > > >> fails.
>>> > > > >> > If your colleague looked into our problem
>>> > > > >> > we would very much appreciate it, thanks,
>>> > > > >> > Mariusz
>>> > > > >> >
>>> > > > >> > models for GridStat processing
>>> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/cams/pll_new
>>> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/merra-
2/pll
>>> > > > >> > /scratch1/BMC/wrf-chem/pagowski/MAPP_2018/MODEL/fv3/pll
>>> > > > >> >
>>> > > > >> > other stuff:
>>> > > > >> >
/home/Mariusz.Pagowski/MAPP_2018/scripts/met/met_grid_stat.sh
>>> > > > >> > /scratch1/BMC/wrf-
chem/pagowski/MAPP_2018/MODEL/metstats/logs
>>> > > > >> >
>>> > /scratch1/BMC/gsd-fv3-
dev/MAPP_2018/pagowski/tmpdir/workdir_gridstat
>>> > > > >> >
>>> > > > >> > On Wed, Jul 8, 2020 at 10:47 AM Julie Prestopnik via RT
<
>>> > > > >> met_help at ucar.edu
>>> > > > >> > >
>>> > > > >> > wrote:
>>> > > > >> >
>>> > > > >> > > Hi Mariusz.
>>> > > > >> > >
>>> > > > >> > > I moved your question over to met_help so that others
could
>>> > > benefit
>>> > > > >> from
>>> > > > >> > > your questions.
>>> > > > >> > >
>>> > > > >> > > I asked a co-worker for some help with this one and
he said
>>> that
>>> > > if
>>> > > > >> the
>>> > > > >> > > data is in NetCDF and in a format that MET can read,
you can
>>> > > specify
>>> > > > >> the
>>> > > > >> > > time to use in the level, i.e. (YYYYMMDD_HHMMSS,*,*).
There
>>> is a
>>> > > > >> METplus
>>> > > > >> > > use case that does this here:
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> >
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>>
parm/use_cases/model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf
>>> > > > >> > >
>>> > > > >> > > FCST_GRID_STAT_VAR1_NAME = pr
>>> > > > >> > > FCST_GRID_STAT_VAR1_LEVELS =
>>> "({valid?fmt=%Y%m01_000000},*,*)"
>>> > > > >> > >
>>> > > > >> > > He said that likely won't work with this data, but
there is
>>> > > likely a
>>> > > > >> > > clever way to use the valid time to create the
appropriate
>>> level
>>> > > > value
>>> > > > >> > for
>>> > > > >> > > each run time in a similar way. If you aren't able to
>>> figure it
>>> > > out,
>>> > > > >> you
>>> > > > >> > > could provide a sample dataset and he could help
figure out
>>> how
>>> > to
>>> > > > >> > > configure it.
>>> > > > >> > >
>>> > > > >> > > Thanks,
>>> > > > >> > > Julie
>>> > > > >> > >
>>> > > > >> > > On Wed Jul 08 10:39:59 2020, jpresto wrote:
>>> > > > >> > > > My start date for GridStat is e.g. 2016060118 for
FCSTs
>>> and
>>> > > OBSs.
>>> > > > >> > > > FCSTs are output at 6-hr intervals (00Z, 06Z,12Z,
18Z) and
>>> > OBSs
>>> > > at
>>> > > > >> 3-
>>> > > > >> > > > hr intervals. For this time GridStat.conf is
configured to
>>> > take
>>> > > > 3rd
>>> > > > >> > > > FCST in the input file [in GridStat.conf: (3,1,*,*)
and
>>> 6th
>>> > OBS
>>> > > > in
>>> > > > >> > > > the other input file: (6,1,*,*) That only works
properly
>>> for
>>> > > > >> > > > FCSTs/OBSs starting at 18z with intervals of 24
hours when
>>> > > running
>>> > > > >> > > > over e.g. a month-long period. However, if I wanted
to run
>>> > > > GridStat
>>> > > > >> to
>>> > > > >> > > > get statistics with data at intervals of 6 hours
e.g.
>>> > > > >> SDATE=2016060118
>>> > > > >> > > > and EDATE=2016063118 that is incorrect since for
00Z
>>> (3,1,*,*)
>>> > > > needs
>>> > > > >> > > > to become (0,1,*,*) etc. Is there a way to pass
times of
>>> > > > FCSTs/OBSs
>>> > > > >> to
>>> > > > >> > > > GridStat.conf when running master_metplus.py ?
>>> > > > >> > > >
>>> > > > >> > > > Thanks again,
>>> > > > >> > > > Mariusz
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> >
>>> > > > >> >
>>> > > > >>
>>> > > > >> --
>>> > > > >> George McCabe - Software Engineer III
>>> > > > >> National Center for Atmospheric Research
>>> > > > >> Research Applications Laboratory
>>> > > > >> 303-497-2768
>>> > > > >> ---
>>> > > > >> My working day may not be your working day. Please do not
feel
>>> > obliged
>>> > > > to
>>> > > > >> reply to this email outside of your normal working hours.
>>> > > > >>
>>> > > > >>
>>> > > >
>>> > > >
>>> > >
>>> > > --
>>> > > George McCabe - Software Engineer III
>>> > > National Center for Atmospheric Research
>>> > > Research Applications Laboratory
>>> > > 303-497-2768
>>> > > ---
>>> > > My working day may not be your working day. Please do not feel
>>> obliged to
>>> > > reply to this email outside of your normal working hours.
>>> > >
>>> > >
>>> >
>>> >
>>>
>>> --
>>> George McCabe - Software Engineer III
>>> National Center for Atmospheric Research
>>> Research Applications Laboratory
>>> 303-497-2768
>>> ---
>>> My working day may not be your working day. Please do not feel
obliged to
>>> reply to this email outside of your normal working hours.
>>>
>>>
>
> --
> Julie Prestopnik (she/her/hers)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
------------------------------------------------
More information about the Met_help
mailing list