[Met_help] [rt.rap.ucar.edu #99910] History for Issue using the PCPCombine derive command
George McCabe via RT
met_help at ucar.edu
Tue May 18 09:01:17 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I'm using PCPCombine/derive to compute a mean among several data files and
the tool is having trouble finding the data files in the provided
directory. An error occurs when using PCPCombine in a config file, but not
when using it without a config file. I've tried a few alternatives, which
I mention below, and I checked online for some tips unsuccessfully, so I'm
hoping someone here may be able to help.
Attached are the config files I used and the resulting log file that
contains the error message. The command I used to feed in the config files
was:
* master_metplus.py -c pcp_combine_derive_metplus.conf -c hera.mcaron.conf*
In *pcp_combine_derive_metplus.conf*, OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
points to a directory that contains only grib files. I can copy those into
the anonymous ftp site if needed.
In addition, I tried this command, which works fine:
*pcp_combine -derive mean
/scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102/* -field
'name="APCP"; level="A24";' derive_mean_APCP.nc*
Other checks I did: I checked if the OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
is accessible and it is. I listed other options to
OBS_PCP_COMBINE_INPUT_ACCUMS (i.e., 1d, 6, 3, 1), which resulted in the
same error. Finally, I made small changes to the config file to see if the
"-sum" method works, and it does.
The reason I'd like to use a config file is because I'd like to follow the
PCPCombine process with GridStat against other data.
Best,
-Marcel
--
*___________________________________*
*Marcel G. Caron*
Physical Scientist, I.M. Systems Group, Inc.
Affiliate, Model Evaluation Group
@NOAA/NWS/NCEP/EMC
*5830 University Research Court,*
*College Park, MD 20740*
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Issue using the PCPCombine derive command
From: John Halley Gotway
Time: Mon May 17 11:20:38 2021
Marcel,
Ah OK, thanks for sending your METplus log file. I do see that the
METplus
wrapper for pcp_combine is erroring out:
05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100) ERROR:
(pcp_combine_wrapper.py:928) Could not find files in
/scratch2/NCEPDEV/ovp/
05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100) ERROR:
(pcp_combine_wrapper.py:626) pcp_combine could not generate command
05/17 15:18:40.002 metplus.PCPCombine (met_util.py:187) ERROR:
PCPCombine
had 2 errors.
The wrapper is looking for data in the directory named
"/scratch2/NCEPDEV/ovp/". However your successful run of pcp_combine
reads
input from:
/scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102
Looking at your METplus config files...
>From pcp_combine_derive_metplus.conf:
OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}
>From hera.mcaron.conf:
INPUT_BASE = /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s
I have a reasonable guess as to what's going on here. You're
running METplus as follows:
master_metplus.py -c pcp_combine_derive_metplus.conf -c
hera.mcaron.conf
The METplus conf files are read in the order that they're provided on
the
command line. But you're referencing INPUT_BASE in
pcp_combine_derive_metplus.conf before it has been defined by
hera.mcaron.conf.
Please try rerunning in the opposite order, like this:
* master_metplus.py -c hera.mcaron.conf -c
pcp_combine_derive_metplus.conf *
Does that produce any better results?
Thanks,
John Halley Gotway
On Mon, May 17, 2021 at 9:56 AM Marcel Caron - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> Mon May 17 09:56:27 2021: Request 99910 was acted upon.
> Transaction: Ticket created by marcel.caron at noaa.gov
> Queue: met_help
> Subject: Issue using the PCPCombine derive command
> Owner: Nobody
> Requestors: marcel.caron at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
>
>
> I'm using PCPCombine/derive to compute a mean among several data
files and
> the tool is having trouble finding the data files in the provided
> directory. An error occurs when using PCPCombine in a config file,
but not
> when using it without a config file. I've tried a few alternatives,
which
> I mention below, and I checked online for some tips unsuccessfully,
so I'm
> hoping someone here may be able to help.
>
> Attached are the config files I used and the resulting log file that
> contains the error message. The command I used to feed in the
config files
> was:
> * master_metplus.py -c pcp_combine_derive_metplus.conf -c
hera.mcaron.conf*
>
> In *pcp_combine_derive_metplus.conf*,
OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> points to a directory that contains only grib files. I can copy
those into
> the anonymous ftp site if needed.
>
> In addition, I tried this command, which works fine:
> *pcp_combine -derive mean
> /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102/* -field
> 'name="APCP"; level="A24";' derive_mean_APCP.nc*
>
> Other checks I did: I checked if the
OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> is accessible and it is. I listed other options to
> OBS_PCP_COMBINE_INPUT_ACCUMS (i.e., 1d, 6, 3, 1), which resulted in
the
> same error. Finally, I made small changes to the config file to see
if the
> "-sum" method works, and it does.
>
> The reason I'd like to use a config file is because I'd like to
follow the
> PCPCombine process with GridStat against other data.
>
> Best,
> -Marcel
>
> --
> *___________________________________*
> *Marcel G. Caron*
> Physical Scientist, I.M. Systems Group, Inc.
> Affiliate, Model Evaluation Group
> @NOAA/NWS/NCEP/EMC
> *5830 University Research Court,*
> *College Park, MD 20740*
>
>
------------------------------------------------
Subject: Issue using the PCPCombine derive command
From: Marcel Caron - NOAA Affiliate
Time: Mon May 17 12:23:09 2021
John:
Thanks for the response! I see that now, and I've switched the order,
feeding in hera.mcaron.conf first. However it still isn't able to
find the
files:
05/17 18:09:17.726 metplus INFO:
****************************************
> 05/17 18:09:17.727 metplus INFO: * Running METplus
> 05/17 18:09:17.727 metplus INFO: * at valid time: 202102281200
> 05/17 18:09:17.727 metplus INFO:
****************************************
> 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing OBS data
> 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing forecast lead
0
> hours
05/17 18:09:17.842 metplus.PCPCombine (command_builder.py:100) ERROR:
> (pcp_combine_wrapper.py:928) Could not find files in
> /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s using template
> st4.{valid?fmt=%Y%m}/*
> 05/17 18:09:17.843 metplus.PCPCombine (command_builder.py:100)
ERROR:
> (pcp_combine_wrapper.py:626) pcp_combine could not generate command
> 05/17 18:09:17.843 metplus.PCPCombine (met_util.py:187) ERROR:
PCPCombine
> had 2 errors.
The directory indicated in the error message is correct as far as I
can
tell. It should be looking
in /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s/st4.202102/ which
does
contain the input files.
Best,
-Marcel
On Mon, May 17, 2021 at 1:20 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Marcel,
>
> Ah OK, thanks for sending your METplus log file. I do see that the
METplus
> wrapper for pcp_combine is erroring out:
>
> 05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100)
ERROR:
> (pcp_combine_wrapper.py:928) Could not find files in
/scratch2/NCEPDEV/ovp/
> 05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100)
ERROR:
> (pcp_combine_wrapper.py:626) pcp_combine could not generate command
> 05/17 15:18:40.002 metplus.PCPCombine (met_util.py:187) ERROR:
PCPCombine
> had 2 errors.
>
> The wrapper is looking for data in the directory named
> "/scratch2/NCEPDEV/ovp/". However your successful run of pcp_combine
reads
> input from:
> /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102
>
> Looking at your METplus config files...
>
> From pcp_combine_derive_metplus.conf:
> OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}
>
> From hera.mcaron.conf:
> INPUT_BASE = /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s
>
> I have a reasonable guess as to what's going on here. You're
> running METplus as follows:
> master_metplus.py -c pcp_combine_derive_metplus.conf -c
hera.mcaron.conf
>
> The METplus conf files are read in the order that they're provided
on the
> command line. But you're referencing INPUT_BASE in
> pcp_combine_derive_metplus.conf before it has been defined by
> hera.mcaron.conf.
>
> Please try rerunning in the opposite order, like this:
> * master_metplus.py -c hera.mcaron.conf -c
> pcp_combine_derive_metplus.conf *
>
> Does that produce any better results?
>
> Thanks,
> John Halley Gotway
>
> On Mon, May 17, 2021 at 9:56 AM Marcel Caron - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Mon May 17 09:56:27 2021: Request 99910 was acted upon.
> > Transaction: Ticket created by marcel.caron at noaa.gov
> > Queue: met_help
> > Subject: Issue using the PCPCombine derive command
> > Owner: Nobody
> > Requestors: marcel.caron at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
> >
> >
> > I'm using PCPCombine/derive to compute a mean among several data
files
> and
> > the tool is having trouble finding the data files in the provided
> > directory. An error occurs when using PCPCombine in a config
file, but
> not
> > when using it without a config file. I've tried a few
alternatives,
> which
> > I mention below, and I checked online for some tips
unsuccessfully, so
> I'm
> > hoping someone here may be able to help.
> >
> > Attached are the config files I used and the resulting log file
that
> > contains the error message. The command I used to feed in the
config
> files
> > was:
> > * master_metplus.py -c pcp_combine_derive_metplus.conf -c
> hera.mcaron.conf*
> >
> > In *pcp_combine_derive_metplus.conf*,
> OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > points to a directory that contains only grib files. I can copy
those
> into
> > the anonymous ftp site if needed.
> >
> > In addition, I tried this command, which works fine:
> > *pcp_combine -derive mean
> > /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102/*
-field
> > 'name="APCP"; level="A24";' derive_mean_APCP.nc*
> >
> > Other checks I did: I checked if the
> OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > is accessible and it is. I listed other options to
> > OBS_PCP_COMBINE_INPUT_ACCUMS (i.e., 1d, 6, 3, 1), which resulted
in the
> > same error. Finally, I made small changes to the config file to
see if
> the
> > "-sum" method works, and it does.
> >
> > The reason I'd like to use a config file is because I'd like to
follow
> the
> > PCPCombine process with GridStat against other data.
> >
> > Best,
> > -Marcel
> >
> > --
> > *___________________________________*
> > *Marcel G. Caron*
> > Physical Scientist, I.M. Systems Group, Inc.
> > Affiliate, Model Evaluation Group
> > @NOAA/NWS/NCEP/EMC
> > *5830 University Research Court,*
> > *College Park, MD 20740*
> >
> >
>
>
------------------------------------------------
Subject: Issue using the PCPCombine derive command
From: John Halley Gotway
Time: Mon May 17 12:44:19 2021
Marcel,
I received word from George McCabe that the order of the config files
actually does not matter. So it makes sense that you're seeing the
behavior. Sorry for pointing you in the wrong direction.
I'm going to reassign this ticket to George. I think he's best suited
to
debug this behavior from the pcp_combine wrapper.
Thanks,
John
On Mon, May 17, 2021 at 12:23 PM Marcel Caron - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
>
> John:
>
> Thanks for the response! I see that now, and I've switched the
order,
> feeding in hera.mcaron.conf first. However it still isn't able to
find the
> files:
>
> 05/17 18:09:17.726 metplus INFO:
****************************************
> > 05/17 18:09:17.727 metplus INFO: * Running METplus
> > 05/17 18:09:17.727 metplus INFO: * at valid time: 202102281200
> > 05/17 18:09:17.727 metplus INFO:
****************************************
> > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing OBS data
> > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing forecast
lead 0
> > hours
>
> 05/17 18:09:17.842 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > (pcp_combine_wrapper.py:928) Could not find files in
> > /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s using template
> > st4.{valid?fmt=%Y%m}/*
> > 05/17 18:09:17.843 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > (pcp_combine_wrapper.py:626) pcp_combine could not generate
command
> > 05/17 18:09:17.843 metplus.PCPCombine (met_util.py:187) ERROR:
PCPCombine
> > had 2 errors.
>
>
> The directory indicated in the error message is correct as far as I
can
> tell. It should be looking
> in /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s/st4.202102/ which
does
> contain the input files.
>
> Best,
> -Marcel
>
> On Mon, May 17, 2021 at 1:20 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Marcel,
> >
> > Ah OK, thanks for sending your METplus log file. I do see that the
> METplus
> > wrapper for pcp_combine is erroring out:
> >
> > 05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > (pcp_combine_wrapper.py:928) Could not find files in
> /scratch2/NCEPDEV/ovp/
> > 05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > (pcp_combine_wrapper.py:626) pcp_combine could not generate
command
> > 05/17 15:18:40.002 metplus.PCPCombine (met_util.py:187) ERROR:
PCPCombine
> > had 2 errors.
> >
> > The wrapper is looking for data in the directory named
> > "/scratch2/NCEPDEV/ovp/". However your successful run of
pcp_combine
> reads
> > input from:
> > /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102
> >
> > Looking at your METplus config files...
> >
> > From pcp_combine_derive_metplus.conf:
> > OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}
> >
> > From hera.mcaron.conf:
> > INPUT_BASE = /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s
> >
> > I have a reasonable guess as to what's going on here. You're
> > running METplus as follows:
> > master_metplus.py -c pcp_combine_derive_metplus.conf -c
> hera.mcaron.conf
> >
> > The METplus conf files are read in the order that they're provided
on the
> > command line. But you're referencing INPUT_BASE in
> > pcp_combine_derive_metplus.conf before it has been defined by
> > hera.mcaron.conf.
> >
> > Please try rerunning in the opposite order, like this:
> > * master_metplus.py -c hera.mcaron.conf -c
> > pcp_combine_derive_metplus.conf *
> >
> > Does that produce any better results?
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Mon, May 17, 2021 at 9:56 AM Marcel Caron - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon May 17 09:56:27 2021: Request 99910 was acted upon.
> > > Transaction: Ticket created by marcel.caron at noaa.gov
> > > Queue: met_help
> > > Subject: Issue using the PCPCombine derive command
> > > Owner: Nobody
> > > Requestors: marcel.caron at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910
> >
> > >
> > >
> > > I'm using PCPCombine/derive to compute a mean among several data
files
> > and
> > > the tool is having trouble finding the data files in the
provided
> > > directory. An error occurs when using PCPCombine in a config
file, but
> > not
> > > when using it without a config file. I've tried a few
alternatives,
> > which
> > > I mention below, and I checked online for some tips
unsuccessfully, so
> > I'm
> > > hoping someone here may be able to help.
> > >
> > > Attached are the config files I used and the resulting log file
that
> > > contains the error message. The command I used to feed in the
config
> > files
> > > was:
> > > * master_metplus.py -c pcp_combine_derive_metplus.conf -c
> > hera.mcaron.conf*
> > >
> > > In *pcp_combine_derive_metplus.conf*,
> > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > > points to a directory that contains only grib files. I can copy
those
> > into
> > > the anonymous ftp site if needed.
> > >
> > > In addition, I tried this command, which works fine:
> > > *pcp_combine -derive mean
> > > /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102/*
-field
> > > 'name="APCP"; level="A24";' derive_mean_APCP.nc*
> > >
> > > Other checks I did: I checked if the
> > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > > is accessible and it is. I listed other options to
> > > OBS_PCP_COMBINE_INPUT_ACCUMS (i.e., 1d, 6, 3, 1), which resulted
in the
> > > same error. Finally, I made small changes to the config file to
see if
> > the
> > > "-sum" method works, and it does.
> > >
> > > The reason I'd like to use a config file is because I'd like to
follow
> > the
> > > PCPCombine process with GridStat against other data.
> > >
> > > Best,
> > > -Marcel
> > >
> > > --
> > > *___________________________________*
> > > *Marcel G. Caron*
> > > Physical Scientist, I.M. Systems Group, Inc.
> > > Affiliate, Model Evaluation Group
> > > @NOAA/NWS/NCEP/EMC
> > > *5830 University Research Court,*
> > > *College Park, MD 20740*
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Issue using the PCPCombine derive command
From: Marcel Caron - NOAA Affiliate
Time: Mon May 17 13:22:16 2021
John:
Okay, no problem. Thank you.
-Marcel
On Mon, May 17, 2021 at 2:44 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Marcel,
>
> I received word from George McCabe that the order of the config
files
> actually does not matter. So it makes sense that you're seeing the
> behavior. Sorry for pointing you in the wrong direction.
>
> I'm going to reassign this ticket to George. I think he's best
suited to
> debug this behavior from the pcp_combine wrapper.
>
> Thanks,
> John
>
> On Mon, May 17, 2021 at 12:23 PM Marcel Caron - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
> >
> > John:
> >
> > Thanks for the response! I see that now, and I've switched the
order,
> > feeding in hera.mcaron.conf first. However it still isn't able to
find
> the
> > files:
> >
> > 05/17 18:09:17.726 metplus INFO:
****************************************
> > > 05/17 18:09:17.727 metplus INFO: * Running METplus
> > > 05/17 18:09:17.727 metplus INFO: * at valid time: 202102281200
> > > 05/17 18:09:17.727 metplus INFO:
> ****************************************
> > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing OBS data
> > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing forecast
lead 0
> > > hours
> >
> > 05/17 18:09:17.842 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > > (pcp_combine_wrapper.py:928) Could not find files in
> > > /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s using template
> > > st4.{valid?fmt=%Y%m}/*
> > > 05/17 18:09:17.843 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > > (pcp_combine_wrapper.py:626) pcp_combine could not generate
command
> > > 05/17 18:09:17.843 metplus.PCPCombine (met_util.py:187) ERROR:
> PCPCombine
> > > had 2 errors.
> >
> >
> > The directory indicated in the error message is correct as far as
I can
> > tell. It should be looking
> > in /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s/st4.202102/
which does
> > contain the input files.
> >
> > Best,
> > -Marcel
> >
> > On Mon, May 17, 2021 at 1:20 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Marcel,
> > >
> > > Ah OK, thanks for sending your METplus log file. I do see that
the
> > METplus
> > > wrapper for pcp_combine is erroring out:
> > >
> > > 05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > > (pcp_combine_wrapper.py:928) Could not find files in
> > /scratch2/NCEPDEV/ovp/
> > > 05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > > (pcp_combine_wrapper.py:626) pcp_combine could not generate
command
> > > 05/17 15:18:40.002 metplus.PCPCombine (met_util.py:187) ERROR:
> PCPCombine
> > > had 2 errors.
> > >
> > > The wrapper is looking for data in the directory named
> > > "/scratch2/NCEPDEV/ovp/". However your successful run of
pcp_combine
> > reads
> > > input from:
> > > /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102
> > >
> > > Looking at your METplus config files...
> > >
> > > From pcp_combine_derive_metplus.conf:
> > > OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}
> > >
> > > From hera.mcaron.conf:
> > > INPUT_BASE = /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s
> > >
> > > I have a reasonable guess as to what's going on here. You're
> > > running METplus as follows:
> > > master_metplus.py -c pcp_combine_derive_metplus.conf -c
> > hera.mcaron.conf
> > >
> > > The METplus conf files are read in the order that they're
provided on
> the
> > > command line. But you're referencing INPUT_BASE in
> > > pcp_combine_derive_metplus.conf before it has been defined by
> > > hera.mcaron.conf.
> > >
> > > Please try rerunning in the opposite order, like this:
> > > * master_metplus.py -c hera.mcaron.conf -c
> > > pcp_combine_derive_metplus.conf *
> > >
> > > Does that produce any better results?
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Mon, May 17, 2021 at 9:56 AM Marcel Caron - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon May 17 09:56:27 2021: Request 99910 was acted upon.
> > > > Transaction: Ticket created by marcel.caron at noaa.gov
> > > > Queue: met_help
> > > > Subject: Issue using the PCPCombine derive command
> > > > Owner: Nobody
> > > > Requestors: marcel.caron at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910
> > >
> > > >
> > > >
> > > > I'm using PCPCombine/derive to compute a mean among several
data
> files
> > > and
> > > > the tool is having trouble finding the data files in the
provided
> > > > directory. An error occurs when using PCPCombine in a config
file,
> but
> > > not
> > > > when using it without a config file. I've tried a few
alternatives,
> > > which
> > > > I mention below, and I checked online for some tips
unsuccessfully,
> so
> > > I'm
> > > > hoping someone here may be able to help.
> > > >
> > > > Attached are the config files I used and the resulting log
file that
> > > > contains the error message. The command I used to feed in the
config
> > > files
> > > > was:
> > > > * master_metplus.py -c pcp_combine_derive_metplus.conf -c
> > > hera.mcaron.conf*
> > > >
> > > > In *pcp_combine_derive_metplus.conf*,
> > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > > > points to a directory that contains only grib files. I can
copy
> those
> > > into
> > > > the anonymous ftp site if needed.
> > > >
> > > > In addition, I tried this command, which works fine:
> > > > *pcp_combine -derive mean
> > > > /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102/*
-field
> > > > 'name="APCP"; level="A24";' derive_mean_APCP.nc*
> > > >
> > > > Other checks I did: I checked if the
> > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > > > is accessible and it is. I listed other options to
> > > > OBS_PCP_COMBINE_INPUT_ACCUMS (i.e., 1d, 6, 3, 1), which
resulted in
> the
> > > > same error. Finally, I made small changes to the config file
to see
> if
> > > the
> > > > "-sum" method works, and it does.
> > > >
> > > > The reason I'd like to use a config file is because I'd like
to
> follow
> > > the
> > > > PCPCombine process with GridStat against other data.
> > > >
> > > > Best,
> > > > -Marcel
> > > >
> > > > --
> > > > *___________________________________*
> > > > *Marcel G. Caron*
> > > > Physical Scientist, I.M. Systems Group, Inc.
> > > > Affiliate, Model Evaluation Group
> > > > @NOAA/NWS/NCEP/EMC
> > > > *5830 University Research Court,*
> > > > *College Park, MD 20740*
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Issue using the PCPCombine derive command
From: George McCabe
Time: Mon May 17 15:13:04 2021
Hi Marcel,
I would first recommend adding:
[config]
LOG_LEVEL = DEBUG
to your hera.macaron.conf file so you can view debugging logs. This
gives
you more information about what the wrapper is actually doing to find
the
files.
The logic for DERIVE and ADD methods of PCPCombine are a little
complicated
and were written with specific use cases in mind. They essentially
look
backwards in time from the current run time to find files that match
to
build the desired accumulation. It expects that you have configured
the
wrapper to specify the template of the files to look for and that
those
files have some sort of accumulation information in the file name. For
example, if you are running at 202102281200 and set:
OBS_PCP_COMBINE_DERIVE_LOOKBACK = 24H
...
OBS_PCP_COMBINE_INPUT_TEMPLATE =
st4.{valid?fmt=%Y%m}/st4_conus.{valid?fmt=%Y%m%d%H}.{level?fmt=%2H}h.grb2
then it would find the file that matches that template, read just that
file
in, and run.
Based on the command you are trying to recreate and the data in your
directory, the current logic for the DERIVE method would need to be
improved to handle a case like this. I can add a config option that
will
skip the lookback logic and just find the files that you specified
with the
wildcard expression. I can put the modified version of the METplus
wrappers
on Hera so you can test that it works as you expect for your use case.
I
will let you know when that is available.
Thanks,
George
On Mon, May 17, 2021 at 1:22 PM Marcel Caron - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
>
> John:
>
> Okay, no problem. Thank you.
>
> -Marcel
>
> On Mon, May 17, 2021 at 2:44 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Marcel,
> >
> > I received word from George McCabe that the order of the config
files
> > actually does not matter. So it makes sense that you're seeing the
> > behavior. Sorry for pointing you in the wrong direction.
> >
> > I'm going to reassign this ticket to George. I think he's best
suited to
> > debug this behavior from the pcp_combine wrapper.
> >
> > Thanks,
> > John
> >
> > On Mon, May 17, 2021 at 12:23 PM Marcel Caron - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
> > >
> > > John:
> > >
> > > Thanks for the response! I see that now, and I've switched the
order,
> > > feeding in hera.mcaron.conf first. However it still isn't able
to find
> > the
> > > files:
> > >
> > > 05/17 18:09:17.726 metplus INFO:
> ****************************************
> > > > 05/17 18:09:17.727 metplus INFO: * Running METplus
> > > > 05/17 18:09:17.727 metplus INFO: * at valid time:
202102281200
> > > > 05/17 18:09:17.727 metplus INFO:
> > ****************************************
> > > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing OBS
data
> > > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing
forecast lead
> 0
> > > > hours
> > >
> > > 05/17 18:09:17.842 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > > > (pcp_combine_wrapper.py:928) Could not find files in
> > > > /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s using template
> > > > st4.{valid?fmt=%Y%m}/*
> > > > 05/17 18:09:17.843 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > > > (pcp_combine_wrapper.py:626) pcp_combine could not generate
command
> > > > 05/17 18:09:17.843 metplus.PCPCombine (met_util.py:187) ERROR:
> > PCPCombine
> > > > had 2 errors.
> > >
> > >
> > > The directory indicated in the error message is correct as far
as I can
> > > tell. It should be looking
> > > in /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s/st4.202102/
which
> does
> > > contain the input files.
> > >
> > > Best,
> > > -Marcel
> > >
> > > On Mon, May 17, 2021 at 1:20 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Marcel,
> > > >
> > > > Ah OK, thanks for sending your METplus log file. I do see that
the
> > > METplus
> > > > wrapper for pcp_combine is erroring out:
> > > >
> > > > 05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > > > (pcp_combine_wrapper.py:928) Could not find files in
> > > /scratch2/NCEPDEV/ovp/
> > > > 05/17 15:18:40.001 metplus.PCPCombine (command_builder.py:100)
ERROR:
> > > > (pcp_combine_wrapper.py:626) pcp_combine could not generate
command
> > > > 05/17 15:18:40.002 metplus.PCPCombine (met_util.py:187) ERROR:
> > PCPCombine
> > > > had 2 errors.
> > > >
> > > > The wrapper is looking for data in the directory named
> > > > "/scratch2/NCEPDEV/ovp/". However your successful run of
pcp_combine
> > > reads
> > > > input from:
> > > > /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102
> > > >
> > > > Looking at your METplus config files...
> > > >
> > > > From pcp_combine_derive_metplus.conf:
> > > > OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}
> > > >
> > > > From hera.mcaron.conf:
> > > > INPUT_BASE = /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s
> > > >
> > > > I have a reasonable guess as to what's going on here. You're
> > > > running METplus as follows:
> > > > master_metplus.py -c pcp_combine_derive_metplus.conf -c
> > > hera.mcaron.conf
> > > >
> > > > The METplus conf files are read in the order that they're
provided on
> > the
> > > > command line. But you're referencing INPUT_BASE in
> > > > pcp_combine_derive_metplus.conf before it has been defined by
> > > > hera.mcaron.conf.
> > > >
> > > > Please try rerunning in the opposite order, like this:
> > > > * master_metplus.py -c hera.mcaron.conf -c
> > > > pcp_combine_derive_metplus.conf *
> > > >
> > > > Does that produce any better results?
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > > On Mon, May 17, 2021 at 9:56 AM Marcel Caron - NOAA Affiliate
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon May 17 09:56:27 2021: Request 99910 was acted upon.
> > > > > Transaction: Ticket created by marcel.caron at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Issue using the PCPCombine derive command
> > > > > Owner: Nobody
> > > > > Requestors: marcel.caron at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910
> > > >
> > > > >
> > > > >
> > > > > I'm using PCPCombine/derive to compute a mean among several
data
> > files
> > > > and
> > > > > the tool is having trouble finding the data files in the
provided
> > > > > directory. An error occurs when using PCPCombine in a
config file,
> > but
> > > > not
> > > > > when using it without a config file. I've tried a few
> alternatives,
> > > > which
> > > > > I mention below, and I checked online for some tips
unsuccessfully,
> > so
> > > > I'm
> > > > > hoping someone here may be able to help.
> > > > >
> > > > > Attached are the config files I used and the resulting log
file
> that
> > > > > contains the error message. The command I used to feed in
the
> config
> > > > files
> > > > > was:
> > > > > * master_metplus.py -c pcp_combine_derive_metplus.conf -c
> > > > hera.mcaron.conf*
> > > > >
> > > > > In *pcp_combine_derive_metplus.conf*,
> > > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > > > > points to a directory that contains only grib files. I can
copy
> > those
> > > > into
> > > > > the anonymous ftp site if needed.
> > > > >
> > > > > In addition, I tried this command, which works fine:
> > > > > *pcp_combine -derive mean
> > > > >
/scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102/*
> -field
> > > > > 'name="APCP"; level="A24";' derive_mean_APCP.nc*
> > > > >
> > > > > Other checks I did: I checked if the
> > > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > > > > is accessible and it is. I listed other options to
> > > > > OBS_PCP_COMBINE_INPUT_ACCUMS (i.e., 1d, 6, 3, 1), which
resulted in
> > the
> > > > > same error. Finally, I made small changes to the config
file to
> see
> > if
> > > > the
> > > > > "-sum" method works, and it does.
> > > > >
> > > > > The reason I'd like to use a config file is because I'd like
to
> > follow
> > > > the
> > > > > PCPCombine process with GridStat against other data.
> > > > >
> > > > > Best,
> > > > > -Marcel
> > > > >
> > > > > --
> > > > > *___________________________________*
> > > > > *Marcel G. Caron*
> > > > > Physical Scientist, I.M. Systems Group, Inc.
> > > > > Affiliate, Model Evaluation Group
> > > > > @NOAA/NWS/NCEP/EMC
> > > > > *5830 University Research Court,*
> > > > > *College Park, MD 20740*
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Issue using the PCPCombine derive command
From: George McCabe
Time: Mon May 17 17:05:15 2021
OK, I added a version of METplus that you can test. You will need to
run
the tests to install METplus 4.0.0 which includes MET 10.0.0 by
running
these commands:
- module use /contrib/METplus/modulefiles
- module load metplus/4.0.0
(this info was taken from this page under NOAA > Hera:
https://dtcenter.org/community-code/metplus/metplus-4-0-existing-
builds)
Instead of calling master_metplus.py from the path, you can run:
/scratch1/BMC/dtc/George.Mccabe/METplus.feature_928/ush/run_metplus.py
(Note: master_metplus.py was replaced with run_metplus.py in 4.0.0,
but
master_metplus.py will still work for a few more versions).
MET_INSTALL_DIR is set in this test version to /contrib/met/10.0.0, so
you
don't need to set MET_INSTALL_DIR in your hera.macaron.conf file
unless you
need to still use the 9.1 version of the MET executables.
You will also need to add a few additional variables to your
pcp_combine_derive_metplus.conf file:
OBS_PCP_COMBINE_INPUT_NAMES = APCP
OBS_PCP_COMBINE_INPUT_LEVELS = A24
These set the -field argument that you specified in your example call
to
pcp_combine.
Let me know if this produces the result you expect. If so, I will
merge the
changes in so it will be available in the next beta release.
Thanks,
George
On Mon, May 17, 2021 at 3:12 PM George McCabe <mccabe at ucar.edu> wrote:
> Hi Marcel,
>
> I would first recommend adding:
>
> [config]
> LOG_LEVEL = DEBUG
>
> to your hera.macaron.conf file so you can view debugging logs. This
gives
> you more information about what the wrapper is actually doing to
find the
> files.
>
> The logic for DERIVE and ADD methods of PCPCombine are a little
> complicated and were written with specific use cases in mind. They
> essentially look backwards in time from the current run time to find
files
> that match to build the desired accumulation. It expects that you
have
> configured the wrapper to specify the template of the files to look
for and
> that those files have some sort of accumulation information in the
file
> name. For example, if you are running at 202102281200 and set:
>
> OBS_PCP_COMBINE_DERIVE_LOOKBACK = 24H
> ...
> OBS_PCP_COMBINE_INPUT_TEMPLATE =
>
st4.{valid?fmt=%Y%m}/st4_conus.{valid?fmt=%Y%m%d%H}.{level?fmt=%2H}h.grb2
>
> then it would find the file that matches that template, read just
that
> file in, and run.
>
> Based on the command you are trying to recreate and the data in your
> directory, the current logic for the DERIVE method would need to be
> improved to handle a case like this. I can add a config option that
will
> skip the lookback logic and just find the files that you specified
with the
> wildcard expression. I can put the modified version of the METplus
wrappers
> on Hera so you can test that it works as you expect for your use
case. I
> will let you know when that is available.
>
> Thanks,
> George
>
>
> On Mon, May 17, 2021 at 1:22 PM Marcel Caron - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
>>
>> John:
>>
>> Okay, no problem. Thank you.
>>
>> -Marcel
>>
>> On Mon, May 17, 2021 at 2:44 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Marcel,
>> >
>> > I received word from George McCabe that the order of the config
files
>> > actually does not matter. So it makes sense that you're seeing
the
>> > behavior. Sorry for pointing you in the wrong direction.
>> >
>> > I'm going to reassign this ticket to George. I think he's best
suited to
>> > debug this behavior from the pcp_combine wrapper.
>> >
>> > Thanks,
>> > John
>> >
>> > On Mon, May 17, 2021 at 12:23 PM Marcel Caron - NOAA Affiliate
via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
>> > >
>> > > John:
>> > >
>> > > Thanks for the response! I see that now, and I've switched the
order,
>> > > feeding in hera.mcaron.conf first. However it still isn't able
to
>> find
>> > the
>> > > files:
>> > >
>> > > 05/17 18:09:17.726 metplus INFO:
>> ****************************************
>> > > > 05/17 18:09:17.727 metplus INFO: * Running METplus
>> > > > 05/17 18:09:17.727 metplus INFO: * at valid time:
202102281200
>> > > > 05/17 18:09:17.727 metplus INFO:
>> > ****************************************
>> > > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing OBS
data
>> > > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing
forecast
>> lead 0
>> > > > hours
>> > >
>> > > 05/17 18:09:17.842 metplus.PCPCombine (command_builder.py:100)
ERROR:
>> > > > (pcp_combine_wrapper.py:928) Could not find files in
>> > > > /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s using
template
>> > > > st4.{valid?fmt=%Y%m}/*
>> > > > 05/17 18:09:17.843 metplus.PCPCombine
(command_builder.py:100)
>> ERROR:
>> > > > (pcp_combine_wrapper.py:626) pcp_combine could not generate
command
>> > > > 05/17 18:09:17.843 metplus.PCPCombine (met_util.py:187)
ERROR:
>> > PCPCombine
>> > > > had 2 errors.
>> > >
>> > >
>> > > The directory indicated in the error message is correct as far
as I
>> can
>> > > tell. It should be looking
>> > > in /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s/st4.202102/
which
>> does
>> > > contain the input files.
>> > >
>> > > Best,
>> > > -Marcel
>> > >
>> > > On Mon, May 17, 2021 at 1:20 PM John Halley Gotway via RT <
>> > > met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Marcel,
>> > > >
>> > > > Ah OK, thanks for sending your METplus log file. I do see
that the
>> > > METplus
>> > > > wrapper for pcp_combine is erroring out:
>> > > >
>> > > > 05/17 15:18:40.001 metplus.PCPCombine
(command_builder.py:100)
>> ERROR:
>> > > > (pcp_combine_wrapper.py:928) Could not find files in
>> > > /scratch2/NCEPDEV/ovp/
>> > > > 05/17 15:18:40.001 metplus.PCPCombine
(command_builder.py:100)
>> ERROR:
>> > > > (pcp_combine_wrapper.py:626) pcp_combine could not generate
command
>> > > > 05/17 15:18:40.002 metplus.PCPCombine (met_util.py:187)
ERROR:
>> > PCPCombine
>> > > > had 2 errors.
>> > > >
>> > > > The wrapper is looking for data in the directory named
>> > > > "/scratch2/NCEPDEV/ovp/". However your successful run of
pcp_combine
>> > > reads
>> > > > input from:
>> > > > /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102
>> > > >
>> > > > Looking at your METplus config files...
>> > > >
>> > > > From pcp_combine_derive_metplus.conf:
>> > > > OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}
>> > > >
>> > > > From hera.mcaron.conf:
>> > > > INPUT_BASE = /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s
>> > > >
>> > > > I have a reasonable guess as to what's going on here. You're
>> > > > running METplus as follows:
>> > > > master_metplus.py -c pcp_combine_derive_metplus.conf -c
>> > > hera.mcaron.conf
>> > > >
>> > > > The METplus conf files are read in the order that they're
provided
>> on
>> > the
>> > > > command line. But you're referencing INPUT_BASE in
>> > > > pcp_combine_derive_metplus.conf before it has been defined by
>> > > > hera.mcaron.conf.
>> > > >
>> > > > Please try rerunning in the opposite order, like this:
>> > > > * master_metplus.py -c hera.mcaron.conf -c
>> > > > pcp_combine_derive_metplus.conf *
>> > > >
>> > > > Does that produce any better results?
>> > > >
>> > > > Thanks,
>> > > > John Halley Gotway
>> > > >
>> > > > On Mon, May 17, 2021 at 9:56 AM Marcel Caron - NOAA Affiliate
via
>> RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Mon May 17 09:56:27 2021: Request 99910 was acted upon.
>> > > > > Transaction: Ticket created by marcel.caron at noaa.gov
>> > > > > Queue: met_help
>> > > > > Subject: Issue using the PCPCombine derive command
>> > > > > Owner: Nobody
>> > > > > Requestors: marcel.caron at noaa.gov
>> > > > > Status: new
>> > > > > Ticket <URL:
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910
>> > > >
>> > > > >
>> > > > >
>> > > > > I'm using PCPCombine/derive to compute a mean among several
data
>> > files
>> > > > and
>> > > > > the tool is having trouble finding the data files in the
provided
>> > > > > directory. An error occurs when using PCPCombine in a
config
>> file,
>> > but
>> > > > not
>> > > > > when using it without a config file. I've tried a few
>> alternatives,
>> > > > which
>> > > > > I mention below, and I checked online for some tips
>> unsuccessfully,
>> > so
>> > > > I'm
>> > > > > hoping someone here may be able to help.
>> > > > >
>> > > > > Attached are the config files I used and the resulting log
file
>> that
>> > > > > contains the error message. The command I used to feed in
the
>> config
>> > > > files
>> > > > > was:
>> > > > > * master_metplus.py -c pcp_combine_derive_metplus.conf -c
>> > > > hera.mcaron.conf*
>> > > > >
>> > > > > In *pcp_combine_derive_metplus.conf*,
>> > > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
>> > > > > points to a directory that contains only grib files. I can
copy
>> > those
>> > > > into
>> > > > > the anonymous ftp site if needed.
>> > > > >
>> > > > > In addition, I tried this command, which works fine:
>> > > > > *pcp_combine -derive mean
>> > > > >
/scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102/*
>> -field
>> > > > > 'name="APCP"; level="A24";' derive_mean_APCP.nc*
>> > > > >
>> > > > > Other checks I did: I checked if the
>> > > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
>> > > > > is accessible and it is. I listed other options to
>> > > > > OBS_PCP_COMBINE_INPUT_ACCUMS (i.e., 1d, 6, 3, 1), which
resulted
>> in
>> > the
>> > > > > same error. Finally, I made small changes to the config
file to
>> see
>> > if
>> > > > the
>> > > > > "-sum" method works, and it does.
>> > > > >
>> > > > > The reason I'd like to use a config file is because I'd
like to
>> > follow
>> > > > the
>> > > > > PCPCombine process with GridStat against other data.
>> > > > >
>> > > > > Best,
>> > > > > -Marcel
>> > > > >
>> > > > > --
>> > > > > *___________________________________*
>> > > > > *Marcel G. Caron*
>> > > > > Physical Scientist, I.M. Systems Group, Inc.
>> > > > > Affiliate, Model Evaluation Group
>> > > > > @NOAA/NWS/NCEP/EMC
>> > > > > *5830 University Research Court,*
>> > > > > *College Park, MD 20740*
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Issue using the PCPCombine derive command
From: Marcel Caron - NOAA Affiliate
Time: Tue May 18 08:53:40 2021
George:
I made the changes you suggested and ran METplus with the test
version, and
it worked great. I could also match the results from the test version
with
my own mean derivation.
That adaptation I think will be useful for my other METplus tasks.
Thanks
for the explanation and the help, and looking forward to the next
release.
Best,
-Marcel
On Mon, May 17, 2021 at 7:05 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> OK, I added a version of METplus that you can test. You will need to
run
> the tests to install METplus 4.0.0 which includes MET 10.0.0 by
running
> these commands:
>
> - module use /contrib/METplus/modulefiles
> - module load metplus/4.0.0
>
> (this info was taken from this page under NOAA > Hera:
> https://dtcenter.org/community-code/metplus/metplus-4-0-existing-
builds)
>
> Instead of calling master_metplus.py from the path, you can run:
>
>
/scratch1/BMC/dtc/George.Mccabe/METplus.feature_928/ush/run_metplus.py
>
> (Note: master_metplus.py was replaced with run_metplus.py in 4.0.0,
but
> master_metplus.py will still work for a few more versions).
>
> MET_INSTALL_DIR is set in this test version to /contrib/met/10.0.0,
so you
> don't need to set MET_INSTALL_DIR in your hera.macaron.conf file
unless you
> need to still use the 9.1 version of the MET executables.
>
> You will also need to add a few additional variables to your
> pcp_combine_derive_metplus.conf file:
>
> OBS_PCP_COMBINE_INPUT_NAMES = APCP
> OBS_PCP_COMBINE_INPUT_LEVELS = A24
>
> These set the -field argument that you specified in your example
call to
> pcp_combine.
>
> Let me know if this produces the result you expect. If so, I will
merge the
> changes in so it will be available in the next beta release.
>
> Thanks,
> George
>
> On Mon, May 17, 2021 at 3:12 PM George McCabe <mccabe at ucar.edu>
wrote:
>
> > Hi Marcel,
> >
> > I would first recommend adding:
> >
> > [config]
> > LOG_LEVEL = DEBUG
> >
> > to your hera.macaron.conf file so you can view debugging logs.
This gives
> > you more information about what the wrapper is actually doing to
find the
> > files.
> >
> > The logic for DERIVE and ADD methods of PCPCombine are a little
> > complicated and were written with specific use cases in mind. They
> > essentially look backwards in time from the current run time to
find
> files
> > that match to build the desired accumulation. It expects that you
have
> > configured the wrapper to specify the template of the files to
look for
> and
> > that those files have some sort of accumulation information in the
file
> > name. For example, if you are running at 202102281200 and set:
> >
> > OBS_PCP_COMBINE_DERIVE_LOOKBACK = 24H
> > ...
> > OBS_PCP_COMBINE_INPUT_TEMPLATE =
> >
st4.{valid?fmt=%Y%m}/st4_conus.{valid?fmt=%Y%m%d%H}.{level?fmt=%2H}h.grb2
> >
> > then it would find the file that matches that template, read just
that
> > file in, and run.
> >
> > Based on the command you are trying to recreate and the data in
your
> > directory, the current logic for the DERIVE method would need to
be
> > improved to handle a case like this. I can add a config option
that will
> > skip the lookback logic and just find the files that you specified
with
> the
> > wildcard expression. I can put the modified version of the METplus
> wrappers
> > on Hera so you can test that it works as you expect for your use
case. I
> > will let you know when that is available.
> >
> > Thanks,
> > George
> >
> >
> > On Mon, May 17, 2021 at 1:22 PM Marcel Caron - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
> >>
> >> John:
> >>
> >> Okay, no problem. Thank you.
> >>
> >> -Marcel
> >>
> >> On Mon, May 17, 2021 at 2:44 PM John Halley Gotway via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Marcel,
> >> >
> >> > I received word from George McCabe that the order of the config
files
> >> > actually does not matter. So it makes sense that you're seeing
the
> >> > behavior. Sorry for pointing you in the wrong direction.
> >> >
> >> > I'm going to reassign this ticket to George. I think he's best
suited
> to
> >> > debug this behavior from the pcp_combine wrapper.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> > On Mon, May 17, 2021 at 12:23 PM Marcel Caron - NOAA Affiliate
via RT
> <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910
>
> >> > >
> >> > > John:
> >> > >
> >> > > Thanks for the response! I see that now, and I've switched
the
> order,
> >> > > feeding in hera.mcaron.conf first. However it still isn't
able to
> >> find
> >> > the
> >> > > files:
> >> > >
> >> > > 05/17 18:09:17.726 metplus INFO:
> >> ****************************************
> >> > > > 05/17 18:09:17.727 metplus INFO: * Running METplus
> >> > > > 05/17 18:09:17.727 metplus INFO: * at valid time:
202102281200
> >> > > > 05/17 18:09:17.727 metplus INFO:
> >> > ****************************************
> >> > > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing OBS
data
> >> > > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing
forecast
> >> lead 0
> >> > > > hours
> >> > >
> >> > > 05/17 18:09:17.842 metplus.PCPCombine
(command_builder.py:100)
> ERROR:
> >> > > > (pcp_combine_wrapper.py:928) Could not find files in
> >> > > > /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s using
template
> >> > > > st4.{valid?fmt=%Y%m}/*
> >> > > > 05/17 18:09:17.843 metplus.PCPCombine
(command_builder.py:100)
> >> ERROR:
> >> > > > (pcp_combine_wrapper.py:626) pcp_combine could not generate
> command
> >> > > > 05/17 18:09:17.843 metplus.PCPCombine (met_util.py:187)
ERROR:
> >> > PCPCombine
> >> > > > had 2 errors.
> >> > >
> >> > >
> >> > > The directory indicated in the error message is correct as
far as I
> >> can
> >> > > tell. It should be looking
> >> > > in
/scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s/st4.202102/ which
> >> does
> >> > > contain the input files.
> >> > >
> >> > > Best,
> >> > > -Marcel
> >> > >
> >> > > On Mon, May 17, 2021 at 1:20 PM John Halley Gotway via RT <
> >> > > met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > > Marcel,
> >> > > >
> >> > > > Ah OK, thanks for sending your METplus log file. I do see
that the
> >> > > METplus
> >> > > > wrapper for pcp_combine is erroring out:
> >> > > >
> >> > > > 05/17 15:18:40.001 metplus.PCPCombine
(command_builder.py:100)
> >> ERROR:
> >> > > > (pcp_combine_wrapper.py:928) Could not find files in
> >> > > /scratch2/NCEPDEV/ovp/
> >> > > > 05/17 15:18:40.001 metplus.PCPCombine
(command_builder.py:100)
> >> ERROR:
> >> > > > (pcp_combine_wrapper.py:626) pcp_combine could not generate
> command
> >> > > > 05/17 15:18:40.002 metplus.PCPCombine (met_util.py:187)
ERROR:
> >> > PCPCombine
> >> > > > had 2 errors.
> >> > > >
> >> > > > The wrapper is looking for data in the directory named
> >> > > > "/scratch2/NCEPDEV/ovp/". However your successful run of
> pcp_combine
> >> > > reads
> >> > > > input from:
> >> > > > /scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102
> >> > > >
> >> > > > Looking at your METplus config files...
> >> > > >
> >> > > > From pcp_combine_derive_metplus.conf:
> >> > > > OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}
> >> > > >
> >> > > > From hera.mcaron.conf:
> >> > > > INPUT_BASE =
/scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s
> >> > > >
> >> > > > I have a reasonable guess as to what's going on here.
You're
> >> > > > running METplus as follows:
> >> > > > master_metplus.py -c pcp_combine_derive_metplus.conf -c
> >> > > hera.mcaron.conf
> >> > > >
> >> > > > The METplus conf files are read in the order that they're
provided
> >> on
> >> > the
> >> > > > command line. But you're referencing INPUT_BASE in
> >> > > > pcp_combine_derive_metplus.conf before it has been defined
by
> >> > > > hera.mcaron.conf.
> >> > > >
> >> > > > Please try rerunning in the opposite order, like this:
> >> > > > * master_metplus.py -c hera.mcaron.conf -c
> >> > > > pcp_combine_derive_metplus.conf *
> >> > > >
> >> > > > Does that produce any better results?
> >> > > >
> >> > > > Thanks,
> >> > > > John Halley Gotway
> >> > > >
> >> > > > On Mon, May 17, 2021 at 9:56 AM Marcel Caron - NOAA
Affiliate via
> >> RT <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > Mon May 17 09:56:27 2021: Request 99910 was acted upon.
> >> > > > > Transaction: Ticket created by marcel.caron at noaa.gov
> >> > > > > Queue: met_help
> >> > > > > Subject: Issue using the PCPCombine derive command
> >> > > > > Owner: Nobody
> >> > > > > Requestors: marcel.caron at noaa.gov
> >> > > > > Status: new
> >> > > > > Ticket <URL:
> >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > I'm using PCPCombine/derive to compute a mean among
several data
> >> > files
> >> > > > and
> >> > > > > the tool is having trouble finding the data files in the
> provided
> >> > > > > directory. An error occurs when using PCPCombine in a
config
> >> file,
> >> > but
> >> > > > not
> >> > > > > when using it without a config file. I've tried a few
> >> alternatives,
> >> > > > which
> >> > > > > I mention below, and I checked online for some tips
> >> unsuccessfully,
> >> > so
> >> > > > I'm
> >> > > > > hoping someone here may be able to help.
> >> > > > >
> >> > > > > Attached are the config files I used and the resulting
log file
> >> that
> >> > > > > contains the error message. The command I used to feed
in the
> >> config
> >> > > > files
> >> > > > > was:
> >> > > > > * master_metplus.py -c pcp_combine_derive_metplus.conf -c
> >> > > > hera.mcaron.conf*
> >> > > > >
> >> > > > > In *pcp_combine_derive_metplus.conf*,
> >> > > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> >> > > > > points to a directory that contains only grib files. I
can copy
> >> > those
> >> > > > into
> >> > > > > the anonymous ftp site if needed.
> >> > > > >
> >> > > > > In addition, I tried this command, which works fine:
> >> > > > > *pcp_combine -derive mean
> >> > > > >
/scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102/*
> >> -field
> >> > > > > 'name="APCP"; level="A24";' derive_mean_APCP.nc*
> >> > > > >
> >> > > > > Other checks I did: I checked if the
> >> > > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> >> > > > > is accessible and it is. I listed other options to
> >> > > > > OBS_PCP_COMBINE_INPUT_ACCUMS (i.e., 1d, 6, 3, 1), which
resulted
> >> in
> >> > the
> >> > > > > same error. Finally, I made small changes to the config
file to
> >> see
> >> > if
> >> > > > the
> >> > > > > "-sum" method works, and it does.
> >> > > > >
> >> > > > > The reason I'd like to use a config file is because I'd
like to
> >> > follow
> >> > > > the
> >> > > > > PCPCombine process with GridStat against other data.
> >> > > > >
> >> > > > > Best,
> >> > > > > -Marcel
> >> > > > >
> >> > > > > --
> >> > > > > *___________________________________*
> >> > > > > *Marcel G. Caron*
> >> > > > > Physical Scientist, I.M. Systems Group, Inc.
> >> > > > > Affiliate, Model Evaluation Group
> >> > > > > @NOAA/NWS/NCEP/EMC
> >> > > > > *5830 University Research Court,*
> >> > > > > *College Park, MD 20740*
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: Issue using the PCPCombine derive command
From: George McCabe
Time: Tue May 18 09:01:13 2021
Great to hear and thanks for confirming that it worked for your use
case.
Here is the GitHub issue for this work in case you are interested in
tracking the progress: https://github.com/dtcenter/METplus/issues/928
The changes should be available in the METplus 4.1.0-beta1 release.
I'll close this ticket now, but feel free to open a new one if you run
into
any issues.
Thanks,
George
On Tue, May 18, 2021 at 8:54 AM Marcel Caron - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
>
> George:
>
> I made the changes you suggested and ran METplus with the test
version, and
> it worked great. I could also match the results from the test
version with
> my own mean derivation.
>
> That adaptation I think will be useful for my other METplus tasks.
Thanks
> for the explanation and the help, and looking forward to the next
release.
>
> Best,
> -Marcel
>
> On Mon, May 17, 2021 at 7:05 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > OK, I added a version of METplus that you can test. You will need
to run
> > the tests to install METplus 4.0.0 which includes MET 10.0.0 by
running
> > these commands:
> >
> > - module use /contrib/METplus/modulefiles
> > - module load metplus/4.0.0
> >
> > (this info was taken from this page under NOAA > Hera:
> > https://dtcenter.org/community-code/metplus/metplus-4-0-existing-
builds)
> >
> > Instead of calling master_metplus.py from the path, you can run:
> >
> >
/scratch1/BMC/dtc/George.Mccabe/METplus.feature_928/ush/run_metplus.py
> >
> > (Note: master_metplus.py was replaced with run_metplus.py in
4.0.0, but
> > master_metplus.py will still work for a few more versions).
> >
> > MET_INSTALL_DIR is set in this test version to
/contrib/met/10.0.0, so
> you
> > don't need to set MET_INSTALL_DIR in your hera.macaron.conf file
unless
> you
> > need to still use the 9.1 version of the MET executables.
> >
> > You will also need to add a few additional variables to your
> > pcp_combine_derive_metplus.conf file:
> >
> > OBS_PCP_COMBINE_INPUT_NAMES = APCP
> > OBS_PCP_COMBINE_INPUT_LEVELS = A24
> >
> > These set the -field argument that you specified in your example
call to
> > pcp_combine.
> >
> > Let me know if this produces the result you expect. If so, I will
merge
> the
> > changes in so it will be available in the next beta release.
> >
> > Thanks,
> > George
> >
> > On Mon, May 17, 2021 at 3:12 PM George McCabe <mccabe at ucar.edu>
wrote:
> >
> > > Hi Marcel,
> > >
> > > I would first recommend adding:
> > >
> > > [config]
> > > LOG_LEVEL = DEBUG
> > >
> > > to your hera.macaron.conf file so you can view debugging logs.
This
> gives
> > > you more information about what the wrapper is actually doing to
find
> the
> > > files.
> > >
> > > The logic for DERIVE and ADD methods of PCPCombine are a little
> > > complicated and were written with specific use cases in mind.
They
> > > essentially look backwards in time from the current run time to
find
> > files
> > > that match to build the desired accumulation. It expects that
you have
> > > configured the wrapper to specify the template of the files to
look for
> > and
> > > that those files have some sort of accumulation information in
the file
> > > name. For example, if you are running at 202102281200 and set:
> > >
> > > OBS_PCP_COMBINE_DERIVE_LOOKBACK = 24H
> > > ...
> > > OBS_PCP_COMBINE_INPUT_TEMPLATE =
> > >
>
st4.{valid?fmt=%Y%m}/st4_conus.{valid?fmt=%Y%m%d%H}.{level?fmt=%2H}h.grb2
> > >
> > > then it would find the file that matches that template, read
just that
> > > file in, and run.
> > >
> > > Based on the command you are trying to recreate and the data in
your
> > > directory, the current logic for the DERIVE method would need to
be
> > > improved to handle a case like this. I can add a config option
that
> will
> > > skip the lookback logic and just find the files that you
specified with
> > the
> > > wildcard expression. I can put the modified version of the
METplus
> > wrappers
> > > on Hera so you can test that it works as you expect for your use
case.
> I
> > > will let you know when that is available.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > > On Mon, May 17, 2021 at 1:22 PM Marcel Caron - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
> > >>
> > >> John:
> > >>
> > >> Okay, no problem. Thank you.
> > >>
> > >> -Marcel
> > >>
> > >> On Mon, May 17, 2021 at 2:44 PM John Halley Gotway via RT <
> > >> met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> > Marcel,
> > >> >
> > >> > I received word from George McCabe that the order of the
config
> files
> > >> > actually does not matter. So it makes sense that you're
seeing the
> > >> > behavior. Sorry for pointing you in the wrong direction.
> > >> >
> > >> > I'm going to reassign this ticket to George. I think he's
best
> suited
> > to
> > >> > debug this behavior from the pcp_combine wrapper.
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >> > On Mon, May 17, 2021 at 12:23 PM Marcel Caron - NOAA
Affiliate via
> RT
> > <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910 >
> > >> > >
> > >> > > John:
> > >> > >
> > >> > > Thanks for the response! I see that now, and I've switched
the
> > order,
> > >> > > feeding in hera.mcaron.conf first. However it still isn't
able to
> > >> find
> > >> > the
> > >> > > files:
> > >> > >
> > >> > > 05/17 18:09:17.726 metplus INFO:
> > >> ****************************************
> > >> > > > 05/17 18:09:17.727 metplus INFO: * Running METplus
> > >> > > > 05/17 18:09:17.727 metplus INFO: * at valid time:
202102281200
> > >> > > > 05/17 18:09:17.727 metplus INFO:
> > >> > ****************************************
> > >> > > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing
OBS data
> > >> > > > 05/17 18:09:17.727 metplus.PCPCombine INFO: Processing
forecast
> > >> lead 0
> > >> > > > hours
> > >> > >
> > >> > > 05/17 18:09:17.842 metplus.PCPCombine
(command_builder.py:100)
> > ERROR:
> > >> > > > (pcp_combine_wrapper.py:928) Could not find files in
> > >> > > > /scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s using
template
> > >> > > > st4.{valid?fmt=%Y%m}/*
> > >> > > > 05/17 18:09:17.843 metplus.PCPCombine
(command_builder.py:100)
> > >> ERROR:
> > >> > > > (pcp_combine_wrapper.py:626) pcp_combine could not
generate
> > command
> > >> > > > 05/17 18:09:17.843 metplus.PCPCombine (met_util.py:187)
ERROR:
> > >> > PCPCombine
> > >> > > > had 2 errors.
> > >> > >
> > >> > >
> > >> > > The directory indicated in the error message is correct as
far as
> I
> > >> can
> > >> > > tell. It should be looking
> > >> > > in
/scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s/st4.202102/
> which
> > >> does
> > >> > > contain the input files.
> > >> > >
> > >> > > Best,
> > >> > > -Marcel
> > >> > >
> > >> > > On Mon, May 17, 2021 at 1:20 PM John Halley Gotway via RT <
> > >> > > met_help at ucar.edu>
> > >> > > wrote:
> > >> > >
> > >> > > > Marcel,
> > >> > > >
> > >> > > > Ah OK, thanks for sending your METplus log file. I do see
that
> the
> > >> > > METplus
> > >> > > > wrapper for pcp_combine is erroring out:
> > >> > > >
> > >> > > > 05/17 15:18:40.001 metplus.PCPCombine
(command_builder.py:100)
> > >> ERROR:
> > >> > > > (pcp_combine_wrapper.py:928) Could not find files in
> > >> > > /scratch2/NCEPDEV/ovp/
> > >> > > > 05/17 15:18:40.001 metplus.PCPCombine
(command_builder.py:100)
> > >> ERROR:
> > >> > > > (pcp_combine_wrapper.py:626) pcp_combine could not
generate
> > command
> > >> > > > 05/17 15:18:40.002 metplus.PCPCombine (met_util.py:187)
ERROR:
> > >> > PCPCombine
> > >> > > > had 2 errors.
> > >> > > >
> > >> > > > The wrapper is looking for data in the directory named
> > >> > > > "/scratch2/NCEPDEV/ovp/". However your successful run of
> > pcp_combine
> > >> > > reads
> > >> > > > input from:
> > >> > > >
/scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102
> > >> > > >
> > >> > > > Looking at your METplus config files...
> > >> > > >
> > >> > > > From pcp_combine_derive_metplus.conf:
> > >> > > > OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}
> > >> > > >
> > >> > > > From hera.mcaron.conf:
> > >> > > > INPUT_BASE =
/scratch2/NCEPDEV/ovp/Marcel.Caron/MEG/data/s2s
> > >> > > >
> > >> > > > I have a reasonable guess as to what's going on here.
You're
> > >> > > > running METplus as follows:
> > >> > > > master_metplus.py -c pcp_combine_derive_metplus.conf
-c
> > >> > > hera.mcaron.conf
> > >> > > >
> > >> > > > The METplus conf files are read in the order that they're
> provided
> > >> on
> > >> > the
> > >> > > > command line. But you're referencing INPUT_BASE in
> > >> > > > pcp_combine_derive_metplus.conf before it has been
defined by
> > >> > > > hera.mcaron.conf.
> > >> > > >
> > >> > > > Please try rerunning in the opposite order, like this:
> > >> > > > * master_metplus.py -c hera.mcaron.conf -c
> > >> > > > pcp_combine_derive_metplus.conf *
> > >> > > >
> > >> > > > Does that produce any better results?
> > >> > > >
> > >> > > > Thanks,
> > >> > > > John Halley Gotway
> > >> > > >
> > >> > > > On Mon, May 17, 2021 at 9:56 AM Marcel Caron - NOAA
Affiliate
> via
> > >> RT <
> > >> > > > met_help at ucar.edu> wrote:
> > >> > > >
> > >> > > > >
> > >> > > > > Mon May 17 09:56:27 2021: Request 99910 was acted upon.
> > >> > > > > Transaction: Ticket created by marcel.caron at noaa.gov
> > >> > > > > Queue: met_help
> > >> > > > > Subject: Issue using the PCPCombine derive command
> > >> > > > > Owner: Nobody
> > >> > > > > Requestors: marcel.caron at noaa.gov
> > >> > > > > Status: new
> > >> > > > > Ticket <URL:
> > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99910
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > I'm using PCPCombine/derive to compute a mean among
several
> data
> > >> > files
> > >> > > > and
> > >> > > > > the tool is having trouble finding the data files in
the
> > provided
> > >> > > > > directory. An error occurs when using PCPCombine in a
config
> > >> file,
> > >> > but
> > >> > > > not
> > >> > > > > when using it without a config file. I've tried a few
> > >> alternatives,
> > >> > > > which
> > >> > > > > I mention below, and I checked online for some tips
> > >> unsuccessfully,
> > >> > so
> > >> > > > I'm
> > >> > > > > hoping someone here may be able to help.
> > >> > > > >
> > >> > > > > Attached are the config files I used and the resulting
log
> file
> > >> that
> > >> > > > > contains the error message. The command I used to feed
in the
> > >> config
> > >> > > > files
> > >> > > > > was:
> > >> > > > > * master_metplus.py -c pcp_combine_derive_metplus.conf
-c
> > >> > > > hera.mcaron.conf*
> > >> > > > >
> > >> > > > > In *pcp_combine_derive_metplus.conf*,
> > >> > > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > >> > > > > points to a directory that contains only grib files. I
can
> copy
> > >> > those
> > >> > > > into
> > >> > > > > the anonymous ftp site if needed.
> > >> > > > >
> > >> > > > > In addition, I tried this command, which works fine:
> > >> > > > > *pcp_combine -derive mean
> > >> > > > >
/scratch2/NCEPDEV/ovp/Marcel.Caron//MEG/data/s2s/st4.202102/*
> > >> -field
> > >> > > > > 'name="APCP"; level="A24";' derive_mean_APCP.nc*
> > >> > > > >
> > >> > > > > Other checks I did: I checked if the
> > >> > > > OBS_PCP_COMBINE_INPUT_[DIR/TEMPLATE]
> > >> > > > > is accessible and it is. I listed other options to
> > >> > > > > OBS_PCP_COMBINE_INPUT_ACCUMS (i.e., 1d, 6, 3, 1), which
> resulted
> > >> in
> > >> > the
> > >> > > > > same error. Finally, I made small changes to the
config file
> to
> > >> see
> > >> > if
> > >> > > > the
> > >> > > > > "-sum" method works, and it does.
> > >> > > > >
> > >> > > > > The reason I'd like to use a config file is because I'd
like
> to
> > >> > follow
> > >> > > > the
> > >> > > > > PCPCombine process with GridStat against other data.
> > >> > > > >
> > >> > > > > Best,
> > >> > > > > -Marcel
> > >> > > > >
> > >> > > > > --
> > >> > > > > *___________________________________*
> > >> > > > > *Marcel G. Caron*
> > >> > > > > Physical Scientist, I.M. Systems Group, Inc.
> > >> > > > > Affiliate, Model Evaluation Group
> > >> > > > > @NOAA/NWS/NCEP/EMC
> > >> > > > > *5830 University Research Court,*
> > >> > > > > *College Park, MD 20740*
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list