[Met_help] [rt.rap.ucar.edu #95640] History for METplus - Grid Stat variable error
George McCabe via RT
met_help at ucar.edu
Wed Aug 5 10:49:28 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi MET help desk,
I’ve come across an issue with running grid_stat on METplus v.3.0.2. I am verifying files created with ensemble_stat in METplus and passing the ensemble NMEP variables through to grid_stat. However, upon looking at the output log, an extra underscore is added to the variable which causes grid_stat to fail. For example, I am passing in the following variable in my METplus config file: “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”. However, the variable that is shown in the log file comes up as: “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level specification also seems to be affected (my input level is “A1” while the grid_stat input level shows up as the generic netCDF input level “(*,*)”). Is there anything I could do to help fix this issue? I’ve gone ahead and attached a truncated log file with the error message as well as my METplus configuration file in case this is needed for troubleshooting. Any help would be greatly appreciated!
Many thanks,
-Brian
—————————————————————
Brian Matilla
Research Fellow— Warn-on-Forecast Team
Cooperative Institute for Mesoscale Meteorological Studies — The University of Oklahoma
NOAA National Severe Storms Laboratory
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: METplus - Grid Stat variable error
From: George McCabe
Time: Wed Jun 17 11:27:31 2020
Hi Brian,
This is unfortunately due to problematic logic used for the GridStat
wrapper. If [FCST/OBS]_PCP_COMBINE_RUN is set to True in the conf
file, the
wrapper automatically sets the field name to <name>_<level> where
<level>
is the value passed in with the letter removed, then sets the level to
(*,*). This is the format of the data generated by PCPCombine. The
logic
assumes that GridStat is run on these data.
I have wanted to remove this bad logic, but it has been left in to
support
legacy configurations. A long time ago, the [FCST/OBS]_VAR<n>_*
variables
controlled settings for PCPCombine, RegridDataPlane, and GridStat, but
we
have since added config variables to control these settings
individually
per tool. It sounds like it is time to resolve this issue. I will
bring it
up at the next METplus meeting and discuss a good solution.
In the meantime, you can get around this by running the PCPCombine
portion
first, then run it again calling EnsembleStat and GridStat with
FCST_PCP_COMBINE_RUN set to False. You should be able to control this
with
command line arguments to master_metplus.py:
/home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
\
-c MP_WOFS_pcp.conf \
-c base_paths.conf \
-c PROCESS_LIST=PCPCombine
/home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
\
-c MP_WOFS_pcp.conf \
-c base_paths.conf \
-c PROCESS_LIST=EnsembleStat
/home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
\
-c MP_WOFS_pcp.conf \
-c base_paths.conf \
-c config.PROCESS_LIST=GridStat \
-c config.FCST_PCP_COMBINE_RUN=False
Another note, you SHOULD be able to define -c
PROCESS_LIST="EnsembleStat,GridStat"to run both, but it looks like
that is
not currently supported due to how the config arguments are processed.
I'll
also make a ticket to enhance the logic to support that.
Let me know if that doesn't work for you and I can take a closer look.
Also, FYI, it is often useful to set [config] LOG_LEVEL = DEBUG. You
get a
lot more information in the logs that are useful when things are not
working correctly.
Thanks,
George
On Wed, Jun 17, 2020 at 11:04 AM Brian Matilla - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Wed Jun 17 11:04:05 2020: Request 95640 was acted upon.
> Transaction: Ticket created by brian.matilla at noaa.gov
> Queue: met_help
> Subject: METplus - Grid Stat variable error
> Owner: Nobody
> Requestors: brian.matilla at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640 >
>
>
> Hi MET help desk,
>
> I’ve come across an issue with running grid_stat on METplus v.3.0.2.
I am
> verifying files created with ensemble_stat in METplus and passing
the
> ensemble NMEP variables through to grid_stat. However, upon looking
at the
> output log, an extra underscore is added to the variable which
causes
> grid_stat to fail. For example, I am passing in the following
variable in
> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
> However, the variable that is shown in the log file comes up as:
> “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
> specification also seems to be affected (my input level is “A1”
while the
> grid_stat input level shows up as the generic netCDF input level
“(*,*)”).
> Is there anything I could do to help fix this issue? I’ve gone ahead
and
> attached a truncated log file with the error message as well as my
METplus
> configuration file in case this is needed for troubleshooting. Any
help
> would be greatly appreciated!
>
> Many thanks,
>
> -Brian
> —————————————————————
> Brian Matilla
> Research Fellow— Warn-on-Forecast Team
> Cooperative Institute for Mesoscale Meteorological Studies — The
> University of Oklahoma
> NOAA National Severe Storms Laboratory
>
> Hi MET help desk,
>
> I’ve come across an issue with running grid_stat on METplus v.3.0.2.
I am
> verifying files created with ensemble_stat in METplus and passing
the
> ensemble NMEP variables through to grid_stat. However, upon looking
at the
> output log, an extra underscore is added to the variable which
causes
> grid_stat to fail. For example, I am passing in the following
variable in
> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
> However, the variable that is shown in the log file comes up as: “
> APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
> specification also seems to be affected (my input level is “A1”
while the
> grid_stat input level shows up as the generic netCDF input level
“(*,*)”).
> Is there anything I could do to help fix this issue? I’ve gone ahead
and
> attached a truncated log file with the error message as well as my
METplus
> configuration file in case this is needed for troubleshooting. Any
help
> would be greatly appreciated!
>
> Many thanks,
>
> -Brian
> —————————————————————
> Brian Matilla
> Research Fellow— Warn-on-Forecast Team
> Cooperative Institute for Mesoscale Meteorological Studies — The
> University of Oklahoma
> NOAA National Severe Storms Laboratory
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: METplus - Grid Stat variable error
From: George McCabe
Time: Wed Jun 17 11:32:24 2020
Hi Brian,
Would you be willing to provide a subset of data used to run this use
case?
Enough data to run one initialization time and one ensemble would be
sufficient. This would make it easier to test the code changes. We
could
also add the use case to the METplus repository so there is an example
of a
successful run with this configuration, since we currently don't have
any
use cases that run PCPCombine > EnsembleStat > GridStat.
Thanks,
George
On Wed, Jun 17, 2020 at 11:27 AM George McCabe <mccabe at ucar.edu>
wrote:
> Hi Brian,
>
> This is unfortunately due to problematic logic used for the GridStat
> wrapper. If [FCST/OBS]_PCP_COMBINE_RUN is set to True in the conf
file, the
> wrapper automatically sets the field name to <name>_<level> where
<level>
> is the value passed in with the letter removed, then sets the level
to
> (*,*). This is the format of the data generated by PCPCombine. The
logic
> assumes that GridStat is run on these data.
>
> I have wanted to remove this bad logic, but it has been left in to
support
> legacy configurations. A long time ago, the [FCST/OBS]_VAR<n>_*
variables
> controlled settings for PCPCombine, RegridDataPlane, and GridStat,
but we
> have since added config variables to control these settings
individually
> per tool. It sounds like it is time to resolve this issue. I will
bring it
> up at the next METplus meeting and discuss a good solution.
>
> In the meantime, you can get around this by running the PCPCombine
portion
> first, then run it again calling EnsembleStat and GridStat with
> FCST_PCP_COMBINE_RUN set to False. You should be able to control
this with
> command line arguments to master_metplus.py:
>
> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
> \
> -c MP_WOFS_pcp.conf \
> -c base_paths.conf \
> -c PROCESS_LIST=PCPCombine
>
> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
> \
> -c MP_WOFS_pcp.conf \
> -c base_paths.conf \
> -c PROCESS_LIST=EnsembleStat
>
> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
> \
> -c MP_WOFS_pcp.conf \
> -c base_paths.conf \
> -c config.PROCESS_LIST=GridStat \
> -c config.FCST_PCP_COMBINE_RUN=False
>
>
>
> Another note, you SHOULD be able to define -c
> PROCESS_LIST="EnsembleStat,GridStat"to run both, but it looks like
that is
> not currently supported due to how the config arguments are
processed. I'll
> also make a ticket to enhance the logic to support that.
>
> Let me know if that doesn't work for you and I can take a closer
look.
> Also, FYI, it is often useful to set [config] LOG_LEVEL = DEBUG. You
get a
> lot more information in the logs that are useful when things are not
> working correctly.
>
> Thanks,
> George
>
> On Wed, Jun 17, 2020 at 11:04 AM Brian Matilla - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> Wed Jun 17 11:04:05 2020: Request 95640 was acted upon.
>> Transaction: Ticket created by brian.matilla at noaa.gov
>> Queue: met_help
>> Subject: METplus - Grid Stat variable error
>> Owner: Nobody
>> Requestors: brian.matilla at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640 >
>>
>>
>> Hi MET help desk,
>>
>> I’ve come across an issue with running grid_stat on METplus
v.3.0.2. I am
>> verifying files created with ensemble_stat in METplus and passing
the
>> ensemble NMEP variables through to grid_stat. However, upon looking
at the
>> output log, an extra underscore is added to the variable which
causes
>> grid_stat to fail. For example, I am passing in the following
variable in
>> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
>> However, the variable that is shown in the log file comes up as:
>> “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
>> specification also seems to be affected (my input level is “A1”
while the
>> grid_stat input level shows up as the generic netCDF input level
“(*,*)”).
>> Is there anything I could do to help fix this issue? I’ve gone
ahead and
>> attached a truncated log file with the error message as well as my
METplus
>> configuration file in case this is needed for troubleshooting. Any
help
>> would be greatly appreciated!
>>
>> Many thanks,
>>
>> -Brian
>> —————————————————————
>> Brian Matilla
>> Research Fellow— Warn-on-Forecast Team
>> Cooperative Institute for Mesoscale Meteorological Studies — The
>> University of Oklahoma
>> NOAA National Severe Storms Laboratory
>>
>> Hi MET help desk,
>>
>> I’ve come across an issue with running grid_stat on METplus
v.3.0.2. I am
>> verifying files created with ensemble_stat in METplus and passing
the
>> ensemble NMEP variables through to grid_stat. However, upon looking
at the
>> output log, an extra underscore is added to the variable which
causes
>> grid_stat to fail. For example, I am passing in the following
variable in
>> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
>> However, the variable that is shown in the log file comes up as: “
>> APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
>> specification also seems to be affected (my input level is “A1”
while
>> the grid_stat input level shows up as the generic netCDF input
level “
>> (*,*)”). Is there anything I could do to help fix this issue? I’ve
gone
>> ahead and attached a truncated log file with the error message as
well as
>> my METplus configuration file in case this is needed for
troubleshooting.
>> Any help would be greatly appreciated!
>>
>> Many thanks,
>>
>> -Brian
>> —————————————————————
>> Brian Matilla
>> Research Fellow— Warn-on-Forecast Team
>> Cooperative Institute for Mesoscale Meteorological Studies — The
>> University of Oklahoma
>> NOAA National Severe Storms Laboratory
>>
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: METplus - Grid Stat variable error
From: Brian Matilla - NOAA Affiliate
Time: Fri Jun 19 11:56:56 2020
Hi George,
Thank you very much for the detailed response. I had been looking
through the wrappers to understand them more and maybe see if I was
specifying the variable name incorrectly but that helps to clear
things up. I was able to work around the issue with the commands you
specified so that’s good. So if I understand right, by having the
[FCST/OBS/_VAR<n>_* variables in the configuration file, does it in a
way supersede any of the custom config variables for each of the
processes in the PROCESS_LIST?
On a slightly separate note, I wanted to propose an addition to the
pcp_combine wrapper. I noticed that when the pcp_combine command is
specified as “USER_DEFINED” and “PCP_COMBINE_SKIP_IF_OUTPUT_EXISTS =
TRUE”, METplus will still execute pcp_combine as if the output did not
exist. I added a check in the wrapper which would acknowledge that if
the expected output file does exist, then it will also skip processing
that forecast lead time. I could send you a piece of the code that I
added that takes care of the logic with existing pcp_combine output
for user defined processes.
Regarding the dataset to send, let me get in touch with my team lead
to see if we can compile a small subset of data to send over. If
possible, please keep this ticket open for a bit longer.
Thanks again!
-Brian
—————————————————————
Brian Matilla
Research Fellow— Warn-on-Forecast Team
Cooperative Institute for Mesoscale Meteorological Studies — The
University of Oklahoma
NOAA National Severe Storms Laboratory
> On Jun 17, 2020, at 12:32 PM, George McCabe via RT
<met_help at ucar.edu> wrote:
>
> Hi Brian,
>
> Would you be willing to provide a subset of data used to run this
use case?
> Enough data to run one initialization time and one ensemble would be
> sufficient. This would make it easier to test the code changes. We
could
> also add the use case to the METplus repository so there is an
example of a
> successful run with this configuration, since we currently don't
have any
> use cases that run PCPCombine > EnsembleStat > GridStat.
>
> Thanks,
> George
>
> On Wed, Jun 17, 2020 at 11:27 AM George McCabe <mccabe at ucar.edu
<mailto:mccabe at ucar.edu>> wrote:
>
>> Hi Brian,
>>
>> This is unfortunately due to problematic logic used for the
GridStat
>> wrapper. If [FCST/OBS]_PCP_COMBINE_RUN is set to True in the conf
file, the
>> wrapper automatically sets the field name to <name>_<level> where
<level>
>> is the value passed in with the letter removed, then sets the level
to
>> (*,*). This is the format of the data generated by PCPCombine. The
logic
>> assumes that GridStat is run on these data.
>>
>> I have wanted to remove this bad logic, but it has been left in to
support
>> legacy configurations. A long time ago, the [FCST/OBS]_VAR<n>_*
variables
>> controlled settings for PCPCombine, RegridDataPlane, and GridStat,
but we
>> have since added config variables to control these settings
individually
>> per tool. It sounds like it is time to resolve this issue. I will
bring it
>> up at the next METplus meeting and discuss a good solution.
>>
>> In the meantime, you can get around this by running the PCPCombine
portion
>> first, then run it again calling EnsembleStat and GridStat with
>> FCST_PCP_COMBINE_RUN set to False. You should be able to control
this with
>> command line arguments to master_metplus.py:
>>
>> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
>> \
>> -c MP_WOFS_pcp.conf \
>> -c base_paths.conf \
>> -c PROCESS_LIST=PCPCombine
>>
>> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
>> \
>> -c MP_WOFS_pcp.conf \
>> -c base_paths.conf \
>> -c PROCESS_LIST=EnsembleStat
>>
>> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
>> \
>> -c MP_WOFS_pcp.conf \
>> -c base_paths.conf \
>> -c config.PROCESS_LIST=GridStat \
>> -c config.FCST_PCP_COMBINE_RUN=False
>>
>>
>>
>> Another note, you SHOULD be able to define -c
>> PROCESS_LIST="EnsembleStat,GridStat"to run both, but it looks like
that is
>> not currently supported due to how the config arguments are
processed. I'll
>> also make a ticket to enhance the logic to support that.
>>
>> Let me know if that doesn't work for you and I can take a closer
look.
>> Also, FYI, it is often useful to set [config] LOG_LEVEL = DEBUG.
You get a
>> lot more information in the logs that are useful when things are
not
>> working correctly.
>>
>> Thanks,
>> George
>>
>> On Wed, Jun 17, 2020 at 11:04 AM Brian Matilla - NOAA Affiliate via
RT <
>> met_help at ucar.edu <mailto:met_help at ucar.edu>> wrote:
>>
>>>
>>> Wed Jun 17 11:04:05 2020: Request 95640 was acted upon.
>>> Transaction: Ticket created by brian.matilla at noaa.gov
<mailto:brian.matilla at noaa.gov>
>>> Queue: met_help
>>> Subject: METplus - Grid Stat variable error
>>> Owner: Nobody
>>> Requestors: brian.matilla at noaa.gov
<mailto:brian.matilla at noaa.gov>
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640
<https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640> >
>>>
>>>
>>> Hi MET help desk,
>>>
>>> I’ve come across an issue with running grid_stat on METplus
v.3.0.2. I am
>>> verifying files created with ensemble_stat in METplus and passing
the
>>> ensemble NMEP variables through to grid_stat. However, upon
looking at the
>>> output log, an extra underscore is added to the variable which
causes
>>> grid_stat to fail. For example, I am passing in the following
variable in
>>> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
>>> However, the variable that is shown in the log file comes up as:
>>> “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
>>> specification also seems to be affected (my input level is “A1”
while the
>>> grid_stat input level shows up as the generic netCDF input level
“(*,*)”).
>>> Is there anything I could do to help fix this issue? I’ve gone
ahead and
>>> attached a truncated log file with the error message as well as my
METplus
>>> configuration file in case this is needed for troubleshooting. Any
help
>>> would be greatly appreciated!
>>>
>>> Many thanks,
>>>
>>> -Brian
>>> —————————————————————
>>> Brian Matilla
>>> Research Fellow— Warn-on-Forecast Team
>>> Cooperative Institute for Mesoscale Meteorological Studies — The
>>> University of Oklahoma
>>> NOAA National Severe Storms Laboratory
>>>
>>> Hi MET help desk,
>>>
>>> I’ve come across an issue with running grid_stat on METplus
v.3.0.2. I am
>>> verifying files created with ensemble_stat in METplus and passing
the
>>> ensemble NMEP variables through to grid_stat. However, upon
looking at the
>>> output log, an extra underscore is added to the variable which
causes
>>> grid_stat to fail. For example, I am passing in the following
variable in
>>> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
>>> However, the variable that is shown in the log file comes up as: “
>>> APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
>>> specification also seems to be affected (my input level is “A1”
while
>>> the grid_stat input level shows up as the generic netCDF input
level “
>>> (*,*)”). Is there anything I could do to help fix this issue? I’ve
gone
>>> ahead and attached a truncated log file with the error message as
well as
>>> my METplus configuration file in case this is needed for
troubleshooting.
>>> Any help would be greatly appreciated!
>>>
>>> Many thanks,
>>>
>>> -Brian
>>> —————————————————————
>>> Brian Matilla
>>> Research Fellow— Warn-on-Forecast Team
>>> Cooperative Institute for Mesoscale Meteorological Studies — The
>>> University of Oklahoma
>>> NOAA National Severe Storms Laboratory
>>>
>>
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
------------------------------------------------
Subject: METplus - Grid Stat variable error
From: George McCabe
Time: Mon Jun 22 08:56:38 2020
Hi Brian,
In general the wrapper-specific variables take precedence over the
[FCST/OBS]_VAR<n> variables, but in the comparison wrappers (GridStat,
EnsembleStat, PointStat, etc.) the name/level passed into the MET
config
file is derived differently when PCPCombine is run on that data set.
The
wrapper-specific variables didn't exist when the use cases were first
created. The "correct" way to configure this case should be to set the
configurations explicitly for each tool separately and chain them
together
(output of tool 1 is input to tool 2), so you were looking at it the
correct way. However, support of this legacy behavior contains hidden
behavior to change the values used based on if another wrapper was
used or
not. I am trying to get away from logic like this because it is
confusing
to users.
For the upcoming 3.1 release (aimed for the end of this month), I am
going
to add a config variable to skip this secret logic and instead use
what the
user provides, which will allow you to run your use case without
breaking
it up into separate runs. For the long term solution, I am hoping to
remove
the logic that changes the name/level if PCPCombine is run and force
the
user to explicitly define the settings for each tool. This will likely
break user's configurations if they based it on the old example, so it
will
require a lengthy explanation of how to update the configs in the
user's
guide to avoid confusion/frustration.
Regarding user-defined PCPCombine runs not using the "skip if output
exists" logic, thank you for bringing that to my attention. I will
look
into that as well.
Thanks,
George
On Fri, Jun 19, 2020 at 11:57 AM Brian Matilla - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640 >
>
> Hi George,
>
> Thank you very much for the detailed response. I had been looking
through
> the wrappers to understand them more and maybe see if I was
specifying the
> variable name incorrectly but that helps to clear things up. I was
able to
> work around the issue with the commands you specified so that’s
good. So if
> I understand right, by having the [FCST/OBS/_VAR<n>_* variables in
the
> configuration file, does it in a way supersede any of the custom
config
> variables for each of the processes in the PROCESS_LIST?
>
> On a slightly separate note, I wanted to propose an addition to the
> pcp_combine wrapper. I noticed that when the pcp_combine command is
> specified as “USER_DEFINED” and “PCP_COMBINE_SKIP_IF_OUTPUT_EXISTS =
TRUE”,
> METplus will still execute pcp_combine as if the output did not
exist. I
> added a check in the wrapper which would acknowledge that if the
expected
> output file does exist, then it will also skip processing that
forecast
> lead time. I could send you a piece of the code that I added that
takes
> care of the logic with existing pcp_combine output for user defined
> processes.
>
> Regarding the dataset to send, let me get in touch with my team lead
to
> see if we can compile a small subset of data to send over. If
possible,
> please keep this ticket open for a bit longer.
>
> Thanks again!
>
> -Brian
> —————————————————————
> Brian Matilla
> Research Fellow— Warn-on-Forecast Team
> Cooperative Institute for Mesoscale Meteorological Studies — The
> University of Oklahoma
> NOAA National Severe Storms Laboratory
>
> > On Jun 17, 2020, at 12:32 PM, George McCabe via RT
<met_help at ucar.edu>
> wrote:
> >
> > Hi Brian,
> >
> > Would you be willing to provide a subset of data used to run this
use
> case?
> > Enough data to run one initialization time and one ensemble would
be
> > sufficient. This would make it easier to test the code changes. We
could
> > also add the use case to the METplus repository so there is an
example
> of a
> > successful run with this configuration, since we currently don't
have any
> > use cases that run PCPCombine > EnsembleStat > GridStat.
> >
> > Thanks,
> > George
> >
> > On Wed, Jun 17, 2020 at 11:27 AM George McCabe <mccabe at ucar.edu
<mailto:
> mccabe at ucar.edu>> wrote:
> >
> >> Hi Brian,
> >>
> >> This is unfortunately due to problematic logic used for the
GridStat
> >> wrapper. If [FCST/OBS]_PCP_COMBINE_RUN is set to True in the conf
file,
> the
> >> wrapper automatically sets the field name to <name>_<level> where
> <level>
> >> is the value passed in with the letter removed, then sets the
level to
> >> (*,*). This is the format of the data generated by PCPCombine.
The logic
> >> assumes that GridStat is run on these data.
> >>
> >> I have wanted to remove this bad logic, but it has been left in
to
> support
> >> legacy configurations. A long time ago, the [FCST/OBS]_VAR<n>_*
> variables
> >> controlled settings for PCPCombine, RegridDataPlane, and
GridStat, but
> we
> >> have since added config variables to control these settings
individually
> >> per tool. It sounds like it is time to resolve this issue. I will
bring
> it
> >> up at the next METplus meeting and discuss a good solution.
> >>
> >> In the meantime, you can get around this by running the
PCPCombine
> portion
> >> first, then run it again calling EnsembleStat and GridStat with
> >> FCST_PCP_COMBINE_RUN set to False. You should be able to control
this
> with
> >> command line arguments to master_metplus.py:
> >>
> >>
> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
> >> \
> >> -c MP_WOFS_pcp.conf \
> >> -c base_paths.conf \
> >> -c PROCESS_LIST=PCPCombine
> >>
> >>
> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
> >> \
> >> -c MP_WOFS_pcp.conf \
> >> -c base_paths.conf \
> >> -c PROCESS_LIST=EnsembleStat
> >>
> >>
> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
> >> \
> >> -c MP_WOFS_pcp.conf \
> >> -c base_paths.conf \
> >> -c config.PROCESS_LIST=GridStat \
> >> -c config.FCST_PCP_COMBINE_RUN=False
> >>
> >>
> >>
> >> Another note, you SHOULD be able to define -c
> >> PROCESS_LIST="EnsembleStat,GridStat"to run both, but it looks
like that
> is
> >> not currently supported due to how the config arguments are
processed.
> I'll
> >> also make a ticket to enhance the logic to support that.
> >>
> >> Let me know if that doesn't work for you and I can take a closer
look.
> >> Also, FYI, it is often useful to set [config] LOG_LEVEL = DEBUG.
You
> get a
> >> lot more information in the logs that are useful when things are
not
> >> working correctly.
> >>
> >> Thanks,
> >> George
> >>
> >> On Wed, Jun 17, 2020 at 11:04 AM Brian Matilla - NOAA Affiliate
via RT <
> >> met_help at ucar.edu <mailto:met_help at ucar.edu>> wrote:
> >>
> >>>
> >>> Wed Jun 17 11:04:05 2020: Request 95640 was acted upon.
> >>> Transaction: Ticket created by brian.matilla at noaa.gov <mailto:
> brian.matilla at noaa.gov>
> >>> Queue: met_help
> >>> Subject: METplus - Grid Stat variable error
> >>> Owner: Nobody
> >>> Requestors: brian.matilla at noaa.gov
<mailto:brian.matilla at noaa.gov>
> >>> Status: new
> >>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640 <
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640> >
> >>>
> >>>
> >>> Hi MET help desk,
> >>>
> >>> I’ve come across an issue with running grid_stat on METplus
v.3.0.2. I
> am
> >>> verifying files created with ensemble_stat in METplus and
passing the
> >>> ensemble NMEP variables through to grid_stat. However, upon
looking at
> the
> >>> output log, an extra underscore is added to the variable which
causes
> >>> grid_stat to fail. For example, I am passing in the following
variable
> in
> >>> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
> >>> However, the variable that is shown in the log file comes up as:
> >>> “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
> >>> specification also seems to be affected (my input level is “A1”
while
> the
> >>> grid_stat input level shows up as the generic netCDF input level
> “(*,*)”).
> >>> Is there anything I could do to help fix this issue? I’ve gone
ahead
> and
> >>> attached a truncated log file with the error message as well as
my
> METplus
> >>> configuration file in case this is needed for troubleshooting.
Any help
> >>> would be greatly appreciated!
> >>>
> >>> Many thanks,
> >>>
> >>> -Brian
> >>> —————————————————————
> >>> Brian Matilla
> >>> Research Fellow— Warn-on-Forecast Team
> >>> Cooperative Institute for Mesoscale Meteorological Studies — The
> >>> University of Oklahoma
> >>> NOAA National Severe Storms Laboratory
> >>>
> >>> Hi MET help desk,
> >>>
> >>> I’ve come across an issue with running grid_stat on METplus
v.3.0.2. I
> am
> >>> verifying files created with ensemble_stat in METplus and
passing the
> >>> ensemble NMEP variables through to grid_stat. However, upon
looking at
> the
> >>> output log, an extra underscore is added to the variable which
causes
> >>> grid_stat to fail. For example, I am passing in the following
variable
> in
> >>> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
> >>> However, the variable that is shown in the log file comes up as:
“
> >>> APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
> >>> specification also seems to be affected (my input level is “A1”
while
> >>> the grid_stat input level shows up as the generic netCDF input
level “
> >>> (*,*)”). Is there anything I could do to help fix this issue?
I’ve gone
> >>> ahead and attached a truncated log file with the error message
as well
> as
> >>> my METplus configuration file in case this is needed for
> troubleshooting.
> >>> Any help would be greatly appreciated!
> >>>
> >>> Many thanks,
> >>>
> >>> -Brian
> >>> —————————————————————
> >>> Brian Matilla
> >>> Research Fellow— Warn-on-Forecast Team
> >>> Cooperative Institute for Mesoscale Meteorological Studies — The
> >>> University of Oklahoma
> >>> NOAA National Severe Storms Laboratory
> >>>
> >>
> >>
> >> --
> >> George McCabe - Software Engineer III
> >> National Center for Atmospheric Research
> >> Research Applications Laboratory
> >> 303-497-2768
> >> ---
> >> My working day may not be your working day. Please do not feel
obliged
> to
> >> reply to this email outside of your normal working hours.
> >>
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: METplus - Grid Stat variable error
From: Brian Matilla - NOAA Affiliate
Time: Wed Jun 24 14:04:36 2020
Hi George,
Ok, I see it now. Thanks a bunch for helping to clear that up. For
now, the workaround suggestions are successful for accomplishing what
I’m trying to do and I’ve written up a quick script that addresses
each process individually.
Thanks,
-Brian
—————————————————————
Brian Matilla
Research Fellow— Warn-on-Forecast Team
Cooperative Institute for Mesoscale Meteorological Studies — The
University of Oklahoma
NOAA National Severe Storms Laboratory
> On Jun 22, 2020, at 9:56 AM, George McCabe via RT
<met_help at ucar.edu> wrote:
>
> Hi Brian,
>
> In general the wrapper-specific variables take precedence over the
> [FCST/OBS]_VAR<n> variables, but in the comparison wrappers
(GridStat,
> EnsembleStat, PointStat, etc.) the name/level passed into the MET
config
> file is derived differently when PCPCombine is run on that data set.
The
> wrapper-specific variables didn't exist when the use cases were
first
> created. The "correct" way to configure this case should be to set
the
> configurations explicitly for each tool separately and chain them
together
> (output of tool 1 is input to tool 2), so you were looking at it the
> correct way. However, support of this legacy behavior contains
hidden
> behavior to change the values used based on if another wrapper was
used or
> not. I am trying to get away from logic like this because it is
confusing
> to users.
>
> For the upcoming 3.1 release (aimed for the end of this month), I am
going
> to add a config variable to skip this secret logic and instead use
what the
> user provides, which will allow you to run your use case without
breaking
> it up into separate runs. For the long term solution, I am hoping to
remove
> the logic that changes the name/level if PCPCombine is run and force
the
> user to explicitly define the settings for each tool. This will
likely
> break user's configurations if they based it on the old example, so
it will
> require a lengthy explanation of how to update the configs in the
user's
> guide to avoid confusion/frustration.
>
> Regarding user-defined PCPCombine runs not using the "skip if output
> exists" logic, thank you for bringing that to my attention. I will
look
> into that as well.
>
> Thanks,
> George
>
> On Fri, Jun 19, 2020 at 11:57 AM Brian Matilla - NOAA Affiliate via
RT <
> met_help at ucar.edu <mailto:met_help at ucar.edu>> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640
<https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640> >
>>
>> Hi George,
>>
>> Thank you very much for the detailed response. I had been looking
through
>> the wrappers to understand them more and maybe see if I was
specifying the
>> variable name incorrectly but that helps to clear things up. I was
able to
>> work around the issue with the commands you specified so that’s
good. So if
>> I understand right, by having the [FCST/OBS/_VAR<n>_* variables in
the
>> configuration file, does it in a way supersede any of the custom
config
>> variables for each of the processes in the PROCESS_LIST?
>>
>> On a slightly separate note, I wanted to propose an addition to the
>> pcp_combine wrapper. I noticed that when the pcp_combine command is
>> specified as “USER_DEFINED” and “PCP_COMBINE_SKIP_IF_OUTPUT_EXISTS
= TRUE”,
>> METplus will still execute pcp_combine as if the output did not
exist. I
>> added a check in the wrapper which would acknowledge that if the
expected
>> output file does exist, then it will also skip processing that
forecast
>> lead time. I could send you a piece of the code that I added that
takes
>> care of the logic with existing pcp_combine output for user defined
>> processes.
>>
>> Regarding the dataset to send, let me get in touch with my team
lead to
>> see if we can compile a small subset of data to send over. If
possible,
>> please keep this ticket open for a bit longer.
>>
>> Thanks again!
>>
>> -Brian
>> —————————————————————
>> Brian Matilla
>> Research Fellow— Warn-on-Forecast Team
>> Cooperative Institute for Mesoscale Meteorological Studies — The
>> University of Oklahoma
>> NOAA National Severe Storms Laboratory
>>
>>> On Jun 17, 2020, at 12:32 PM, George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>>
>>> Hi Brian,
>>>
>>> Would you be willing to provide a subset of data used to run this
use
>> case?
>>> Enough data to run one initialization time and one ensemble would
be
>>> sufficient. This would make it easier to test the code changes. We
could
>>> also add the use case to the METplus repository so there is an
example
>> of a
>>> successful run with this configuration, since we currently don't
have any
>>> use cases that run PCPCombine > EnsembleStat > GridStat.
>>>
>>> Thanks,
>>> George
>>>
>>> On Wed, Jun 17, 2020 at 11:27 AM George McCabe <mccabe at ucar.edu
<mailto:mccabe at ucar.edu> <mailto:
>> mccabe at ucar.edu <mailto:mccabe at ucar.edu>>> wrote:
>>>
>>>> Hi Brian,
>>>>
>>>> This is unfortunately due to problematic logic used for the
GridStat
>>>> wrapper. If [FCST/OBS]_PCP_COMBINE_RUN is set to True in the conf
file,
>> the
>>>> wrapper automatically sets the field name to <name>_<level> where
>> <level>
>>>> is the value passed in with the letter removed, then sets the
level to
>>>> (*,*). This is the format of the data generated by PCPCombine.
The logic
>>>> assumes that GridStat is run on these data.
>>>>
>>>> I have wanted to remove this bad logic, but it has been left in
to
>> support
>>>> legacy configurations. A long time ago, the [FCST/OBS]_VAR<n>_*
>> variables
>>>> controlled settings for PCPCombine, RegridDataPlane, and
GridStat, but
>> we
>>>> have since added config variables to control these settings
individually
>>>> per tool. It sounds like it is time to resolve this issue. I will
bring
>> it
>>>> up at the next METplus meeting and discuss a good solution.
>>>>
>>>> In the meantime, you can get around this by running the
PCPCombine
>> portion
>>>> first, then run it again calling EnsembleStat and GridStat with
>>>> FCST_PCP_COMBINE_RUN set to False. You should be able to control
this
>> with
>>>> command line arguments to master_metplus.py:
>>>>
>>>>
>> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
>>>> \
>>>> -c MP_WOFS_pcp.conf \
>>>> -c base_paths.conf \
>>>> -c PROCESS_LIST=PCPCombine
>>>>
>>>>
>> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
>>>> \
>>>> -c MP_WOFS_pcp.conf \
>>>> -c base_paths.conf \
>>>> -c PROCESS_LIST=EnsembleStat
>>>>
>>>>
>> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
3.0.2//ush/master_metplus.py
>>>> \
>>>> -c MP_WOFS_pcp.conf \
>>>> -c base_paths.conf \
>>>> -c config.PROCESS_LIST=GridStat \
>>>> -c config.FCST_PCP_COMBINE_RUN=False
>>>>
>>>>
>>>>
>>>> Another note, you SHOULD be able to define -c
>>>> PROCESS_LIST="EnsembleStat,GridStat"to run both, but it looks
like that
>> is
>>>> not currently supported due to how the config arguments are
processed.
>> I'll
>>>> also make a ticket to enhance the logic to support that.
>>>>
>>>> Let me know if that doesn't work for you and I can take a closer
look.
>>>> Also, FYI, it is often useful to set [config] LOG_LEVEL = DEBUG.
You
>> get a
>>>> lot more information in the logs that are useful when things are
not
>>>> working correctly.
>>>>
>>>> Thanks,
>>>> George
>>>>
>>>> On Wed, Jun 17, 2020 at 11:04 AM Brian Matilla - NOAA Affiliate
via RT <
>>>> met_help at ucar.edu <mailto:met_help at ucar.edu>
<mailto:met_help at ucar.edu <mailto:met_help at ucar.edu>>> wrote:
>>>>
>>>>>
>>>>> Wed Jun 17 11:04:05 2020: Request 95640 was acted upon.
>>>>> Transaction: Ticket created by brian.matilla at noaa.gov
<mailto:brian.matilla at noaa.gov> <mailto:
>> brian.matilla at noaa.gov <mailto:brian.matilla at noaa.gov>>
>>>>> Queue: met_help
>>>>> Subject: METplus - Grid Stat variable error
>>>>> Owner: Nobody
>>>>> Requestors: brian.matilla at noaa.gov
<mailto:brian.matilla at noaa.gov> <mailto:brian.matilla at noaa.gov
<mailto:brian.matilla at noaa.gov>>
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640
<https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640> <
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640> >
>>>>>
>>>>>
>>>>> Hi MET help desk,
>>>>>
>>>>> I’ve come across an issue with running grid_stat on METplus
v.3.0.2. I
>> am
>>>>> verifying files created with ensemble_stat in METplus and
passing the
>>>>> ensemble NMEP variables through to grid_stat. However, upon
looking at
>> the
>>>>> output log, an extra underscore is added to the variable which
causes
>>>>> grid_stat to fail. For example, I am passing in the following
variable
>> in
>>>>> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
>>>>> However, the variable that is shown in the log file comes up as:
>>>>> “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
>>>>> specification also seems to be affected (my input level is “A1”
while
>> the
>>>>> grid_stat input level shows up as the generic netCDF input level
>> “(*,*)”).
>>>>> Is there anything I could do to help fix this issue? I’ve gone
ahead
>> and
>>>>> attached a truncated log file with the error message as well as
my
>> METplus
>>>>> configuration file in case this is needed for troubleshooting.
Any help
>>>>> would be greatly appreciated!
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> -Brian
>>>>> —————————————————————
>>>>> Brian Matilla
>>>>> Research Fellow— Warn-on-Forecast Team
>>>>> Cooperative Institute for Mesoscale Meteorological Studies — The
>>>>> University of Oklahoma
>>>>> NOAA National Severe Storms Laboratory
>>>>>
>>>>> Hi MET help desk,
>>>>>
>>>>> I’ve come across an issue with running grid_stat on METplus
v.3.0.2. I
>> am
>>>>> verifying files created with ensemble_stat in METplus and
passing the
>>>>> ensemble NMEP variables through to grid_stat. However, upon
looking at
>> the
>>>>> output log, an extra underscore is added to the variable which
causes
>>>>> grid_stat to fail. For example, I am passing in the following
variable
>> in
>>>>> my METplus config file:
“APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
>>>>> However, the variable that is shown in the log file comes up as:
“
>>>>> APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
>>>>> specification also seems to be affected (my input level is “A1”
while
>>>>> the grid_stat input level shows up as the generic netCDF input
level “
>>>>> (*,*)”). Is there anything I could do to help fix this issue?
I’ve gone
>>>>> ahead and attached a truncated log file with the error message
as well
>> as
>>>>> my METplus configuration file in case this is needed for
>> troubleshooting.
>>>>> Any help would be greatly appreciated!
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> -Brian
>>>>> —————————————————————
>>>>> Brian Matilla
>>>>> Research Fellow— Warn-on-Forecast Team
>>>>> Cooperative Institute for Mesoscale Meteorological Studies — The
>>>>> University of Oklahoma
>>>>> NOAA National Severe Storms Laboratory
>>>>>
>>>>
>>>>
>>>> --
>>>> George McCabe - Software Engineer III
>>>> National Center for Atmospheric Research
>>>> Research Applications Laboratory
>>>> 303-497-2768
>>>> ---
>>>> My working day may not be your working day. Please do not feel
obliged
>> to
>>>> reply to this email outside of your normal working hours.
>>>>
>>>
>>>
>>> --
>>> George McCabe - Software Engineer III
>>> National Center for Atmospheric Research
>>> Research Applications Laboratory
>>> 303-497-2768
>>> ---
>>> My working day may not be your working day. Please do not feel
obliged to
>>> reply to this email outside of your normal working hours.
>>
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
------------------------------------------------
Subject: METplus - Grid Stat variable error
From: George McCabe
Time: Wed Aug 05 10:49:02 2020
Hi Brian,
Just following up on this issue. We are aiming to release METplus 3.1
by next week, so these changes will be available soon.
It wouldn't make it into this release, but if you are still interested
in providing some sample data and config files for your use case we
could include it in a future release.
I will close this ticket, but feel free to open a new one if you have
any other questions.
Thanks,
George
On Wed Jun 24 14:04:36 2020, brian.matilla at noaa.gov wrote:
> Hi George,
>
> Ok, I see it now. Thanks a bunch for helping to clear that up. For
> now, the workaround suggestions are successful for accomplishing
what
> I’m trying to do and I’ve written up a quick script that addresses
> each process individually.
>
> Thanks,
>
> -Brian
> —————————————————————
> Brian Matilla
> Research Fellow— Warn-on-Forecast Team
> Cooperative Institute for Mesoscale Meteorological Studies — The
> University of Oklahoma
> NOAA National Severe Storms Laboratory
>
> > On Jun 22, 2020, at 9:56 AM, George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > Hi Brian,
> >
> > In general the wrapper-specific variables take precedence over the
> > [FCST/OBS]_VAR<n> variables, but in the comparison wrappers
> > (GridStat,
> > EnsembleStat, PointStat, etc.) the name/level passed into the MET
> > config
> > file is derived differently when PCPCombine is run on that data
set.
> > The
> > wrapper-specific variables didn't exist when the use cases were
first
> > created. The "correct" way to configure this case should be to set
> > the
> > configurations explicitly for each tool separately and chain them
> > together
> > (output of tool 1 is input to tool 2), so you were looking at it
the
> > correct way. However, support of this legacy behavior contains
hidden
> > behavior to change the values used based on if another wrapper was
> > used or
> > not. I am trying to get away from logic like this because it is
> > confusing
> > to users.
> >
> > For the upcoming 3.1 release (aimed for the end of this month), I
am
> > going
> > to add a config variable to skip this secret logic and instead use
> > what the
> > user provides, which will allow you to run your use case without
> > breaking
> > it up into separate runs. For the long term solution, I am hoping
to
> > remove
> > the logic that changes the name/level if PCPCombine is run and
force
> > the
> > user to explicitly define the settings for each tool. This will
> > likely
> > break user's configurations if they based it on the old example,
so
> > it will
> > require a lengthy explanation of how to update the configs in the
> > user's
> > guide to avoid confusion/frustration.
> >
> > Regarding user-defined PCPCombine runs not using the "skip if
output
> > exists" logic, thank you for bringing that to my attention. I will
> > look
> > into that as well.
> >
> > Thanks,
> > George
> >
> > On Fri, Jun 19, 2020 at 11:57 AM Brian Matilla - NOAA Affiliate
via
> > RT <
> > met_help at ucar.edu <mailto:met_help at ucar.edu>> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640
> >> <https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640> >
> >>
> >> Hi George,
> >>
> >> Thank you very much for the detailed response. I had been looking
> >> through
> >> the wrappers to understand them more and maybe see if I was
> >> specifying the
> >> variable name incorrectly but that helps to clear things up. I
was
> >> able to
> >> work around the issue with the commands you specified so that’s
> >> good. So if
> >> I understand right, by having the [FCST/OBS/_VAR<n>_* variables
in
> >> the
> >> configuration file, does it in a way supersede any of the custom
> >> config
> >> variables for each of the processes in the PROCESS_LIST?
> >>
> >> On a slightly separate note, I wanted to propose an addition to
the
> >> pcp_combine wrapper. I noticed that when the pcp_combine command
is
> >> specified as “USER_DEFINED” and
“PCP_COMBINE_SKIP_IF_OUTPUT_EXISTS =
> >> TRUE”,
> >> METplus will still execute pcp_combine as if the output did not
> >> exist. I
> >> added a check in the wrapper which would acknowledge that if the
> >> expected
> >> output file does exist, then it will also skip processing that
> >> forecast
> >> lead time. I could send you a piece of the code that I added that
> >> takes
> >> care of the logic with existing pcp_combine output for user
defined
> >> processes.
> >>
> >> Regarding the dataset to send, let me get in touch with my team
lead
> >> to
> >> see if we can compile a small subset of data to send over. If
> >> possible,
> >> please keep this ticket open for a bit longer.
> >>
> >> Thanks again!
> >>
> >> -Brian
> >> —————————————————————
> >> Brian Matilla
> >> Research Fellow— Warn-on-Forecast Team
> >> Cooperative Institute for Mesoscale Meteorological Studies — The
> >> University of Oklahoma
> >> NOAA National Severe Storms Laboratory
> >>
> >>> On Jun 17, 2020, at 12:32 PM, George McCabe via RT
> >>> <met_help at ucar.edu>
> >> wrote:
> >>>
> >>> Hi Brian,
> >>>
> >>> Would you be willing to provide a subset of data used to run
this
> >>> use
> >> case?
> >>> Enough data to run one initialization time and one ensemble
would
> >>> be
> >>> sufficient. This would make it easier to test the code changes.
We
> >>> could
> >>> also add the use case to the METplus repository so there is an
> >>> example
> >> of a
> >>> successful run with this configuration, since we currently don't
> >>> have any
> >>> use cases that run PCPCombine > EnsembleStat > GridStat.
> >>>
> >>> Thanks,
> >>> George
> >>>
> >>> On Wed, Jun 17, 2020 at 11:27 AM George McCabe <mccabe at ucar.edu
> >>> <mailto:mccabe at ucar.edu> <mailto:
> >> mccabe at ucar.edu <mailto:mccabe at ucar.edu>>> wrote:
> >>>
> >>>> Hi Brian,
> >>>>
> >>>> This is unfortunately due to problematic logic used for the
> >>>> GridStat
> >>>> wrapper. If [FCST/OBS]_PCP_COMBINE_RUN is set to True in the
conf
> >>>> file,
> >> the
> >>>> wrapper automatically sets the field name to <name>_<level>
where
> >> <level>
> >>>> is the value passed in with the letter removed, then sets the
> >>>> level to
> >>>> (*,*). This is the format of the data generated by PCPCombine.
The
> >>>> logic
> >>>> assumes that GridStat is run on these data.
> >>>>
> >>>> I have wanted to remove this bad logic, but it has been left in
to
> >> support
> >>>> legacy configurations. A long time ago, the [FCST/OBS]_VAR<n>_*
> >> variables
> >>>> controlled settings for PCPCombine, RegridDataPlane, and
GridStat,
> >>>> but
> >> we
> >>>> have since added config variables to control these settings
> >>>> individually
> >>>> per tool. It sounds like it is time to resolve this issue. I
will
> >>>> bring
> >> it
> >>>> up at the next METplus meeting and discuss a good solution.
> >>>>
> >>>> In the meantime, you can get around this by running the
PCPCombine
> >> portion
> >>>> first, then run it again calling EnsembleStat and GridStat with
> >>>> FCST_PCP_COMBINE_RUN set to False. You should be able to
control
> >>>> this
> >> with
> >>>> command line arguments to master_metplus.py:
> >>>>
> >>>>
> >> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
> >> 3.0.2//ush/master_metplus.py
> >>>> \
> >>>> -c MP_WOFS_pcp.conf \
> >>>> -c base_paths.conf \
> >>>> -c PROCESS_LIST=PCPCombine
> >>>>
> >>>>
> >> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
> >> 3.0.2//ush/master_metplus.py
> >>>> \
> >>>> -c MP_WOFS_pcp.conf \
> >>>> -c base_paths.conf \
> >>>> -c PROCESS_LIST=EnsembleStat
> >>>>
> >>>>
> >> /home/brian.matilla/WoFS_2020/MET_scripts/METplus-
> >> 3.0.2//ush/master_metplus.py
> >>>> \
> >>>> -c MP_WOFS_pcp.conf \
> >>>> -c base_paths.conf \
> >>>> -c config.PROCESS_LIST=GridStat \
> >>>> -c config.FCST_PCP_COMBINE_RUN=False
> >>>>
> >>>>
> >>>>
> >>>> Another note, you SHOULD be able to define -c
> >>>> PROCESS_LIST="EnsembleStat,GridStat"to run both, but it looks
like
> >>>> that
> >> is
> >>>> not currently supported due to how the config arguments are
> >>>> processed.
> >> I'll
> >>>> also make a ticket to enhance the logic to support that.
> >>>>
> >>>> Let me know if that doesn't work for you and I can take a
closer
> >>>> look.
> >>>> Also, FYI, it is often useful to set [config] LOG_LEVEL =
DEBUG.
> >>>> You
> >> get a
> >>>> lot more information in the logs that are useful when things
are
> >>>> not
> >>>> working correctly.
> >>>>
> >>>> Thanks,
> >>>> George
> >>>>
> >>>> On Wed, Jun 17, 2020 at 11:04 AM Brian Matilla - NOAA Affiliate
> >>>> via RT <
> >>>> met_help at ucar.edu <mailto:met_help at ucar.edu>
> >>>> <mailto:met_help at ucar.edu <mailto:met_help at ucar.edu>>> wrote:
> >>>>
> >>>>>
> >>>>> Wed Jun 17 11:04:05 2020: Request 95640 was acted upon.
> >>>>> Transaction: Ticket created by brian.matilla at noaa.gov
> >>>>> <mailto:brian.matilla at noaa.gov> <mailto:
> >> brian.matilla at noaa.gov <mailto:brian.matilla at noaa.gov>>
> >>>>> Queue: met_help
> >>>>> Subject: METplus - Grid Stat variable error
> >>>>> Owner: Nobody
> >>>>> Requestors: brian.matilla at noaa.gov
> >>>>> <mailto:brian.matilla at noaa.gov> <mailto:brian.matilla at noaa.gov
> >>>>> <mailto:brian.matilla at noaa.gov>>
> >>>>> Status: new
> >>>>> Ticket <URL:
> >>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640
> >>>>> <https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640> <
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95640> >
> >>>>>
> >>>>>
> >>>>> Hi MET help desk,
> >>>>>
> >>>>> I’ve come across an issue with running grid_stat on METplus
> >>>>> v.3.0.2. I
> >> am
> >>>>> verifying files created with ensemble_stat in METplus and
passing
> >>>>> the
> >>>>> ensemble NMEP variables through to grid_stat. However, upon
> >>>>> looking at
> >> the
> >>>>> output log, an extra underscore is added to the variable which
> >>>>> causes
> >>>>> grid_stat to fail. For example, I am passing in the following
> >>>>> variable
> >> in
> >>>>> my METplus config file:
> >>>>> “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
> >>>>> However, the variable that is shown in the log file comes up
as:
> >>>>> “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
> >>>>> specification also seems to be affected (my input level is
“A1”
> >>>>> while
> >> the
> >>>>> grid_stat input level shows up as the generic netCDF input
level
> >> “(*,*)”).
> >>>>> Is there anything I could do to help fix this issue? I’ve gone
> >>>>> ahead
> >> and
> >>>>> attached a truncated log file with the error message as well
as
> >>>>> my
> >> METplus
> >>>>> configuration file in case this is needed for troubleshooting.
> >>>>> Any help
> >>>>> would be greatly appreciated!
> >>>>>
> >>>>> Many thanks,
> >>>>>
> >>>>> -Brian
> >>>>> —————————————————————
> >>>>> Brian Matilla
> >>>>> Research Fellow— Warn-on-Forecast Team
> >>>>> Cooperative Institute for Mesoscale Meteorological Studies —
The
> >>>>> University of Oklahoma
> >>>>> NOAA National Severe Storms Laboratory
> >>>>>
> >>>>> Hi MET help desk,
> >>>>>
> >>>>> I’ve come across an issue with running grid_stat on METplus
> >>>>> v.3.0.2. I
> >> am
> >>>>> verifying files created with ensemble_stat in METplus and
passing
> >>>>> the
> >>>>> ensemble NMEP variables through to grid_stat. However, upon
> >>>>> looking at
> >> the
> >>>>> output log, an extra underscore is added to the variable which
> >>>>> causes
> >>>>> grid_stat to fail. For example, I am passing in the following
> >>>>> variable
> >> in
> >>>>> my METplus config file:
> >>>>> “APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1”.
> >>>>> However, the variable that is shown in the log file comes up
as:
> >>>>> “
> >>>>> APCP_01_A1_ENS_NMEP_ge12.7_NBRHD1_NEAREST1_1(*,*)”. The level
> >>>>> specification also seems to be affected (my input level is
“A1”
> >>>>> while
> >>>>> the grid_stat input level shows up as the generic netCDF input
> >>>>> level “
> >>>>> (*,*)”). Is there anything I could do to help fix this issue?
> >>>>> I’ve gone
> >>>>> ahead and attached a truncated log file with the error message
as
> >>>>> well
> >> as
> >>>>> my METplus configuration file in case this is needed for
> >> troubleshooting.
> >>>>> Any help would be greatly appreciated!
> >>>>>
> >>>>> Many thanks,
> >>>>>
> >>>>> -Brian
> >>>>> —————————————————————
> >>>>> Brian Matilla
> >>>>> Research Fellow— Warn-on-Forecast Team
> >>>>> Cooperative Institute for Mesoscale Meteorological Studies —
The
> >>>>> University of Oklahoma
> >>>>> NOAA National Severe Storms Laboratory
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> George McCabe - Software Engineer III
> >>>> National Center for Atmospheric Research
> >>>> Research Applications Laboratory
> >>>> 303-497-2768
> >>>> ---
> >>>> My working day may not be your working day. Please do not feel
> >>>> obliged
> >> to
> >>>> reply to this email outside of your normal working hours.
> >>>>
> >>>
> >>>
> >>> --
> >>> George McCabe - Software Engineer III
> >>> National Center for Atmospheric Research
> >>> Research Applications Laboratory
> >>> 303-497-2768
> >>> ---
> >>> My working day may not be your working day. Please do not feel
> >>> obliged to
> >>> reply to this email outside of your normal working hours.
> >>
> >>
> >>
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
> > obliged to
> > reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list