[Met_help] [rt.rap.ucar.edu #100072] History for Does METplus support python embedding of climo fields?
George McCabe via RT
met_help at ucar.edu
Wed Jun 16 15:19:56 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi metplus help,
I have another quick question regarding MET-10.0.0 and METplus-4.0. After looking through the documentation, it looks like grid_stat does not support python embedding scripts for the climatology file? It seems to ignore setting “GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = PYTHON_NUMPY” and “CLIMO_MEAN_VAR1_NAME = my_clim2xr.py ….” similar to the behavior of FCST and OBS fields. Can you please confirm if metplus supports python embedding for climatology files, or I’m just not using the correct syntax. If not, this would be really useful. If METplus cannot automatically read level information in netcdf files (similar to level=P500 for grib files), python embedding becomes a really important alternate as some script needs to first search for the correct indices or more easily return a PYTHON_NUMPY or XARRAY array.
Thanks,
Scott
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Does METplus support python embedding of climo fields?
From: George McCabe
Time: Tue Jun 01 13:53:27 2021
Hi Scott,
Python embedding is supported in climo fields in the MET tools.
GRID_STAT_CLIMO_MEAN_FIELD is the variable you will want to use to
specify
the list of fields to pass into the climo_mean dictionary. This does
not
currently support the logic of specifying a name then applying the
current
level for a list of items to each item, so you will have to write out
each
item in the list.
I don't think that use of this variable has been tested using Python
embedding in this way, so it may need some slight modification to get
it to
format correctly. My guess is it will try to format each item as a
string
without adding the "{ name=" syntax around it. Adding that text
yourself
may be automatically surrounded by quotation marks, which would break
the
parsing.
Could you confirm that this is the case? If so, I could see the logic
modified to check if GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE =
PYTHON_NUMPY and
format the fields accordingly if so.
Initial testing of running METplus using Python embedding for climo
fields
utilized the *_MET_CONFIG_OVERRIDES variables (more information here:
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#overriding-
unsupported-met-configuration-variables)
to set the field list. Here is an example of how this was used in that
test:
GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = PYTHON_NUMPY
GRID_STAT_CLIMO_STDEV_INPUT_TEMPLATE = PYTHON_NUMPY
GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = {field = [{name=
"/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
/d1/projects/CPC_data/input/M
ET_precip_climos/precip_clim_mean_unsmoothed_07d.nc:{valid?fmt=%d%m}";},{name=
"/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
/d1/projects/CPC_data/input/MET_precip_climos/precip_clim_mean_unsmoothed_07d.nc:{valid?fmt=%d%m}";}];}
climo_stdev = {field = [{name=
"/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
/d1/projects/CPC_data/input/MET_precip_climos/precip_clim_std_unsmoothed_07d.nc:{valid?fmt=%d%m}";},{name=
"/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
/d1/projects/CPC_data/input/MET_precip_climos/precip_clim_std_unsmoothed_07d.nc:
{valid?fmt=%d%m}";}];}
The basic idea of the MET overrides variable is you can specify any
MET
config overrides that will be added to the end of the wrapped MET
config
file so it takes precedence over any values set previously. If
configuring
using GRID_STAT_CLIMO_MEAN_FIELD does not work here as I suspect, then
you
can get around this using the overrides variable for now. I can let
you
know when we have a version that you can test the changes.
Thanks,
George
On Tue, Jun 1, 2021 at 1:10 PM Rabenhorst, Scott D. (GSFC-
610.1)[SCIENCE
SYSTEMS AND APPLICATIONS INC] via RT <met_help at ucar.edu> wrote:
>
> Tue Jun 01 13:10:13 2021: Request 100072 was acted upon.
> Transaction: Ticket created by scott.d.rabenhorst at nasa.gov
> Queue: met_help
> Subject: Does METplus support python embedding of climo fields?
> Owner: Nobody
> Requestors: scott.d.rabenhorst at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100072 >
>
>
> Hi metplus help,
>
> I have another quick question regarding MET-10.0.0 and METplus-4.0.
After
> looking through the documentation, it looks like grid_stat does not
support
> python embedding scripts for the climatology file? It seems to
ignore
> setting “GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = PYTHON_NUMPY” and
> “CLIMO_MEAN_VAR1_NAME = my_clim2xr.py ….” similar to the behavior of
FCST
> and OBS fields. Can you please confirm if metplus supports python
embedding
> for climatology files, or I’m just not using the correct syntax. If
not,
> this would be really useful. If METplus cannot automatically read
level
> information in netcdf files (similar to level=P500 for grib files),
python
> embedding becomes a really important alternate as some script needs
to
> first search for the correct indices or more easily return a
PYTHON_NUMPY
> or XARRAY array.
>
> Thanks,
> Scott
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [EXTERNAL] Re: [rt.rap.ucar.edu #100072] Does METplus support python embedding of climo fields?
From: Rabenhorst, Scott D. (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]
Time: Wed Jun 02 09:11:37 2021
Hi George,
This is a very interesting approach and something I will test later
today/tomorrow. It is good to know that python embedding is supported
in the CLIMO fields too, albeit a bit different. Are there plans to
enable CLIMO to behave similar to FCST and OBS and loop over levels
etc? This would be really helpful since I imagine it would be
impossible to mix logic and loop over FCST and OBS but list out levels
for CLIMO? I'll have to rethink my scripting. Again, it appears as if
METplus interfaces much more easily with grib files, but
unfortunately, we do not use grib files (nor do we ever plan to do
so). Thus python embedding seems like the best approach for now.
Although, if met was able to read netcdf level information according
to CF-standards, this would mitigate some of the issues.
Thanks,
Scott
On 6/1/21, 3:53 PM, "George McCabe via RT" <met_help at ucar.edu> wrote:
Hi Scott,
Python embedding is supported in climo fields in the MET tools.
GRID_STAT_CLIMO_MEAN_FIELD is the variable you will want to use to
specify
the list of fields to pass into the climo_mean dictionary. This
does not
currently support the logic of specifying a name then applying the
current
level for a list of items to each item, so you will have to write
out each
item in the list.
I don't think that use of this variable has been tested using
Python
embedding in this way, so it may need some slight modification to
get it to
format correctly. My guess is it will try to format each item as a
string
without adding the "{ name=" syntax around it. Adding that text
yourself
may be automatically surrounded by quotation marks, which would
break the
parsing.
Could you confirm that this is the case? If so, I could see the
logic
modified to check if GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE =
PYTHON_NUMPY and
format the fields accordingly if so.
Initial testing of running METplus using Python embedding for
climo fields
utilized the *_MET_CONFIG_OVERRIDES variables (more information
here:
https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmetplus.readthedocs.io%2Fen%2Flatest%2FUsers_Guide%2Fsystemconfiguration.html%23overriding-
unsupported-met-configuration-
variables&data=04%7C01%7Cscott.d.rabenhorst%40nasa.gov%7C3408925c2469488d72a508d92536f0cb%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637581740165876780%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KOLi09Kl3rT6hgdz1ppm4fP5pYXYRbYfyLqGg3QT3Zw%3D&reserved=0)
to set the field list. Here is an example of how this was used in
that test:
GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = PYTHON_NUMPY
GRID_STAT_CLIMO_STDEV_INPUT_TEMPLATE = PYTHON_NUMPY
GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = {field = [{name=
"/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
/d1/projects/CPC_data/input/M
ET_precip_climos/precip_clim_mean_unsmoothed_07d.nc:{valid?fmt=%d%m}";},{name=
"/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
/d1/projects/CPC_data/input/MET_precip_climos/precip_clim_mean_unsmoothed_07d.nc:{valid?fmt=%d%m}";}];}
climo_stdev = {field = [{name=
"/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
/d1/projects/CPC_data/input/MET_precip_climos/precip_clim_std_unsmoothed_07d.nc:{valid?fmt=%d%m}";},{name=
"/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
/d1/projects/CPC_data/input/MET_precip_climos/precip_clim_std_unsmoothed_07d.nc:
{valid?fmt=%d%m}";}];}
The basic idea of the MET overrides variable is you can specify
any MET
config overrides that will be added to the end of the wrapped MET
config
file so it takes precedence over any values set previously. If
configuring
using GRID_STAT_CLIMO_MEAN_FIELD does not work here as I suspect,
then you
can get around this using the overrides variable for now. I can
let you
know when we have a version that you can test the changes.
Thanks,
George
On Tue, Jun 1, 2021 at 1:10 PM Rabenhorst, Scott D. (GSFC-
610.1)[SCIENCE
SYSTEMS AND APPLICATIONS INC] via RT <met_help at ucar.edu> wrote:
>
> Tue Jun 01 13:10:13 2021: Request 100072 was acted upon.
> Transaction: Ticket created by scott.d.rabenhorst at nasa.gov
> Queue: met_help
> Subject: Does METplus support python embedding of climo
fields?
> Owner: Nobody
> Requestors: scott.d.rabenhorst at nasa.gov
> Status: new
> Ticket <URL:
https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frt.rap.ucar.edu%2Frt%2FTicket%2FDisplay.html%3Fid%3D100072&data=04%7C01%7Cscott.d.rabenhorst%40nasa.gov%7C3408925c2469488d72a508d92536f0cb%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637581740165876780%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0Lds0qy36A0UDyMuHKy06zsoBZZFYBVffS7UmF6JyPQ%3D&reserved=0
>
>
>
> Hi metplus help,
>
> I have another quick question regarding MET-10.0.0 and METplus-
4.0. After
> looking through the documentation, it looks like grid_stat does
not support
> python embedding scripts for the climatology file? It seems to
ignore
> setting “GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = PYTHON_NUMPY” and
> “CLIMO_MEAN_VAR1_NAME = my_clim2xr.py ….” similar to the
behavior of FCST
> and OBS fields. Can you please confirm if metplus supports
python embedding
> for climatology files, or I’m just not using the correct syntax.
If not,
> this would be really useful. If METplus cannot automatically
read level
> information in netcdf files (similar to level=P500 for grib
files), python
> embedding becomes a really important alternate as some script
needs to
> first search for the correct indices or more easily return a
PYTHON_NUMPY
> or XARRAY array.
>
> Thanks,
> Scott
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Does METplus support python embedding of climo fields?
From: George McCabe
Time: Wed Jun 02 10:02:51 2021
Hi Scott,
I created GitHub issue for improving handling of climo info when using
Python embedding: https://github.com/dtcenter/METplus/issues/944
I agree that it would be good to implement similar logic to fcst/obs
to
reference the current level in the name value. Another engineer is
creating
a GitHub issue in the MET repository to implement logic to reference
values
in NetCDF dimensions to extract data.
I think these are both great enhancements to the current logic.
Funding
from our sponsors typically dictates the priority of our tasking, so I
am
unsure of the timeline to complete this work unless specific funding
to
complete this work is allocated. You can contact Tara Jensen, the
project
manager, at jensen at ucar.edu if you'd like to discuss further.
Thanks,
George
On Wed, Jun 2, 2021 at 9:11 AM Rabenhorst, Scott D. (GSFC-
610.1)[SCIENCE
SYSTEMS AND APPLICATIONS INC] via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100072 >
>
> Hi George,
>
> This is a very interesting approach and something I will test later
> today/tomorrow. It is good to know that python embedding is
supported in
> the CLIMO fields too, albeit a bit different. Are there plans to
enable
> CLIMO to behave similar to FCST and OBS and loop over levels etc?
This
> would be really helpful since I imagine it would be impossible to
mix logic
> and loop over FCST and OBS but list out levels for CLIMO? I'll have
to
> rethink my scripting. Again, it appears as if METplus interfaces
much more
> easily with grib files, but unfortunately, we do not use grib files
(nor do
> we ever plan to do so). Thus python embedding seems like the best
approach
> for now. Although, if met was able to read netcdf level information
> according to CF-standards, this would mitigate some of the issues.
>
> Thanks,
> Scott
>
>
>
> On 6/1/21, 3:53 PM, "George McCabe via RT" <met_help at ucar.edu>
wrote:
>
> Hi Scott,
>
> Python embedding is supported in climo fields in the MET tools.
> GRID_STAT_CLIMO_MEAN_FIELD is the variable you will want to use
to
> specify
> the list of fields to pass into the climo_mean dictionary. This
does
> not
> currently support the logic of specifying a name then applying
the
> current
> level for a list of items to each item, so you will have to
write out
> each
> item in the list.
>
> I don't think that use of this variable has been tested using
Python
> embedding in this way, so it may need some slight modification
to get
> it to
> format correctly. My guess is it will try to format each item as
a
> string
> without adding the "{ name=" syntax around it. Adding that text
> yourself
> may be automatically surrounded by quotation marks, which would
break
> the
> parsing.
>
> Could you confirm that this is the case? If so, I could see the
logic
> modified to check if GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE =
> PYTHON_NUMPY and
> format the fields accordingly if so.
>
> Initial testing of running METplus using Python embedding for
climo
> fields
> utilized the *_MET_CONFIG_OVERRIDES variables (more information
here:
>
>
https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmetplus.readthedocs.io%2Fen%2Flatest%2FUsers_Guide%2Fsystemconfiguration.html%23overriding-
unsupported-met-configuration-
variables&data=04%7C01%7Cscott.d.rabenhorst%40nasa.gov%7C3408925c2469488d72a508d92536f0cb%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637581740165876780%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KOLi09Kl3rT6hgdz1ppm4fP5pYXYRbYfyLqGg3QT3Zw%3D&reserved=0
> )
> to set the field list. Here is an example of how this was used
in that
> test:
>
>
> GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = PYTHON_NUMPY
> GRID_STAT_CLIMO_STDEV_INPUT_TEMPLATE = PYTHON_NUMPY
>
> GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = {field = [{name=
> "/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
> /d1/projects/CPC_data/input/M
> ET_precip_climos/precip_clim_mean_unsmoothed_07d.nc:
> {valid?fmt=%d%m}";},{name=
> "/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
>
>
/d1/projects/CPC_data/input/MET_precip_climos/precip_clim_mean_unsmoothed_07d.nc:
> {valid?fmt=%d%m}";}];}
> climo_stdev = {field = [{name=
> "/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
>
>
/d1/projects/CPC_data/input/MET_precip_climos/precip_clim_std_unsmoothed_07d.nc:
> {valid?fmt=%d%m}";},{name=
> "/d1/projects/CPC_data/scripts/precip_climo_mean_embedded.py
> /d1/projects/CPC_data/input/MET_precip_climos/
> precip_clim_std_unsmoothed_07d.nc:
> {valid?fmt=%d%m}";}];}
>
> The basic idea of the MET overrides variable is you can specify
any MET
> config overrides that will be added to the end of the wrapped
MET
> config
> file so it takes precedence over any values set previously. If
> configuring
> using GRID_STAT_CLIMO_MEAN_FIELD does not work here as I
suspect, then
> you
> can get around this using the overrides variable for now. I can
let you
> know when we have a version that you can test the changes.
>
> Thanks,
> George
>
> On Tue, Jun 1, 2021 at 1:10 PM Rabenhorst, Scott D.
> (GSFC-610.1)[SCIENCE
> SYSTEMS AND APPLICATIONS INC] via RT <met_help at ucar.edu> wrote:
>
> >
> > Tue Jun 01 13:10:13 2021: Request 100072 was acted upon.
> > Transaction: Ticket created by scott.d.rabenhorst at nasa.gov
> > Queue: met_help
> > Subject: Does METplus support python embedding of climo
fields?
> > Owner: Nobody
> > Requestors: scott.d.rabenhorst at nasa.gov
> > Status: new
> > Ticket <URL:
>
https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frt.rap.ucar.edu%2Frt%2FTicket%2FDisplay.html%3Fid%3D100072&data=04%7C01%7Cscott.d.rabenhorst%40nasa.gov%7C3408925c2469488d72a508d92536f0cb%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637581740165876780%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0Lds0qy36A0UDyMuHKy06zsoBZZFYBVffS7UmF6JyPQ%3D&reserved=0
> >
> >
> >
> > Hi metplus help,
> >
> > I have another quick question regarding MET-10.0.0 and
METplus-4.0.
> After
> > looking through the documentation, it looks like grid_stat
does not
> support
> > python embedding scripts for the climatology file? It seems to
ignore
> > setting “GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = PYTHON_NUMPY”
and
> > “CLIMO_MEAN_VAR1_NAME = my_clim2xr.py ….” similar to the
behavior of
> FCST
> > and OBS fields. Can you please confirm if metplus supports
python
> embedding
> > for climatology files, or I’m just not using the correct
syntax. If
> not,
> > this would be really useful. If METplus cannot automatically
read
> level
> > information in netcdf files (similar to level=P500 for grib
files),
> python
> > embedding becomes a really important alternate as some script
needs
> to
> > first search for the correct indices or more easily return a
> PYTHON_NUMPY
> > or XARRAY array.
> >
> > Thanks,
> > Scott
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged
> to
> reply to this email outside of your normal working hours.
>
>
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list