[Met_help] [rt.rap.ucar.edu #100321] History for Reading in ensemble netcdf
John Opatz via RT
met_help at ucar.edu
Wed Aug 4 12:00:38 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I have some S2S ensemble forecasts where 40 ensemble members are written into the same netcdf file (as realizations). I want to apply a variety of MET tools to these. I can't get this file read in with plot_data_plane so I suspect that I won't get any of the MET tools to read this in (note I haven't tried). I know enough to know that if plot_data_plane doesn't work I usually won't get grid_stat etc to work either. These files are constructed to make dealing with the forecasts easier... as the ensemble is a lagged one with the oldest members of the ensemble for any given lead time horizon being 10 days older than the newest ones... they are rebadged to make it look more homogeneous and easier to handle, i.e. all the members that are valid for a particular forecast horizon are put together in one file (here a day 18 forecast). I wouldn't know how to extract a given ensemble member from the ensemble in a config file either... any thoughts? Will MET handle this kind of file (i.!
e. all members in the same file together?) or do I have to break this up again?! I have to add that I have no experience of running ensemble_stat and know very little about how to get MET to ingest ensemble forecasts but all the examples suggest that ensemble members are in separate files (even separate directories!) and it looks like MET expects n files for n ensemble members, i.e. if I gave it 1 file with n members it won't work.
TIA
Marion
--
Dr Marion Mittermaier Verification & diagnostics expert scientist (Global Model Evaluation & Development)
Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
Tel: +44 (0) 330 135 1604
E-mail: marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk> Website: http://www.metoffice.gov.uk<http://www.metoffice.gov.uk/>
http://www.metoffice.gov.uk/research/people/marion-mittermaier
Associate Editor for 2021: Monthly Weather Review<https://www.ametsoc.org/index.cfm/ams/publications/journals/monthly-weather-review/>
My working day may not be your working day. Please do not feel obliged to reply to this email outside of your normal working hours.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Reading in ensemble netcdf
From: Minna Win
Time: Thu Jun 24 11:33:23 2021
Hi Marion,
I'm looking for someone who can assist you with this issue. John
Halley
Gotway is out of the office until July 6.
Regards,
Minna
---------------
Minna Win
Pronouns: she/her
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
---------------
On Thu, Jun 24, 2021 at 4:40 AM marion.mittermaier at metoffice.gov.uk
via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 24 04:33:12 2021: Request 100321 was acted upon.
> Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> Queue: met_help
> Subject: Reading in ensemble netcdf
> Owner: Nobody
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100321 >
>
>
> Hi,
>
> I have some S2S ensemble forecasts where 40 ensemble members are
written
> into the same netcdf file (as realizations). I want to apply a
variety of
> MET tools to these. I can't get this file read in with
plot_data_plane so
> I suspect that I won't get any of the MET tools to read this in
(note I
> haven't tried). I know enough to know that if plot_data_plane
doesn't work
> I usually won't get grid_stat etc to work either. These files are
> constructed to make dealing with the forecasts easier... as the
ensemble is
> a lagged one with the oldest members of the ensemble for any given
lead
> time horizon being 10 days older than the newest ones... they are
rebadged
> to make it look more homogeneous and easier to handle, i.e. all the
members
> that are valid for a particular forecast horizon are put together in
one
> file (here a day 18 forecast). I wouldn't know how to extract a
given
> ensemble member from the ensemble in a config file either... any
thoughts?
> Will MET handle this kind of file (i.!
> e. all members in the same file together?) or do I have to break
this up
> again?! I have to add that I have no experience of running
ensemble_stat
> and know very little about how to get MET to ingest ensemble
forecasts but
> all the examples suggest that ensemble members are in separate files
(even
> separate directories!) and it looks like MET expects n files for n
ensemble
> members, i.e. if I gave it 1 file with n members it won't work.
>
> TIA
> Marion
>
> --
> Dr Marion Mittermaier Verification & diagnostics expert
scientist
> (Global Model Evaluation & Development)
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0) 330 135 1604
> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> Website:
http://www.metoffice.gov.uk
> <http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> Associate Editor for 2021: Monthly Weather Review<
> https://www.ametsoc.org/index.cfm/ams/publications/journals/monthly-
weather-review/
> >
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
>
------------------------------------------------
Subject: Reading in ensemble netcdf
From: John Opatz
Time: Thu Jun 24 13:59:57 2021
Hi Marion,
Your suspicions are correct; as it currently stands, MET is unable to
pull
apart netCDF files to access the individual ensemble members outside
of
manually telling it, via:
field = [ { name = "precipitation_amount";
level = "(0,n,*,*)";
where n is the number of the ensemble member. We do plan on adjusting
this in the future, with a github issue
<https://github.com/dtcenter/MET/issues/1695> already in the queue for
the EnsembleStat reorganization.
I was able to access an individual (0th) member via plot_data_plane
using
plot_data_plane sub5226_20190909_d18.nc sub_test.ps
'name="precipitation_amount"; level="(0,0,*,*)";'
But as with EnsembleStat, that will only show you a single member.
In the meantime, there is a way to use Python embedding and METplus to
dissect your ensemble netCDF without having to fill up your disk space
with a netCDF for each member. Take a look at the EnsembleStat use
case for Air Quality and Composition here
<https://metplus.readthedocs.io/en/latest/generated/model_applications/air_quality_and_comp/EnsembleStat_fcstICAP_obsMODIS_aod.html#sphx-
glr-generated-model-applications-air-quality-and-comp-ensemblestat-
fcsticap-obsmodis-aod-py>.
This use case reads in a 7 member ensemble using a Python script and
listing the number of the ensemble members (0 to 6) as the input
template. Note that the script accesses this list of numbers by the
MET_PYTHON_INPUT_ARG command at the end of the ENS_VAR1_NAME and
FCST_VAR1_NAME variable input.
You may also be able to utilize the CUSTOM_LOOP_LIST METplus variable
to accomplish the same task. Take a look at the entry for it here
<https://metplus.readthedocs.io/en/latest/Users_Guide/glossary.html?highlight=custom_list#term-
custom-loop-list>.
I would think setting that equal to the 0-39 for your member count,
then doing something along the lines of
ENS_VAR1_LEVELS = "(0,{custom?fmt=%d},*,*)"
FCST_VAR1_LEVELS = "(0, {custom?fmt=%d},*,*)"
I'm not as certain about that solution, but since your ensemble
members can be read in directly via plot_data_plane, it's worth a
chance.
-John O.
On Thu, Jun 24, 2021 at 11:33 AM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100321 >
>
> Hi Marion,
>
> I'm looking for someone who can assist you with this issue. John
Halley
> Gotway is out of the office until July 6.
>
> Regards,
> Minna
> ---------------
> Minna Win
> Pronouns: she/her
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax: 303-497-8401
> ---------------
>
>
>
> On Thu, Jun 24, 2021 at 4:40 AM marion.mittermaier at metoffice.gov.uk
via
> RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 24 04:33:12 2021: Request 100321 was acted upon.
> > Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> > Queue: met_help
> > Subject: Reading in ensemble netcdf
> > Owner: Nobody
> > Requestors: marion.mittermaier at metoffice.gov.uk
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100321 >
> >
> >
> > Hi,
> >
> > I have some S2S ensemble forecasts where 40 ensemble members are
written
> > into the same netcdf file (as realizations). I want to apply a
variety of
> > MET tools to these. I can't get this file read in with
plot_data_plane
> so
> > I suspect that I won't get any of the MET tools to read this in
(note I
> > haven't tried). I know enough to know that if plot_data_plane
doesn't
> work
> > I usually won't get grid_stat etc to work either. These files are
> > constructed to make dealing with the forecasts easier... as the
ensemble
> is
> > a lagged one with the oldest members of the ensemble for any given
lead
> > time horizon being 10 days older than the newest ones... they are
> rebadged
> > to make it look more homogeneous and easier to handle, i.e. all
the
> members
> > that are valid for a particular forecast horizon are put together
in one
> > file (here a day 18 forecast). I wouldn't know how to extract a
given
> > ensemble member from the ensemble in a config file either... any
> thoughts?
> > Will MET handle this kind of file (i.!
> > e. all members in the same file together?) or do I have to break
this up
> > again?! I have to add that I have no experience of running
ensemble_stat
> > and know very little about how to get MET to ingest ensemble
forecasts
> but
> > all the examples suggest that ensemble members are in separate
files
> (even
> > separate directories!) and it looks like MET expects n files for n
> ensemble
> > members, i.e. if I gave it 1 file with n members it won't work.
> >
> > TIA
> > Marion
> >
> > --
> > Dr Marion Mittermaier Verification & diagnostics expert
scientist
> > (Global Model Evaluation & Development)
> >
> > Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> > Tel: +44 (0) 330 135 1604
> > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> > marion.mittermaier at metoffice.gov.uk> Website:
> http://www.metoffice.gov.uk
> > <http://www.metoffice.gov.uk/>
> >
> > http://www.metoffice.gov.uk/research/people/marion-mittermaier
> >
> > Associate Editor for 2021: Monthly Weather Review<
> >
> https://www.ametsoc.org/index.cfm/ams/publications/journals/monthly-
weather-review/
> > >
> >
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
> >
>
>
--
John Opatz
Associate Scientist III
NCAR RAL and DTC
Boulder, Colorado
+1 303-497-8305
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #100321] Reading in ensemble netcdf
From: marion.mittermaier at metoffice.gov.uk
Time: Thu Jul 22 08:36:41 2021
Thanks John O!.... I have only just got back to this so this should be
useful as a next step. I will have a look at the AQ example and the
python embedding.
Marion
-----Original Message-----
From: John Opatz via RT <met_help at ucar.edu>
Sent: 24 June 2021 21:00
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #100321] Reading in ensemble netcdf
This email was received from an external source. Always check sender
details, links & attachments.
Hi Marion,
Your suspicions are correct; as it currently stands, MET is unable to
pull apart netCDF files to access the individual ensemble members
outside of manually telling it, via:
field = [ { name = "precipitation_amount";
level = "(0,n,*,*)";
where n is the number of the ensemble member. We do plan on adjusting
this in the future, with a github issue
<https://github.com/dtcenter/MET/issues/1695> already in the queue for
the EnsembleStat reorganization.
I was able to access an individual (0th) member via plot_data_plane
using
plot_data_plane sub5226_20190909_d18.nc sub_test.ps
'name="precipitation_amount"; level="(0,0,*,*)";'
But as with EnsembleStat, that will only show you a single member.
In the meantime, there is a way to use Python embedding and METplus to
dissect your ensemble netCDF without having to fill up your disk space
with a netCDF for each member. Take a look at the EnsembleStat use
case for Air Quality and Composition here
<https://metplus.readthedocs.io/en/latest/generated/model_applications/air_quality_and_comp/EnsembleStat_fcstICAP_obsMODIS_aod.html#sphx-
glr-generated-model-applications-air-quality-and-comp-ensemblestat-
fcsticap-obsmodis-aod-py>.
This use case reads in a 7 member ensemble using a Python script and
listing the number of the ensemble members (0 to 6) as the input
template. Note that the script accesses this list of numbers by the
MET_PYTHON_INPUT_ARG command at the end of the ENS_VAR1_NAME and
FCST_VAR1_NAME variable input.
You may also be able to utilize the CUSTOM_LOOP_LIST METplus variable
to accomplish the same task. Take a look at the entry for it here
<https://metplus.readthedocs.io/en/latest/Users_Guide/glossary.html?highlight=custom_list#term-
custom-loop-list>.
I would think setting that equal to the 0-39 for your member count,
then doing something along the lines of
ENS_VAR1_LEVELS = "(0,{custom?fmt=%d},*,*)"
FCST_VAR1_LEVELS = "(0, {custom?fmt=%d},*,*)"
I'm not as certain about that solution, but since your ensemble
members can be read in directly via plot_data_plane, it's worth a
chance.
-John O.
On Thu, Jun 24, 2021 at 11:33 AM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100321 >
>
> Hi Marion,
>
> I'm looking for someone who can assist you with this issue. John
> Halley Gotway is out of the office until July 6.
>
> Regards,
> Minna
> ---------------
> Minna Win
> Pronouns: she/her
> National Center for Atmospheric Research Developmental Testbed
Center
> Phone: 303-497-8423
> Fax: 303-497-8401
> ---------------
>
>
>
> On Thu, Jun 24, 2021 at 4:40 AM marion.mittermaier at metoffice.gov.uk
via
> RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 24 04:33:12 2021: Request 100321 was acted upon.
> > Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> > Queue: met_help
> > Subject: Reading in ensemble netcdf
> > Owner: Nobody
> > Requestors: marion.mittermaier at metoffice.gov.uk
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100321 >
> >
> >
> > Hi,
> >
> > I have some S2S ensemble forecasts where 40 ensemble members are
written
> > into the same netcdf file (as realizations). I want to apply a
variety of
> > MET tools to these. I can't get this file read in with
plot_data_plane
> so
> > I suspect that I won't get any of the MET tools to read this in
(note I
> > haven't tried). I know enough to know that if plot_data_plane
doesn't
> work
> > I usually won't get grid_stat etc to work either. These files are
> > constructed to make dealing with the forecasts easier... as the
ensemble
> is
> > a lagged one with the oldest members of the ensemble for any given
lead
> > time horizon being 10 days older than the newest ones... they are
> rebadged
> > to make it look more homogeneous and easier to handle, i.e. all
the
> members
> > that are valid for a particular forecast horizon are put together
in one
> > file (here a day 18 forecast). I wouldn't know how to extract a
given
> > ensemble member from the ensemble in a config file either... any
> thoughts?
> > Will MET handle this kind of file (i.!
> > e. all members in the same file together?) or do I have to break
this up
> > again?! I have to add that I have no experience of running
ensemble_stat
> > and know very little about how to get MET to ingest ensemble
forecasts
> but
> > all the examples suggest that ensemble members are in separate
files
> (even
> > separate directories!) and it looks like MET expects n files for n
> ensemble
> > members, i.e. if I gave it 1 file with n members it won't work.
> >
> > TIA
> > Marion
> >
> > --
> > Dr Marion Mittermaier Verification & diagnostics expert
scientist
> > (Global Model Evaluation & Development)
> >
> > Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> > Tel: +44 (0) 330 135 1604
> > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> > marion.mittermaier at metoffice.gov.uk> Website:
> http://www.metoffice.gov.uk
> > <http://www.metoffice.gov.uk/>
> >
> > http://www.metoffice.gov.uk/research/people/marion-mittermaier
> >
> > Associate Editor for 2021: Monthly Weather Review<
> >
> https://www.ametsoc.org/index.cfm/ams/publications/journals/monthly-
weather-review/
> > >
> >
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
> >
>
>
--
John Opatz
Associate Scientist III
NCAR RAL and DTC
Boulder, Colorado
+1 303-497-8305
------------------------------------------------
More information about the Met_help
mailing list