[Met_help] [rt.rap.ucar.edu #98391] History for Regrid_data_plane with precipitation accumulations from a NetCDF file
John Halley Gotway via RT
met_help at ucar.edu
Wed Feb 17 10:48:09 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I've had an issue when running regrid_data_plane on some NetCDF files containing GPM precipitation accumulations.
The daily precipitation accumulations have been computed from the original half-hourly snapshots which are downloaded into the Met Office as a NetCDF format file. I do not know whether this process is CF-compliant. The conversion from half-hourly snapshot to daily accumulation is done using the SciTools Iris python package (https://github.com/SciTools/iris) and claims to output NetCDF as CF-1.5 compliant files.
I then needed to regrid files of this type onto a global forecast model grid, example given in fcst.nc attached, which has been created using wgrib and the NCO tools, I think, but does not appear to be CF-compliant.
Using METv9.1, I get the regridded gpm file as attached, in gpm_2019060103_2019060203_nepsgrid.nc. This file states that the init_time is the same as the valid_time, which is the mid-point of the accumulation window.
I used the following command:
$ regrid_data_plane gpm_2019060103_2019060203.nc neps_fcst.nc gpm_2019060103_2019060203_nepsgrid.nc \
-field 'name="precipitation_amount"; level="(*,*)";' -method NEAREST
Is this due to something I am missing from the NetCDF headers in the original observation file? In addition, I can't see anything in the NetCDF output from regrid_data_plane which indicates it is a 24-hour total. When I then use this in a downstream grid_stat job, I get a warning message that the forecast and observed times are not the same.
As a general question, related to this, what headers does MET need in order to identify accumulations from variables in NetCDF files? (And thus be able to sensibly name MET tool output files, say from grid_stat. As context for this, I am attempting to use the regrid_data_plane output, attached, as input to grid_stat, but currently the output files generated by grid_stat do not have the expected dates/times/lead times in them. There is definitely an issue with the forecast file headers as well, I just want to confirm what dimensions and attributes MET needs.)
At the Met Office we tend to have bounds on the time coordinate indicating the start and end of the accumulation period, with the time point being the mid-point of those bounds. Then there might be an attribute ('cell_method') indicating a 24-hour sum. There might also be a forecast_reference_time and a forecast_period. Would this be sufficient?
Kind Regards,
Rachel North
Dr Rachel North, Research Scientist
Met Office, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
Tel: +44 (0)3301352097 Fax: +44 (0)1392 885681
rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.uk> https://www.metoffice.gov.uk/
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Regrid_data_plane with precipitation accumulations from a NetCDF file
From: Julie Prestopnik
Time: Tue Jan 26 12:23:19 2021
Hi Rachel.
I am assigning this ticket to John Halley Gotway. Please allow a few
days
for a response.
Julie
On Tue, Jan 26, 2021 at 11:11 AM North, Rachel via RT
<met_help at ucar.edu>
wrote:
>
> Tue Jan 26 11:11:37 2021: Request 98391 was acted upon.
> Transaction: Ticket created by rachel.north at metoffice.gov.uk
> Queue: met_help
> Subject: Regrid_data_plane with precipitation accumulations
from a
> NetCDF file
> Owner: Nobody
> Requestors: rachel.north at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98391 >
>
>
> Hi,
>
> I've had an issue when running regrid_data_plane on some NetCDF
files
> containing GPM precipitation accumulations.
> The daily precipitation accumulations have been computed from the
original
> half-hourly snapshots which are downloaded into the Met Office as a
NetCDF
> format file. I do not know whether this process is CF-compliant. The
> conversion from half-hourly snapshot to daily accumulation is done
using
> the SciTools Iris python package (https://github.com/SciTools/iris)
and
> claims to output NetCDF as CF-1.5 compliant files.
>
> I then needed to regrid files of this type onto a global forecast
model
> grid, example given in fcst.nc attached, which has been created
using
> wgrib and the NCO tools, I think, but does not appear to be CF-
compliant.
>
> Using METv9.1, I get the regridded gpm file as attached, in
> gpm_2019060103_2019060203_nepsgrid.nc. This file states that the
> init_time is the same as the valid_time, which is the mid-point of
the
> accumulation window.
>
> I used the following command:
> $ regrid_data_plane gpm_2019060103_2019060203.nc neps_fcst.nc
> gpm_2019060103_2019060203_nepsgrid.nc \
> -field
'name="precipitation_amount";
> level="(*,*)";' -method NEAREST
>
> Is this due to something I am missing from the NetCDF headers in the
> original observation file? In addition, I can't see anything in the
NetCDF
> output from regrid_data_plane which indicates it is a 24-hour total.
When I
> then use this in a downstream grid_stat job, I get a warning message
that
> the forecast and observed times are not the same.
>
> As a general question, related to this, what headers does MET need
in
> order to identify accumulations from variables in NetCDF files? (And
thus
> be able to sensibly name MET tool output files, say from grid_stat.
As
> context for this, I am attempting to use the regrid_data_plane
output,
> attached, as input to grid_stat, but currently the output files
generated
> by grid_stat do not have the expected dates/times/lead times in
them. There
> is definitely an issue with the forecast file headers as well, I
just want
> to confirm what dimensions and attributes MET needs.)
>
> At the Met Office we tend to have bounds on the time coordinate
indicating
> the start and end of the accumulation period, with the time point
being the
> mid-point of those bounds. Then there might be an attribute
('cell_method')
> indicating a 24-hour sum. There might also be a
forecast_reference_time
> and a forecast_period. Would this be sufficient?
>
> Kind Regards,
> Rachel North
>
> Dr Rachel North, Research Scientist
> Met Office, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
> Tel: +44 (0)3301352097 Fax: +44 (0)1392 885681
> rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.uk>
> https://www.metoffice.gov.uk/
>
>
>
--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Regrid_data_plane with precipitation accumulations from a NetCDF file
From: John Halley Gotway
Time: Thu Jan 28 14:33:35 2021
Hi Rachel,
Thanks for sending along sample data to illustrate your question. That
helps a lot.
I'm copying Howard Soh, since he's the person who knows the most about
MET's support for the NetCDF CF-convention.
So the input file is named "gpm_2019060103_2019060203.nc". I see 2
variables in that file related to time:
*double time ;*
* time:bounds = "time_bnds" ;*
* time:units = "hours since 1970-01-01" ; time:standard_name =
"time"
; time:calendar = "gregorian" ;*
* double time_bnds(bnds) ;*
As you report, the value of "time" is the mid-point of the current
time
interval. At first, I thought this wasn't CF-compliant, but checking
the
documentation I see a very similar example here:
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html
Search for "Example 7.5"
Howard, it looks like MET is *NOT* parsing the "*bounds*" attribute
for the
time variable and handling it in a CF-compliant way. Can you please
confirm
that's the case? And when we add that support, it seems like we should
define the valid time of the data as max(time_bnds)... and the
accumulation
interval for the data at max(time_bnds) - min(time_bnds).
Rachel, does that approach sound correct to you?
In addition, I wanted to mention that we *usually* see CF-compliant
files
where time is defined as a coordinate variable... so there's a time
dimension, time variable, and then the data variables include that
time
dimension. Howard, can you comment as to whether or not that's a CF
rule...
or is that just a common practice? Looks like MET is at least parsing
the
singular time variable fine anyway, even though it's not a coordinate
variable.
*I'd recommend that we prioritize correcting this support for the time
bounds attribute for MET version 10.0.0.*
In the meantime, let me demonstrate a temporary workaround. Running
plot_data_plane at verbosity 4 to dump out metadata info:
*MET-main_v9.1/met/bin/plot_data_plane gpm_2019060103_2019060203.nc
<http://gpm_2019060103_2019060203.nc> gpm_2019060103_2019060203.ps
<http://gpm_2019060103_2019060203.ps> 'name="precipitation_amount";
level="(*,*)";' -v 4*
*DEBUG 4: Data plane information:DEBUG 4: plane min: 0DEBUG 4:
plane max: 247.173DEBUG 4: valid time: 20190601_145959DEBUG 4:
lead time: 000000DEBUG 4: init time: 20190601_145959DEBUG 4:
accum time: 000000*
And then adding "set_attr" options to manually override the valid
time:
*MET-main_v9.1/met/bin/plot_data_plane gpm_2019060103_2019060203.nc
<http://gpm_2019060103_2019060203.nc> gpm_2019060103_2019060203.ps
<http://gpm_2019060103_2019060203.ps> 'name="precipitation_amount";
level="(*,*)"; set_attr_init="20190602_03";
set_attr_valid="20190602_03";'
-v 4*
*DEBUG 4: Data plane information:DEBUG 4: Data plane information:DEBUG
4:
plane min: 0DEBUG 4: plane max: 247.173DEBUG 4: valid
time:
20190602_030000DEBUG 4: lead time: 000000DEBUG 4: init
time:
20190602_030000DEBUG 4: accum time: 000000*
So you're welcome to use those set_attr options (search for set_attr
here
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html) to
override
the metadata. Obviously that's not a great longterm fix, but should
hopefully enable you to make progress while Howard and I figure out
the
better fix!
Do note, that for some reason "set_attr_accum" did NOT work as I
expected.
I'll figure out why and get that fixed up.
Thanks,
John
On Tue, Jan 26, 2021 at 12:23 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> Tue Jan 26 12:23:26 2021: Request 98391 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by jpresto
> Queue: met_help
> Subject: Regrid_data_plane with precipitation accumulations
from a
> NetCDF file
> Owner: johnhg
> Requestors: rachel.north at metoffice.gov.uk
> Status: open
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98391 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: Regrid_data_plane with precipitation accumulations from a NetCDF file
From: John Halley Gotway
Time: Thu Jan 28 16:24:42 2021
Rachel,
Just FYI, I did indeed find a bug that caused set_attr_lead to not
work as
expected:
https://github.com/dtcenter/MET/issues/1646
It's an easy fix that'll be included in met-10.0.0 and met-9.1.2 (if
we
actually do that bugfix release).
And Howard will get back to us about enhancing MET to handle the time
"bounds" attribute.
Thanks,
John
On Thu, Jan 28, 2021 at 2:33 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Hi Rachel,
>
> Thanks for sending along sample data to illustrate your question.
That
> helps a lot.
>
> I'm copying Howard Soh, since he's the person who knows the most
about
> MET's support for the NetCDF CF-convention.
>
> So the input file is named "gpm_2019060103_2019060203.nc". I see 2
> variables in that file related to time:
> *double time ;*
> * time:bounds = "time_bnds" ;*
>
>
>
> * time:units = "hours since 1970-01-01" ; time:standard_name =
"time"
> ; time:calendar = "gregorian" ;*
> * double time_bnds(bnds) ;*
>
> As you report, the value of "time" is the mid-point of the current
time
> interval. At first, I thought this wasn't CF-compliant, but checking
the
> documentation I see a very similar example here:
>
>
> https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html
> Search for "Example 7.5"
>
> Howard, it looks like MET is *NOT* parsing the "*bounds*" attribute
for
> the time variable and handling it in a CF-compliant way. Can you
please
> confirm that's the case? And when we add that support, it seems like
we
> should define the valid time of the data as max(time_bnds)... and
the
> accumulation interval for the data at max(time_bnds) -
min(time_bnds).
>
> Rachel, does that approach sound correct to you?
>
> In addition, I wanted to mention that we *usually* see CF-compliant
files
> where time is defined as a coordinate variable... so there's a time
> dimension, time variable, and then the data variables include that
time
> dimension. Howard, can you comment as to whether or not that's a CF
rule...
> or is that just a common practice? Looks like MET is at least
parsing the
> singular time variable fine anyway, even though it's not a
coordinate
> variable.
>
> *I'd recommend that we prioritize correcting this support for the
time
> bounds attribute for MET version 10.0.0.*
>
> In the meantime, let me demonstrate a temporary workaround. Running
> plot_data_plane at verbosity 4 to dump out metadata info:
>
>
> *MET-main_v9.1/met/bin/plot_data_plane gpm_2019060103_2019060203.nc
> <http://gpm_2019060103_2019060203.nc> gpm_2019060103_2019060203.ps
> <http://gpm_2019060103_2019060203.ps> 'name="precipitation_amount";
> level="(*,*)";' -v 4*
>
>
>
>
>
>
> *DEBUG 4: Data plane information:DEBUG 4: plane min: 0DEBUG 4:
> plane max: 247.173DEBUG 4: valid time: 20190601_145959DEBUG 4:
> lead time: 000000DEBUG 4: init time: 20190601_145959DEBUG 4:
> accum time: 000000*
>
> And then adding "set_attr" options to manually override the valid
time:
>
>
> *MET-main_v9.1/met/bin/plot_data_plane gpm_2019060103_2019060203.nc
> <http://gpm_2019060103_2019060203.nc> gpm_2019060103_2019060203.ps
> <http://gpm_2019060103_2019060203.ps> 'name="precipitation_amount";
> level="(*,*)"; set_attr_init="20190602_03";
set_attr_valid="20190602_03";'
> -v 4*
>
>
>
>
>
>
>
> *DEBUG 4: Data plane information:DEBUG 4: Data plane
information:DEBUG 4:
> plane min: 0DEBUG 4: plane max: 247.173DEBUG 4:
valid
> time: 20190602_030000DEBUG 4: lead time: 000000DEBUG 4:
init
> time: 20190602_030000DEBUG 4: accum time: 000000*
>
> So you're welcome to use those set_attr options (search for set_attr
here
> https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html) to
> override the metadata. Obviously that's not a great longterm fix,
but
> should hopefully enable you to make progress while Howard and I
figure out
> the better fix!
>
> Do note, that for some reason "set_attr_accum" did NOT work as I
expected.
> I'll figure out why and get that fixed up.
>
> Thanks,
> John
>
> On Tue, Jan 26, 2021 at 12:23 PM Julie Prestopnik via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Tue Jan 26 12:23:26 2021: Request 98391 was acted upon.
>> Transaction: Given to johnhg (John Halley Gotway) by jpresto
>> Queue: met_help
>> Subject: Regrid_data_plane with precipitation accumulations
from a
>> NetCDF file
>> Owner: johnhg
>> Requestors: rachel.north at metoffice.gov.uk
>> Status: open
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98391 >
>>
>>
>> This transaction appears to have no content
>>
>
------------------------------------------------
Subject: Regrid_data_plane with precipitation accumulations from a NetCDF file
From: Howard Soh
Time: Thu Jan 28 22:01:11 2021
MET does not process "bounds' attribute. MET just uses the time in the
"time" coordinate variable which is the choice (mid-point or end-
point) of product generator over the bounds variable. The "bounds"
attribute is used at the coordinate variables. So it's not limited to
the accumulation (as you said, cell_method attribute shows additional
information like the accumulation). There is no rule nor
recommendation on how to use bounds variable. It's an option for
users. User can use the end-point from the bounds variable instead of
the mid-point time or vise versa.
If there is "cell_method" attribute. MET might process the bounds
variable based on settings at the call_method attribute. For example,
pick the latest time if "cell_method" indicates accumulation or other
cases. But how does MET know the user wants latest_time or mid-point
time over the time variable?
What are the requirements for MET (regrid_data_plane only or others,
too for accumulation info)?
- setting the mid-point time (into valid_time) from the bounds
variable instead of using time variable
- setting the end-point time (into valid_time) from the bounds
variable instead of using time variable
- adding new attribute for the accumulation?
[CF complaint]
The units and long_name attributes were carried over COARDS. CF added
the standard_name attribute as optional. The "units" attribute is
required, but not standard_name and long_name. The use of at least one
of them is strongly recommended. Some CF checker gives an error
message if both are missing.
The "axis" and "coordinates" attributes were introduced to identify
the coordinate variables.
The coordinate variable with the same dimension name is the CF. It was
the convention of COARDS and carried to CF. This is not required but
supported (as a CF standard) for backward compatibility.
Cheers,
Howard
On Thu Jan 28 16:24:42 2021, johnhg wrote:
> Rachel,
>
> Just FYI, I did indeed find a bug that caused set_attr_lead to not
> work as
> expected:
> https://github.com/dtcenter/MET/issues/1646
>
> It's an easy fix that'll be included in met-10.0.0 and met-9.1.2 (if
> we
> actually do that bugfix release).
>
> And Howard will get back to us about enhancing MET to handle the
time
> "bounds" attribute.
>
> Thanks,
> John
>
> On Thu, Jan 28, 2021 at 2:33 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Hi Rachel,
> >
> > Thanks for sending along sample data to illustrate your question.
> > That
> > helps a lot.
> >
> > I'm copying Howard Soh, since he's the person who knows the most
> > about
> > MET's support for the NetCDF CF-convention.
> >
> > So the input file is named "gpm_2019060103_2019060203.nc". I see 2
> > variables in that file related to time:
> > *double time ;*
> > * time:bounds = "time_bnds" ;*
> >
> >
> >
> > * time:units = "hours since 1970-01-01" ; time:standard_name =
> > "time"
> > ; time:calendar = "gregorian" ;*
> > * double time_bnds(bnds) ;*
> >
> > As you report, the value of "time" is the mid-point of the current
> > time
> > interval. At first, I thought this wasn't CF-compliant, but
checking
> > the
> > documentation I see a very similar example here:
> >
> >
> > https://cfconventions.org/Data/cf-conventions/cf-conventions-
1.8/cf-
> > conventions.html
> > Search for "Example 7.5"
> >
> > Howard, it looks like MET is *NOT* parsing the "*bounds*"
attribute
> > for
> > the time variable and handling it in a CF-compliant way. Can you
> > please
> > confirm that's the case? And when we add that support, it seems
like
> > we
> > should define the valid time of the data as max(time_bnds)... and
the
> > accumulation interval for the data at max(time_bnds) -
> > min(time_bnds).
> >
> > Rachel, does that approach sound correct to you?
> >
> > In addition, I wanted to mention that we *usually* see CF-
compliant
> > files
> > where time is defined as a coordinate variable... so there's a
time
> > dimension, time variable, and then the data variables include that
> > time
> > dimension. Howard, can you comment as to whether or not that's a
CF
> > rule...
> > or is that just a common practice? Looks like MET is at least
parsing
> > the
> > singular time variable fine anyway, even though it's not a
coordinate
> > variable.
> >
> > *I'd recommend that we prioritize correcting this support for the
> > time
> > bounds attribute for MET version 10.0.0.*
> >
> > In the meantime, let me demonstrate a temporary workaround.
Running
> > plot_data_plane at verbosity 4 to dump out metadata info:
> >
> >
> > *MET-main_v9.1/met/bin/plot_data_plane
gpm_2019060103_2019060203.nc
> > <http://gpm_2019060103_2019060203.nc> gpm_2019060103_2019060203.ps
> > <http://gpm_2019060103_2019060203.ps>
'name="precipitation_amount";
> > level="(*,*)";' -v 4*
> >
> >
> >
> >
> >
> >
> > *DEBUG 4: Data plane information:DEBUG 4: plane min: 0DEBUG
4:
> > plane max: 247.173DEBUG 4: valid time: 20190601_145959DEBUG
4:
> > lead time: 000000DEBUG 4: init time: 20190601_145959DEBUG 4:
> > accum time: 000000*
> >
> > And then adding "set_attr" options to manually override the valid
> > time:
> >
> >
> > *MET-main_v9.1/met/bin/plot_data_plane
gpm_2019060103_2019060203.nc
> > <http://gpm_2019060103_2019060203.nc> gpm_2019060103_2019060203.ps
> > <http://gpm_2019060103_2019060203.ps>
'name="precipitation_amount";
> > level="(*,*)"; set_attr_init="20190602_03";
> > set_attr_valid="20190602_03";'
> > -v 4*
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 4: Data plane information:DEBUG 4: Data plane
> > information:DEBUG 4:
> > plane min: 0DEBUG 4: plane max: 247.173DEBUG 4:
> > valid
> > time: 20190602_030000DEBUG 4: lead time: 000000DEBUG 4:
> > init
> > time: 20190602_030000DEBUG 4: accum time: 000000*
> >
> > So you're welcome to use those set_attr options (search for
set_attr
> > here
> > https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html) to
> > override the metadata. Obviously that's not a great longterm fix,
but
> > should hopefully enable you to make progress while Howard and I
> > figure out
> > the better fix!
> >
> > Do note, that for some reason "set_attr_accum" did NOT work as I
> > expected.
> > I'll figure out why and get that fixed up.
> >
> > Thanks,
> > John
> >
> > On Tue, Jan 26, 2021 at 12:23 PM Julie Prestopnik via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> Tue Jan 26 12:23:26 2021: Request 98391 was acted upon.
> >> Transaction: Given to johnhg (John Halley Gotway) by jpresto
> >> Queue: met_help
> >> Subject: Regrid_data_plane with precipitation accumulations
> >> from a
> >> NetCDF file
> >> Owner: johnhg
> >> Requestors: rachel.north at metoffice.gov.uk
> >> Status: open
> >> Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98391 >
> >>
> >>
> >> This transaction appears to have no content
> >>
> >
------------------------------------------------
Subject: Regrid_data_plane with precipitation accumulations from a NetCDF file
From: John Halley Gotway
Time: Fri Jan 29 15:52:28 2021
Howard,
I am proposing that you add a GitHub issue to add the following logic
to
MET's NetCDF CF library:
- When parsing time variables, check for the "bounds" attribute.
- If present, it should specify the name of the variable containing
the
time bounds.
- Parse the times from that variable.
- Define valid_time = max of those times
- Define accum_time = max - min of those times
We need Rachel to confirm that logic is correct.
And to you, does this seem appropriate within the context of the
CF-convention? Or do we need to clarify logic for handling the
"cell_method" information?
John
On Thu, Jan 28, 2021 at 10:01 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98391 >
>
> MET does not process "bounds' attribute. MET just uses the time in
the
> "time" coordinate variable which is the choice (mid-point or end-
point) of
> product generator over the bounds variable. The "bounds" attribute
is used
> at the coordinate variables. So it's not limited to the accumulation
(as
> you said, cell_method attribute shows additional information like
the
> accumulation). There is no rule nor recommendation on how to use
bounds
> variable. It's an option for users. User can use the end-point from
the
> bounds variable instead of the mid-point time or vise versa.
>
> If there is "cell_method" attribute. MET might process the bounds
variable
> based on settings at the call_method attribute. For example, pick
the
> latest time if "cell_method" indicates accumulation or other cases.
But how
> does MET know the user wants latest_time or mid-point time over the
time
> variable?
>
> What are the requirements for MET (regrid_data_plane only or others,
too
> for accumulation info)?
> - setting the mid-point time (into valid_time) from the bounds
variable
> instead of using time variable
> - setting the end-point time (into valid_time) from the bounds
variable
> instead of using time variable
> - adding new attribute for the accumulation?
>
> [CF complaint]
>
> The units and long_name attributes were carried over COARDS. CF
added the
> standard_name attribute as optional. The "units" attribute is
required, but
> not standard_name and long_name. The use of at least one of them is
> strongly recommended. Some CF checker gives an error message if both
are
> missing.
>
> The "axis" and "coordinates" attributes were introduced to identify
the
> coordinate variables.
>
> The coordinate variable with the same dimension name is the CF. It
was the
> convention of COARDS and carried to CF. This is not required but
supported
> (as a CF standard) for backward compatibility.
>
> Cheers,
> Howard
>
> On Thu Jan 28 16:24:42 2021, johnhg wrote:
> > Rachel,
> >
> > Just FYI, I did indeed find a bug that caused set_attr_lead to not
> > work as
> > expected:
> > https://github.com/dtcenter/MET/issues/1646
> >
> > It's an easy fix that'll be included in met-10.0.0 and met-9.1.2
(if
> > we
> > actually do that bugfix release).
> >
> > And Howard will get back to us about enhancing MET to handle the
time
> > "bounds" attribute.
> >
> > Thanks,
> > John
> >
> > On Thu, Jan 28, 2021 at 2:33 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Hi Rachel,
> > >
> > > Thanks for sending along sample data to illustrate your
question.
> > > That
> > > helps a lot.
> > >
> > > I'm copying Howard Soh, since he's the person who knows the most
> > > about
> > > MET's support for the NetCDF CF-convention.
> > >
> > > So the input file is named "gpm_2019060103_2019060203.nc". I see
2
> > > variables in that file related to time:
> > > *double time ;*
> > > * time:bounds = "time_bnds" ;*
> > >
> > >
> > >
> > > * time:units = "hours since 1970-01-01" ; time:standard_name
=
> > > "time"
> > > ; time:calendar = "gregorian" ;*
> > > * double time_bnds(bnds) ;*
> > >
> > > As you report, the value of "time" is the mid-point of the
current
> > > time
> > > interval. At first, I thought this wasn't CF-compliant, but
checking
> > > the
> > > documentation I see a very similar example here:
> > >
> > >
> > > https://cfconventions.org/Data/cf-conventions/cf-conventions-
1.8/cf-
> > > conventions.html
> > > Search for "Example 7.5"
> > >
> > > Howard, it looks like MET is *NOT* parsing the "*bounds*"
attribute
> > > for
> > > the time variable and handling it in a CF-compliant way. Can you
> > > please
> > > confirm that's the case? And when we add that support, it seems
like
> > > we
> > > should define the valid time of the data as max(time_bnds)...
and the
> > > accumulation interval for the data at max(time_bnds) -
> > > min(time_bnds).
> > >
> > > Rachel, does that approach sound correct to you?
> > >
> > > In addition, I wanted to mention that we *usually* see CF-
compliant
> > > files
> > > where time is defined as a coordinate variable... so there's a
time
> > > dimension, time variable, and then the data variables include
that
> > > time
> > > dimension. Howard, can you comment as to whether or not that's a
CF
> > > rule...
> > > or is that just a common practice? Looks like MET is at least
parsing
> > > the
> > > singular time variable fine anyway, even though it's not a
coordinate
> > > variable.
> > >
> > > *I'd recommend that we prioritize correcting this support for
the
> > > time
> > > bounds attribute for MET version 10.0.0.*
> > >
> > > In the meantime, let me demonstrate a temporary workaround.
Running
> > > plot_data_plane at verbosity 4 to dump out metadata info:
> > >
> > >
> > > *MET-main_v9.1/met/bin/plot_data_plane
gpm_2019060103_2019060203.nc
> > > <http://gpm_2019060103_2019060203.nc>
gpm_2019060103_2019060203.ps
> > > <http://gpm_2019060103_2019060203.ps>
'name="precipitation_amount";
> > > level="(*,*)";' -v 4*
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 4: Data plane information:DEBUG 4: plane min:
0DEBUG 4:
> > > plane max: 247.173DEBUG 4: valid time: 20190601_145959DEBUG
4:
> > > lead time: 000000DEBUG 4: init time: 20190601_145959DEBUG
4:
> > > accum time: 000000*
> > >
> > > And then adding "set_attr" options to manually override the
valid
> > > time:
> > >
> > >
> > > *MET-main_v9.1/met/bin/plot_data_plane
gpm_2019060103_2019060203.nc
> > > <http://gpm_2019060103_2019060203.nc>
gpm_2019060103_2019060203.ps
> > > <http://gpm_2019060103_2019060203.ps>
'name="precipitation_amount";
> > > level="(*,*)"; set_attr_init="20190602_03";
> > > set_attr_valid="20190602_03";'
> > > -v 4*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 4: Data plane information:DEBUG 4: Data plane
> > > information:DEBUG 4:
> > > plane min: 0DEBUG 4: plane max: 247.173DEBUG 4:
> > > valid
> > > time: 20190602_030000DEBUG 4: lead time: 000000DEBUG 4:
> > > init
> > > time: 20190602_030000DEBUG 4: accum time: 000000*
> > >
> > > So you're welcome to use those set_attr options (search for
set_attr
> > > here
> > > https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html)
to
> > > override the metadata. Obviously that's not a great longterm
fix, but
> > > should hopefully enable you to make progress while Howard and I
> > > figure out
> > > the better fix!
> > >
> > > Do note, that for some reason "set_attr_accum" did NOT work as I
> > > expected.
> > > I'll figure out why and get that fixed up.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Jan 26, 2021 at 12:23 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> Tue Jan 26 12:23:26 2021: Request 98391 was acted upon.
> > >> Transaction: Given to johnhg (John Halley Gotway) by jpresto
> > >> Queue: met_help
> > >> Subject: Regrid_data_plane with precipitation
accumulations
> > >> from a
> > >> NetCDF file
> > >> Owner: johnhg
> > >> Requestors: rachel.north at metoffice.gov.uk
> > >> Status: open
> > >> Ticket <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98391 >
> > >>
> > >>
> > >> This transaction appears to have no content
> > >>
> > >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list