[Met_help] [rt.rap.ucar.edu #89970] History for HDF5 Obs readable by MET?
John Halley Gotway via RT
met_help at ucar.edu
Tue May 7 17:05:35 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I was wondering if there is a way to ingest HDF5 Observations files to
verify my forecast fields using any of the stat functions?
Justin
Justin Tsu
Marine Meteorology Division
Data Assimilation/Mesoscale Modeling
Building 704 Room 212
Naval Research Laboratory, Code 7531
7 Grace Hopper Avenue
Monterey, CA 93943-5502
Ph. (831) 656-4111
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: HDF5 Obs readable by MET?
From: John Halley Gotway
Time: Thu May 02 17:37:42 2019
Justin,
By HDF5 observations, do you mean gridded or point data? Either way
the
answer is currently no. But for gridded data you could write a python
script to read the data, define the metadata, and pass it into MET via
memory.
I’m curious to know what dataset you’re trying to read though. MET
already
links to the HDF5 library since NetCDF4 is built upon HDF5. If
there’s a
dataset which follows well-defined conventions, we could consider
adding
native support to MET for handling it directly rather than having to
go
through python.
Thanks
John Halley Gotway
On Mon, Apr 29, 2019 at 3:42 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> Mon Apr 29 15:42:16 2019: Request 89970 was acted upon.
> Transaction: Ticket created by justin.tsu at nrlmry.navy.mil
> Queue: met_help
> Subject: HDF5 Obs readable by MET?
> Owner: Nobody
> Requestors: justin.tsu at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89970 >
>
>
> Hello,
>
>
>
> I was wondering if there is a way to ingest HDF5 Observations files
to
> verify my forecast fields using any of the stat functions?
>
>
>
> Justin
>
>
>
> Justin Tsu
>
> Marine Meteorology Division
>
> Data Assimilation/Mesoscale Modeling
>
> Building 704 Room 212
>
> Naval Research Laboratory, Code 7531
>
> 7 Grace Hopper Avenue
>
> Monterey, CA 93943-5502
>
>
>
> Ph. (831) 656-4111
>
>
>
>
>
------------------------------------------------
Subject: HDF5 Obs readable by MET?
From: Tsu, Mr. Justin
Time: Thu May 02 18:46:32 2019
Hi John,
We are using two separate hdf5 files containing raob and surface point
obs and are in the process of developing a well-defined convention.
Would you, by chance, have an example of what you would consider a
well-defined convention for obs that MET can ingest as an HDF5?
Thanks,
Justin
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, May 2, 2019 4:38 PM
To: Tsu, Mr. Justin
Subject: Re: [rt.rap.ucar.edu #89970] HDF5 Obs readable by MET?
Justin,
By HDF5 observations, do you mean gridded or point data? Either way
the
answer is currently no. But for gridded data you could write a python
script to read the data, define the metadata, and pass it into MET via
memory.
I’m curious to know what dataset you’re trying to read though. MET
already
links to the HDF5 library since NetCDF4 is built upon HDF5. If
there’s a
dataset which follows well-defined conventions, we could consider
adding
native support to MET for handling it directly rather than having to
go
through python.
Thanks
John Halley Gotway
On Mon, Apr 29, 2019 at 3:42 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> Mon Apr 29 15:42:16 2019: Request 89970 was acted upon.
> Transaction: Ticket created by justin.tsu at nrlmry.navy.mil
> Queue: met_help
> Subject: HDF5 Obs readable by MET?
> Owner: Nobody
> Requestors: justin.tsu at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89970 >
>
>
> Hello,
>
>
>
> I was wondering if there is a way to ingest HDF5 Observations files
to
> verify my forecast fields using any of the stat functions?
>
>
>
> Justin
>
>
>
> Justin Tsu
>
> Marine Meteorology Division
>
> Data Assimilation/Mesoscale Modeling
>
> Building 704 Room 212
>
> Naval Research Laboratory, Code 7531
>
> 7 Grace Hopper Avenue
>
> Monterey, CA 93943-5502
>
>
>
> Ph. (831) 656-4111
>
>
>
>
>
------------------------------------------------
Subject: HDF5 Obs readable by MET?
From: John Halley Gotway
Time: Fri May 03 09:52:36 2019
Justin,
Unfortunately, no, I don't have a specific HDF5 convention to
recommend.
Here's some related information about what MET currently supports...
- The pb2nc tool reads point observations from PREPBUFR and BUFR
formats.
But writing BUFR files isn't very palatable.
- The ascii2nc tool reads point observations from the following ascii
formats... and these are easy to extend: "met_point", "little_r",
"surfrad", "wwsis", "aeronet", "aeronetv2", or "aeronetv3"
- The madis2nc tool reads point observations from the following
flavors of
NetCDF MADIS data files: metar, raob, profiler, maritime, mesonet, or
acarsProfiles
- The lidar2nc tool extracts data from HDF files and formats them for
use
in MET (this is pretty specific to CALIPSO data right now).
- All of the tools are converters. They read data from an input file
and
reformat them into a common NetCDF file format that the rest of the
MET
tools know how to read. If you'd like, you could write directly to
the
NetCDF format that MET is already reading.
And here's what's on the horizon...
- In met-8.0, we added python embedding to support running a python
script
to read a 2d plane of data and pass it in to MET via memory. We'd
like to
extend this capability to also support the passing of point
observations
from MET to python in memory. The tricky part, as you've found, is
defining the conventions for doing so.
- MET currently reads gridded data which follows the NetCDF
Climate-Forecast conventions (
http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-
conventions.html).
The CF convention also includes conventions for point observations...
however, to my knowledge, it is not very widely used. It's always
seemed
to me that adding support for point observations in this format would
be a
logical extension.
- We are aware of the ongoing efforts of JEDI, to store point
observations
in a database. And we will need to enhance MET to work with that
database
in some way. Will your point observations live in JEDI?
Hope that helps.
Thanks,
John
On Thu, May 2, 2019 at 6:47 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89970 >
>
> Hi John,
>
> We are using two separate hdf5 files containing raob and surface
point obs
> and are in the process of developing a well-defined convention.
Would you,
> by chance, have an example of what you would consider a well-defined
> convention for obs that MET can ingest as an HDF5?
>
> Thanks,
> Justin
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, May 2, 2019 4:38 PM
> To: Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #89970] HDF5 Obs readable by MET?
>
> Justin,
>
> By HDF5 observations, do you mean gridded or point data? Either way
the
> answer is currently no. But for gridded data you could write a
python
> script to read the data, define the metadata, and pass it into MET
via
> memory.
>
> I’m curious to know what dataset you’re trying to read though. MET
already
> links to the HDF5 library since NetCDF4 is built upon HDF5. If
there’s a
> dataset which follows well-defined conventions, we could consider
adding
> native support to MET for handling it directly rather than having to
go
> through python.
>
> Thanks
> John Halley Gotway
>
> On Mon, Apr 29, 2019 at 3:42 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Mon Apr 29 15:42:16 2019: Request 89970 was acted upon.
> > Transaction: Ticket created by justin.tsu at nrlmry.navy.mil
> > Queue: met_help
> > Subject: HDF5 Obs readable by MET?
> > Owner: Nobody
> > Requestors: justin.tsu at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89970 >
> >
> >
> > Hello,
> >
> >
> >
> > I was wondering if there is a way to ingest HDF5 Observations
files to
> > verify my forecast fields using any of the stat functions?
> >
> >
> >
> > Justin
> >
> >
> >
> > Justin Tsu
> >
> > Marine Meteorology Division
> >
> > Data Assimilation/Mesoscale Modeling
> >
> > Building 704 Room 212
> >
> > Naval Research Laboratory, Code 7531
> >
> > 7 Grace Hopper Avenue
> >
> > Monterey, CA 93943-5502
> >
> >
> >
> > Ph. (831) 656-4111
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list