[Met_help] [rt.rap.ucar.edu #95225] History for read in Satellite data
John Halley Gotway via RT
met_help at ucar.edu
Mon Jun 8 12:48:25 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello John,
Thank you for helping me with reading in the satellite data using MET. I
have another set of satellite data which also needs to be read in by MET
for verification. Those two datasets are different but similar, so I think
you are the better helper to ask. The datasets are located below.
"ash_mass" is the variable needed to be verified.
/gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
Thank you.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Mon May 11 17:40:22 2020
Binyu,
OK, I logged onto WCOSS and retrieved one of these sample files.
I used ncview to display the latitude and longitude variable values
and
have attached a screenshot.
This does not look like gridded data to me. Instead, it's a view of
satellite pixels. MET currently does evaluations on gridded data, not
satellite pixels. So the question is if/how we can get this satellite
data
onto a grid.
MET does include a tool named point2grid which is intended to do
exactly
that. But it's pretty limited on the input files it can accept. Howard
Soh,
cc'ed here, is the developer who's worked most on that tool.
Howard, do you have any suggestions on how Binyu could interpolate the
data
from this NetCDF file onto a grid?
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
Thanks,
John
On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: read in Satellite data
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>
>
> Hello John,
>
> Thank you for helping me with reading in the satellite data using
MET. I
> have another set of satellite data which also needs to be read in by
MET
> for verification. Those two datasets are different but similar, so I
think
> you are the better helper to ask. The datasets are located below.
> "ash_mass" is the variable needed to be verified.
>
> /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
>
> Thank you.
> Binyu
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Mon May 18 13:35:15 2020
Hello,
Thank you John.
I look forward to Howard's response.
Binyu
On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> OK, I logged onto WCOSS and retrieved one of these sample files.
>
> I used ncview to display the latitude and longitude variable values
and
> have attached a screenshot.
>
> This does not look like gridded data to me. Instead, it's a view of
> satellite pixels. MET currently does evaluations on gridded data,
not
> satellite pixels. So the question is if/how we can get this
satellite data
> onto a grid.
>
> MET does include a tool named point2grid which is intended to do
exactly
> that. But it's pretty limited on the input files it can accept.
Howard Soh,
> cc'ed here, is the developer who's worked most on that tool.
>
> Howard, do you have any suggestions on how Binyu could interpolate
the data
> from this NetCDF file onto a grid?
>
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
>
> Thanks,
> John
>
>
> On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > Transaction: Ticket created by binyu.wang at noaa.gov
> > Queue: met_help
> > Subject: read in Satellite data
> > Owner: Nobody
> > Requestors: binyu.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> >
> >
> > Hello John,
> >
> > Thank you for helping me with reading in the satellite data using
MET. I
> > have another set of satellite data which also needs to be read in
by MET
> > for verification. Those two datasets are different but similar, so
I
> think
> > you are the better helper to ask. The datasets are located below.
> > "ash_mass" is the variable needed to be verified.
> >
> > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> >
> > Thank you.
> > Binyu
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Mon May 18 13:55:05 2020
Binyu,
I ran into a somewhat similar situation last week. We have data on a
tri-polar grid which is not supported in MET. So I did the following
steps:
(1) Write a python script to reformat the tri-polar data as if it were
lat/lon point observations.
(2) Run that script using ascii2nc to reformat the data into MET's
NetCDF
point observation format.
(3) Run that NetCDF point obs file through the point2grid tool to
interpolate the data to a regular grids.
And that process did (eventually) work, but it took well over 3 hours
for
all those steps since the input data consists of over 6 million
points! So
not very realistic! We need to figure out how to get lat/lon data into
the
point2grid logic much more efficiently. And I think that same solution
would work well for your data. But unfortunately, that would require
new
development.
John
On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>
> Hello,
>
> Thank you John.
>
> I look forward to Howard's response.
>
> Binyu
>
>
>
> On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > OK, I logged onto WCOSS and retrieved one of these sample files.
> >
> > I used ncview to display the latitude and longitude variable
values and
> > have attached a screenshot.
> >
> > This does not look like gridded data to me. Instead, it's a view
of
> > satellite pixels. MET currently does evaluations on gridded data,
not
> > satellite pixels. So the question is if/how we can get this
satellite
> data
> > onto a grid.
> >
> > MET does include a tool named point2grid which is intended to do
exactly
> > that. But it's pretty limited on the input files it can accept.
Howard
> Soh,
> > cc'ed here, is the developer who's worked most on that tool.
> >
> > Howard, do you have any suggestions on how Binyu could interpolate
the
> data
> > from this NetCDF file onto a grid?
> >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> >
> > Thanks,
> > John
> >
> >
> > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > Queue: met_help
> > > Subject: read in Satellite data
> > > Owner: Nobody
> > > Requestors: binyu.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> >
> > >
> > >
> > > Hello John,
> > >
> > > Thank you for helping me with reading in the satellite data
using MET.
> I
> > > have another set of satellite data which also needs to be read
in by
> MET
> > > for verification. Those two datasets are different but similar,
so I
> > think
> > > you are the better helper to ask. The datasets are located
below.
> > > "ash_mass" is the variable needed to be verified.
> > >
> > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > >
> > > Thank you.
> > > Binyu
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Mon May 18 20:31:39 2020
Thank you for the update, John. Can you share your scripts/code to me?
I
guess my data won't take so long because there is not so much data
inside.
Do you have an estimate how long it will take for the new development?
I have three sets of satellite data, except the previous two, I also
need
to work on the third set below, is this one easier to modify to be
read in
by MET?
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
Binyu
On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I ran into a somewhat similar situation last week. We have data on a
> tri-polar grid which is not supported in MET. So I did the following
steps:
> (1) Write a python script to reformat the tri-polar data as if it
were
> lat/lon point observations.
> (2) Run that script using ascii2nc to reformat the data into MET's
NetCDF
> point observation format.
> (3) Run that NetCDF point obs file through the point2grid tool to
> interpolate the data to a regular grids.
>
> And that process did (eventually) work, but it took well over 3
hours for
> all those steps since the input data consists of over 6 million
points! So
> not very realistic! We need to figure out how to get lat/lon data
into the
> point2grid logic much more efficiently. And I think that same
solution
> would work well for your data. But unfortunately, that would require
new
> development.
>
> John
>
> On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> >
> > Hello,
> >
> > Thank you John.
> >
> > I look forward to Howard's response.
> >
> > Binyu
> >
> >
> >
> > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > OK, I logged onto WCOSS and retrieved one of these sample files.
> > >
> > > I used ncview to display the latitude and longitude variable
values and
> > > have attached a screenshot.
> > >
> > > This does not look like gridded data to me. Instead, it's a view
of
> > > satellite pixels. MET currently does evaluations on gridded
data, not
> > > satellite pixels. So the question is if/how we can get this
satellite
> > data
> > > onto a grid.
> > >
> > > MET does include a tool named point2grid which is intended to do
> exactly
> > > that. But it's pretty limited on the input files it can accept.
Howard
> > Soh,
> > > cc'ed here, is the developer who's worked most on that tool.
> > >
> > > Howard, do you have any suggestions on how Binyu could
interpolate the
> > data
> > > from this NetCDF file onto a grid?
> > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > Queue: met_help
> > > > Subject: read in Satellite data
> > > > Owner: Nobody
> > > > Requestors: binyu.wang at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > >
> > > >
> > > >
> > > > Hello John,
> > > >
> > > > Thank you for helping me with reading in the satellite data
using
> MET.
> > I
> > > > have another set of satellite data which also needs to be read
in by
> > MET
> > > > for verification. Those two datasets are different but
similar, so I
> > > think
> > > > you are the better helper to ask. The datasets are located
below.
> > > > "ash_mass" is the variable needed to be verified.
> > > >
> > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > >
> > > > Thank you.
> > > > Binyu
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Tue May 19 09:50:22 2020
Binyu,
The point2grid tool already includes direct support for processing
GOES-16
satellite data.
Please take a look at chapter 4.5 of the MET User's Guide and try
passing
this data directly to point2grid:
https://dtcenter.org/sites/default/files/community-code/met/docs/user-
guide/MET_Users_Guide_v9.0.1.pdf
This functionality was developed using GOES16 aerosol data so it may
or may
not work. If not, please let me know, and I'll grab that file and ask
our
developer, Howard Soh, to take a look.
Listed below are the steps I took to process tri-polar grid data as
individual points. But this all took over 3 hours. I don't have an
estimate
yet for enhancing point2grid to process this data more directly. We're
discussing internally but do so how it'd be useful for at least 2
projects.
FYI, I posted the input data for the commands listed below to:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/for_binyu_data/
-------
I ran ascii2nc as we discussed and would say that this is not really a
viable option. Processing 6,232,499 grid points as individual point
observations just takes way too long!
I ran the following commands on kiowa.
*> cd /d1/projects/SeaIce/tripolar*
*> export MET_PYTHON_EXE=/usr/local/python3/bin/python*
*> time /usr/local/met-9.0.1/bin/ascii2nc 'tripolar_to_lat_lon.py
input/20200127/rtofs_glo_2ds_n048_daily_diag.nc
<http://rtofs_glo_2ds_n048_daily_diag.nc/> ice_coverage
north' tripolar_points.nc <http://tripolar_points.nc/> -format python*
This takes a whopping *2 hours and 25 minutes* to run ascii2nc on a
single
field of data:
(1) Read the data and reformat the 6.2 million values into a list of
lists.
(2) Write the result to a temporary pickle file.
(3) Read that pickle file into memory.
(4) Write those point obs back out to a 286MB NetCDF file.
By comparison, the input rtofs data file, with multiple variables, is
154MB
in size.
Next, run point2grid on that NetCDF file to regrid to the same Polar
Stereographic
grid as the IMS data, and this step took 55 minutes:
*> time /usr/local/met-9.0.1/bin/point2grid tripolar_points.nc
<http://tripolar_points.nc/> imssnow96.20190201.grb.grib2
tripolar_points_IMS.nc -field 'name="ice_coverage"; level="L0";'
-method
MAX*
Then plot the result:
*> time /usr/local/met-9.0.1/bin/plot_data_plane
tripolar_points_IMS.nc
tripolar_points_IMS.ps 'name="ice_coverage"; level="(*,*)";'*
On Mon, May 18, 2020 at 8:32 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
> Thank you for the update, John. Can you share your scripts/code to
me? I
> guess my data won't take so long because there is not so much data
inside.
> Do you have an estimate how long it will take for the new
development?
>
> I have three sets of satellite data, except the previous two, I also
need
> to work on the third set below, is this one easier to modify to be
read in
> by MET?
>
> /gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
> geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
>
>
> Binyu
>
> On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I ran into a somewhat similar situation last week. We have data on
a
> > tri-polar grid which is not supported in MET. So I did the
following
> steps:
> > (1) Write a python script to reformat the tri-polar data as if it
were
> > lat/lon point observations.
> > (2) Run that script using ascii2nc to reformat the data into MET's
NetCDF
> > point observation format.
> > (3) Run that NetCDF point obs file through the point2grid tool to
> > interpolate the data to a regular grids.
> >
> > And that process did (eventually) work, but it took well over 3
hours for
> > all those steps since the input data consists of over 6 million
points!
> So
> > not very realistic! We need to figure out how to get lat/lon data
into
> the
> > point2grid logic much more efficiently. And I think that same
solution
> > would work well for your data. But unfortunately, that would
require new
> > development.
> >
> > John
> >
> > On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> > >
> > > Hello,
> > >
> > > Thank you John.
> > >
> > > I look forward to Howard's response.
> > >
> > > Binyu
> > >
> > >
> > >
> > > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > OK, I logged onto WCOSS and retrieved one of these sample
files.
> > > >
> > > > I used ncview to display the latitude and longitude variable
values
> and
> > > > have attached a screenshot.
> > > >
> > > > This does not look like gridded data to me. Instead, it's a
view of
> > > > satellite pixels. MET currently does evaluations on gridded
data, not
> > > > satellite pixels. So the question is if/how we can get this
satellite
> > > data
> > > > onto a grid.
> > > >
> > > > MET does include a tool named point2grid which is intended to
do
> > exactly
> > > > that. But it's pretty limited on the input files it can
accept.
> Howard
> > > Soh,
> > > > cc'ed here, is the developer who's worked most on that tool.
> > > >
> > > > Howard, do you have any suggestions on how Binyu could
interpolate
> the
> > > data
> > > > from this NetCDF file onto a grid?
> > > >
> > > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: read in Satellite data
> > > > > Owner: Nobody
> > > > > Requestors: binyu.wang at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > > >
> > > > >
> > > > >
> > > > > Hello John,
> > > > >
> > > > > Thank you for helping me with reading in the satellite data
using
> > MET.
> > > I
> > > > > have another set of satellite data which also needs to be
read in
> by
> > > MET
> > > > > for verification. Those two datasets are different but
similar, so
> I
> > > > think
> > > > > you are the better helper to ask. The datasets are located
below.
> > > > > "ash_mass" is the variable needed to be verified.
> > > > >
> > > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > > >
> > > > > Thank you.
> > > > > Binyu
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Thu Jun 04 15:49:33 2020
Hello John,
I tried point2grid; but it didn't work. Here is what I did ( Due to
an
account issue, I am working on Hera now) :
#! /bin/sh
DIR=/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado
point2grid \
$DIR/geocatL2.GOES-16.Full_Disk.2019056.150030.hdf \
G212 \
test.nc \
-field 'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";'
\
-qc 0,1,2
-method MAX -v 1
Actually I am not sure what I should put as "to_grid argument"? Should
I
just use one of my forecasting members (grib2 format)? My forecasting
files
are under:
/scratch2/NCEPDEV/naqfc/Binyu.Wang/VA_ense/Out_Reven.21.0p25
Thank you.
Binyu
On Tue, May 19, 2020 at 11:51 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Binyu,
>
> The point2grid tool already includes direct support for processing
GOES-16
> satellite data.
>
> Please take a look at chapter 4.5 of the MET User's Guide and try
passing
> this data directly to point2grid:
>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.1.pdf
>
> This functionality was developed using GOES16 aerosol data so it may
or may
> not work. If not, please let me know, and I'll grab that file and
ask our
> developer, Howard Soh, to take a look.
>
> Listed below are the steps I took to process tri-polar grid data as
> individual points. But this all took over 3 hours. I don't have an
estimate
> yet for enhancing point2grid to process this data more directly.
We're
> discussing internally but do so how it'd be useful for at least 2
projects.
>
> FYI, I posted the input data for the commands listed below to:
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/for_binyu_data/
>
> -------
>
> I ran ascii2nc as we discussed and would say that this is not really
a
> viable option. Processing 6,232,499 grid points as individual point
> observations just takes way too long!
>
> I ran the following commands on kiowa.
>
> *> cd /d1/projects/SeaIce/tripolar*
> *> export MET_PYTHON_EXE=/usr/local/python3/bin/python*
> *> time /usr/local/met-9.0.1/bin/ascii2nc 'tripolar_to_lat_lon.py
> input/20200127/rtofs_glo_2ds_n048_daily_diag.nc
> <http://rtofs_glo_2ds_n048_daily_diag.nc/> ice_coverage
> north' tripolar_points.nc <http://tripolar_points.nc/> -format
python*
>
> This takes a whopping *2 hours and 25 minutes* to run ascii2nc on a
single
> field of data:
> (1) Read the data and reformat the 6.2 million values into a list of
lists.
> (2) Write the result to a temporary pickle file.
> (3) Read that pickle file into memory.
> (4) Write those point obs back out to a 286MB NetCDF file.
> By comparison, the input rtofs data file, with multiple variables,
is 154MB
> in size.
>
> Next, run point2grid on that NetCDF file to regrid to the same Polar
> Stereographic
> grid as the IMS data, and this step took 55 minutes:
> *> time /usr/local/met-9.0.1/bin/point2grid tripolar_points.nc
> <http://tripolar_points.nc/> imssnow96.20190201.grb.grib2
> tripolar_points_IMS.nc -field 'name="ice_coverage"; level="L0";'
-method
> MAX*
>
> Then plot the result:
> *> time /usr/local/met-9.0.1/bin/plot_data_plane
tripolar_points_IMS.nc
> tripolar_points_IMS.ps 'name="ice_coverage"; level="(*,*)";'*
>
> On Mon, May 18, 2020 at 8:32 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> > Thank you for the update, John. Can you share your scripts/code to
me? I
> > guess my data won't take so long because there is not so much data
> inside.
> > Do you have an estimate how long it will take for the new
development?
> >
> > I have three sets of satellite data, except the previous two, I
also need
> > to work on the third set below, is this one easier to modify to be
read
> in
> > by MET?
> >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
> > geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
> >
> >
> > Binyu
> >
> > On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > I ran into a somewhat similar situation last week. We have data
on a
> > > tri-polar grid which is not supported in MET. So I did the
following
> > steps:
> > > (1) Write a python script to reformat the tri-polar data as if
it were
> > > lat/lon point observations.
> > > (2) Run that script using ascii2nc to reformat the data into
MET's
> NetCDF
> > > point observation format.
> > > (3) Run that NetCDF point obs file through the point2grid tool
to
> > > interpolate the data to a regular grids.
> > >
> > > And that process did (eventually) work, but it took well over 3
hours
> for
> > > all those steps since the input data consists of over 6 million
points!
> > So
> > > not very realistic! We need to figure out how to get lat/lon
data into
> > the
> > > point2grid logic much more efficiently. And I think that same
solution
> > > would work well for your data. But unfortunately, that would
require
> new
> > > development.
> > >
> > > John
> > >
> > > On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
>
> > > >
> > > > Hello,
> > > >
> > > > Thank you John.
> > > >
> > > > I look forward to Howard's response.
> > > >
> > > > Binyu
> > > >
> > > >
> > > >
> > > > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > OK, I logged onto WCOSS and retrieved one of these sample
files.
> > > > >
> > > > > I used ncview to display the latitude and longitude variable
values
> > and
> > > > > have attached a screenshot.
> > > > >
> > > > > This does not look like gridded data to me. Instead, it's a
view of
> > > > > satellite pixels. MET currently does evaluations on gridded
data,
> not
> > > > > satellite pixels. So the question is if/how we can get this
> satellite
> > > > data
> > > > > onto a grid.
> > > > >
> > > > > MET does include a tool named point2grid which is intended
to do
> > > exactly
> > > > > that. But it's pretty limited on the input files it can
accept.
> > Howard
> > > > Soh,
> > > > > cc'ed here, is the developer who's worked most on that tool.
> > > > >
> > > > > Howard, do you have any suggestions on how Binyu could
interpolate
> > the
> > > > data
> > > > > from this NetCDF file onto a grid?
> > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: read in Satellite data
> > > > > > Owner: Nobody
> > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > > > >
> > > > > >
> > > > > >
> > > > > > Hello John,
> > > > > >
> > > > > > Thank you for helping me with reading in the satellite
data using
> > > MET.
> > > > I
> > > > > > have another set of satellite data which also needs to be
read in
> > by
> > > > MET
> > > > > > for verification. Those two datasets are different but
similar,
> so
> > I
> > > > > think
> > > > > > you are the better helper to ask. The datasets are located
below.
> > > > > > "ash_mass" is the variable needed to be verified.
> > > > > >
> > > > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > > > >
> > > > > > Thank you.
> > > > > > Binyu
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Thu Jun 04 17:24:33 2020
On hera, I ran:
*module load intel anaconda/latest met/9.0_anacondal*
*point2grid
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
G212 test.nc <http://test.nc> -field
'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";' -qc
0,1,2
-method MAX*
ERROR :
ERROR : process_data_file() ->
"/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
not a valid data file
ERROR :
So then I looked in the MET User's Guide at section 4.5 about the
Point2Grid tool. See attached screenshot.
This tool processes GOES16/17 data in NetCDF format. You have passed
an HDF
file as input. That's the explanation for the error message you're
seeing.
Is this GOES-16 data also available in NetCDF format?
John
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Fri Jun 05 08:49:34 2020
I don't think so. What will be your suggestion then? Convert GOES16
HDF to
GOES NetCDF? Thank you.
Binyu
On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> On hera, I ran:
>
>
> *module load intel anaconda/latest met/9.0_anacondal*
> *point2grid
>
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> G212 test.nc <http://test.nc> -field
> 'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";' -qc
0,1,2
> -method MAX*
>
> ERROR :
> ERROR : process_data_file() ->
>
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> not a valid data file
> ERROR :
>
> So then I looked in the MET User's Guide at section 4.5 about the
> Point2Grid tool. See attached screenshot.
>
> This tool processes GOES16/17 data in NetCDF format. You have passed
an HDF
> file as input. That's the explanation for the error message you're
seeing.
>
> Is this GOES-16 data also available in NetCDF format?
>
> John
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Fri Jun 05 12:16:32 2020
Binyu,
I'd recommend asking Ho-Chun Huang for information on GOES-16 data
sources.
I believe he worked with Howard Soh to test the initial support for
these
datasets. And he would likely have more information on where the
NetCDF
files came from.
Thanks,
John
On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
> I don't think so. What will be your suggestion then? Convert GOES16
HDF to
> GOES NetCDF? Thank you.
>
> Binyu
>
> On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > On hera, I ran:
> >
> >
> > *module load intel anaconda/latest met/9.0_anacondal*
> > *point2grid
> >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > G212 test.nc <http://test.nc> -field
> > 'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";'
-qc 0,1,2
> > -method MAX*
> >
> > ERROR :
> > ERROR : process_data_file() ->
> >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > not a valid data file
> > ERROR :
> >
> > So then I looked in the MET User's Guide at section 4.5 about the
> > Point2Grid tool. See attached screenshot.
> >
> > This tool processes GOES16/17 data in NetCDF format. You have
passed an
> HDF
> > file as input. That's the explanation for the error message you're
> seeing.
> >
> > Is this GOES-16 data also available in NetCDF format?
> >
> > John
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Fri Jun 05 16:11:15 2020
John,
In case I can not get GOES16 NetCDF files, we convert HDF to NetCDF,
but
MET can not read it as I expected. What should be added to the NetCDF
obtained from GOES-16 HDF files?
Thank you.
Binyu
On Fri, Jun 5, 2020 at 2:16 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I'd recommend asking Ho-Chun Huang for information on GOES-16 data
sources.
> I believe he worked with Howard Soh to test the initial support for
these
> datasets. And he would likely have more information on where the
NetCDF
> files came from.
>
> Thanks,
> John
>
> On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> > I don't think so. What will be your suggestion then? Convert
GOES16 HDF
> to
> > GOES NetCDF? Thank you.
> >
> > Binyu
> >
> > On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > On hera, I ran:
> > >
> > >
> > > *module load intel anaconda/latest met/9.0_anacondal*
> > > *point2grid
> > >
> > >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > > G212 test.nc <http://test.nc> -field
> > > 'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";'
-qc
> 0,1,2
> > > -method MAX*
> > >
> > > ERROR :
> > > ERROR : process_data_file() ->
> > >
> > >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > > not a valid data file
> > > ERROR :
> > >
> > > So then I looked in the MET User's Guide at section 4.5 about
the
> > > Point2Grid tool. See attached screenshot.
> > >
> > > This tool processes GOES16/17 data in NetCDF format. You have
passed an
> > HDF
> > > file as input. That's the explanation for the error message
you're
> > seeing.
> > >
> > > Is this GOES-16 data also available in NetCDF format?
> > >
> > > John
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Fri Jun 05 16:23:17 2020
Binyu,
I do not know, and I would not recommend trying to write a converter
yourself. There is no easy simple thing to be added to the NetCDF file
to make it readable by MET. Instead, Howard Soh worked with Ho-Chun
over
several weeks to get all the details straight. So I'd recommend
figuring
out how those other NetCDF GOES16 files were created and following a
similar path.
The only other alternative would be to write a python script yourself
to
read the HDF data, interpolate those pixels onto a regular grid, and
pass
that data in memory using MET's python-embedding functionality. But
that
would require a significant level of effort. You can find example
python
scripts on the MET website (
https://dtcenter.org/community-code/model-evaluation-tools-met/sample-
analysis-scripts).
These sample scripts just read data from one format and serve them up
to
MET. But none of them do the step of interpolating point data to a
regular
grid. Perhaps there are existing Python packages that could help you
with
that conversion, but I really don't know.
John
On Fri, Jun 5, 2020 at 4:11 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>
> John,
>
> In case I can not get GOES16 NetCDF files, we convert HDF to
NetCDF, but
> MET can not read it as I expected. What should be added to the
NetCDF
> obtained from GOES-16 HDF files?
>
> Thank you.
> Binyu
>
> On Fri, Jun 5, 2020 at 2:16 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I'd recommend asking Ho-Chun Huang for information on GOES-16 data
> sources.
> > I believe he worked with Howard Soh to test the initial support
for these
> > datasets. And he would likely have more information on where the
NetCDF
> > files came from.
> >
> > Thanks,
> > John
> >
> > On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > I don't think so. What will be your suggestion then? Convert
GOES16 HDF
> > to
> > > GOES NetCDF? Thank you.
> > >
> > > Binyu
> > >
> > > On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > On hera, I ran:
> > > >
> > > >
> > > > *module load intel anaconda/latest met/9.0_anacondal*
> > > > *point2grid
> > > >
> > > >
> > >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > > > G212 test.nc <http://test.nc> -field
> > > > 'name="vcat_ashprop_13_15_16_ash_mass_loading";
level="(*,*)";' -qc
> > 0,1,2
> > > > -method MAX*
> > > >
> > > > ERROR :
> > > > ERROR : process_data_file() ->
> > > >
> > > >
> > >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > > > not a valid data file
> > > > ERROR :
> > > >
> > > > So then I looked in the MET User's Guide at section 4.5 about
the
> > > > Point2Grid tool. See attached screenshot.
> > > >
> > > > This tool processes GOES16/17 data in NetCDF format. You have
passed
> an
> > > HDF
> > > > file as input. That's the explanation for the error message
you're
> > > seeing.
> > > >
> > > > Is this GOES-16 data also available in NetCDF format?
> > > >
> > > > John
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Fri Jun 05 21:20:22 2020
Thank you, John. I will look for the NetCDF files as suggested.
Binyu
On Fri, Jun 5, 2020 at 6:23 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I do not know, and I would not recommend trying to write a converter
> yourself. There is no easy simple thing to be added to the NetCDF
file
> to make it readable by MET. Instead, Howard Soh worked with Ho-Chun
over
> several weeks to get all the details straight. So I'd recommend
figuring
> out how those other NetCDF GOES16 files were created and following a
> similar path.
>
> The only other alternative would be to write a python script
yourself to
> read the HDF data, interpolate those pixels onto a regular grid, and
pass
> that data in memory using MET's python-embedding functionality. But
that
> would require a significant level of effort. You can find example
python
> scripts on the MET website (
>
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> ).
> These sample scripts just read data from one format and serve them
up to
> MET. But none of them do the step of interpolating point data to a
regular
> grid. Perhaps there are existing Python packages that could help you
with
> that conversion, but I really don't know.
>
> John
>
> On Fri, Jun 5, 2020 at 4:11 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> >
> > John,
> >
> > In case I can not get GOES16 NetCDF files, we convert HDF to
NetCDF, but
> > MET can not read it as I expected. What should be added to the
NetCDF
> > obtained from GOES-16 HDF files?
> >
> > Thank you.
> > Binyu
> >
> > On Fri, Jun 5, 2020 at 2:16 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > I'd recommend asking Ho-Chun Huang for information on GOES-16
data
> > sources.
> > > I believe he worked with Howard Soh to test the initial support
for
> these
> > > datasets. And he would likely have more information on where the
NetCDF
> > > files came from.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > I don't think so. What will be your suggestion then? Convert
GOES16
> HDF
> > > to
> > > > GOES NetCDF? Thank you.
> > > >
> > > > Binyu
> > > >
> > > > On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > On hera, I ran:
> > > > >
> > > > >
> > > > > *module load intel anaconda/latest met/9.0_anacondal*
> > > > > *point2grid
> > > > >
> > > > >
> > > >
> > >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > > > > G212 test.nc <http://test.nc> -field
> > > > > 'name="vcat_ashprop_13_15_16_ash_mass_loading";
level="(*,*)";' -qc
> > > 0,1,2
> > > > > -method MAX*
> > > > >
> > > > > ERROR :
> > > > > ERROR : process_data_file() ->
> > > > >
> > > > >
> > > >
> > >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > > > > not a valid data file
> > > > > ERROR :
> > > > >
> > > > > So then I looked in the MET User's Guide at section 4.5
about the
> > > > > Point2Grid tool. See attached screenshot.
> > > > >
> > > > > This tool processes GOES16/17 data in NetCDF format. You
have
> passed
> > an
> > > > HDF
> > > > > file as input. That's the explanation for the error message
you're
> > > > seeing.
> > > > >
> > > > > Is this GOES-16 data also available in NetCDF format?
> > > > >
> > > > > John
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Mon Jun 08 12:48:23 2020
Binyu,
Sounds good. I'll go ahead and resolve this ticket for now. But if
issues
arise in your use of these NetCDF files, feel free to write a new
question
to met-help.
Thanks,
John
On Fri, Jun 5, 2020 at 9:20 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
> Thank you, John. I will look for the NetCDF files as suggested.
>
> Binyu
>
>
> On Fri, Jun 5, 2020 at 6:23 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I do not know, and I would not recommend trying to write a
converter
> > yourself. There is no easy simple thing to be added to the NetCDF
file
> > to make it readable by MET. Instead, Howard Soh worked with Ho-
Chun over
> > several weeks to get all the details straight. So I'd recommend
figuring
> > out how those other NetCDF GOES16 files were created and following
a
> > similar path.
> >
> > The only other alternative would be to write a python script
yourself to
> > read the HDF data, interpolate those pixels onto a regular grid,
and pass
> > that data in memory using MET's python-embedding functionality.
But that
> > would require a significant level of effort. You can find example
python
> > scripts on the MET website (
> >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> > ).
> > These sample scripts just read data from one format and serve them
up to
> > MET. But none of them do the step of interpolating point data to a
> regular
> > grid. Perhaps there are existing Python packages that could help
you with
> > that conversion, but I really don't know.
> >
> > John
> >
> > On Fri, Jun 5, 2020 at 4:11 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> > >
> > > John,
> > >
> > > In case I can not get GOES16 NetCDF files, we convert HDF to
NetCDF,
> but
> > > MET can not read it as I expected. What should be added to the
NetCDF
> > > obtained from GOES-16 HDF files?
> > >
> > > Thank you.
> > > Binyu
> > >
> > > On Fri, Jun 5, 2020 at 2:16 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > I'd recommend asking Ho-Chun Huang for information on GOES-16
data
> > > sources.
> > > > I believe he worked with Howard Soh to test the initial
support for
> > these
> > > > datasets. And he would likely have more information on where
the
> NetCDF
> > > > files came from.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > I don't think so. What will be your suggestion then? Convert
GOES16
> > HDF
> > > > to
> > > > > GOES NetCDF? Thank you.
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > On hera, I ran:
> > > > > >
> > > > > >
> > > > > > *module load intel anaconda/latest met/9.0_anacondal*
> > > > > > *point2grid
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > > > > > G212 test.nc <http://test.nc> -field
> > > > > > 'name="vcat_ashprop_13_15_16_ash_mass_loading";
level="(*,*)";'
> -qc
> > > > 0,1,2
> > > > > > -method MAX*
> > > > > >
> > > > > > ERROR :
> > > > > > ERROR : process_data_file() ->
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > > > > > not a valid data file
> > > > > > ERROR :
> > > > > >
> > > > > > So then I looked in the MET User's Guide at section 4.5
about the
> > > > > > Point2Grid tool. See attached screenshot.
> > > > > >
> > > > > > This tool processes GOES16/17 data in NetCDF format. You
have
> > passed
> > > an
> > > > > HDF
> > > > > > file as input. That's the explanation for the error
message
> you're
> > > > > seeing.
> > > > > >
> > > > > > Is this GOES-16 data also available in NetCDF format?
> > > > > >
> > > > > > John
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list