[Met_help] [rt.rap.ucar.edu #95225] History for read in Satellite data
John Halley Gotway via RT
met_help at ucar.edu
Mon Jul 20 10:52:30 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello John,
Thank you for helping me with reading in the satellite data using MET. I
have another set of satellite data which also needs to be read in by MET
for verification. Those two datasets are different but similar, so I think
you are the better helper to ask. The datasets are located below.
"ash_mass" is the variable needed to be verified.
/gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
Thank you.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Mon May 11 17:40:22 2020
Binyu,
OK, I logged onto WCOSS and retrieved one of these sample files.
I used ncview to display the latitude and longitude variable values
and
have attached a screenshot.
This does not look like gridded data to me. Instead, it's a view of
satellite pixels. MET currently does evaluations on gridded data, not
satellite pixels. So the question is if/how we can get this satellite
data
onto a grid.
MET does include a tool named point2grid which is intended to do
exactly
that. But it's pretty limited on the input files it can accept. Howard
Soh,
cc'ed here, is the developer who's worked most on that tool.
Howard, do you have any suggestions on how Binyu could interpolate the
data
from this NetCDF file onto a grid?
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
Thanks,
John
On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: read in Satellite data
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>
>
> Hello John,
>
> Thank you for helping me with reading in the satellite data using
MET. I
> have another set of satellite data which also needs to be read in by
MET
> for verification. Those two datasets are different but similar, so I
think
> you are the better helper to ask. The datasets are located below.
> "ash_mass" is the variable needed to be verified.
>
> /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
>
> Thank you.
> Binyu
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Mon May 18 13:35:15 2020
Hello,
Thank you John.
I look forward to Howard's response.
Binyu
On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> OK, I logged onto WCOSS and retrieved one of these sample files.
>
> I used ncview to display the latitude and longitude variable values
and
> have attached a screenshot.
>
> This does not look like gridded data to me. Instead, it's a view of
> satellite pixels. MET currently does evaluations on gridded data,
not
> satellite pixels. So the question is if/how we can get this
satellite data
> onto a grid.
>
> MET does include a tool named point2grid which is intended to do
exactly
> that. But it's pretty limited on the input files it can accept.
Howard Soh,
> cc'ed here, is the developer who's worked most on that tool.
>
> Howard, do you have any suggestions on how Binyu could interpolate
the data
> from this NetCDF file onto a grid?
>
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
>
> Thanks,
> John
>
>
> On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > Transaction: Ticket created by binyu.wang at noaa.gov
> > Queue: met_help
> > Subject: read in Satellite data
> > Owner: Nobody
> > Requestors: binyu.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> >
> >
> > Hello John,
> >
> > Thank you for helping me with reading in the satellite data using
MET. I
> > have another set of satellite data which also needs to be read in
by MET
> > for verification. Those two datasets are different but similar, so
I
> think
> > you are the better helper to ask. The datasets are located below.
> > "ash_mass" is the variable needed to be verified.
> >
> > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> >
> > Thank you.
> > Binyu
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Mon May 18 13:55:05 2020
Binyu,
I ran into a somewhat similar situation last week. We have data on a
tri-polar grid which is not supported in MET. So I did the following
steps:
(1) Write a python script to reformat the tri-polar data as if it were
lat/lon point observations.
(2) Run that script using ascii2nc to reformat the data into MET's
NetCDF
point observation format.
(3) Run that NetCDF point obs file through the point2grid tool to
interpolate the data to a regular grids.
And that process did (eventually) work, but it took well over 3 hours
for
all those steps since the input data consists of over 6 million
points! So
not very realistic! We need to figure out how to get lat/lon data into
the
point2grid logic much more efficiently. And I think that same solution
would work well for your data. But unfortunately, that would require
new
development.
John
On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>
> Hello,
>
> Thank you John.
>
> I look forward to Howard's response.
>
> Binyu
>
>
>
> On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > OK, I logged onto WCOSS and retrieved one of these sample files.
> >
> > I used ncview to display the latitude and longitude variable
values and
> > have attached a screenshot.
> >
> > This does not look like gridded data to me. Instead, it's a view
of
> > satellite pixels. MET currently does evaluations on gridded data,
not
> > satellite pixels. So the question is if/how we can get this
satellite
> data
> > onto a grid.
> >
> > MET does include a tool named point2grid which is intended to do
exactly
> > that. But it's pretty limited on the input files it can accept.
Howard
> Soh,
> > cc'ed here, is the developer who's worked most on that tool.
> >
> > Howard, do you have any suggestions on how Binyu could interpolate
the
> data
> > from this NetCDF file onto a grid?
> >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> >
> > Thanks,
> > John
> >
> >
> > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > Queue: met_help
> > > Subject: read in Satellite data
> > > Owner: Nobody
> > > Requestors: binyu.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> >
> > >
> > >
> > > Hello John,
> > >
> > > Thank you for helping me with reading in the satellite data
using MET.
> I
> > > have another set of satellite data which also needs to be read
in by
> MET
> > > for verification. Those two datasets are different but similar,
so I
> > think
> > > you are the better helper to ask. The datasets are located
below.
> > > "ash_mass" is the variable needed to be verified.
> > >
> > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > >
> > > Thank you.
> > > Binyu
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Mon May 18 20:31:39 2020
Thank you for the update, John. Can you share your scripts/code to me?
I
guess my data won't take so long because there is not so much data
inside.
Do you have an estimate how long it will take for the new development?
I have three sets of satellite data, except the previous two, I also
need
to work on the third set below, is this one easier to modify to be
read in
by MET?
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
Binyu
On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I ran into a somewhat similar situation last week. We have data on a
> tri-polar grid which is not supported in MET. So I did the following
steps:
> (1) Write a python script to reformat the tri-polar data as if it
were
> lat/lon point observations.
> (2) Run that script using ascii2nc to reformat the data into MET's
NetCDF
> point observation format.
> (3) Run that NetCDF point obs file through the point2grid tool to
> interpolate the data to a regular grids.
>
> And that process did (eventually) work, but it took well over 3
hours for
> all those steps since the input data consists of over 6 million
points! So
> not very realistic! We need to figure out how to get lat/lon data
into the
> point2grid logic much more efficiently. And I think that same
solution
> would work well for your data. But unfortunately, that would require
new
> development.
>
> John
>
> On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> >
> > Hello,
> >
> > Thank you John.
> >
> > I look forward to Howard's response.
> >
> > Binyu
> >
> >
> >
> > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > OK, I logged onto WCOSS and retrieved one of these sample files.
> > >
> > > I used ncview to display the latitude and longitude variable
values and
> > > have attached a screenshot.
> > >
> > > This does not look like gridded data to me. Instead, it's a view
of
> > > satellite pixels. MET currently does evaluations on gridded
data, not
> > > satellite pixels. So the question is if/how we can get this
satellite
> > data
> > > onto a grid.
> > >
> > > MET does include a tool named point2grid which is intended to do
> exactly
> > > that. But it's pretty limited on the input files it can accept.
Howard
> > Soh,
> > > cc'ed here, is the developer who's worked most on that tool.
> > >
> > > Howard, do you have any suggestions on how Binyu could
interpolate the
> > data
> > > from this NetCDF file onto a grid?
> > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > Queue: met_help
> > > > Subject: read in Satellite data
> > > > Owner: Nobody
> > > > Requestors: binyu.wang at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > >
> > > >
> > > >
> > > > Hello John,
> > > >
> > > > Thank you for helping me with reading in the satellite data
using
> MET.
> > I
> > > > have another set of satellite data which also needs to be read
in by
> > MET
> > > > for verification. Those two datasets are different but
similar, so I
> > > think
> > > > you are the better helper to ask. The datasets are located
below.
> > > > "ash_mass" is the variable needed to be verified.
> > > >
> > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > >
> > > > Thank you.
> > > > Binyu
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Tue May 19 09:50:22 2020
Binyu,
The point2grid tool already includes direct support for processing
GOES-16
satellite data.
Please take a look at chapter 4.5 of the MET User's Guide and try
passing
this data directly to point2grid:
https://dtcenter.org/sites/default/files/community-code/met/docs/user-
guide/MET_Users_Guide_v9.0.1.pdf
This functionality was developed using GOES16 aerosol data so it may
or may
not work. If not, please let me know, and I'll grab that file and ask
our
developer, Howard Soh, to take a look.
Listed below are the steps I took to process tri-polar grid data as
individual points. But this all took over 3 hours. I don't have an
estimate
yet for enhancing point2grid to process this data more directly. We're
discussing internally but do so how it'd be useful for at least 2
projects.
FYI, I posted the input data for the commands listed below to:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/for_binyu_data/
-------
I ran ascii2nc as we discussed and would say that this is not really a
viable option. Processing 6,232,499 grid points as individual point
observations just takes way too long!
I ran the following commands on kiowa.
*> cd /d1/projects/SeaIce/tripolar*
*> export MET_PYTHON_EXE=/usr/local/python3/bin/python*
*> time /usr/local/met-9.0.1/bin/ascii2nc 'tripolar_to_lat_lon.py
input/20200127/rtofs_glo_2ds_n048_daily_diag.nc
<http://rtofs_glo_2ds_n048_daily_diag.nc/> ice_coverage
north' tripolar_points.nc <http://tripolar_points.nc/> -format python*
This takes a whopping *2 hours and 25 minutes* to run ascii2nc on a
single
field of data:
(1) Read the data and reformat the 6.2 million values into a list of
lists.
(2) Write the result to a temporary pickle file.
(3) Read that pickle file into memory.
(4) Write those point obs back out to a 286MB NetCDF file.
By comparison, the input rtofs data file, with multiple variables, is
154MB
in size.
Next, run point2grid on that NetCDF file to regrid to the same Polar
Stereographic
grid as the IMS data, and this step took 55 minutes:
*> time /usr/local/met-9.0.1/bin/point2grid tripolar_points.nc
<http://tripolar_points.nc/> imssnow96.20190201.grb.grib2
tripolar_points_IMS.nc -field 'name="ice_coverage"; level="L0";'
-method
MAX*
Then plot the result:
*> time /usr/local/met-9.0.1/bin/plot_data_plane
tripolar_points_IMS.nc
tripolar_points_IMS.ps 'name="ice_coverage"; level="(*,*)";'*
On Mon, May 18, 2020 at 8:32 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
> Thank you for the update, John. Can you share your scripts/code to
me? I
> guess my data won't take so long because there is not so much data
inside.
> Do you have an estimate how long it will take for the new
development?
>
> I have three sets of satellite data, except the previous two, I also
need
> to work on the third set below, is this one easier to modify to be
read in
> by MET?
>
> /gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
> geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
>
>
> Binyu
>
> On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I ran into a somewhat similar situation last week. We have data on
a
> > tri-polar grid which is not supported in MET. So I did the
following
> steps:
> > (1) Write a python script to reformat the tri-polar data as if it
were
> > lat/lon point observations.
> > (2) Run that script using ascii2nc to reformat the data into MET's
NetCDF
> > point observation format.
> > (3) Run that NetCDF point obs file through the point2grid tool to
> > interpolate the data to a regular grids.
> >
> > And that process did (eventually) work, but it took well over 3
hours for
> > all those steps since the input data consists of over 6 million
points!
> So
> > not very realistic! We need to figure out how to get lat/lon data
into
> the
> > point2grid logic much more efficiently. And I think that same
solution
> > would work well for your data. But unfortunately, that would
require new
> > development.
> >
> > John
> >
> > On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> > >
> > > Hello,
> > >
> > > Thank you John.
> > >
> > > I look forward to Howard's response.
> > >
> > > Binyu
> > >
> > >
> > >
> > > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > OK, I logged onto WCOSS and retrieved one of these sample
files.
> > > >
> > > > I used ncview to display the latitude and longitude variable
values
> and
> > > > have attached a screenshot.
> > > >
> > > > This does not look like gridded data to me. Instead, it's a
view of
> > > > satellite pixels. MET currently does evaluations on gridded
data, not
> > > > satellite pixels. So the question is if/how we can get this
satellite
> > > data
> > > > onto a grid.
> > > >
> > > > MET does include a tool named point2grid which is intended to
do
> > exactly
> > > > that. But it's pretty limited on the input files it can
accept.
> Howard
> > > Soh,
> > > > cc'ed here, is the developer who's worked most on that tool.
> > > >
> > > > Howard, do you have any suggestions on how Binyu could
interpolate
> the
> > > data
> > > > from this NetCDF file onto a grid?
> > > >
> > > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: read in Satellite data
> > > > > Owner: Nobody
> > > > > Requestors: binyu.wang at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > > >
> > > > >
> > > > >
> > > > > Hello John,
> > > > >
> > > > > Thank you for helping me with reading in the satellite data
using
> > MET.
> > > I
> > > > > have another set of satellite data which also needs to be
read in
> by
> > > MET
> > > > > for verification. Those two datasets are different but
similar, so
> I
> > > > think
> > > > > you are the better helper to ask. The datasets are located
below.
> > > > > "ash_mass" is the variable needed to be verified.
> > > > >
> > > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > > >
> > > > > Thank you.
> > > > > Binyu
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Thu Jun 04 15:49:33 2020
Hello John,
I tried point2grid; but it didn't work. Here is what I did ( Due to
an
account issue, I am working on Hera now) :
#! /bin/sh
DIR=/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado
point2grid \
$DIR/geocatL2.GOES-16.Full_Disk.2019056.150030.hdf \
G212 \
test.nc \
-field 'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";'
\
-qc 0,1,2
-method MAX -v 1
Actually I am not sure what I should put as "to_grid argument"? Should
I
just use one of my forecasting members (grib2 format)? My forecasting
files
are under:
/scratch2/NCEPDEV/naqfc/Binyu.Wang/VA_ense/Out_Reven.21.0p25
Thank you.
Binyu
On Tue, May 19, 2020 at 11:51 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Binyu,
>
> The point2grid tool already includes direct support for processing
GOES-16
> satellite data.
>
> Please take a look at chapter 4.5 of the MET User's Guide and try
passing
> this data directly to point2grid:
>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.1.pdf
>
> This functionality was developed using GOES16 aerosol data so it may
or may
> not work. If not, please let me know, and I'll grab that file and
ask our
> developer, Howard Soh, to take a look.
>
> Listed below are the steps I took to process tri-polar grid data as
> individual points. But this all took over 3 hours. I don't have an
estimate
> yet for enhancing point2grid to process this data more directly.
We're
> discussing internally but do so how it'd be useful for at least 2
projects.
>
> FYI, I posted the input data for the commands listed below to:
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/for_binyu_data/
>
> -------
>
> I ran ascii2nc as we discussed and would say that this is not really
a
> viable option. Processing 6,232,499 grid points as individual point
> observations just takes way too long!
>
> I ran the following commands on kiowa.
>
> *> cd /d1/projects/SeaIce/tripolar*
> *> export MET_PYTHON_EXE=/usr/local/python3/bin/python*
> *> time /usr/local/met-9.0.1/bin/ascii2nc 'tripolar_to_lat_lon.py
> input/20200127/rtofs_glo_2ds_n048_daily_diag.nc
> <http://rtofs_glo_2ds_n048_daily_diag.nc/> ice_coverage
> north' tripolar_points.nc <http://tripolar_points.nc/> -format
python*
>
> This takes a whopping *2 hours and 25 minutes* to run ascii2nc on a
single
> field of data:
> (1) Read the data and reformat the 6.2 million values into a list of
lists.
> (2) Write the result to a temporary pickle file.
> (3) Read that pickle file into memory.
> (4) Write those point obs back out to a 286MB NetCDF file.
> By comparison, the input rtofs data file, with multiple variables,
is 154MB
> in size.
>
> Next, run point2grid on that NetCDF file to regrid to the same Polar
> Stereographic
> grid as the IMS data, and this step took 55 minutes:
> *> time /usr/local/met-9.0.1/bin/point2grid tripolar_points.nc
> <http://tripolar_points.nc/> imssnow96.20190201.grb.grib2
> tripolar_points_IMS.nc -field 'name="ice_coverage"; level="L0";'
-method
> MAX*
>
> Then plot the result:
> *> time /usr/local/met-9.0.1/bin/plot_data_plane
tripolar_points_IMS.nc
> tripolar_points_IMS.ps 'name="ice_coverage"; level="(*,*)";'*
>
> On Mon, May 18, 2020 at 8:32 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> > Thank you for the update, John. Can you share your scripts/code to
me? I
> > guess my data won't take so long because there is not so much data
> inside.
> > Do you have an estimate how long it will take for the new
development?
> >
> > I have three sets of satellite data, except the previous two, I
also need
> > to work on the third set below, is this one easier to modify to be
read
> in
> > by MET?
> >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
> > geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
> >
> >
> > Binyu
> >
> > On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > I ran into a somewhat similar situation last week. We have data
on a
> > > tri-polar grid which is not supported in MET. So I did the
following
> > steps:
> > > (1) Write a python script to reformat the tri-polar data as if
it were
> > > lat/lon point observations.
> > > (2) Run that script using ascii2nc to reformat the data into
MET's
> NetCDF
> > > point observation format.
> > > (3) Run that NetCDF point obs file through the point2grid tool
to
> > > interpolate the data to a regular grids.
> > >
> > > And that process did (eventually) work, but it took well over 3
hours
> for
> > > all those steps since the input data consists of over 6 million
points!
> > So
> > > not very realistic! We need to figure out how to get lat/lon
data into
> > the
> > > point2grid logic much more efficiently. And I think that same
solution
> > > would work well for your data. But unfortunately, that would
require
> new
> > > development.
> > >
> > > John
> > >
> > > On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
>
> > > >
> > > > Hello,
> > > >
> > > > Thank you John.
> > > >
> > > > I look forward to Howard's response.
> > > >
> > > > Binyu
> > > >
> > > >
> > > >
> > > > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > OK, I logged onto WCOSS and retrieved one of these sample
files.
> > > > >
> > > > > I used ncview to display the latitude and longitude variable
values
> > and
> > > > > have attached a screenshot.
> > > > >
> > > > > This does not look like gridded data to me. Instead, it's a
view of
> > > > > satellite pixels. MET currently does evaluations on gridded
data,
> not
> > > > > satellite pixels. So the question is if/how we can get this
> satellite
> > > > data
> > > > > onto a grid.
> > > > >
> > > > > MET does include a tool named point2grid which is intended
to do
> > > exactly
> > > > > that. But it's pretty limited on the input files it can
accept.
> > Howard
> > > > Soh,
> > > > > cc'ed here, is the developer who's worked most on that tool.
> > > > >
> > > > > Howard, do you have any suggestions on how Binyu could
interpolate
> > the
> > > > data
> > > > > from this NetCDF file onto a grid?
> > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: read in Satellite data
> > > > > > Owner: Nobody
> > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > > > >
> > > > > >
> > > > > >
> > > > > > Hello John,
> > > > > >
> > > > > > Thank you for helping me with reading in the satellite
data using
> > > MET.
> > > > I
> > > > > > have another set of satellite data which also needs to be
read in
> > by
> > > > MET
> > > > > > for verification. Those two datasets are different but
similar,
> so
> > I
> > > > > think
> > > > > > you are the better helper to ask. The datasets are located
below.
> > > > > > "ash_mass" is the variable needed to be verified.
> > > > > >
> > > > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > > > >
> > > > > > Thank you.
> > > > > > Binyu
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Thu Jun 04 17:24:33 2020
On hera, I ran:
*module load intel anaconda/latest met/9.0_anacondal*
*point2grid
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
G212 test.nc <http://test.nc> -field
'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";' -qc
0,1,2
-method MAX*
ERROR :
ERROR : process_data_file() ->
"/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
not a valid data file
ERROR :
So then I looked in the MET User's Guide at section 4.5 about the
Point2Grid tool. See attached screenshot.
This tool processes GOES16/17 data in NetCDF format. You have passed
an HDF
file as input. That's the explanation for the error message you're
seeing.
Is this GOES-16 data also available in NetCDF format?
John
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Fri Jun 05 08:49:34 2020
I don't think so. What will be your suggestion then? Convert GOES16
HDF to
GOES NetCDF? Thank you.
Binyu
On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> On hera, I ran:
>
>
> *module load intel anaconda/latest met/9.0_anacondal*
> *point2grid
>
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> G212 test.nc <http://test.nc> -field
> 'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";' -qc
0,1,2
> -method MAX*
>
> ERROR :
> ERROR : process_data_file() ->
>
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> not a valid data file
> ERROR :
>
> So then I looked in the MET User's Guide at section 4.5 about the
> Point2Grid tool. See attached screenshot.
>
> This tool processes GOES16/17 data in NetCDF format. You have passed
an HDF
> file as input. That's the explanation for the error message you're
seeing.
>
> Is this GOES-16 data also available in NetCDF format?
>
> John
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Fri Jun 05 12:16:32 2020
Binyu,
I'd recommend asking Ho-Chun Huang for information on GOES-16 data
sources.
I believe he worked with Howard Soh to test the initial support for
these
datasets. And he would likely have more information on where the
NetCDF
files came from.
Thanks,
John
On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
> I don't think so. What will be your suggestion then? Convert GOES16
HDF to
> GOES NetCDF? Thank you.
>
> Binyu
>
> On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > On hera, I ran:
> >
> >
> > *module load intel anaconda/latest met/9.0_anacondal*
> > *point2grid
> >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > G212 test.nc <http://test.nc> -field
> > 'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";'
-qc 0,1,2
> > -method MAX*
> >
> > ERROR :
> > ERROR : process_data_file() ->
> >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > not a valid data file
> > ERROR :
> >
> > So then I looked in the MET User's Guide at section 4.5 about the
> > Point2Grid tool. See attached screenshot.
> >
> > This tool processes GOES16/17 data in NetCDF format. You have
passed an
> HDF
> > file as input. That's the explanation for the error message you're
> seeing.
> >
> > Is this GOES-16 data also available in NetCDF format?
> >
> > John
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Fri Jun 05 16:11:15 2020
John,
In case I can not get GOES16 NetCDF files, we convert HDF to NetCDF,
but
MET can not read it as I expected. What should be added to the NetCDF
obtained from GOES-16 HDF files?
Thank you.
Binyu
On Fri, Jun 5, 2020 at 2:16 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I'd recommend asking Ho-Chun Huang for information on GOES-16 data
sources.
> I believe he worked with Howard Soh to test the initial support for
these
> datasets. And he would likely have more information on where the
NetCDF
> files came from.
>
> Thanks,
> John
>
> On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> > I don't think so. What will be your suggestion then? Convert
GOES16 HDF
> to
> > GOES NetCDF? Thank you.
> >
> > Binyu
> >
> > On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > On hera, I ran:
> > >
> > >
> > > *module load intel anaconda/latest met/9.0_anacondal*
> > > *point2grid
> > >
> > >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > > G212 test.nc <http://test.nc> -field
> > > 'name="vcat_ashprop_13_15_16_ash_mass_loading"; level="(*,*)";'
-qc
> 0,1,2
> > > -method MAX*
> > >
> > > ERROR :
> > > ERROR : process_data_file() ->
> > >
> > >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > > not a valid data file
> > > ERROR :
> > >
> > > So then I looked in the MET User's Guide at section 4.5 about
the
> > > Point2Grid tool. See attached screenshot.
> > >
> > > This tool processes GOES16/17 data in NetCDF format. You have
passed an
> > HDF
> > > file as input. That's the explanation for the error message
you're
> > seeing.
> > >
> > > Is this GOES-16 data also available in NetCDF format?
> > >
> > > John
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Fri Jun 05 16:23:17 2020
Binyu,
I do not know, and I would not recommend trying to write a converter
yourself. There is no easy simple thing to be added to the NetCDF file
to make it readable by MET. Instead, Howard Soh worked with Ho-Chun
over
several weeks to get all the details straight. So I'd recommend
figuring
out how those other NetCDF GOES16 files were created and following a
similar path.
The only other alternative would be to write a python script yourself
to
read the HDF data, interpolate those pixels onto a regular grid, and
pass
that data in memory using MET's python-embedding functionality. But
that
would require a significant level of effort. You can find example
python
scripts on the MET website (
https://dtcenter.org/community-code/model-evaluation-tools-met/sample-
analysis-scripts).
These sample scripts just read data from one format and serve them up
to
MET. But none of them do the step of interpolating point data to a
regular
grid. Perhaps there are existing Python packages that could help you
with
that conversion, but I really don't know.
John
On Fri, Jun 5, 2020 at 4:11 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>
> John,
>
> In case I can not get GOES16 NetCDF files, we convert HDF to
NetCDF, but
> MET can not read it as I expected. What should be added to the
NetCDF
> obtained from GOES-16 HDF files?
>
> Thank you.
> Binyu
>
> On Fri, Jun 5, 2020 at 2:16 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I'd recommend asking Ho-Chun Huang for information on GOES-16 data
> sources.
> > I believe he worked with Howard Soh to test the initial support
for these
> > datasets. And he would likely have more information on where the
NetCDF
> > files came from.
> >
> > Thanks,
> > John
> >
> > On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > I don't think so. What will be your suggestion then? Convert
GOES16 HDF
> > to
> > > GOES NetCDF? Thank you.
> > >
> > > Binyu
> > >
> > > On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > On hera, I ran:
> > > >
> > > >
> > > > *module load intel anaconda/latest met/9.0_anacondal*
> > > > *point2grid
> > > >
> > > >
> > >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > > > G212 test.nc <http://test.nc> -field
> > > > 'name="vcat_ashprop_13_15_16_ash_mass_loading";
level="(*,*)";' -qc
> > 0,1,2
> > > > -method MAX*
> > > >
> > > > ERROR :
> > > > ERROR : process_data_file() ->
> > > >
> > > >
> > >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > > > not a valid data file
> > > > ERROR :
> > > >
> > > > So then I looked in the MET User's Guide at section 4.5 about
the
> > > > Point2Grid tool. See attached screenshot.
> > > >
> > > > This tool processes GOES16/17 data in NetCDF format. You have
passed
> an
> > > HDF
> > > > file as input. That's the explanation for the error message
you're
> > > seeing.
> > > >
> > > > Is this GOES-16 data also available in NetCDF format?
> > > >
> > > > John
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Fri Jun 05 21:20:22 2020
Thank you, John. I will look for the NetCDF files as suggested.
Binyu
On Fri, Jun 5, 2020 at 6:23 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I do not know, and I would not recommend trying to write a converter
> yourself. There is no easy simple thing to be added to the NetCDF
file
> to make it readable by MET. Instead, Howard Soh worked with Ho-Chun
over
> several weeks to get all the details straight. So I'd recommend
figuring
> out how those other NetCDF GOES16 files were created and following a
> similar path.
>
> The only other alternative would be to write a python script
yourself to
> read the HDF data, interpolate those pixels onto a regular grid, and
pass
> that data in memory using MET's python-embedding functionality. But
that
> would require a significant level of effort. You can find example
python
> scripts on the MET website (
>
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> ).
> These sample scripts just read data from one format and serve them
up to
> MET. But none of them do the step of interpolating point data to a
regular
> grid. Perhaps there are existing Python packages that could help you
with
> that conversion, but I really don't know.
>
> John
>
> On Fri, Jun 5, 2020 at 4:11 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> >
> > John,
> >
> > In case I can not get GOES16 NetCDF files, we convert HDF to
NetCDF, but
> > MET can not read it as I expected. What should be added to the
NetCDF
> > obtained from GOES-16 HDF files?
> >
> > Thank you.
> > Binyu
> >
> > On Fri, Jun 5, 2020 at 2:16 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > I'd recommend asking Ho-Chun Huang for information on GOES-16
data
> > sources.
> > > I believe he worked with Howard Soh to test the initial support
for
> these
> > > datasets. And he would likely have more information on where the
NetCDF
> > > files came from.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > I don't think so. What will be your suggestion then? Convert
GOES16
> HDF
> > > to
> > > > GOES NetCDF? Thank you.
> > > >
> > > > Binyu
> > > >
> > > > On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > On hera, I ran:
> > > > >
> > > > >
> > > > > *module load intel anaconda/latest met/9.0_anacondal*
> > > > > *point2grid
> > > > >
> > > > >
> > > >
> > >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > > > > G212 test.nc <http://test.nc> -field
> > > > > 'name="vcat_ashprop_13_15_16_ash_mass_loading";
level="(*,*)";' -qc
> > > 0,1,2
> > > > > -method MAX*
> > > > >
> > > > > ERROR :
> > > > > ERROR : process_data_file() ->
> > > > >
> > > > >
> > > >
> > >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > > > > not a valid data file
> > > > > ERROR :
> > > > >
> > > > > So then I looked in the MET User's Guide at section 4.5
about the
> > > > > Point2Grid tool. See attached screenshot.
> > > > >
> > > > > This tool processes GOES16/17 data in NetCDF format. You
have
> passed
> > an
> > > > HDF
> > > > > file as input. That's the explanation for the error message
you're
> > > > seeing.
> > > > >
> > > > > Is this GOES-16 data also available in NetCDF format?
> > > > >
> > > > > John
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Mon Jun 08 12:48:23 2020
Binyu,
Sounds good. I'll go ahead and resolve this ticket for now. But if
issues
arise in your use of these NetCDF files, feel free to write a new
question
to met-help.
Thanks,
John
On Fri, Jun 5, 2020 at 9:20 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
> Thank you, John. I will look for the NetCDF files as suggested.
>
> Binyu
>
>
> On Fri, Jun 5, 2020 at 6:23 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I do not know, and I would not recommend trying to write a
converter
> > yourself. There is no easy simple thing to be added to the NetCDF
file
> > to make it readable by MET. Instead, Howard Soh worked with Ho-
Chun over
> > several weeks to get all the details straight. So I'd recommend
figuring
> > out how those other NetCDF GOES16 files were created and following
a
> > similar path.
> >
> > The only other alternative would be to write a python script
yourself to
> > read the HDF data, interpolate those pixels onto a regular grid,
and pass
> > that data in memory using MET's python-embedding functionality.
But that
> > would require a significant level of effort. You can find example
python
> > scripts on the MET website (
> >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> > ).
> > These sample scripts just read data from one format and serve them
up to
> > MET. But none of them do the step of interpolating point data to a
> regular
> > grid. Perhaps there are existing Python packages that could help
you with
> > that conversion, but I really don't know.
> >
> > John
> >
> > On Fri, Jun 5, 2020 at 4:11 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> > >
> > > John,
> > >
> > > In case I can not get GOES16 NetCDF files, we convert HDF to
NetCDF,
> but
> > > MET can not read it as I expected. What should be added to the
NetCDF
> > > obtained from GOES-16 HDF files?
> > >
> > > Thank you.
> > > Binyu
> > >
> > > On Fri, Jun 5, 2020 at 2:16 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > I'd recommend asking Ho-Chun Huang for information on GOES-16
data
> > > sources.
> > > > I believe he worked with Howard Soh to test the initial
support for
> > these
> > > > datasets. And he would likely have more information on where
the
> NetCDF
> > > > files came from.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Jun 5, 2020 at 8:50 AM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > I don't think so. What will be your suggestion then? Convert
GOES16
> > HDF
> > > > to
> > > > > GOES NetCDF? Thank you.
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Thu, Jun 4, 2020 at 7:24 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > On hera, I ran:
> > > > > >
> > > > > >
> > > > > > *module load intel anaconda/latest met/9.0_anacondal*
> > > > > > *point2grid
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf
> > > > > > G212 test.nc <http://test.nc> -field
> > > > > > 'name="vcat_ashprop_13_15_16_ash_mass_loading";
level="(*,*)";'
> -qc
> > > > 0,1,2
> > > > > > -method MAX*
> > > > > >
> > > > > > ERROR :
> > > > > > ERROR : process_data_file() ->
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> "/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Reventado/geocatL2.GOES-
16.Full_Disk.2019056.150030.hdf"
> > > > > > not a valid data file
> > > > > > ERROR :
> > > > > >
> > > > > > So then I looked in the MET User's Guide at section 4.5
about the
> > > > > > Point2Grid tool. See attached screenshot.
> > > > > >
> > > > > > This tool processes GOES16/17 data in NetCDF format. You
have
> > passed
> > > an
> > > > > HDF
> > > > > > file as input. That's the explanation for the error
message
> you're
> > > > > seeing.
> > > > > >
> > > > > > Is this GOES-16 data also available in NetCDF format?
> > > > > >
> > > > > > John
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Wed Jul 01 15:43:13 2020
Hello John,
Back to the Satellite data that I have, I don't expect I could get the
data
with the right format very soon. So I just follow the steps:
1.convert the satellite data to ascII
2.convert the ascII file to grid file using ascii2nc and point2grid
3.get hourly average using "cdo ensmean" (the satellite data is every
10
mins, while model is hourly average)
Fortunately I don't have a lot of valid grids, so the whole process is
pretty fast. Anyhow I did a test and used"plot_data_plane" to make a
plot,which looks reasonable. However I am not sure if I made a mistake
using the conversion:
1. I am using all default config file for both "ascii2nc and
point2grid",
is that correct?
Here is my script to do the conversion from ascII to nc:
/scratch2/NCEPDEV/naqfc/Binyu.Wang/Scripts/MET/asc2grid.sh
2.After I finished the conversion, I did the ensemble verification as
before:
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_Raikoke.sh
Then I tried to make a HIST plot (see the attachment 1). The plot
looks
weird to me because the plot indicates that model under-predict obs.
However the overlay plot shows that model prediction is close to obs
(attachment 2). What is the problem then?
Here are the warnings I got, I guess this will cause some problems.
But I
am not sure how to fix the problem.
from "run_es.log"
WARNING: NcCfFile::open() -> could not determine the valid time, using
0
from "run_gs.log"
WARNING: process_scores() -> Forecast and observation valid times do
not
match 20190623_000000 != 19700101_000000 for
VAFTD_L18000-0_ENS_FREQ_ge0.05(*,*) versus ashL0
Just in case: here is the example of original satellite obs:
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Raikoke/SCOPE_NWC_ASH-L2-
ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-193000-fv2.nc
Thank you.
On Tue, May 19, 2020 at 11:51 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Binyu,
>
> The point2grid tool already includes direct support for processing
GOES-16
> satellite data.
>
> Please take a look at chapter 4.5 of the MET User's Guide and try
passing
> this data directly to point2grid:
>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.1.pdf
>
> This functionality was developed using GOES16 aerosol data so it may
or may
> not work. If not, please let me know, and I'll grab that file and
ask our
> developer, Howard Soh, to take a look.
>
> Listed below are the steps I took to process tri-polar grid data as
> individual points. But this all took over 3 hours. I don't have an
estimate
> yet for enhancing point2grid to process this data more directly.
We're
> discussing internally but do so how it'd be useful for at least 2
projects.
>
> FYI, I posted the input data for the commands listed below to:
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/for_binyu_data/
>
> -------
>
> I ran ascii2nc as we discussed and would say that this is not really
a
> viable option. Processing 6,232,499 grid points as individual point
> observations just takes way too long!
>
> I ran the following commands on kiowa.
>
> *> cd /d1/projects/SeaIce/tripolar*
> *> export MET_PYTHON_EXE=/usr/local/python3/bin/python*
> *> time /usr/local/met-9.0.1/bin/ascii2nc 'tripolar_to_lat_lon.py
> input/20200127/rtofs_glo_2ds_n048_daily_diag.nc
> <http://rtofs_glo_2ds_n048_daily_diag.nc/> ice_coverage
> north' tripolar_points.nc <http://tripolar_points.nc/> -format
python*
>
> This takes a whopping *2 hours and 25 minutes* to run ascii2nc on a
single
> field of data:
> (1) Read the data and reformat the 6.2 million values into a list of
lists.
> (2) Write the result to a temporary pickle file.
> (3) Read that pickle file into memory.
> (4) Write those point obs back out to a 286MB NetCDF file.
> By comparison, the input rtofs data file, with multiple variables,
is 154MB
> in size.
>
> Next, run point2grid on that NetCDF file to regrid to the same Polar
> Stereographic
> grid as the IMS data, and this step took 55 minutes:
> *> time /usr/local/met-9.0.1/bin/point2grid tripolar_points.nc
> <http://tripolar_points.nc/> imssnow96.20190201.grb.grib2
> tripolar_points_IMS.nc -field 'name="ice_coverage"; level="L0";'
-method
> MAX*
>
> Then plot the result:
> *> time /usr/local/met-9.0.1/bin/plot_data_plane
tripolar_points_IMS.nc
> tripolar_points_IMS.ps 'name="ice_coverage"; level="(*,*)";'*
>
> On Mon, May 18, 2020 at 8:32 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> > Thank you for the update, John. Can you share your scripts/code to
me? I
> > guess my data won't take so long because there is not so much data
> inside.
> > Do you have an estimate how long it will take for the new
development?
> >
> > I have three sets of satellite data, except the previous two, I
also need
> > to work on the third set below, is this one easier to modify to be
read
> in
> > by MET?
> >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
> > geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
> >
> >
> > Binyu
> >
> > On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > I ran into a somewhat similar situation last week. We have data
on a
> > > tri-polar grid which is not supported in MET. So I did the
following
> > steps:
> > > (1) Write a python script to reformat the tri-polar data as if
it were
> > > lat/lon point observations.
> > > (2) Run that script using ascii2nc to reformat the data into
MET's
> NetCDF
> > > point observation format.
> > > (3) Run that NetCDF point obs file through the point2grid tool
to
> > > interpolate the data to a regular grids.
> > >
> > > And that process did (eventually) work, but it took well over 3
hours
> for
> > > all those steps since the input data consists of over 6 million
points!
> > So
> > > not very realistic! We need to figure out how to get lat/lon
data into
> > the
> > > point2grid logic much more efficiently. And I think that same
solution
> > > would work well for your data. But unfortunately, that would
require
> new
> > > development.
> > >
> > > John
> > >
> > > On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
>
> > > >
> > > > Hello,
> > > >
> > > > Thank you John.
> > > >
> > > > I look forward to Howard's response.
> > > >
> > > > Binyu
> > > >
> > > >
> > > >
> > > > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > OK, I logged onto WCOSS and retrieved one of these sample
files.
> > > > >
> > > > > I used ncview to display the latitude and longitude variable
values
> > and
> > > > > have attached a screenshot.
> > > > >
> > > > > This does not look like gridded data to me. Instead, it's a
view of
> > > > > satellite pixels. MET currently does evaluations on gridded
data,
> not
> > > > > satellite pixels. So the question is if/how we can get this
> satellite
> > > > data
> > > > > onto a grid.
> > > > >
> > > > > MET does include a tool named point2grid which is intended
to do
> > > exactly
> > > > > that. But it's pretty limited on the input files it can
accept.
> > Howard
> > > > Soh,
> > > > > cc'ed here, is the developer who's worked most on that tool.
> > > > >
> > > > > Howard, do you have any suggestions on how Binyu could
interpolate
> > the
> > > > data
> > > > > from this NetCDF file onto a grid?
> > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: read in Satellite data
> > > > > > Owner: Nobody
> > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > > > >
> > > > > >
> > > > > >
> > > > > > Hello John,
> > > > > >
> > > > > > Thank you for helping me with reading in the satellite
data using
> > > MET.
> > > > I
> > > > > > have another set of satellite data which also needs to be
read in
> > by
> > > > MET
> > > > > > for verification. Those two datasets are different but
similar,
> so
> > I
> > > > > think
> > > > > > you are the better helper to ask. The datasets are located
below.
> > > > > > "ash_mass" is the variable needed to be verified.
> > > > > >
> > > > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > > > >
> > > > > > Thank you.
> > > > > > Binyu
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Mon Jul 06 20:50:20 2020
Vinyl,
Sorry I never got a chance to take a look at this issue before leaving
for
vacation for 2 weeks.
Perhaps one of the other engineers or scientists on our team could
take a
look.
Tara or George, any thoughts?
John
On Wed, Jul 1, 2020 at 5:43 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>
> Hello John,
>
> Back to the Satellite data that I have, I don't expect I could get
the data
> with the right format very soon. So I just follow the steps:
> 1.convert the satellite data to ascII
> 2.convert the ascII file to grid file using ascii2nc and point2grid
> 3.get hourly average using "cdo ensmean" (the satellite data is
every 10
> mins, while model is hourly average)
>
> Fortunately I don't have a lot of valid grids, so the whole process
is
> pretty fast. Anyhow I did a test and used"plot_data_plane" to make a
> plot,which looks reasonable. However I am not sure if I made a
mistake
> using the conversion:
>
> 1. I am using all default config file for both "ascii2nc and
point2grid",
> is that correct?
> Here is my script to do the conversion from ascII to nc:
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/Scripts/MET/asc2grid.sh
> 2.After I finished the conversion, I did the ensemble verification
as
> before:
>
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_Raikoke.sh
>
> Then I tried to make a HIST plot (see the attachment 1). The plot
looks
> weird to me because the plot indicates that model under-predict obs.
> However the overlay plot shows that model prediction is close to obs
> (attachment 2). What is the problem then?
>
> Here are the warnings I got, I guess this will cause some problems.
But I
> am not sure how to fix the problem.
> from "run_es.log"
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0
> from "run_gs.log"
> WARNING: process_scores() -> Forecast and observation valid times do
not
> match 20190623_000000 != 19700101_000000 for
> VAFTD_L18000-0_ENS_FREQ_ge0.05(*,*) versus ashL0
>
> Just in case: here is the example of original satellite obs:
>
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Raikoke/SCOPE_NWC_ASH-L2-
ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-
> 20190624-193000-fv2.nc
>
> Thank you.
>
> On Tue, May 19, 2020 at 11:51 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Binyu,
> >
> > The point2grid tool already includes direct support for processing
> GOES-16
> > satellite data.
> >
> > Please take a look at chapter 4.5 of the MET User's Guide and try
passing
> > this data directly to point2grid:
> >
> >
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.1.pdf
> >
> > This functionality was developed using GOES16 aerosol data so it
may or
> may
> > not work. If not, please let me know, and I'll grab that file and
ask
> our
> > developer, Howard Soh, to take a look.
> >
> > Listed below are the steps I took to process tri-polar grid data
as
> > individual points. But this all took over 3 hours. I don't have an
> estimate
> > yet for enhancing point2grid to process this data more directly.
We're
> > discussing internally but do so how it'd be useful for at least 2
> projects.
> >
> > FYI, I posted the input data for the commands listed below to:
> > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/for_binyu_data/
> >
> > -------
> >
> > I ran ascii2nc as we discussed and would say that this is not
really a
> > viable option. Processing 6,232,499 grid points as individual
point
> > observations just takes way too long!
> >
> > I ran the following commands on kiowa.
> >
> > *> cd /d1/projects/SeaIce/tripolar*
> > *> export MET_PYTHON_EXE=/usr/local/python3/bin/python*
> > *> time /usr/local/met-9.0.1/bin/ascii2nc 'tripolar_to_lat_lon.py
> > input/20200127/rtofs_glo_2ds_n048_daily_diag.nc
> > <http://rtofs_glo_2ds_n048_daily_diag.nc/> ice_coverage
> > north' tripolar_points.nc <http://tripolar_points.nc/> -format
python*
> >
> > This takes a whopping *2 hours and 25 minutes* to run ascii2nc on
a
> single
> > field of data:
> > (1) Read the data and reformat the 6.2 million values into a list
of
> lists.
> > (2) Write the result to a temporary pickle file.
> > (3) Read that pickle file into memory.
> > (4) Write those point obs back out to a 286MB NetCDF file.
> > By comparison, the input rtofs data file, with multiple variables,
is
> 154MB
> > in size.
> >
> > Next, run point2grid on that NetCDF file to regrid to the same
Polar
> > Stereographic
> > grid as the IMS data, and this step took 55 minutes:
> > *> time /usr/local/met-9.0.1/bin/point2grid tripolar_points.nc
> > <http://tripolar_points.nc/> imssnow96.20190201.grb.grib2
> > tripolar_points_IMS.nc -field 'name="ice_coverage"; level="L0";'
-method
> > MAX*
> >
> > Then plot the result:
> > *> time /usr/local/met-9.0.1/bin/plot_data_plane
tripolar_points_IMS.nc
> > tripolar_points_IMS.ps 'name="ice_coverage"; level="(*,*)";'*
> >
> > On Mon, May 18, 2020 at 8:32 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Thank you for the update, John. Can you share your scripts/code
to me?
> I
> > > guess my data won't take so long because there is not so much
data
> > inside.
> > > Do you have an estimate how long it will take for the new
development?
> > >
> > > I have three sets of satellite data, except the previous two, I
also
> need
> > > to work on the third set below, is this one easier to modify to
be read
> > in
> > > by MET?
> > >
> > >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
> > > geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
> > >
> > >
> > > Binyu
> > >
> > > On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > I ran into a somewhat similar situation last week. We have
data on a
> > > > tri-polar grid which is not supported in MET. So I did the
following
> > > steps:
> > > > (1) Write a python script to reformat the tri-polar data as if
it
> were
> > > > lat/lon point observations.
> > > > (2) Run that script using ascii2nc to reformat the data into
MET's
> > NetCDF
> > > > point observation format.
> > > > (3) Run that NetCDF point obs file through the point2grid tool
to
> > > > interpolate the data to a regular grids.
> > > >
> > > > And that process did (eventually) work, but it took well over
3 hours
> > for
> > > > all those steps since the input data consists of over 6
million
> points!
> > > So
> > > > not very realistic! We need to figure out how to get lat/lon
data
> into
> > > the
> > > > point2grid logic much more efficiently. And I think that same
> solution
> > > > would work well for your data. But unfortunately, that would
require
> > new
> > > > development.
> > > >
> > > > John
> > > >
> > > > On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> > > > >
> > > > > Hello,
> > > > >
> > > > > Thank you John.
> > > > >
> > > > > I look forward to Howard's response.
> > > > >
> > > > > Binyu
> > > > >
> > > > >
> > > > >
> > > > > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Binyu,
> > > > > >
> > > > > > OK, I logged onto WCOSS and retrieved one of these sample
files.
> > > > > >
> > > > > > I used ncview to display the latitude and longitude
variable
> values
> > > and
> > > > > > have attached a screenshot.
> > > > > >
> > > > > > This does not look like gridded data to me. Instead, it's
a view
> of
> > > > > > satellite pixels. MET currently does evaluations on
gridded data,
> > not
> > > > > > satellite pixels. So the question is if/how we can get
this
> > satellite
> > > > > data
> > > > > > onto a grid.
> > > > > >
> > > > > > MET does include a tool named point2grid which is intended
to do
> > > > exactly
> > > > > > that. But it's pretty limited on the input files it can
accept.
> > > Howard
> > > > > Soh,
> > > > > > cc'ed here, is the developer who's worked most on that
tool.
> > > > > >
> > > > > > Howard, do you have any suggestions on how Binyu could
> interpolate
> > > the
> > > > > data
> > > > > > from this NetCDF file onto a grid?
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via
RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
> > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > Queue: met_help
> > > > > > > Subject: read in Satellite data
> > > > > > > Owner: Nobody
> > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hello John,
> > > > > > >
> > > > > > > Thank you for helping me with reading in the satellite
data
> using
> > > > MET.
> > > > > I
> > > > > > > have another set of satellite data which also needs to
be read
> in
> > > by
> > > > > MET
> > > > > > > for verification. Those two datasets are different but
similar,
> > so
> > > I
> > > > > > think
> > > > > > > you are the better helper to ask. The datasets are
located
> below.
> > > > > > > "ash_mass" is the variable needed to be verified.
> > > > > > >
> > > > > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > > > > >
> > > > > > > Thank you.
> > > > > > > Binyu
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Mon Jul 06 20:51:03 2020
And sorry that my phone autocorrected your name to Vinyl!
John
On Mon, Jul 6, 2020 at 10:50 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Vinyl,
>
> Sorry I never got a chance to take a look at this issue before
leaving for
> vacation for 2 weeks.
>
> Perhaps one of the other engineers or scientists on our team could
take a
> look.
>
> Tara or George, any thoughts?
>
> John
>
> On Wed, Jul 1, 2020 at 5:43 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>>
>> Hello John,
>>
>> Back to the Satellite data that I have, I don't expect I could get
the
>> data
>> with the right format very soon. So I just follow the steps:
>> 1.convert the satellite data to ascII
>> 2.convert the ascII file to grid file using ascii2nc and point2grid
>> 3.get hourly average using "cdo ensmean" (the satellite data is
every 10
>> mins, while model is hourly average)
>>
>> Fortunately I don't have a lot of valid grids, so the whole process
is
>> pretty fast. Anyhow I did a test and used"plot_data_plane" to make
a
>> plot,which looks reasonable. However I am not sure if I made a
mistake
>> using the conversion:
>>
>> 1. I am using all default config file for both "ascii2nc and
point2grid",
>> is that correct?
>> Here is my script to do the conversion from ascII to nc:
>> /scratch2/NCEPDEV/naqfc/Binyu.Wang/Scripts/MET/asc2grid.sh
>> 2.After I finished the conversion, I did the ensemble verification
as
>> before:
>>
>>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_Raikoke.sh
>>
>> Then I tried to make a HIST plot (see the attachment 1). The plot
looks
>> weird to me because the plot indicates that model under-predict
obs.
>> However the overlay plot shows that model prediction is close to
obs
>> (attachment 2). What is the problem then?
>>
>> Here are the warnings I got, I guess this will cause some problems.
But I
>> am not sure how to fix the problem.
>> from "run_es.log"
>> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0
>> from "run_gs.log"
>> WARNING: process_scores() -> Forecast and observation valid times
do not
>> match 20190623_000000 != 19700101_000000 for
>> VAFTD_L18000-0_ENS_FREQ_ge0.05(*,*) versus ashL0
>>
>> Just in case: here is the example of original satellite obs:
>>
>> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Raikoke/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-
>> 20190624-193000-fv2.nc
>>
>> Thank you.
>>
>> On Tue, May 19, 2020 at 11:51 AM John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>> > Binyu,
>> >
>> > The point2grid tool already includes direct support for
processing
>> GOES-16
>> > satellite data.
>> >
>> > Please take a look at chapter 4.5 of the MET User's Guide and try
>> passing
>> > this data directly to point2grid:
>> >
>> >
>> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.1.pdf
>> >
>> > This functionality was developed using GOES16 aerosol data so it
may or
>> may
>> > not work. If not, please let me know, and I'll grab that file
and ask
>> our
>> > developer, Howard Soh, to take a look.
>> >
>> > Listed below are the steps I took to process tri-polar grid data
as
>> > individual points. But this all took over 3 hours. I don't have
an
>> estimate
>> > yet for enhancing point2grid to process this data more directly.
We're
>> > discussing internally but do so how it'd be useful for at least 2
>> projects.
>> >
>> > FYI, I posted the input data for the commands listed below to:
>> > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/for_binyu_data/
>> >
>> > -------
>> >
>> > I ran ascii2nc as we discussed and would say that this is not
really a
>> > viable option. Processing 6,232,499 grid points as individual
point
>> > observations just takes way too long!
>> >
>> > I ran the following commands on kiowa.
>> >
>> > *> cd /d1/projects/SeaIce/tripolar*
>> > *> export MET_PYTHON_EXE=/usr/local/python3/bin/python*
>> > *> time /usr/local/met-9.0.1/bin/ascii2nc 'tripolar_to_lat_lon.py
>> > input/20200127/rtofs_glo_2ds_n048_daily_diag.nc
>> > <http://rtofs_glo_2ds_n048_daily_diag.nc/> ice_coverage
>> > north' tripolar_points.nc <http://tripolar_points.nc/> -format
python*
>> >
>> > This takes a whopping *2 hours and 25 minutes* to run ascii2nc on
a
>> single
>> > field of data:
>> > (1) Read the data and reformat the 6.2 million values into a list
of
>> lists.
>> > (2) Write the result to a temporary pickle file.
>> > (3) Read that pickle file into memory.
>> > (4) Write those point obs back out to a 286MB NetCDF file.
>> > By comparison, the input rtofs data file, with multiple
variables, is
>> 154MB
>> > in size.
>> >
>> > Next, run point2grid on that NetCDF file to regrid to the same
Polar
>> > Stereographic
>> > grid as the IMS data, and this step took 55 minutes:
>> > *> time /usr/local/met-9.0.1/bin/point2grid tripolar_points.nc
>> > <http://tripolar_points.nc/> imssnow96.20190201.grb.grib2
>> > tripolar_points_IMS.nc -field 'name="ice_coverage"; level="L0";'
-method
>> > MAX*
>> >
>> > Then plot the result:
>> > *> time /usr/local/met-9.0.1/bin/plot_data_plane
tripolar_points_IMS.nc
>> > tripolar_points_IMS.ps 'name="ice_coverage"; level="(*,*)";'*
>> >
>> > On Mon, May 18, 2020 at 8:32 PM binyu.wang at noaa.gov via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > > Thank you for the update, John. Can you share your scripts/code
to
>> me? I
>> > > guess my data won't take so long because there is not so much
data
>> > inside.
>> > > Do you have an estimate how long it will take for the new
development?
>> > >
>> > > I have three sets of satellite data, except the previous two, I
also
>> need
>> > > to work on the third set below, is this one easier to modify to
be
>> read
>> > in
>> > > by MET?
>> > >
>> > >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
>> > > geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
>> > >
>> > >
>> > > Binyu
>> > >
>> > > On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT <
>> > > met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Binyu,
>> > > >
>> > > > I ran into a somewhat similar situation last week. We have
data on a
>> > > > tri-polar grid which is not supported in MET. So I did the
following
>> > > steps:
>> > > > (1) Write a python script to reformat the tri-polar data as
if it
>> were
>> > > > lat/lon point observations.
>> > > > (2) Run that script using ascii2nc to reformat the data into
MET's
>> > NetCDF
>> > > > point observation format.
>> > > > (3) Run that NetCDF point obs file through the point2grid
tool to
>> > > > interpolate the data to a regular grids.
>> > > >
>> > > > And that process did (eventually) work, but it took well over
3
>> hours
>> > for
>> > > > all those steps since the input data consists of over 6
million
>> points!
>> > > So
>> > > > not very realistic! We need to figure out how to get lat/lon
data
>> into
>> > > the
>> > > > point2grid logic much more efficiently. And I think that same
>> solution
>> > > > would work well for your data. But unfortunately, that would
require
>> > new
>> > > > development.
>> > > >
>> > > > John
>> > > >
>> > > > On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>> > > > >
>> > > > > Hello,
>> > > > >
>> > > > > Thank you John.
>> > > > >
>> > > > > I look forward to Howard's response.
>> > > > >
>> > > > > Binyu
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT <
>> > > > > met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > > Binyu,
>> > > > > >
>> > > > > > OK, I logged onto WCOSS and retrieved one of these sample
files.
>> > > > > >
>> > > > > > I used ncview to display the latitude and longitude
variable
>> values
>> > > and
>> > > > > > have attached a screenshot.
>> > > > > >
>> > > > > > This does not look like gridded data to me. Instead, it's
a
>> view of
>> > > > > > satellite pixels. MET currently does evaluations on
gridded
>> data,
>> > not
>> > > > > > satellite pixels. So the question is if/how we can get
this
>> > satellite
>> > > > > data
>> > > > > > onto a grid.
>> > > > > >
>> > > > > > MET does include a tool named point2grid which is
intended to do
>> > > > exactly
>> > > > > > that. But it's pretty limited on the input files it can
accept.
>> > > Howard
>> > > > > Soh,
>> > > > > > cc'ed here, is the developer who's worked most on that
tool.
>> > > > > >
>> > > > > > Howard, do you have any suggestions on how Binyu could
>> interpolate
>> > > the
>> > > > > data
>> > > > > > from this NetCDF file onto a grid?
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
>> > > > > >
>> > > > > > Thanks,
>> > > > > > John
>> > > > > >
>> > > > > >
>> > > > > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via
RT <
>> > > > > > met_help at ucar.edu> wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > Mon May 11 12:41:58 2020: Request 95225 was acted upon.
>> > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
>> > > > > > > Queue: met_help
>> > > > > > > Subject: read in Satellite data
>> > > > > > > Owner: Nobody
>> > > > > > > Requestors: binyu.wang at noaa.gov
>> > > > > > > Status: new
>> > > > > > > Ticket <URL:
>> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
>> > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > Hello John,
>> > > > > > >
>> > > > > > > Thank you for helping me with reading in the satellite
data
>> using
>> > > > MET.
>> > > > > I
>> > > > > > > have another set of satellite data which also needs to
be
>> read in
>> > > by
>> > > > > MET
>> > > > > > > for verification. Those two datasets are different but
>> similar,
>> > so
>> > > I
>> > > > > > think
>> > > > > > > you are the better helper to ask. The datasets are
located
>> below.
>> > > > > > > "ash_mass" is the variable needed to be verified.
>> > > > > > >
>> > > > > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
>> > > > > > >
>> > > > > > > Thank you.
>> > > > > > > Binyu
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
------------------------------------------------
Subject: read in Satellite data
From: binyu.wang at noaa.gov
Time: Mon Jul 06 21:38:12 2020
Hello John,
I wish you had a wonderful vacation.
I didn't get any reply for the previous email. However I sent in
another
ticket about the "valid time" warning today and Howard provided
solution
for me and he created a github issue "
https://github.com/NCAR/MET/issues/1407"
But there are two things I didn't ask:
1. As you know MET can not read in the pixel data, so I had to start
from
converting that to ascII and then to *nc using point2nc utility,
fortunately it is pretty fast b/c I don't have a lot of valid grid.
However
I am not sure if this will bring some bias or error?
2. Another question is about the plots attached in the previous email.
( by
the way although I got the "valid time" warning before Howard gave me
the
solution, it seems the *stat files are correct.) I didn't understand
why
the magnitude of obs. seems close to the forecast based on the second
plot,
but the HIST indicates a series under-estimate? Is this reasonable
(because
the second plot is using the mean of ensemble members) or it indicates
I
made a mistake somewhere?
Thank you.
Binyu
On Mon, Jul 6, 2020 at 10:51 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Vinyl,
>
> Sorry I never got a chance to take a look at this issue before
leaving for
> vacation for 2 weeks.
>
> Perhaps one of the other engineers or scientists on our team could
take a
> look.
>
> Tara or George, any thoughts?
>
> John
>
> On Wed, Jul 1, 2020 at 5:43 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> >
> > Hello John,
> >
> > Back to the Satellite data that I have, I don't expect I could get
the
> data
> > with the right format very soon. So I just follow the steps:
> > 1.convert the satellite data to ascII
> > 2.convert the ascII file to grid file using ascii2nc and
point2grid
> > 3.get hourly average using "cdo ensmean" (the satellite data is
every 10
> > mins, while model is hourly average)
> >
> > Fortunately I don't have a lot of valid grids, so the whole
process is
> > pretty fast. Anyhow I did a test and used"plot_data_plane" to make
a
> > plot,which looks reasonable. However I am not sure if I made a
mistake
> > using the conversion:
> >
> > 1. I am using all default config file for both "ascii2nc and
point2grid",
> > is that correct?
> > Here is my script to do the conversion from ascII to nc:
> > /scratch2/NCEPDEV/naqfc/Binyu.Wang/Scripts/MET/asc2grid.sh
> > 2.After I finished the conversion, I did the ensemble
verification as
> > before:
> >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_Raikoke.sh
> >
> > Then I tried to make a HIST plot (see the attachment 1). The plot
looks
> > weird to me because the plot indicates that model under-predict
obs.
> > However the overlay plot shows that model prediction is close to
obs
> > (attachment 2). What is the problem then?
> >
> > Here are the warnings I got, I guess this will cause some
problems. But I
> > am not sure how to fix the problem.
> > from "run_es.log"
> > WARNING: NcCfFile::open() -> could not determine the valid time,
using 0
> > from "run_gs.log"
> > WARNING: process_scores() -> Forecast and observation valid times
do not
> > match 20190623_000000 != 19700101_000000 for
> > VAFTD_L18000-0_ENS_FREQ_ge0.05(*,*) versus ashL0
> >
> > Just in case: here is the example of original satellite obs:
> >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Raikoke/SCOPE_NWC_ASH-L2-
ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-
> > 20190624-193000-fv2.nc
> >
> > Thank you.
> >
> > On Tue, May 19, 2020 at 11:51 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Binyu,
> > >
> > > The point2grid tool already includes direct support for
processing
> > GOES-16
> > > satellite data.
> > >
> > > Please take a look at chapter 4.5 of the MET User's Guide and
try
> passing
> > > this data directly to point2grid:
> > >
> > >
> >
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.1.pdf
> > >
> > > This functionality was developed using GOES16 aerosol data so it
may or
> > may
> > > not work. If not, please let me know, and I'll grab that file
and ask
> > our
> > > developer, Howard Soh, to take a look.
> > >
> > > Listed below are the steps I took to process tri-polar grid data
as
> > > individual points. But this all took over 3 hours. I don't have
an
> > estimate
> > > yet for enhancing point2grid to process this data more directly.
We're
> > > discussing internally but do so how it'd be useful for at least
2
> > projects.
> > >
> > > FYI, I posted the input data for the commands listed below to:
> > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/for_binyu_data/
> > >
> > > -------
> > >
> > > I ran ascii2nc as we discussed and would say that this is not
really a
> > > viable option. Processing 6,232,499 grid points as individual
point
> > > observations just takes way too long!
> > >
> > > I ran the following commands on kiowa.
> > >
> > > *> cd /d1/projects/SeaIce/tripolar*
> > > *> export MET_PYTHON_EXE=/usr/local/python3/bin/python*
> > > *> time /usr/local/met-9.0.1/bin/ascii2nc
'tripolar_to_lat_lon.py
> > > input/20200127/rtofs_glo_2ds_n048_daily_diag.nc
> > > <http://rtofs_glo_2ds_n048_daily_diag.nc/> ice_coverage
> > > north' tripolar_points.nc <http://tripolar_points.nc/> -format
python*
> > >
> > > This takes a whopping *2 hours and 25 minutes* to run ascii2nc
on a
> > single
> > > field of data:
> > > (1) Read the data and reformat the 6.2 million values into a
list of
> > lists.
> > > (2) Write the result to a temporary pickle file.
> > > (3) Read that pickle file into memory.
> > > (4) Write those point obs back out to a 286MB NetCDF file.
> > > By comparison, the input rtofs data file, with multiple
variables, is
> > 154MB
> > > in size.
> > >
> > > Next, run point2grid on that NetCDF file to regrid to the same
Polar
> > > Stereographic
> > > grid as the IMS data, and this step took 55 minutes:
> > > *> time /usr/local/met-9.0.1/bin/point2grid tripolar_points.nc
> > > <http://tripolar_points.nc/> imssnow96.20190201.grb.grib2
> > > tripolar_points_IMS.nc -field 'name="ice_coverage"; level="L0";'
> -method
> > > MAX*
> > >
> > > Then plot the result:
> > > *> time /usr/local/met-9.0.1/bin/plot_data_plane
tripolar_points_IMS.nc
> > > tripolar_points_IMS.ps 'name="ice_coverage"; level="(*,*)";'*
> > >
> > > On Mon, May 18, 2020 at 8:32 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Thank you for the update, John. Can you share your
scripts/code to
> me?
> > I
> > > > guess my data won't take so long because there is not so much
data
> > > inside.
> > > > Do you have an estimate how long it will take for the new
> development?
> > > >
> > > > I have three sets of satellite data, except the previous two,
I also
> > need
> > > > to work on the third set below, is this one easier to modify
to be
> read
> > > in
> > > > by MET?
> > > >
> > > >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
> > > > geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
> > > >
> > > >
> > > > Binyu
> > > >
> > > > On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > I ran into a somewhat similar situation last week. We have
data on
> a
> > > > > tri-polar grid which is not supported in MET. So I did the
> following
> > > > steps:
> > > > > (1) Write a python script to reformat the tri-polar data as
if it
> > were
> > > > > lat/lon point observations.
> > > > > (2) Run that script using ascii2nc to reformat the data into
MET's
> > > NetCDF
> > > > > point observation format.
> > > > > (3) Run that NetCDF point obs file through the point2grid
tool to
> > > > > interpolate the data to a regular grids.
> > > > >
> > > > > And that process did (eventually) work, but it took well
over 3
> hours
> > > for
> > > > > all those steps since the input data consists of over 6
million
> > points!
> > > > So
> > > > > not very realistic! We need to figure out how to get lat/lon
data
> > into
> > > > the
> > > > > point2grid logic much more efficiently. And I think that
same
> > solution
> > > > > would work well for your data. But unfortunately, that would
> require
> > > new
> > > > > development.
> > > > >
> > > > > John
> > > > >
> > > > > On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Thank you John.
> > > > > >
> > > > > > I look forward to Howard's response.
> > > > > >
> > > > > > Binyu
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Binyu,
> > > > > > >
> > > > > > > OK, I logged onto WCOSS and retrieved one of these
sample
> files.
> > > > > > >
> > > > > > > I used ncview to display the latitude and longitude
variable
> > values
> > > > and
> > > > > > > have attached a screenshot.
> > > > > > >
> > > > > > > This does not look like gridded data to me. Instead,
it's a
> view
> > of
> > > > > > > satellite pixels. MET currently does evaluations on
gridded
> data,
> > > not
> > > > > > > satellite pixels. So the question is if/how we can get
this
> > > satellite
> > > > > > data
> > > > > > > onto a grid.
> > > > > > >
> > > > > > > MET does include a tool named point2grid which is
intended to
> do
> > > > > exactly
> > > > > > > that. But it's pretty limited on the input files it can
accept.
> > > > Howard
> > > > > > Soh,
> > > > > > > cc'ed here, is the developer who's worked most on that
tool.
> > > > > > >
> > > > > > > Howard, do you have any suggestions on how Binyu could
> > interpolate
> > > > the
> > > > > > data
> > > > > > > from this NetCDF file onto a grid?
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Mon May 11 12:41:58 2020: Request 95225 was acted
upon.
> > > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > > Queue: met_help
> > > > > > > > Subject: read in Satellite data
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hello John,
> > > > > > > >
> > > > > > > > Thank you for helping me with reading in the satellite
data
> > using
> > > > > MET.
> > > > > > I
> > > > > > > > have another set of satellite data which also needs to
be
> read
> > in
> > > > by
> > > > > > MET
> > > > > > > > for verification. Those two datasets are different but
> similar,
> > > so
> > > > I
> > > > > > > think
> > > > > > > > you are the better helper to ask. The datasets are
located
> > below.
> > > > > > > > "ash_mass" is the variable needed to be verified.
> > > > > > > >
> > > > > > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > > > > > >
> > > > > > > > Thank you.
> > > > > > > > Binyu
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: read in Satellite data
From: John Halley Gotway
Time: Wed Jul 08 07:13:04 2020
Binyu,
I do see the ranked histogram that you sent. I do think it's a good
sign
that the obs value sometimes falls within the range of the forecast
values.
In the past, you had issues getting the units correct, but this ranked
histogram indicates to me that you solved that problem.
As for the processing and interpretation of your data, that's up to
you and
your colleagues. Yes, the last bar of the ranked histogram is very
high,
indicating that the obs value is often greater than all of the
forecast
values. But figuring out why that is the case is really up to you.
I'm glad you were able to use the point2grid tool to process your
data.
This is still a relatively new tool, and I don't have any advice as to
what
sort of "bias or error" it will introduce into your processing.
Our goal in supporting the MET software is to answer questions about
how
the tools work and fix the code when users find a bug. The
statisticians on
our team can give some recommendations about when certain statistics
do or
do not make sense. But interpreting the results for your specific
dataset
is really up to you and your team.
Glad to hear you were able to make more progress!
John
On Mon, Jul 6, 2020 at 9:38 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
>
> Hello John,
>
> I wish you had a wonderful vacation.
>
> I didn't get any reply for the previous email. However I sent in
another
> ticket about the "valid time" warning today and Howard provided
solution
> for me and he created a github issue "
> https://github.com/NCAR/MET/issues/1407"
>
> But there are two things I didn't ask:
> 1. As you know MET can not read in the pixel data, so I had to start
from
> converting that to ascII and then to *nc using point2nc utility,
> fortunately it is pretty fast b/c I don't have a lot of valid grid.
However
> I am not sure if this will bring some bias or error?
>
> 2. Another question is about the plots attached in the previous
email. ( by
> the way although I got the "valid time" warning before Howard gave
me the
> solution, it seems the *stat files are correct.) I didn't understand
why
> the magnitude of obs. seems close to the forecast based on the
second plot,
> but the HIST indicates a series under-estimate? Is this reasonable
(because
> the second plot is using the mean of ensemble members) or it
indicates I
> made a mistake somewhere?
>
> Thank you.
> Binyu
>
> On Mon, Jul 6, 2020 at 10:51 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Vinyl,
> >
> > Sorry I never got a chance to take a look at this issue before
leaving
> for
> > vacation for 2 weeks.
> >
> > Perhaps one of the other engineers or scientists on our team could
take a
> > look.
> >
> > Tara or George, any thoughts?
> >
> > John
> >
> > On Wed, Jul 1, 2020 at 5:43 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225 >
> > >
> > > Hello John,
> > >
> > > Back to the Satellite data that I have, I don't expect I could
get the
> > data
> > > with the right format very soon. So I just follow the steps:
> > > 1.convert the satellite data to ascII
> > > 2.convert the ascII file to grid file using ascii2nc and
point2grid
> > > 3.get hourly average using "cdo ensmean" (the satellite data is
every
> 10
> > > mins, while model is hourly average)
> > >
> > > Fortunately I don't have a lot of valid grids, so the whole
process is
> > > pretty fast. Anyhow I did a test and used"plot_data_plane" to
make a
> > > plot,which looks reasonable. However I am not sure if I made a
mistake
> > > using the conversion:
> > >
> > > 1. I am using all default config file for both "ascii2nc and
> point2grid",
> > > is that correct?
> > > Here is my script to do the conversion from ascII to nc:
> > > /scratch2/NCEPDEV/naqfc/Binyu.Wang/Scripts/MET/asc2grid.sh
> > > 2.After I finished the conversion, I did the ensemble
verification as
> > > before:
> > >
> > >
> >
>
/scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_Raikoke.sh
> > >
> > > Then I tried to make a HIST plot (see the attachment 1). The
plot looks
> > > weird to me because the plot indicates that model under-predict
obs.
> > > However the overlay plot shows that model prediction is close to
obs
> > > (attachment 2). What is the problem then?
> > >
> > > Here are the warnings I got, I guess this will cause some
problems.
> But I
> > > am not sure how to fix the problem.
> > > from "run_es.log"
> > > WARNING: NcCfFile::open() -> could not determine the valid time,
using
> 0
> > > from "run_gs.log"
> > > WARNING: process_scores() -> Forecast and observation valid
times do
> not
> > > match 20190623_000000 != 19700101_000000 for
> > > VAFTD_L18000-0_ENS_FREQ_ge0.05(*,*) versus ashL0
> > >
> > > Just in case: here is the example of original satellite obs:
> > >
> > >
> >
> /scratch2/NCEPDEV/naqfc/Binyu.Wang/MET/Obs/Raikoke/SCOPE_NWC_ASH-L2-
ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-
> > > 20190624-193000-fv2.nc
> > >
> > > Thank you.
> > >
> > > On Tue, May 19, 2020 at 11:51 AM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Binyu,
> > > >
> > > > The point2grid tool already includes direct support for
processing
> > > GOES-16
> > > > satellite data.
> > > >
> > > > Please take a look at chapter 4.5 of the MET User's Guide and
try
> > passing
> > > > this data directly to point2grid:
> > > >
> > > >
> > >
> >
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.1.pdf
> > > >
> > > > This functionality was developed using GOES16 aerosol data so
it may
> or
> > > may
> > > > not work. If not, please let me know, and I'll grab that file
and
> ask
> > > our
> > > > developer, Howard Soh, to take a look.
> > > >
> > > > Listed below are the steps I took to process tri-polar grid
data as
> > > > individual points. But this all took over 3 hours. I don't
have an
> > > estimate
> > > > yet for enhancing point2grid to process this data more
directly.
> We're
> > > > discussing internally but do so how it'd be useful for at
least 2
> > > projects.
> > > >
> > > > FYI, I posted the input data for the commands listed below to:
> > > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/for_binyu_data/
> > > >
> > > > -------
> > > >
> > > > I ran ascii2nc as we discussed and would say that this is not
really
> a
> > > > viable option. Processing 6,232,499 grid points as individual
point
> > > > observations just takes way too long!
> > > >
> > > > I ran the following commands on kiowa.
> > > >
> > > > *> cd /d1/projects/SeaIce/tripolar*
> > > > *> export MET_PYTHON_EXE=/usr/local/python3/bin/python*
> > > > *> time /usr/local/met-9.0.1/bin/ascii2nc
'tripolar_to_lat_lon.py
> > > > input/20200127/rtofs_glo_2ds_n048_daily_diag.nc
> > > > <http://rtofs_glo_2ds_n048_daily_diag.nc/> ice_coverage
> > > > north' tripolar_points.nc <http://tripolar_points.nc/> -format
> python*
> > > >
> > > > This takes a whopping *2 hours and 25 minutes* to run ascii2nc
on a
> > > single
> > > > field of data:
> > > > (1) Read the data and reformat the 6.2 million values into a
list of
> > > lists.
> > > > (2) Write the result to a temporary pickle file.
> > > > (3) Read that pickle file into memory.
> > > > (4) Write those point obs back out to a 286MB NetCDF file.
> > > > By comparison, the input rtofs data file, with multiple
variables, is
> > > 154MB
> > > > in size.
> > > >
> > > > Next, run point2grid on that NetCDF file to regrid to the same
Polar
> > > > Stereographic
> > > > grid as the IMS data, and this step took 55 minutes:
> > > > *> time /usr/local/met-9.0.1/bin/point2grid tripolar_points.nc
> > > > <http://tripolar_points.nc/> imssnow96.20190201.grb.grib2
> > > > tripolar_points_IMS.nc -field 'name="ice_coverage";
level="L0";'
> > -method
> > > > MAX*
> > > >
> > > > Then plot the result:
> > > > *> time /usr/local/met-9.0.1/bin/plot_data_plane
> tripolar_points_IMS.nc
> > > > tripolar_points_IMS.ps 'name="ice_coverage"; level="(*,*)";'*
> > > >
> > > > On Mon, May 18, 2020 at 8:32 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Thank you for the update, John. Can you share your
scripts/code to
> > me?
> > > I
> > > > > guess my data won't take so long because there is not so
much data
> > > > inside.
> > > > > Do you have an estimate how long it will take for the new
> > development?
> > > > >
> > > > > I have three sets of satellite data, except the previous
two, I
> also
> > > need
> > > > > to work on the third set below, is this one easier to modify
to be
> > read
> > > > in
> > > > > by MET?
> > > > >
> > > > >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Reventado/Obs/
> > > > > geocatL2.GOES-16.Full_Disk.2019056.150030.hdf
> > > > >
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Mon, May 18, 2020 at 3:55 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Binyu,
> > > > > >
> > > > > > I ran into a somewhat similar situation last week. We have
data
> on
> > a
> > > > > > tri-polar grid which is not supported in MET. So I did the
> > following
> > > > > steps:
> > > > > > (1) Write a python script to reformat the tri-polar data
as if it
> > > were
> > > > > > lat/lon point observations.
> > > > > > (2) Run that script using ascii2nc to reformat the data
into
> MET's
> > > > NetCDF
> > > > > > point observation format.
> > > > > > (3) Run that NetCDF point obs file through the point2grid
tool to
> > > > > > interpolate the data to a regular grids.
> > > > > >
> > > > > > And that process did (eventually) work, but it took well
over 3
> > hours
> > > > for
> > > > > > all those steps since the input data consists of over 6
million
> > > points!
> > > > > So
> > > > > > not very realistic! We need to figure out how to get
lat/lon data
> > > into
> > > > > the
> > > > > > point2grid logic much more efficiently. And I think that
same
> > > solution
> > > > > > would work well for your data. But unfortunately, that
would
> > require
> > > > new
> > > > > > development.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Mon, May 18, 2020 at 1:35 PM binyu.wang at noaa.gov via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> >
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > Thank you John.
> > > > > > >
> > > > > > > I look forward to Howard's response.
> > > > > > >
> > > > > > > Binyu
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, May 11, 2020 at 7:40 PM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Binyu,
> > > > > > > >
> > > > > > > > OK, I logged onto WCOSS and retrieved one of these
sample
> > files.
> > > > > > > >
> > > > > > > > I used ncview to display the latitude and longitude
variable
> > > values
> > > > > and
> > > > > > > > have attached a screenshot.
> > > > > > > >
> > > > > > > > This does not look like gridded data to me. Instead,
it's a
> > view
> > > of
> > > > > > > > satellite pixels. MET currently does evaluations on
gridded
> > data,
> > > > not
> > > > > > > > satellite pixels. So the question is if/how we can get
this
> > > > satellite
> > > > > > > data
> > > > > > > > onto a grid.
> > > > > > > >
> > > > > > > > MET does include a tool named point2grid which is
intended to
> > do
> > > > > > exactly
> > > > > > > > that. But it's pretty limited on the input files it
can
> accept.
> > > > > Howard
> > > > > > > Soh,
> > > > > > > > cc'ed here, is the developer who's worked most on that
tool.
> > > > > > > >
> > > > > > > > Howard, do you have any suggestions on how Binyu could
> > > interpolate
> > > > > the
> > > > > > > data
> > > > > > > > from this NetCDF file onto a grid?
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data/SCOPE_NWC_ASH-
L2-ASH_PRODUCTS-HIMAWARI8_NOAA-RAIKOKE-20190624-203000-fv2.nc
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, May 11, 2020 at 12:42 PM binyu.wang at noaa.gov
via RT
> <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Mon May 11 12:41:58 2020: Request 95225 was acted
upon.
> > > > > > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: read in Satellite data
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95225
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hello John,
> > > > > > > > >
> > > > > > > > > Thank you for helping me with reading in the
satellite data
> > > using
> > > > > > MET.
> > > > > > > I
> > > > > > > > > have another set of satellite data which also needs
to be
> > read
> > > in
> > > > > by
> > > > > > > MET
> > > > > > > > > for verification. Those two datasets are different
but
> > similar,
> > > > so
> > > > > I
> > > > > > > > think
> > > > > > > > > you are the better helper to ask. The datasets are
located
> > > below.
> > > > > > > > > "ash_mass" is the variable needed to be verified.
> > > > > > > > >
> > > > > > > > > /gpfs/dell2/emc/modeling/save/Binyu.Wang/VA/Raikoke
> > > > > > > > >
> > > > > > > > > Thank you.
> > > > > > > > > Binyu
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list