[Met_help] [rt.rap.ucar.edu #93612] History for modis_regrid question

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

MET-Help,

I put the MODIS HDF format files under "/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Kasatochi/Obs", and I tried to read those with the script "/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/Kasatochi/Obs/test.sh" ,but got error as:

DEBUG 1: Opening data file: geocatL2.Aqua.2008221.134000.hdf
ERROR  :
ERROR  : modis_regrid -> file "geocatL2.Aqua.2008221.134000.hdf" not a valid data file
ERROR  :

Any clue here? Thank you.

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: modis_regrid question
From: John Halley Gotway
Time: Fri Dec 27 16:35:24 2019

Binyu,

Sorry for delay.  I moved this question over to met-help and am
responding through that.

Thanks for sending the path to the sample data.  I retrieved the file
and saw the same behavior your did when running it through
modis_regrid.  I do see that the file is in the expected HDF4 format.

One point to clarify, and this is little confusing, the "-data_file"
command line argument specifies a gridded file that's already on the
desired output grid.  For testing, I just used a GFS file to put the
MODIS data on a global grid:

modis_regrid \
-field ash_mass_loading \
-data_file gfs_2012040900_F012.grib \
-out t2.nc \
-units percent \
-scale 0.01 \
-offset 0 \
-fill 127 \
geocatL2.Aqua.2008221.134000.hdf

This gets past the error message you saw, but the call still fails.  I
ran it through the debugger and see that it's not parsing the number
of swaths from the metadata.  And looking at a header dump of the Aqua
file versus the sample MODIS file we use in our nightly build
(MYD06_L2.A2013032.0630.051.2013032185634.hdf), there are significant
differences in the names of dimensions and attributes.

So I'm honestly not surprised that it didn't work because it hasn't
been tested extensively.

So where do we go from here?  You started by sending me a NetCDF file
that was post-processed satellite output.  It sounds like you'd like
to get that data onto a regularly spaced grid.  I'm sure there's
multiple ways this could be accomplished.  Any thoughts on how you'd
like to proceed?

One option would be figuring out how to use your post-processed NetCDF
output in MET.

Another option would be enhancing the MODIS-regrid tool to handle the
Aqua data.

FYI, I believe this is the dataset we used during development:
https://ladsweb.modaps.eosdis.nasa.gov/missions-and-
measurements/products/MYD06_L2/

Thanks,
John

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #93612] modis_regrid question
From: binyu.wang at noaa.gov
Time: Mon Dec 30 10:03:25 2019

Hello John,

Thank you very much for help. So it looks that convert the NetCDF
output
will be easier, isn't it?

It seems my current NetCDF files are in "classic" format, while MET
only
read V3.6. But I can't figure out how to do the conversion to meet MET
standard.

Binyu


On 12/27/2019 6:35 PM, John Halley Gotway via RT wrote:
> Binyu,
>
> Sorry for delay.  I moved this question over to met-help and am
responding through that.
>
> Thanks for sending the path to the sample data.  I retrieved the
file and saw the same behavior your did when running it through
modis_regrid.  I do see that the file is in the expected HDF4 format.
>
> One point to clarify, and this is little confusing, the "-data_file"
command line argument specifies a gridded file that's already on the
desired output grid.  For testing, I just used a GFS file to put the
MODIS data on a global grid:
>
> modis_regrid \
> -field ash_mass_loading \
> -data_file gfs_2012040900_F012.grib \
> -out t2.nc \
> -units percent \
> -scale 0.01 \
> -offset 0 \
> -fill 127 \
> geocatL2.Aqua.2008221.134000.hdf
>
> This gets past the error message you saw, but the call still fails.
I ran it through the debugger and see that it's not parsing the number
of swaths from the metadata.  And looking at a header dump of the Aqua
file versus the sample MODIS file we use in our nightly build
(MYD06_L2.A2013032.0630.051.2013032185634.hdf), there are significant
differences in the names of dimensions and attributes.
>
> So I'm honestly not surprised that it didn't work because it hasn't
been tested extensively.
>
> So where do we go from here?  You started by sending me a NetCDF
file that was post-processed satellite output.  It sounds like you'd
like to get that data onto a regularly spaced grid.  I'm sure there's
multiple ways this could be accomplished.  Any thoughts on how you'd
like to proceed?
>
> One option would be figuring out how to use your post-processed
NetCDF output in MET.
>
> Another option would be enhancing the MODIS-regrid tool to handle
the Aqua data.
>
> FYI, I believe this is the dataset we used during development:
> https://ladsweb.modaps.eosdis.nasa.gov/missions-and-
measurements/products/MYD06_L2/
>
> Thanks,
> John

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #93612] modis_regrid question
From: binyu.wang at noaa.gov
Time: Mon Dec 30 14:33:57 2019

Hello,

Based on the MET user guide, on page 39:  "MET package can handle
gridded input data in one of three formats: GRIB version 1, GRIB
version
2, or netCDF classic format (CF compliant)", then why it can not read
my
NetCDF file? It is classic format.

However on page 390, it said " Currently, only NetCDF version 3.6 can
be
used with MET". So which statement is correct?  I am confused now.

Thank you.

Binyu

On 12/27/2019 6:35 PM, John Halley Gotway via RT wrote:
> Binyu,
>
> Sorry for delay.  I moved this question over to met-help and am
responding through that.
>
> Thanks for sending the path to the sample data.  I retrieved the
file and saw the same behavior your did when running it through
modis_regrid.  I do see that the file is in the expected HDF4 format.
>
> One point to clarify, and this is little confusing, the "-data_file"
command line argument specifies a gridded file that's already on the
desired output grid.  For testing, I just used a GFS file to put the
MODIS data on a global grid:
>
> modis_regrid \
> -field ash_mass_loading \
> -data_file gfs_2012040900_F012.grib \
> -out t2.nc \
> -units percent \
> -scale 0.01 \
> -offset 0 \
> -fill 127 \
> geocatL2.Aqua.2008221.134000.hdf
>
> This gets past the error message you saw, but the call still fails.
I ran it through the debugger and see that it's not parsing the number
of swaths from the metadata.  And looking at a header dump of the Aqua
file versus the sample MODIS file we use in our nightly build
(MYD06_L2.A2013032.0630.051.2013032185634.hdf), there are significant
differences in the names of dimensions and attributes.
>
> So I'm honestly not surprised that it didn't work because it hasn't
been tested extensively.
>
> So where do we go from here?  You started by sending me a NetCDF
file that was post-processed satellite output.  It sounds like you'd
like to get that data onto a regularly spaced grid.  I'm sure there's
multiple ways this could be accomplished.  Any thoughts on how you'd
like to proceed?
>
> One option would be figuring out how to use your post-processed
NetCDF output in MET.
>
> Another option would be enhancing the MODIS-regrid tool to handle
the Aqua data.
>
> FYI, I believe this is the dataset we used during development:
> https://ladsweb.modaps.eosdis.nasa.gov/missions-and-
measurements/products/MYD06_L2/
>
> Thanks,
> John

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #93612] modis_regrid question
From: binyu.wang at noaa.gov
Time: Tue Dec 31 12:31:29 2019

Hello John,

Since I need to do ensemble verification, maybe the best way  is to
convert the NetCDF to grib2 to match the forecasting grids?

Thank you.

Binyu

On 12/27/2019 6:35 PM, John Halley Gotway via RT wrote:
> Binyu,
>
> Sorry for delay.  I moved this question over to met-help and am
responding through that.
>
> Thanks for sending the path to the sample data.  I retrieved the
file and saw the same behavior your did when running it through
modis_regrid.  I do see that the file is in the expected HDF4 format.
>
> One point to clarify, and this is little confusing, the "-data_file"
command line argument specifies a gridded file that's already on the
desired output grid.  For testing, I just used a GFS file to put the
MODIS data on a global grid:
>
> modis_regrid \
> -field ash_mass_loading \
> -data_file gfs_2012040900_F012.grib \
> -out t2.nc \
> -units percent \
> -scale 0.01 \
> -offset 0 \
> -fill 127 \
> geocatL2.Aqua.2008221.134000.hdf
>
> This gets past the error message you saw, but the call still fails.
I ran it through the debugger and see that it's not parsing the number
of swaths from the metadata.  And looking at a header dump of the Aqua
file versus the sample MODIS file we use in our nightly build
(MYD06_L2.A2013032.0630.051.2013032185634.hdf), there are significant
differences in the names of dimensions and attributes.
>
> So I'm honestly not surprised that it didn't work because it hasn't
been tested extensively.
>
> So where do we go from here?  You started by sending me a NetCDF
file that was post-processed satellite output.  It sounds like you'd
like to get that data onto a regularly spaced grid.  I'm sure there's
multiple ways this could be accomplished.  Any thoughts on how you'd
like to proceed?
>
> One option would be figuring out how to use your post-processed
NetCDF output in MET.
>
> Another option would be enhancing the MODIS-regrid tool to handle
the Aqua data.
>
> FYI, I believe this is the dataset we used during development:
> https://ladsweb.modaps.eosdis.nasa.gov/missions-and-
measurements/products/MYD06_L2/
>
> Thanks,
> John

------------------------------------------------
Subject: modis_regrid question
From: John Halley Gotway
Time: Wed Apr 01 11:11:48 2020

Hi Binyu,

I'm going back through some old MET-Help tickets and ran across this
one about ensemble verification.  It looks like I failed to answer
your question from a while ago, and I apologize for that.

Do you have any remaining questions about ensemble verification in
MET?

Thanks,
John

------------------------------------------------