[Met_help] [rt.rap.ucar.edu #94885] History for Can MET read compress netCDF file

Thu Apr 16 14:45:49 MDT 2020

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

EMC is in a discussion with NOAA NESDIS aerosol group on the L3 gridded
VIIRS AOD file for regional and global AOD verification.

We are considering requesting a fine resolution such as 1kmx1km and hourly
data.  The file size can be huge as the number of grids is 36000x18000.
But at any hour, only a limited area of the globe has VIIRS observation.
The rest of the globe are null values.  We are considering the use of
netCDF compress utilities (nccopy) to greatly reduce the filesize for
dataflow between NESDIS and NCO, e.g.,

nccopy -dN test.nc testd1.nc for N in [1,9].

Please do web search of nccopy for more detail (I found info at UCAR
Unidata site)

My question to the MET group is can MET handle the compress netCDF data? at
what level -d, -k, -s, ...etc. do you suggest us to use without losing
accuracy?

Thank you for your help.

Ho-Chun Huang

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2792

College Park, MD 20740

Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>

301-683-3958

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Can MET read compress netCDF file
From: George McCabe
Time: Thu Apr 16 08:55:25 2020

Hi Ho-Chun,

I am sorry for the delayed response. I assigned this ticket to John
Halley Gotway, who is more familiar with how MET handles compressed
NetCDF files.

Thanks,
George

On Fri Apr 10 07:51:01 2020, ho-chun.huang at noaa.gov wrote:
> Hi,
>
> EMC is in a discussion with NOAA NESDIS aerosol group on the L3
gridded
> VIIRS AOD file for regional and global AOD verification.
>
> We are considering requesting a fine resolution such as 1kmx1km and
hourly
> data.  The file size can be huge as the number of grids is
36000x18000.
> But at any hour, only a limited area of the globe has VIIRS
observation.
> The rest of the globe are null values.  We are considering the use
of
> netCDF compress utilities (nccopy) to greatly reduce the filesize
for
> dataflow between NESDIS and NCO, e.g.,
>
> nccopy -dN test.nc testd1.nc for N in [1,9].
>
> Please do web search of nccopy for more detail (I found info at UCAR
> Unidata site)
>
> My question to the MET group is can MET handle the compress netCDF
data? at
> what level -d, -k, -s, ...etc. do you suggest us to use without
losing
> accuracy?
>
> Thank you for your help.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958

------------------------------------------------
Subject: Can MET read compress netCDF file
From: Howard Soh
Time: Thu Apr 16 10:54:19 2020

The compressed NetCDF files are supported. MET can produce the
compressed NetCDF with "-compress level" option.

Cheers,
Howard

On Thu Apr 16 08:55:25 2020, mccabe wrote:
> Hi Ho-Chun,
>
> I am sorry for the delayed response. I assigned this ticket to John
> Halley Gotway, who is more familiar with how MET handles compressed
> NetCDF files.
>
> Thanks,
> George
>
>
> On Fri Apr 10 07:51:01 2020, ho-chun.huang at noaa.gov wrote:
> > Hi,
> >
> > EMC is in a discussion with NOAA NESDIS aerosol group on the L3
> > gridded
> > VIIRS AOD file for regional and global AOD verification.
> >
> > We are considering requesting a fine resolution such as 1kmx1km
and
> > hourly
> > data.  The file size can be huge as the number of grids is
> > 36000x18000.
> > But at any hour, only a limited area of the globe has VIIRS
> > observation.
> > The rest of the globe are null values.  We are considering the use
of
> > netCDF compress utilities (nccopy) to greatly reduce the filesize
for
> > dataflow between NESDIS and NCO, e.g.,
> >
> > nccopy -dN test.nc testd1.nc for N in [1,9].
> >
> > Please do web search of nccopy for more detail (I found info at
UCAR
> > Unidata site)
> >
> > My question to the MET group is can MET handle the compress netCDF
> > data? at
> > what level -d, -k, -s, ...etc. do you suggest us to use without
> > losing
> > accuracy?
> >
> > Thank you for your help.
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958

------------------------------------------------
Subject: Can MET read compress netCDF file
From: Ho-Chun Huang - NOAA Affiliate
Time: Thu Apr 16 11:26:18 2020

Hi, Howard:

Thanks for the information.  MET will support compress netCDf at any
level
of compression?  for example -d9 (although I do not think I will use
it).
 Most likely, -d1 or -d3 can reduce the file size by factor.  Just
want to
be sure before I talk to the satellite group about the requirement
tomorrow.

Ho-Chun Huang

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2792

College Park, MD 20740

Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>

301-683-3958

On Thu, Apr 16, 2020 at 12:54 PM Howard Soh via RT <met_help at ucar.edu>
wrote:

> The compressed NetCDF files are supported. MET can produce the
compressed
> NetCDF with "-compress level" option.
>
> Cheers,
> Howard
>
> On Thu Apr 16 08:55:25 2020, mccabe wrote:
> > Hi Ho-Chun,
> >
> > I am sorry for the delayed response. I assigned this ticket to
John
> > Halley Gotway, who is more familiar with how MET handles
compressed
> > NetCDF files.
> >
> > Thanks,
> > George
> >
> >
> > On Fri Apr 10 07:51:01 2020, ho-chun.huang at noaa.gov wrote:
> > > Hi,
> > >
> > > EMC is in a discussion with NOAA NESDIS aerosol group on the L3
> > > gridded
> > > VIIRS AOD file for regional and global AOD verification.
> > >
> > > We are considering requesting a fine resolution such as 1kmx1km
and
> > > hourly
> > > data.  The file size can be huge as the number of grids is
> > > 36000x18000.
> > > But at any hour, only a limited area of the globe has VIIRS
> > > observation.
> > > The rest of the globe are null values.  We are considering the
use of
> > > netCDF compress utilities (nccopy) to greatly reduce the
filesize for
> > > dataflow between NESDIS and NCO, e.g.,
> > >
> > > nccopy -dN test.nc testd1.nc for N in [1,9].
> > >
> > > Please do web search of nccopy for more detail (I found info at
UCAR
> > > Unidata site)
> > >
> > > My question to the MET group is can MET handle the compress
netCDF
> > > data? at
> > > what level -d, -k, -s, ...etc. do you suggest us to use without
> > > losing
> > > accuracy?
> > >
> > > Thank you for your help.
> > >
> > > Ho-Chun Huang
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct., Rm. 2792
> > >
> > > College Park, MD 20740
> > >
> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >
> > > 301-683-3958
>
>
>
>

------------------------------------------------
Subject: Can MET read compress netCDF file
From: Howard Soh
Time: Thu Apr 16 13:18:54 2020

Yes, MET supports any level of compression. I did not compare the
compressed file size recently but there was no big saving with 4+
compression level.

Cheers,
Howard

On Thu Apr 16 11:26:18 2020, ho-chun.huang at noaa.gov wrote:
> Hi, Howard:
>
> Thanks for the information.  MET will support compress netCDf at any
level
> of compression?  for example -d9 (although I do not think I will use
it).
>  Most likely, -d1 or -d3 can reduce the file size by factor.  Just
want to
> be sure before I talk to the satellite group about the requirement
tomorrow.
>
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Thu, Apr 16, 2020 at 12:54 PM Howard Soh via RT
<met_help at ucar.edu>
> wrote:
>
> > The compressed NetCDF files are supported. MET can produce the
compressed
> > NetCDF with "-compress level" option.
> >
> > Cheers,
> > Howard
> >
> > On Thu Apr 16 08:55:25 2020, mccabe wrote:
> > > Hi Ho-Chun,
> > >
> > > I am sorry for the delayed response. I assigned this ticket to
John
> > > Halley Gotway, who is more familiar with how MET handles
compressed
> > > NetCDF files.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > > On Fri Apr 10 07:51:01 2020, ho-chun.huang at noaa.gov wrote:
> > > > Hi,
> > > >
> > > > EMC is in a discussion with NOAA NESDIS aerosol group on the
L3
> > > > gridded
> > > > VIIRS AOD file for regional and global AOD verification.
> > > >
> > > > We are considering requesting a fine resolution such as
1kmx1km and
> > > > hourly
> > > > data.  The file size can be huge as the number of grids is
> > > > 36000x18000.
> > > > But at any hour, only a limited area of the globe has VIIRS
> > > > observation.
> > > > The rest of the globe are null values.  We are considering the
use of
> > > > netCDF compress utilities (nccopy) to greatly reduce the
filesize for
> > > > dataflow between NESDIS and NCO, e.g.,
> > > >
> > > > nccopy -dN test.nc testd1.nc for N in [1,9].
> > > >
> > > > Please do web search of nccopy for more detail (I found info
at UCAR
> > > > Unidata site)
> > > >
> > > > My question to the MET group is can MET handle the compress
netCDF
> > > > data? at
> > > > what level -d, -k, -s, ...etc. do you suggest us to use
without
> > > > losing
> > > > accuracy?
> > > >
> > > > Thank you for your help.
> > > >
> > > > Ho-Chun Huang
> > > >
> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > >
> > > > 5830 University Research Ct., Rm. 2792
> > > >
> > > > College Park, MD 20740
> > > >
> > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >
> > > > 301-683-3958
> >
> >
> >
> >

------------------------------------------------
Subject: Can MET read compress netCDF file
From: Ho-Chun Huang - NOAA Affiliate
Time: Thu Apr 16 13:53:47 2020

Howard:

Thanks.  Great information from you.  Please close the ticket.

Ho-Chun Huang

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2792

College Park, MD 20740

Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>

301-683-3958

On Thu, Apr 16, 2020 at 3:19 PM Howard Soh via RT <met_help at ucar.edu>
wrote:

> Yes, MET supports any level of compression. I did not compare the
> compressed file size recently but there was no big saving with 4+
> compression level.
>
> Cheers,
> Howard
>
> On Thu Apr 16 11:26:18 2020, ho-chun.huang at noaa.gov wrote:
> > Hi, Howard:
> >
> > Thanks for the information.  MET will support compress netCDf at
any
> level
> > of compression?  for example -d9 (although I do not think I will
use it).
> >  Most likely, -d1 or -d3 can reduce the file size by factor.  Just
want
> to
> > be sure before I talk to the satellite group about the requirement
> tomorrow.
> >
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Thu, Apr 16, 2020 at 12:54 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > The compressed NetCDF files are supported. MET can produce the
> compressed
> > > NetCDF with "-compress level" option.
> > >
> > > Cheers,
> > > Howard
> > >
> > > On Thu Apr 16 08:55:25 2020, mccabe wrote:
> > > > Hi Ho-Chun,
> > > >
> > > > I am sorry for the delayed response. I assigned this ticket to
John
> > > > Halley Gotway, who is more familiar with how MET handles
compressed
> > > > NetCDF files.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > > On Fri Apr 10 07:51:01 2020, ho-chun.huang at noaa.gov wrote:
> > > > > Hi,
> > > > >
> > > > > EMC is in a discussion with NOAA NESDIS aerosol group on the
L3
> > > > > gridded
> > > > > VIIRS AOD file for regional and global AOD verification.
> > > > >
> > > > > We are considering requesting a fine resolution such as
1kmx1km and
> > > > > hourly
> > > > > data.  The file size can be huge as the number of grids is
> > > > > 36000x18000.
> > > > > But at any hour, only a limited area of the globe has VIIRS
> > > > > observation.
> > > > > The rest of the globe are null values.  We are considering
the use
> of
> > > > > netCDF compress utilities (nccopy) to greatly reduce the
filesize
> for
> > > > > dataflow between NESDIS and NCO, e.g.,
> > > > >
> > > > > nccopy -dN test.nc testd1.nc for N in [1,9].
> > > > >
> > > > > Please do web search of nccopy for more detail (I found info
at
> UCAR
> > > > > Unidata site)
> > > > >
> > > > > My question to the MET group is can MET handle the compress
netCDF
> > > > > data? at
> > > > > what level -d, -k, -s, ...etc. do you suggest us to use
without
> > > > > losing
> > > > > accuracy?
> > > > >
> > > > > Thank you for your help.
> > > > >
> > > > > Ho-Chun Huang
> > > > >
> > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >
> > > > > 5830 University Research Ct., Rm. 2792
> > > > >
> > > > > College Park, MD 20740
> > > > >
> > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > >
> > > > > 301-683-3958
> > >
> > >
> > >
> > >
>
>
>
>

------------------------------------------------