[Met_help] [rt.rap.ucar.edu #82926] History for not seeing impact from -compress option in ascii2nc
Howard Soh via RT
met_help at ucar.edu
Tue Nov 14 12:13:55 MST 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear MET Help,
I'm trying to compress my output netcdf files produced by ascii2nc (per the MET v6.0 user's guide), but I'm not seeing any difference in the size of the resulting netcdf files. I tried two things:
(1) Including the '-compress 9' option in the ascii2nc command, and
(2) Setting the environmental variable MET_NC_COMPRESS beforehand.
Neither test changed the output netcdf file size.
Please advise,
Jonathan Case
*****************************************************
Jonathan Case (Jon)
ENSCO, Inc./NASA SPoRT Center
National Space Science and Technology Center
320 Sparkman Drive, Room 3000B
Huntsville, AL 35805
Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt: case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
*****************************************************
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #82926] not seeing impact from -compress option in ascii2nc
From: Howard Soh
Time: Tue Nov 07 14:31:19 2017
Hi Jon,
I see that you are having problems running ascii2nc (met-6.0).
Many cases, the output size is not reduced significantly by higher
compression.
Would you compare the file size in byte unit?
29M Nov 7 13:55 clear_pvwatts.comp0.nc (no compression)
13M Nov 7 13:55 clear_pvwatts.comp1.nc (compression level 1)
13M Nov 7 14:09 clear_pvwatts.comp9.nc (compression level 9)
30038670 Nov 7 13:55 clear_pvwatts.comp0.nc
13304742 Nov 7 13:55 clear_pvwatts.comp1.nc
13222906 Nov 7 13:55 clear_pvwatts.comp9.nc
If the file size is the same, I would check how many observations were
processed.
Would you run ascii2nc with "-v 3 -log ascii2nc_compression.log"
options and send me the log file, ascii2nc_compression.log?
Cheers,
Howard
FYI:
Note: The command line option has the higher priority over the
environment variable.
Check list:
1. The output netcdf format version (should be "netCDF-4")
ncdump -k <netcdf_file_name>
2. The NetCDF dimensions (nhdr and nobs should not be zero)
ncdump -h <netcdf_file_name>
On 11/7/2017 12:59 PM, Case, Jonathan[ENSCO INC] via RT wrote:
> Tue Nov 07 12:59:00 2017: Request 82926 was acted upon.
> Transaction: Ticket created by jonathan.case-1 at nasa.gov
> Queue: met_help
> Subject: not seeing impact from -compress option in ascii2nc
> Owner: Nobody
> Requestors: jonathan.case-1 at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=82926 >
>
>
> Dear MET Help,
>
> I'm trying to compress my output netcdf files produced by ascii2nc
(per the MET v6.0 user's guide), but I'm not seeing any difference in
the size of the resulting netcdf files. I tried two things:
>
> (1) Including the '-compress 9' option in the ascii2nc command,
and
>
> (2) Setting the environmental variable MET_NC_COMPRESS beforehand.
>
> Neither test changed the output netcdf file size.
>
> Please advise,
> Jonathan Case
>
> *****************************************************
> Jonathan Case (Jon)
> ENSCO, Inc./NASA SPoRT Center
> National Space Science and Technology Center
> 320 Sparkman Drive, Room 3000B
> Huntsville, AL 35805
> Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> *****************************************************
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #82926] not seeing impact from -compress option in ascii2nc
From: Case, Jonathan[ENSCO INC]
Time: Wed Nov 08 10:56:07 2017
Hello Howard,
I think that somehow the default compression was set to something
higher than 0 for my initial runs.
Regardless, I ran the tests for compression 0, 1, 5, and 9. The only
real gains appear to be from level 0 to 1.
Best,
Jon
Resulting netcdf file sizes:
> ls -l COMPRESS*/*.nc
-rw-r--r-- 1 jlcase s0746 41765817 Nov 8 12:38
COMPRESS0/sfcobs_20160713_1800.nc
-rw-r--r-- 1 jlcase s0746 31210914 Nov 8 12:42
COMPRESS1/sfcobs_20160713_1800.nc
-rw-r--r-- 1 jlcase s0746 31127436 Nov 8 12:44
COMPRESS5/sfcobs_20160713_1800.nc
-rw-r--r-- 1 jlcase s0746 31090493 Nov 8 12:46
COMPRESS9/sfcobs_20160713_1800.nc
Log outputs:
Compress0:
DEBUG 1: Config File: /usr/local/other/SLES11.3/met/6.0/gcc-
5.3/share/met/config/Ascii2NcConfig_default
DEBUG 2: Rejected 0 observations off the masking grid.
DEBUG 2: Rejected 0 observations outside the masking polyline.
DEBUG 2: Rejected 0 observations not matched with station ID's.
DEBUG 2: Processing observations for 50502 headers.
DEBUG 1: Creating NetCDF Observation file:
/discover/nobackup/jlcase/MET/pointData/sfcobs_20160713_1800.nc
DEBUG 2: Finished processing 228441 observations for 50502 headers.
Compress1:
DEBUG 1: Config File: /usr/local/other/SLES11.3/met/6.0/gcc-
5.3/share/met/config/Ascii2NcConfig_default
DEBUG 2: Rejected 0 observations off the masking grid.
DEBUG 2: Rejected 0 observations outside the masking polyline.
DEBUG 2: Rejected 0 observations not matched with station ID's.
DEBUG 2: Processing observations for 50502 headers.
DEBUG 1: Creating NetCDF Observation file:
/discover/nobackup/jlcase/MET/pointData/sfcobs_20160713_1800.nc
DEBUG 3: nc_utils.add_var() deflate_level: 1
DEBUG 3: nc_utils.add_var() deflate_level: 1
DEBUG 3: nc_utils.add_var() deflate_level: 1
DEBUG 3: nc_utils.add_var() deflate_level: 1
DEBUG 3: nc_utils.add_var() deflate_level: 1
DEBUG 3: nc_utils.add_var() deflate_level: 1
DEBUG 2: Finished processing 228441 observations for 50502 headers.
Compress5:
DEBUG 1: Config File: /usr/local/other/SLES11.3/met/6.0/gcc-
5.3/share/met/config/Ascii2NcConfig_default
DEBUG 2: Rejected 0 observations off the masking grid.
DEBUG 2: Rejected 0 observations outside the masking polyline.
DEBUG 2: Rejected 0 observations not matched with station ID's.
DEBUG 2: Processing observations for 50502 headers.
DEBUG 1: Creating NetCDF Observation file:
/discover/nobackup/jlcase/MET/pointData/sfcobs_20160713_1800.nc
DEBUG 3: nc_utils.add_var() deflate_level: 5
DEBUG 3: nc_utils.add_var() deflate_level: 5
DEBUG 3: nc_utils.add_var() deflate_level: 5
DEBUG 3: nc_utils.add_var() deflate_level: 5
DEBUG 3: nc_utils.add_var() deflate_level: 5
DEBUG 3: nc_utils.add_var() deflate_level: 5
DEBUG 2: Finished processing 228441 observations for 50502 headers.
Compress9:
DEBUG 1: Config File: /usr/local/other/SLES11.3/met/6.0/gcc-
5.3/share/met/config/Ascii2NcConfig_default
DEBUG 2: Rejected 0 observations off the masking grid.
DEBUG 2: Rejected 0 observations outside the masking polyline.
DEBUG 2: Rejected 0 observations not matched with station ID's.
DEBUG 2: Processing observations for 50502 headers.
DEBUG 1: Creating NetCDF Observation file:
/discover/nobackup/jlcase/MET/pointData/sfcobs_20160713_1800.nc
DEBUG 3: nc_utils.add_var() deflate_level: 9
DEBUG 3: nc_utils.add_var() deflate_level: 9
DEBUG 3: nc_utils.add_var() deflate_level: 9
DEBUG 3: nc_utils.add_var() deflate_level: 9
DEBUG 3: nc_utils.add_var() deflate_level: 9
DEBUG 3: nc_utils.add_var() deflate_level: 9
DEBUG 2: Finished processing 228441 observations for 50502 headers.
-----Original Message-----
From: Howard Soh via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, November 7, 2017 3:31 PM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
Subject: Re: [rt.rap.ucar.edu #82926] not seeing impact from -compress
option in ascii2nc
Hi Jon,
I see that you are having problems running ascii2nc (met-6.0).
Many cases, the output size is not reduced significantly by higher
compression.
Would you compare the file size in byte unit?
29M Nov 7 13:55 clear_pvwatts.comp0.nc (no compression) 13M Nov 7
13:55 clear_pvwatts.comp1.nc (compression level 1) 13M Nov 7 14:09
clear_pvwatts.comp9.nc (compression level 9)
30038670 Nov 7 13:55 clear_pvwatts.comp0.nc
13304742 Nov 7 13:55 clear_pvwatts.comp1.nc
13222906 Nov 7 13:55 clear_pvwatts.comp9.nc
If the file size is the same, I would check how many observations were
processed.
Would you run ascii2nc with "-v 3 -log ascii2nc_compression.log"
options and send me the log file, ascii2nc_compression.log?
Cheers,
Howard
FYI:
Note: The command line option has the higher priority over the
environment variable.
Check list:
1. The output netcdf format version (should be "netCDF-4")
ncdump -k <netcdf_file_name>
2. The NetCDF dimensions (nhdr and nobs should not be zero)
ncdump -h <netcdf_file_name>
On 11/7/2017 12:59 PM, Case, Jonathan[ENSCO INC] via RT wrote:
> Tue Nov 07 12:59:00 2017: Request 82926 was acted upon.
> Transaction: Ticket created by jonathan.case-1 at nasa.gov
> Queue: met_help
> Subject: not seeing impact from -compress option in ascii2nc
> Owner: Nobody
> Requestors: jonathan.case-1 at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=82926
> >
>
>
> Dear MET Help,
>
> I'm trying to compress my output netcdf files produced by ascii2nc
(per the MET v6.0 user's guide), but I'm not seeing any difference in
the size of the resulting netcdf files. I tried two things:
>
> (1) Including the '-compress 9' option in the ascii2nc command,
and
>
> (2) Setting the environmental variable MET_NC_COMPRESS beforehand.
>
> Neither test changed the output netcdf file size.
>
> Please advise,
> Jonathan Case
>
> *****************************************************
> Jonathan Case (Jon)
> ENSCO, Inc./NASA SPoRT Center
> National Space Science and Technology Center
> 320 Sparkman Drive, Room 3000B
> Huntsville, AL 35805
> Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
> case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> *****************************************************
>
>
------------------------------------------------
Subject: not seeing impact from -compress option in ascii2nc
From: Howard Soh
Time: Wed Nov 08 13:07:24 2017
Thank you for update.
I will close the ticket if no more issues.
Cheers,
Howard
FYI: How to set the compression level:
1. By using system wide default configuration file. If not defined,
the
compression is disabled. This can be overridden by user defined
configuration file.
nc_compression = d;
d: 0 to 9 (0 to disable compression)
2. By using environment variable, MET_NC_COMPRESS, which overrides the
setting from configuration file
3. By using command line option "-compress d" which overrides the
previous setting by the environment variable or the configuration
file.
On 11/8/2017 10:56 AM, Case, Jonathan[ENSCO INC] via RT wrote:
> <URL:https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=82926 >
>
> Hello Howard,
>
> I think that somehow the default compression was set to something
higher than 0 for my initial runs.
> Regardless, I ran the tests for compression 0, 1, 5, and 9. The only
real gains appear to be from level 0 to 1.
>
> Best,
> Jon
>
> Resulting netcdf file sizes:
> > ls -l COMPRESS*/*.nc
> -rw-r--r-- 1 jlcase s0746 41765817 Nov 8 12:38
COMPRESS0/sfcobs_20160713_1800.nc
> -rw-r--r-- 1 jlcase s0746 31210914 Nov 8 12:42
COMPRESS1/sfcobs_20160713_1800.nc
> -rw-r--r-- 1 jlcase s0746 31127436 Nov 8 12:44
COMPRESS5/sfcobs_20160713_1800.nc
> -rw-r--r-- 1 jlcase s0746 31090493 Nov 8 12:46
COMPRESS9/sfcobs_20160713_1800.nc
>
> Log outputs:
>
> Compress0:
> DEBUG 1: Config File: /usr/local/other/SLES11.3/met/6.0/gcc-
5.3/share/met/config/Ascii2NcConfig_default
> DEBUG 2: Rejected 0 observations off the masking grid.
> DEBUG 2: Rejected 0 observations outside the masking polyline.
> DEBUG 2: Rejected 0 observations not matched with station ID's.
> DEBUG 2: Processing observations for 50502 headers.
> DEBUG 1: Creating NetCDF Observation file:
/discover/nobackup/jlcase/MET/pointData/sfcobs_20160713_1800.nc
> DEBUG 2: Finished processing 228441 observations for 50502 headers.
>
> Compress1:
> DEBUG 1: Config File: /usr/local/other/SLES11.3/met/6.0/gcc-
5.3/share/met/config/Ascii2NcConfig_default
> DEBUG 2: Rejected 0 observations off the masking grid.
> DEBUG 2: Rejected 0 observations outside the masking polyline.
> DEBUG 2: Rejected 0 observations not matched with station ID's.
> DEBUG 2: Processing observations for 50502 headers.
> DEBUG 1: Creating NetCDF Observation file:
/discover/nobackup/jlcase/MET/pointData/sfcobs_20160713_1800.nc
> DEBUG 3: nc_utils.add_var() deflate_level: 1
> DEBUG 3: nc_utils.add_var() deflate_level: 1
> DEBUG 3: nc_utils.add_var() deflate_level: 1
> DEBUG 3: nc_utils.add_var() deflate_level: 1
> DEBUG 3: nc_utils.add_var() deflate_level: 1
> DEBUG 3: nc_utils.add_var() deflate_level: 1
> DEBUG 2: Finished processing 228441 observations for 50502 headers.
>
> Compress5:
> DEBUG 1: Config File: /usr/local/other/SLES11.3/met/6.0/gcc-
5.3/share/met/config/Ascii2NcConfig_default
> DEBUG 2: Rejected 0 observations off the masking grid.
> DEBUG 2: Rejected 0 observations outside the masking polyline.
> DEBUG 2: Rejected 0 observations not matched with station ID's.
> DEBUG 2: Processing observations for 50502 headers.
> DEBUG 1: Creating NetCDF Observation file:
/discover/nobackup/jlcase/MET/pointData/sfcobs_20160713_1800.nc
> DEBUG 3: nc_utils.add_var() deflate_level: 5
> DEBUG 3: nc_utils.add_var() deflate_level: 5
> DEBUG 3: nc_utils.add_var() deflate_level: 5
> DEBUG 3: nc_utils.add_var() deflate_level: 5
> DEBUG 3: nc_utils.add_var() deflate_level: 5
> DEBUG 3: nc_utils.add_var() deflate_level: 5
> DEBUG 2: Finished processing 228441 observations for 50502 headers.
>
> Compress9:
> DEBUG 1: Config File: /usr/local/other/SLES11.3/met/6.0/gcc-
5.3/share/met/config/Ascii2NcConfig_default
> DEBUG 2: Rejected 0 observations off the masking grid.
> DEBUG 2: Rejected 0 observations outside the masking polyline.
> DEBUG 2: Rejected 0 observations not matched with station ID's.
> DEBUG 2: Processing observations for 50502 headers.
> DEBUG 1: Creating NetCDF Observation file:
/discover/nobackup/jlcase/MET/pointData/sfcobs_20160713_1800.nc
> DEBUG 3: nc_utils.add_var() deflate_level: 9
> DEBUG 3: nc_utils.add_var() deflate_level: 9
> DEBUG 3: nc_utils.add_var() deflate_level: 9
> DEBUG 3: nc_utils.add_var() deflate_level: 9
> DEBUG 3: nc_utils.add_var() deflate_level: 9
> DEBUG 3: nc_utils.add_var() deflate_level: 9
> DEBUG 2: Finished processing 228441 observations for 50502 headers.
>
>
> -----Original Message-----
> From: Howard Soh via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, November 7, 2017 3:31 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC]<jonathan.case-1 at nasa.gov>
> Subject: Re: [rt.rap.ucar.edu #82926] not seeing impact from
-compress option in ascii2nc
>
> Hi Jon,
>
> I see that you are having problems running ascii2nc (met-6.0).
>
> Many cases, the output size is not reduced significantly by higher
compression.
> Would you compare the file size in byte unit?
>
> 29M Nov 7 13:55 clear_pvwatts.comp0.nc (no compression) 13M Nov 7
13:55 clear_pvwatts.comp1.nc (compression level 1) 13M Nov 7 14:09
clear_pvwatts.comp9.nc (compression level 9)
>
> 30038670 Nov 7 13:55 clear_pvwatts.comp0.nc
> 13304742 Nov 7 13:55 clear_pvwatts.comp1.nc
> 13222906 Nov 7 13:55 clear_pvwatts.comp9.nc
>
> If the file size is the same, I would check how many observations
were processed.
> Would you run ascii2nc with "-v 3 -log ascii2nc_compression.log"
options and send me the log file, ascii2nc_compression.log?
>
> Cheers,
> Howard
>
> FYI:
>
> Note: The command line option has the higher priority over the
environment variable.
>
> Check list:
> 1. The output netcdf format version (should be "netCDF-4")
> ncdump -k <netcdf_file_name>
> 2. The NetCDF dimensions (nhdr and nobs should not be zero)
> ncdump -h <netcdf_file_name>
>
>
>
> On 11/7/2017 12:59 PM, Case, Jonathan[ENSCO INC] via RT wrote:
>> Tue Nov 07 12:59:00 2017: Request 82926 was acted upon.
>> Transaction: Ticket created byjonathan.case-1 at nasa.gov
>> Queue: met_help
>> Subject: not seeing impact from -compress option in ascii2nc
>> Owner: Nobody
>> Requestors:jonathan.case-1 at nasa.gov
>> Status: new
>> Ticket
<URL:https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=82926
>> Dear MET Help,
>>
>> I'm trying to compress my output netcdf files produced by ascii2nc
(per the MET v6.0 user's guide), but I'm not seeing any difference in
the size of the resulting netcdf files. I tried two things:
>>
>> (1) Including the '-compress 9' option in the ascii2nc command,
and
>>
>> (2) Setting the environmental variable MET_NC_COMPRESS
beforehand.
>>
>> Neither test changed the output netcdf file size.
>>
>> Please advise,
>> Jonathan Case
>>
>> *****************************************************
>> Jonathan Case (Jon)
>> ENSCO, Inc./NASA SPoRT Center
>> National Space Science and Technology Center
>> 320 Sparkman Drive, Room 3000B
>> Huntsville, AL 35805
>> Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
>> case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
>> *****************************************************
>>
>>
>
>
------------------------------------------------
More information about the Met_help
mailing list