[ncl-talk] Reading WRF compressed files -- how to?

Wed Apr 10 08:40:42 MDT 2019

Hi Barry,

offline ....

Sorry for the one word response last night. It was sent from my cell phone.
--

The 'timing hit' is when the file is *created. *Reading a compressed file
has minimal overhead and is quite efficient.

Creating a file  large with low compression incurs minimal overhead while
higher the compression level can have significant overhead.

*http://www.ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml*
<http://www.ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml>

contains:

*CompressionLevel* *Default value*: -1

Specify the level of data compression as an integer in the range 0 through
9. Increasing values indicate greater compression. Compression is lossless.
There are tradeoffs between the time spent compressing the file, versus the
amount of compression achieved. Informal tests show that compression level
9 results in a file only a few percent smaller than a compression level 5
file, but it requires 4 or 5 times the amount of time to create it. (This
option is ignored unless the *Format* option is set to *"NetCDF4Classic"*
or *"NetCDF4"*.)
Cheers
D

On Tue, Apr 9, 2019 at 9:16 PM Dennis Shea <shea at ucar.edu> wrote:

> No.
>
> Sent from my iPhone
>
> On Apr 9, 2019, at 8:49 PM, Barry Lynn <barry.h.lynn at gmail.com> wrote:
>
> Hi Dennis, Rick, Alan.
>
> Thank you for looking into this.
>
> It was noted that NCL knows how to uncompress the file when reading it.
> Does this imply that the read takes a lot longer?
>
> Barry
>
> On Tue, Apr 9, 2019 at 9:12 PM Dennis Shea <shea at ucar.edu> wrote:
>
>> Yikes!!  Rick and Alan are much quicker than me!!!  Probably smarter
>> too!! Not nearly as good-looking though!
>>
>> Still, I'll add what I was writing:
>> ============================================
>> Hi Barry,
>>
>> For reading a compressed file, the user need not set any file
>> information. NCL detects the file type and if compression is being used.
>> 'knows' how to uncompress.
>> This holds for 6.3.0 (maybe earlier) ===> 6.5.0
>>
>> ======
>> *%> ncl*
>>  NCAR Command Language Version* 6.3.0*
>>
>> ncl 0> f = *addfile*("wrfout_d03_2018-01-05_00_00_00*.nc*","r")   ; or
>> *.nc4*
>> ncl 1> print(f)    ; <=== see attached
>> =========
>>
>>
>> %> *ncdump -k* wrfout_d03_2018-01-05_00_00_00
>> netCDF-4
>>
>> %> *ncl -V*
>> NCL 6.5.0                   [ Also: NCL 6.3.0 ]
>>
>> *%> ncl_filedump* wrfout_d03_2018-01-05_00_00_00.nc >&!
>> out.wrfout_nc4_compressed
>> *%> less* out.wrfout_nc4_compressed
>>
>> See attached
>>
>> Note:*  Chunking Info*:
>>
>> Variable: XLONG
>> [ 1 <Time> x 237 <south_north> x 150 <west_east> ]
>>
>> Variable: ZNW
>> Chunking Info:      [ 1 <Time> x 44 <bottom_top> ]
>>
>> Variable: ZS
>> Chunking Info:      [ 1 <Time> x 4 <soil_layers_stag> ]
>>
>>
>>
>> On Tue, Apr 9, 2019 at 11:50 AM Rick Brownrigg <brownrig at ucar.edu> wrote:
>>
>>> Yes, to add to what Alan just said, I've been able to open that file
>>> just fine with a simple addfile().  However, if I say
>>>
>>>    f = addfile(.....)
>>>   print(f)
>>>
>>> and then quit out of the pager before all the output has been written to
>>> screen, the code hands indefinitely.  Very odd behavior, and I've not seen
>>> that with other files...Still looking into this one.
>>>
>>>
>>>
>>> On Tue, Apr 9, 2019 at 11:46 AM Brammer, Alan P <abrammer at albany.edu>
>>> wrote:
>>>
>>>> To read a compressed NetCDF you shouldn't need to do anything special.
>>>> Just use the addfile() as normal, no need to specify the setfileoption
>>>> parameters, they will only work on creating (maybe writing) to a file.
>>>>
>>>> In your wrf namelist you can tell wrf to put the ".nc" suffix on the
>>>> file names, so it's more obvious what file it is.  Otherwise just add it as
>>>> a suffix in the ncl addfile command so NCL knows what kind of file it's
>>>> dealing with.
>>>>
>>>> ncl 0> filename = "wrfout_d03_2018-01-05_00_00_00"
>>>> ncl 1> print(addfile(filename+".nc", "r"))
>>>>
>>>>
>>>>
>>>> Alan
>>>>
>>>>
>>>> On Tue, Apr 9, 2019 at 11:28 AM Barry Lynn <barry.h.lynn at gmail.com>
>>>> wrote:
>>>>
>>>>>  wrfout_d03_2018-01-05_00:00:00
>>>>> <https://drive.google.com/file/d/1_C3tTQj3rwnfiCM4TBUYPdD8jOwUynt8/view?usp=drive_web>
>>>>> Please try this one. It is definitely 1 time step of output from WRF.
>>>>>
>>>>> On Tue, Apr 9, 2019 at 8:11 PM Rick Brownrigg <brownrig at ucar.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi Barry,
>>>>>>
>>>>>> This looks like a truncated file?  Its only 336K, and the name is
>>>>>> suspect.?
>>>>>>
>>>>>> RB
>>>>>>
>>>>>> On Tue, Apr 9, 2019 at 10:55 AM Barry Lynn <barry.h.lynn at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> begin
>>>>>>>
>>>>>>>
>>>>>>> ; filename="wrfout_d03_2018-01-05_00:00:00"
>>>>>>>
>>>>>>>   filename="wrfout_compresed.one_time_step.nc.pid19740.ncks.tmp"
>>>>>>>
>>>>>>>
>>>>>>> ;
>>>>>>>
>>>>>>> ;   mpdatabaseversion= "Earth..4"
>>>>>>>
>>>>>>>  setfileoption("nc","format","netcdf4")
>>>>>>>
>>>>>>>  print("got to here 1")
>>>>>>>
>>>>>>> setfileoption("nc","CompressionLevel",5)
>>>>>>>
>>>>>>>  print("got to here 2")
>>>>>>>
>>>>>>> a = addfile(filename,"r")
>>>>>>>
>>>>>>>
>>>>>>> IT DOES NOT PASS THE LINE ABOVE (ncl 6.5)
>>>>>>>
>>>>>>>  print("got to here 3")
>>>>>>>
>>>>>>>  end
>>>>>>>
>>>>>>> On Tue, Apr 9, 2019 at 7:12 PM Barry Lynn <barry.h.lynn at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi:
>>>>>>>>
>>>>>>>> I am working on it.  It will take a bit of time, as I am missing a
>>>>>>>> library.
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>> On Tue, Apr 9, 2019 at 7:09 PM Rick Brownrigg <brownrig at ucar.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Barry,
>>>>>>>>>
>>>>>>>>> Are you able to send me that file?
>>>>>>>>>
>>>>>>>>> Rick
>>>>>>>>>
>>>>>>>>> On Tue, Apr 9, 2019 at 12:40 AM Barry Lynn <barry.h.lynn at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hello:
>>>>>>>>>>
>>>>>>>>>> I am at a loss trying to find an example to read a WRF file
>>>>>>>>>> defined this way.
>>>>>>>>>>
>>>>>>>>>> This build of WRF will use NETCDF4 with HDF5 compression
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I tried the following, but the ncl program just hangs up after
>>>>>>>>>> passing "here 2"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  setfileoption("nc","format","netcdf4")
>>>>>>>>>>
>>>>>>>>>>  print("got to here 1")
>>>>>>>>>>
>>>>>>>>>> setfileoption("nc","CompressionLevel",5)
>>>>>>>>>>
>>>>>>>>>>  print("got to here 2")
>>>>>>>>>>
>>>>>>>>>> a = addfile(filename,"r")
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Barry H. Lynn, Ph.D
>>>>>>>>>> Senior Associate Scientist, Lecturer,
>>>>>>>>>> The Institute of the Earth Science,
>>>>>>>>>> The Hebrew University of Jerusalem,
>>>>>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>>>>>> Tel: 972 547 231 170
>>>>>>>>>> Fax: (972)-25662581
>>>>>>>>>>
>>>>>>>>>> C.E.O, Weather It Is, LTD
>>>>>>>>>> Weather and Climate Focus
>>>>>>>>>> http://weather-it-is.com
>>>>>>>>>> Jerusalem, Israel
>>>>>>>>>> Local: 02 930 9525
>>>>>>>>>> Cell: 054 7 231 170
>>>>>>>>>> Int-IS: x972 2 930 9525
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> ncl-talk mailing list
>>>>>>>>>> ncl-talk at ucar.edu
>>>>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Barry H. Lynn, Ph.D
>>>>>>>> Senior Associate Scientist, Lecturer,
>>>>>>>> The Institute of the Earth Science,
>>>>>>>> The Hebrew University of Jerusalem,
>>>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>>>> Tel: 972 547 231 170
>>>>>>>> Fax: (972)-25662581
>>>>>>>>
>>>>>>>> C.E.O, Weather It Is, LTD
>>>>>>>> Weather and Climate Focus
>>>>>>>> http://weather-it-is.com
>>>>>>>> Jerusalem, Israel
>>>>>>>> Local: 02 930 9525
>>>>>>>> Cell: 054 7 231 170
>>>>>>>> Int-IS: x972 2 930 9525
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Barry H. Lynn, Ph.D
>>>>>>> Senior Associate Scientist, Lecturer,
>>>>>>> The Institute of the Earth Science,
>>>>>>> The Hebrew University of Jerusalem,
>>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>>> Tel: 972 547 231 170
>>>>>>> Fax: (972)-25662581
>>>>>>>
>>>>>>> C.E.O, Weather It Is, LTD
>>>>>>> Weather and Climate Focus
>>>>>>> http://weather-it-is.com
>>>>>>> Jerusalem, Israel
>>>>>>> Local: 02 930 9525
>>>>>>> Cell: 054 7 231 170
>>>>>>> Int-IS: x972 2 930 9525
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Barry H. Lynn, Ph.D
>>>>> Senior Associate Scientist, Lecturer,
>>>>> The Institute of the Earth Science,
>>>>> The Hebrew University of Jerusalem,
>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>> Tel: 972 547 231 170
>>>>> Fax: (972)-25662581
>>>>>
>>>>> C.E.O, Weather It Is, LTD
>>>>> Weather and Climate Focus
>>>>> http://weather-it-is.com
>>>>> Jerusalem, Israel
>>>>> Local: 02 930 9525
>>>>> Cell: 054 7 231 170
>>>>> Int-IS: x972 2 930 9525
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> ncl-talk at ucar.edu
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>
>>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
>
> --
> Barry H. Lynn, Ph.D
> Senior Associate Scientist, Lecturer,
> The Institute of the Earth Science,
> The Hebrew University of Jerusalem,
> Givat Ram, Jerusalem 91904, Israel
> Tel: 972 547 231 170
> Fax: (972)-25662581
>
> C.E.O, Weather It Is, LTD
> Weather and Climate Focus
> http://weather-it-is.com
> Jerusalem, Israel
> Local: 02 930 9525
> Cell: 054 7 231 170
> Int-IS: x972 2 930 9525
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190410/1cddae84/attachment.html>