[ncl-talk] Reading WRF compressed files -- how to?

Tue Apr 9 20:49:20 MDT 2019

Hi Dennis, Rick, Alan.

Thank you for looking into this.

It was noted that NCL knows how to uncompress the file when reading it.
Does this imply that the read takes a lot longer?

Barry

On Tue, Apr 9, 2019 at 9:12 PM Dennis Shea <shea at ucar.edu> wrote:

> Yikes!!  Rick and Alan are much quicker than me!!!  Probably smarter too!!
> Not nearly as good-looking though!
>
> Still, I'll add what I was writing:
> ============================================
> Hi Barry,
>
> For reading a compressed file, the user need not set any file information.
> NCL detects the file type and if compression is being used. 'knows' how to
> uncompress.
> This holds for 6.3.0 (maybe earlier) ===> 6.5.0
>
> ======
> *%> ncl*
>  NCAR Command Language Version* 6.3.0*
>
> ncl 0> f = *addfile*("wrfout_d03_2018-01-05_00_00_00*.nc*","r")   ; or
> *.nc4*
> ncl 1> print(f)    ; <=== see attached
> =========
>
>
> %> *ncdump -k* wrfout_d03_2018-01-05_00_00_00
> netCDF-4
>
> %> *ncl -V*
> NCL 6.5.0                   [ Also: NCL 6.3.0 ]
>
> *%> ncl_filedump* wrfout_d03_2018-01-05_00_00_00.nc >&!
> out.wrfout_nc4_compressed
> *%> less* out.wrfout_nc4_compressed
>
> See attached
>
> Note:*  Chunking Info*:
>
> Variable: XLONG
> [ 1 <Time> x 237 <south_north> x 150 <west_east> ]
>
> Variable: ZNW
> Chunking Info:      [ 1 <Time> x 44 <bottom_top> ]
>
> Variable: ZS
> Chunking Info:      [ 1 <Time> x 4 <soil_layers_stag> ]
>
>
>
> On Tue, Apr 9, 2019 at 11:50 AM Rick Brownrigg <brownrig at ucar.edu> wrote:
>
>> Yes, to add to what Alan just said, I've been able to open that file just
>> fine with a simple addfile().  However, if I say
>>
>>    f = addfile(.....)
>>   print(f)
>>
>> and then quit out of the pager before all the output has been written to
>> screen, the code hands indefinitely.  Very odd behavior, and I've not seen
>> that with other files...Still looking into this one.
>>
>>
>>
>> On Tue, Apr 9, 2019 at 11:46 AM Brammer, Alan P <abrammer at albany.edu>
>> wrote:
>>
>>> To read a compressed NetCDF you shouldn't need to do anything special.
>>> Just use the addfile() as normal, no need to specify the setfileoption
>>> parameters, they will only work on creating (maybe writing) to a file.
>>>
>>> In your wrf namelist you can tell wrf to put the ".nc" suffix on the
>>> file names, so it's more obvious what file it is.  Otherwise just add it as
>>> a suffix in the ncl addfile command so NCL knows what kind of file it's
>>> dealing with.
>>>
>>> ncl 0> filename = "wrfout_d03_2018-01-05_00_00_00"
>>> ncl 1> print(addfile(filename+".nc", "r"))
>>>
>>>
>>>
>>> Alan
>>>
>>>
>>> On Tue, Apr 9, 2019 at 11:28 AM Barry Lynn <barry.h.lynn at gmail.com>
>>> wrote:
>>>
>>>>  wrfout_d03_2018-01-05_00:00:00
>>>> <https://drive.google.com/file/d/1_C3tTQj3rwnfiCM4TBUYPdD8jOwUynt8/view?usp=drive_web>
>>>> Please try this one. It is definitely 1 time step of output from WRF.
>>>>
>>>> On Tue, Apr 9, 2019 at 8:11 PM Rick Brownrigg <brownrig at ucar.edu>
>>>> wrote:
>>>>
>>>>> Hi Barry,
>>>>>
>>>>> This looks like a truncated file?  Its only 336K, and the name is
>>>>> suspect.?
>>>>>
>>>>> RB
>>>>>
>>>>> On Tue, Apr 9, 2019 at 10:55 AM Barry Lynn <barry.h.lynn at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> begin
>>>>>>
>>>>>>
>>>>>> ; filename="wrfout_d03_2018-01-05_00:00:00"
>>>>>>
>>>>>>   filename="wrfout_compresed.one_time_step.nc.pid19740.ncks.tmp"
>>>>>>
>>>>>>
>>>>>> ;
>>>>>>
>>>>>> ;   mpdatabaseversion= "Earth..4"
>>>>>>
>>>>>>  setfileoption("nc","format","netcdf4")
>>>>>>
>>>>>>  print("got to here 1")
>>>>>>
>>>>>> setfileoption("nc","CompressionLevel",5)
>>>>>>
>>>>>>  print("got to here 2")
>>>>>>
>>>>>> a = addfile(filename,"r")
>>>>>>
>>>>>>
>>>>>> IT DOES NOT PASS THE LINE ABOVE (ncl 6.5)
>>>>>>
>>>>>>  print("got to here 3")
>>>>>>
>>>>>>  end
>>>>>>
>>>>>> On Tue, Apr 9, 2019 at 7:12 PM Barry Lynn <barry.h.lynn at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi:
>>>>>>>
>>>>>>> I am working on it.  It will take a bit of time, as I am missing a
>>>>>>> library.
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>> On Tue, Apr 9, 2019 at 7:09 PM Rick Brownrigg <brownrig at ucar.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Barry,
>>>>>>>>
>>>>>>>> Are you able to send me that file?
>>>>>>>>
>>>>>>>> Rick
>>>>>>>>
>>>>>>>> On Tue, Apr 9, 2019 at 12:40 AM Barry Lynn <barry.h.lynn at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello:
>>>>>>>>>
>>>>>>>>> I am at a loss trying to find an example to read a WRF file
>>>>>>>>> defined this way.
>>>>>>>>>
>>>>>>>>> This build of WRF will use NETCDF4 with HDF5 compression
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I tried the following, but the ncl program just hangs up after
>>>>>>>>> passing "here 2"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  setfileoption("nc","format","netcdf4")
>>>>>>>>>
>>>>>>>>>  print("got to here 1")
>>>>>>>>>
>>>>>>>>> setfileoption("nc","CompressionLevel",5)
>>>>>>>>>
>>>>>>>>>  print("got to here 2")
>>>>>>>>>
>>>>>>>>> a = addfile(filename,"r")
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Barry H. Lynn, Ph.D
>>>>>>>>> Senior Associate Scientist, Lecturer,
>>>>>>>>> The Institute of the Earth Science,
>>>>>>>>> The Hebrew University of Jerusalem,
>>>>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>>>>> Tel: 972 547 231 170
>>>>>>>>> Fax: (972)-25662581
>>>>>>>>>
>>>>>>>>> C.E.O, Weather It Is, LTD
>>>>>>>>> Weather and Climate Focus
>>>>>>>>> http://weather-it-is.com
>>>>>>>>> Jerusalem, Israel
>>>>>>>>> Local: 02 930 9525
>>>>>>>>> Cell: 054 7 231 170
>>>>>>>>> Int-IS: x972 2 930 9525
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> ncl-talk mailing list
>>>>>>>>> ncl-talk at ucar.edu
>>>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Barry H. Lynn, Ph.D
>>>>>>> Senior Associate Scientist, Lecturer,
>>>>>>> The Institute of the Earth Science,
>>>>>>> The Hebrew University of Jerusalem,
>>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>>> Tel: 972 547 231 170
>>>>>>> Fax: (972)-25662581
>>>>>>>
>>>>>>> C.E.O, Weather It Is, LTD
>>>>>>> Weather and Climate Focus
>>>>>>> http://weather-it-is.com
>>>>>>> Jerusalem, Israel
>>>>>>> Local: 02 930 9525
>>>>>>> Cell: 054 7 231 170
>>>>>>> Int-IS: x972 2 930 9525
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Barry H. Lynn, Ph.D
>>>>>> Senior Associate Scientist, Lecturer,
>>>>>> The Institute of the Earth Science,
>>>>>> The Hebrew University of Jerusalem,
>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>> Tel: 972 547 231 170
>>>>>> Fax: (972)-25662581
>>>>>>
>>>>>> C.E.O, Weather It Is, LTD
>>>>>> Weather and Climate Focus
>>>>>> http://weather-it-is.com
>>>>>> Jerusalem, Israel
>>>>>> Local: 02 930 9525
>>>>>> Cell: 054 7 231 170
>>>>>> Int-IS: x972 2 930 9525
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Barry H. Lynn, Ph.D
>>>> Senior Associate Scientist, Lecturer,
>>>> The Institute of the Earth Science,
>>>> The Hebrew University of Jerusalem,
>>>> Givat Ram, Jerusalem 91904, Israel
>>>> Tel: 972 547 231 170
>>>> Fax: (972)-25662581
>>>>
>>>> C.E.O, Weather It Is, LTD
>>>> Weather and Climate Focus
>>>> http://weather-it-is.com
>>>> Jerusalem, Israel
>>>> Local: 02 930 9525
>>>> Cell: 054 7 231 170
>>>> Int-IS: x972 2 930 9525
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> ncl-talk at ucar.edu
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

-- 
Barry H. Lynn, Ph.D
Senior Associate Scientist, Lecturer,
The Institute of the Earth Science,
The Hebrew University of Jerusalem,
Givat Ram, Jerusalem 91904, Israel
Tel: 972 547 231 170
Fax: (972)-25662581

C.E.O, Weather It Is, LTD
Weather and Climate Focus
http://weather-it-is.com
Jerusalem, Israel
Local: 02 930 9525
Cell: 054 7 231 170
Int-IS: x972 2 930 9525
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190410/16ae6f7a/attachment-0001.html>