[ncl-talk] Reading WRF compressed files -- how to?
Barry Lynn
barry.h.lynn at gmail.com
Thu Apr 11 06:46:21 MDT 2019
Hi:
The good news is that now we can read the file. The "bad" news is that we
had to use a newer computer with updated software to do it (still with NCL
6.5).
So, we may never know the mystery of why we couldn't read this software.
As a side note, do you know of a way to compress our old files to have the
same format as the new ones? Our new files are netcdf hdp5.
I was thinking of using these commands when writing:
setfileoption("nc","format","netcdf4")
print("got to here 1")
setfileoption("nc","CompressionLevel",5)
But, I am not really sure how to read an entire file (standard, classic)
and write it entirely (these are 100 GB files).
I just know how to read individual variables.
Thank you for your suggestion.
Barry
On Wed, Apr 10, 2019 at 10:13 PM Rick Brownrigg <brownrig at ucar.edu> wrote:
> I also have not been able to duplicate the problem. I do see that weird
> issue where NCL hangs if the output of f=addfile/print(f) is interrupted,
> so *something* is odd about that file.
>
> Barry, not sure what to suggest. I'm wondering where it is hanging and
> what its doing. We can get some insight if you are willing to do the
> following (caveat, I can't guarantee a solution will emerge):
>
> Run ncl under gdb:
> gdb ncl
>
> when get the (gdb) prompt, type:
> run nameOfYourScript.ncl
>
> let that run for a sufficiently long time until your certain its hung, and
> type CTRL-C. You get the (gdb) prompt back, and type:
> bt
>
> Copy that output into an email, and type
> cont
>
> The program will resume running; let it run a minute more and repeat the
> CRTL-C/bt/copy output step again and please the outputs to me. Type "quit"
> to get out of gdb. (the reason for doing the CTRL-C business twice is I'm
> trying to see if its stuck solidly in one place, or still actively
> executing something)
>
> Another thing you might try is running:
> strace ncl nameOfYourScript.ncl
>
> This will generate tons of output -- I'm curious if this settles into the
> same message being printed repeatedly. If so, could you also send that to
> me? Type CTRL-C to get out strace.
>
> Again, not sure any of this will help, but short of running on your
> machine, I don't know what to suggest.
>
> Rick
>
> On Wed, Apr 10, 2019 at 12:54 PM Dennis Shea <shea at ucar.edu> wrote:
>
>> Hi Barry,
>>
>> I am not the person to ask about this type of issue. I believe it is a
>> missing library or, perhaps, a 'screwed-up' library.
>> I can run that file on my MAC and two separate systems [divisional and
>> NCAR main] without any problem.
>> As previously noted, I used 6.3.0 & 6.5.0 without issues.
>>
>> I am cc'ing Kevin Hallock and Rick Brownrigg. Perhaps, they can help
>> determine the issue.
>>
>> You will have to provide information from the following:
>>
>> %> uname -a
>> %> ncl -V
>>
>> Good luck
>> D
>>
>> On Wed, Apr 10, 2019 at 10:37 AM Barry Lynn <barry.h.lynn at gmail.com>
>> wrote:
>>
>>> Hi Dennis:
>>>
>>> I wasn't sure if I should respond to you or generally.
>>>
>>> I am unable to read that one time step file I sent. The program just
>>> hangs.
>>>
>>> I noticed that the file takes up about 1/5 the space, and each disk set
>>> costs us about $6000. So, if we could read that file we can really save on
>>> energy and other resources.
>>>
>>> Barry
>>>
>>> On Wed, Apr 10, 2019 at 5:40 PM Dennis Shea <shea at ucar.edu> wrote:
>>>
>>>> Hi Barry,
>>>>
>>>> offline ....
>>>>
>>>> Sorry for the one word response last night. It was sent from my cell
>>>> phone.
>>>> --
>>>>
>>>> The 'timing hit' is when the file is *created. *Reading a compressed
>>>> file has minimal overhead and is quite efficient.
>>>>
>>>> Creating a file large with low compression incurs minimal overhead
>>>> while higher the compression level can have significant overhead.
>>>>
>>>>
>>>>
>>>> *http://www.ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml*
>>>> <http://www.ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml>
>>>>
>>>> contains:
>>>>
>>>> *CompressionLevel* *Default value*: -1
>>>>
>>>> Specify the level of data compression as an integer in the range 0
>>>> through 9. Increasing values indicate greater compression. Compression is
>>>> lossless. There are tradeoffs between the time spent compressing the file,
>>>> versus the amount of compression achieved. Informal tests show that
>>>> compression level 9 results in a file only a few percent smaller than a
>>>> compression level 5 file, but it requires 4 or 5 times the amount of time
>>>> to create it. (This option is ignored unless the *Format* option is
>>>> set to *"NetCDF4Classic"* or *"NetCDF4"*.)
>>>> Cheers
>>>> D
>>>>
>>>> On Tue, Apr 9, 2019 at 9:16 PM Dennis Shea <shea at ucar.edu> wrote:
>>>>
>>>>> No.
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On Apr 9, 2019, at 8:49 PM, Barry Lynn <barry.h.lynn at gmail.com> wrote:
>>>>>
>>>>> Hi Dennis, Rick, Alan.
>>>>>
>>>>> Thank you for looking into this.
>>>>>
>>>>> It was noted that NCL knows how to uncompress the file when reading
>>>>> it. Does this imply that the read takes a lot longer?
>>>>>
>>>>> Barry
>>>>>
>>>>> On Tue, Apr 9, 2019 at 9:12 PM Dennis Shea <shea at ucar.edu> wrote:
>>>>>
>>>>>> Yikes!! Rick and Alan are much quicker than me!!! Probably smarter
>>>>>> too!! Not nearly as good-looking though!
>>>>>>
>>>>>> Still, I'll add what I was writing:
>>>>>> ============================================
>>>>>> Hi Barry,
>>>>>>
>>>>>> For reading a compressed file, the user need not set any file
>>>>>> information. NCL detects the file type and if compression is being used.
>>>>>> 'knows' how to uncompress.
>>>>>> This holds for 6.3.0 (maybe earlier) ===> 6.5.0
>>>>>>
>>>>>> ======
>>>>>> *%> ncl*
>>>>>> NCAR Command Language Version* 6.3.0*
>>>>>>
>>>>>> ncl 0> f = *addfile*("wrfout_d03_2018-01-05_00_00_00*.nc*","r") ;
>>>>>> or *.nc4*
>>>>>> ncl 1> print(f) ; <=== see attached
>>>>>> =========
>>>>>>
>>>>>>
>>>>>> %> *ncdump -k* wrfout_d03_2018-01-05_00_00_00
>>>>>> netCDF-4
>>>>>>
>>>>>> %> *ncl -V*
>>>>>> NCL 6.5.0 [ Also: NCL 6.3.0 ]
>>>>>>
>>>>>> *%> ncl_filedump* wrfout_d03_2018-01-05_00_00_00.nc >&!
>>>>>> out.wrfout_nc4_compressed
>>>>>> *%> less* out.wrfout_nc4_compressed
>>>>>>
>>>>>> See attached
>>>>>>
>>>>>> Note:* Chunking Info*:
>>>>>>
>>>>>> Variable: XLONG
>>>>>> [ 1 <Time> x 237 <south_north> x 150 <west_east> ]
>>>>>>
>>>>>> Variable: ZNW
>>>>>> Chunking Info: [ 1 <Time> x 44 <bottom_top> ]
>>>>>>
>>>>>> Variable: ZS
>>>>>> Chunking Info: [ 1 <Time> x 4 <soil_layers_stag> ]
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 9, 2019 at 11:50 AM Rick Brownrigg <brownrig at ucar.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, to add to what Alan just said, I've been able to open that file
>>>>>>> just fine with a simple addfile(). However, if I say
>>>>>>>
>>>>>>> f = addfile(.....)
>>>>>>> print(f)
>>>>>>>
>>>>>>> and then quit out of the pager before all the output has been
>>>>>>> written to screen, the code hands indefinitely. Very odd behavior, and
>>>>>>> I've not seen that with other files...Still looking into this one.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 9, 2019 at 11:46 AM Brammer, Alan P <abrammer at albany.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> To read a compressed NetCDF you shouldn't need to do anything
>>>>>>>> special. Just use the addfile() as normal, no need to specify the
>>>>>>>> setfileoption parameters, they will only work on creating (maybe writing)
>>>>>>>> to a file.
>>>>>>>>
>>>>>>>> In your wrf namelist you can tell wrf to put the ".nc" suffix on
>>>>>>>> the file names, so it's more obvious what file it is. Otherwise just add
>>>>>>>> it as a suffix in the ncl addfile command so NCL knows what kind of file
>>>>>>>> it's dealing with.
>>>>>>>>
>>>>>>>> ncl 0> filename = "wrfout_d03_2018-01-05_00_00_00"
>>>>>>>> ncl 1> print(addfile(filename+".nc", "r"))
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Alan
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 9, 2019 at 11:28 AM Barry Lynn <barry.h.lynn at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> wrfout_d03_2018-01-05_00:00:00
>>>>>>>>> <https://drive.google.com/file/d/1_C3tTQj3rwnfiCM4TBUYPdD8jOwUynt8/view?usp=drive_web>
>>>>>>>>> Please try this one. It is definitely 1 time step of output from
>>>>>>>>> WRF.
>>>>>>>>>
>>>>>>>>> On Tue, Apr 9, 2019 at 8:11 PM Rick Brownrigg <brownrig at ucar.edu>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Barry,
>>>>>>>>>>
>>>>>>>>>> This looks like a truncated file? Its only 336K, and the name is
>>>>>>>>>> suspect.?
>>>>>>>>>>
>>>>>>>>>> RB
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 9, 2019 at 10:55 AM Barry Lynn <
>>>>>>>>>> barry.h.lynn at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> begin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ; filename="wrfout_d03_2018-01-05_00:00:00"
>>>>>>>>>>>
>>>>>>>>>>> filename="wrfout_compresed.one_time_step.nc.pid19740.ncks.tmp"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ;
>>>>>>>>>>>
>>>>>>>>>>> ; mpdatabaseversion= "Earth..4"
>>>>>>>>>>>
>>>>>>>>>>> setfileoption("nc","format","netcdf4")
>>>>>>>>>>>
>>>>>>>>>>> print("got to here 1")
>>>>>>>>>>>
>>>>>>>>>>> setfileoption("nc","CompressionLevel",5)
>>>>>>>>>>>
>>>>>>>>>>> print("got to here 2")
>>>>>>>>>>>
>>>>>>>>>>> a = addfile(filename,"r")
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> IT DOES NOT PASS THE LINE ABOVE (ncl 6.5)
>>>>>>>>>>>
>>>>>>>>>>> print("got to here 3")
>>>>>>>>>>>
>>>>>>>>>>> end
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 9, 2019 at 7:12 PM Barry Lynn <
>>>>>>>>>>> barry.h.lynn at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi:
>>>>>>>>>>>>
>>>>>>>>>>>> I am working on it. It will take a bit of time, as I am
>>>>>>>>>>>> missing a library.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 9, 2019 at 7:09 PM Rick Brownrigg <
>>>>>>>>>>>> brownrig at ucar.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Barry,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Are you able to send me that file?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Rick
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Apr 9, 2019 at 12:40 AM Barry Lynn <
>>>>>>>>>>>>> barry.h.lynn at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am at a loss trying to find an example to read a WRF file
>>>>>>>>>>>>>> defined this way.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This build of WRF will use NETCDF4 with HDF5 compression
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I tried the following, but the ncl program just hangs up
>>>>>>>>>>>>>> after passing "here 2"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> setfileoption("nc","format","netcdf4")
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> print("got to here 1")
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> setfileoption("nc","CompressionLevel",5)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> print("got to here 2")
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> a = addfile(filename,"r")
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Barry H. Lynn, Ph.D
>>>>>>>>>>>>>> Senior Associate Scientist, Lecturer,
>>>>>>>>>>>>>> The Institute of the Earth Science,
>>>>>>>>>>>>>> The Hebrew University of Jerusalem,
>>>>>>>>>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>>>>>>>>>> Tel: 972 547 231 170
>>>>>>>>>>>>>> Fax: (972)-25662581
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> C.E.O, Weather It Is, LTD
>>>>>>>>>>>>>> Weather and Climate Focus
>>>>>>>>>>>>>> http://weather-it-is.com
>>>>>>>>>>>>>> Jerusalem, Israel
>>>>>>>>>>>>>> Local: 02 930 9525
>>>>>>>>>>>>>> Cell: 054 7 231 170
>>>>>>>>>>>>>> Int-IS: x972 2 930 9525
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> ncl-talk mailing list
>>>>>>>>>>>>>> ncl-talk at ucar.edu
>>>>>>>>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Barry H. Lynn, Ph.D
>>>>>>>>>>>> Senior Associate Scientist, Lecturer,
>>>>>>>>>>>> The Institute of the Earth Science,
>>>>>>>>>>>> The Hebrew University of Jerusalem,
>>>>>>>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>>>>>>>> Tel: 972 547 231 170
>>>>>>>>>>>> Fax: (972)-25662581
>>>>>>>>>>>>
>>>>>>>>>>>> C.E.O, Weather It Is, LTD
>>>>>>>>>>>> Weather and Climate Focus
>>>>>>>>>>>> http://weather-it-is.com
>>>>>>>>>>>> Jerusalem, Israel
>>>>>>>>>>>> Local: 02 930 9525
>>>>>>>>>>>> Cell: 054 7 231 170
>>>>>>>>>>>> Int-IS: x972 2 930 9525
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Barry H. Lynn, Ph.D
>>>>>>>>>>> Senior Associate Scientist, Lecturer,
>>>>>>>>>>> The Institute of the Earth Science,
>>>>>>>>>>> The Hebrew University of Jerusalem,
>>>>>>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>>>>>>> Tel: 972 547 231 170
>>>>>>>>>>> Fax: (972)-25662581
>>>>>>>>>>>
>>>>>>>>>>> C.E.O, Weather It Is, LTD
>>>>>>>>>>> Weather and Climate Focus
>>>>>>>>>>> http://weather-it-is.com
>>>>>>>>>>> Jerusalem, Israel
>>>>>>>>>>> Local: 02 930 9525
>>>>>>>>>>> Cell: 054 7 231 170
>>>>>>>>>>> Int-IS: x972 2 930 9525
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Barry H. Lynn, Ph.D
>>>>>>>>> Senior Associate Scientist, Lecturer,
>>>>>>>>> The Institute of the Earth Science,
>>>>>>>>> The Hebrew University of Jerusalem,
>>>>>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>>>>>> Tel: 972 547 231 170
>>>>>>>>> Fax: (972)-25662581
>>>>>>>>>
>>>>>>>>> C.E.O, Weather It Is, LTD
>>>>>>>>> Weather and Climate Focus
>>>>>>>>> http://weather-it-is.com
>>>>>>>>> Jerusalem, Israel
>>>>>>>>> Local: 02 930 9525
>>>>>>>>> Cell: 054 7 231 170
>>>>>>>>> Int-IS: x972 2 930 9525
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> ncl-talk mailing list
>>>>>>>>> ncl-talk at ucar.edu
>>>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>> ncl-talk mailing list
>>>>>>> ncl-talk at ucar.edu
>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>
>>>>>> _______________________________________________
>>>>>> ncl-talk mailing list
>>>>>> ncl-talk at ucar.edu
>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Barry H. Lynn, Ph.D
>>>>> Senior Associate Scientist, Lecturer,
>>>>> The Institute of the Earth Science,
>>>>> The Hebrew University of Jerusalem,
>>>>> Givat Ram, Jerusalem 91904, Israel
>>>>> Tel: 972 547 231 170
>>>>> Fax: (972)-25662581
>>>>>
>>>>> C.E.O, Weather It Is, LTD
>>>>> Weather and Climate Focus
>>>>> http://weather-it-is.com
>>>>> Jerusalem, Israel
>>>>> Local: 02 930 9525
>>>>> Cell: 054 7 231 170
>>>>> Int-IS: x972 2 930 9525
>>>>>
>>>>>
>>>
>>> --
>>> Barry H. Lynn, Ph.D
>>> Senior Associate Scientist, Lecturer,
>>> The Institute of the Earth Science,
>>> The Hebrew University of Jerusalem,
>>> Givat Ram, Jerusalem 91904, Israel
>>> Tel: 972 547 231 170
>>> Fax: (972)-25662581
>>>
>>> C.E.O, Weather It Is, LTD
>>> Weather and Climate Focus
>>> http://weather-it-is.com
>>> Jerusalem, Israel
>>> Local: 02 930 9525
>>> Cell: 054 7 231 170
>>> Int-IS: x972 2 930 9525
>>>
>>>
--
Barry H. Lynn, Ph.D
Senior Associate Scientist, Lecturer,
The Institute of the Earth Science,
The Hebrew University of Jerusalem,
Givat Ram, Jerusalem 91904, Israel
Tel: 972 547 231 170
Fax: (972)-25662581
C.E.O, Weather It Is, LTD
Weather and Climate Focus
http://weather-it-is.com
Jerusalem, Israel
Local: 02 930 9525
Cell: 054 7 231 170
Int-IS: x972 2 930 9525
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190411/2f0f7f95/attachment.html>
More information about the ncl-talk
mailing list