[Wrf-users] Modifying WRF I/O code to do Lossy compression

[Alex Fierro]

Hi Nikhil:

As far as I know there are two ways to compress/reduce the size of wrf.out
files:

(i) The easy method is via the namelist.input files:

 &time_control
…
 iofields_filename                   =
"my_iofields_list.txt","my_iofields_list.txt",
 ignore_iofields_warning             = .true.,

Where my_iofields_list.txt is an ascii file containing e.g.,:

-:h:0:tlwdn,tlwup,slwdn,slwup,tswdn,tswup,sswdn,sswup,p_hyd
-:h:0:hailnc,hml,huml,hvml,isltyp,ivgtyp,lai,lu_index,mapfac_m,mapfac_mx

which removes (-:h) the variables in the list (tlwdn,tlwup,slwdn...etc) rom
the history stream 0 (:0) at runtime.

(ii) NCO operators. One can get a factor of 12 decrease in file size by
using packing (converting reals to scaled 16-bit integers) + compression,
or about a factor 4-5 for just compression.

ncpdq -O -7 -L 2 wrfout_d02_2012-08-28_12:01:00 wrfoutest.nc
Result:
-rw-r--r--  1 joe  staff   3.4G Mar 23  2014 wrfout_d02_2012-08-28_12:01:00
-rw-r--r--  1 joe  staff   254M Oct 24 13:41 wrfoutest.nc


ncks -7 -L 2  wrfout_d02_2012-08-28_12:01:00 wrfoutest_nopack.nc
Result:
-rw-r--r--  1 joe  staff   3.4G Mar 23  2014 wrfout_d02_2012-08-28_12:01:00
-rw-r--r--  1 joe  staff   699M Oct 24 13:46 wrfoutest_nopack.nc


Combining (i) and (ii) would naturally yield the best results.

Cheers and hope this helps,

-- 
-------------------------------------------------------------
Alexandre Fierro, PhD
Research Scientist-
National Severe Storms Laboratory (NSSL/NOAA)
*The Cooperative Institute for Mesoscale Meteorological Studies* (OU/NOAA)
Los Alamos National Laboratory, Los Alamos, NM (LANL)

"Yesterday is History, Tomorrow is a Mystery and Today is a Gift; That is
why it is called the Present"

"There are only 10 types of people in the world:
Those who understand binary, and those who don't"

"My opinions are my own and not representative of OU, NSSL,
AOML, HRD, LANL or any affiliates."
         ^.^
       (o  o)
     /(   V   )\
   ---m---m----

On 29 July 2015 at 03:53, Nikhil Garg <nikhilgarg.gju at gmail.com> wrote:

>
> I am assuming that you want to save physical space, I was wondering why
> not try netcdf4 for wrf model. It works with WRF model version 3.5 and
> above. Although WRF 3.4.1 can be compiled with nc4 option it doesnt seem to
> work atleast for my case and I use nccopy for compression, but i have
> tested and it works fine in 3.6 version.
>
> I know that it is not really an answer to your question but it does
> save you effort needed for compression using nccopy.
>
> On 28 July 2015 at 16:31, Jatin Kala <jatin.kala.jk at gmail.com> wrote:
>
>> Hi all,
>> Just wondering if anyone has any experience in modifying the WRF I/O
>> source code to do "Lossy" compression for the wrfout_d0* files, i.e.,
>> something along the lines of storing say temperatures as integer with a
>> scale factor to convert to float again later when analyzing the outputs.
>>
>> I already know about NetCDF's nccopy, just wondering if someone has been
>> brave enough to try "Lossy" type compression within the WRF source code,
>> excluding wrfrst_d0* files.
>>
>> Cheers,
>> Jatin.
>> _______________________________________________
>> Wrf-users mailing list
>> Wrf-users at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/wrf-users
>>
>
>
>
> --
> Regards
>
> Nikhil
>
> -------------------------------------------------------------------
> Big whirls have little whirls,
> Which feed on their velocity,
> And little whirls have lesser whirls,
> And so on to viscosity
> (Richardson, 1922)
>
> _______________________________________________
> Wrf-users mailing list
> Wrf-users at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/wrf-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/wrf-users/attachments/20150729/ecb1730e/attachment.html