[ncl-talk] Convert unpacked netCDF file to packed one

[Dennis Shea]

I was just typing a response that contained the same information as Dave's.

Likely, you have no control "the model I am running only accept packed nc
file with scale factor."
As noted by DaveA, the file you are using is netCDF-4
[ Perhaps too much info but netCDF-4's compression is described *here ]*
<https://www.unidata.ucar.edu/blogs/developer/entry/netcdf_compression>
===============

%> *ncdump -k* air.2019.nc
netCDF-4 classic model

In fact, I believe *all* the netCDF files created by "NCEP-DOE AMIP-II
Reanalysis" (and others) will be netCDF-4.

Maybe you could have the model developers alter the code.
Really, they should have checked if the 'scale_factor' and/or 'add_offset'
attributes are present.
If not, use the variable directly.
====================

            f = addfile("air.2019.nc","r")
            x = f->air
            printVarSummary(x)                 ; full uncompressed values
                                                             ; *** LOOK AT
THE SIZE****
                                                             ; It is MUCH
LARGER than the ENTIRE netCDF file

      xShort   = *pack_values*(xFloat, "short", False)
      *printVarSummary*
<http://www.ncl.ucar.edu/Document/Functions/Built-in/printVarSummary.shtml>(xShort)

      xShort   = pack_values(x, "short", False)
      printVarSummary(xShort)  ; *** LOOK AT THE SIZE****
                               ; HALF the size of 'x'
                               ; but much larger than the compressed value!

 The use (say) the following to write 'xShort' a new (much larger)
file that the original

*http://www.ncl.ucar.edu/Applications/method_1.shtml*
<http://www.ncl.ucar.edu/Applications/method_1.shtml>

Then use that new file for the model.

Likely,* EVERY *variable will have to be done like this for the "model
you are using


*===*



*It is much easier for the model code to be altered.*


*Good Luck*


On Mon, Jan 11, 2021 at 12:46 PM Dave Allured - NOAA Affiliate via ncl-talk
<ncl-talk at mailman.ucar.edu> wrote:

> I see.  I suggest the best solution would be a simple upgrade to the model
> code, so that it could read both packed and unpacked air temp files.  This
> will enable the model to use future data updates without extra processing.
>
> In fortran, this would be an if statement to check whether there is a
> scale_factor attribute.  If not, then skip reading scale_factor and
> add_offset, also skip the unpacking statement or loop that converts packed
> data values to unpacked.  Just read in the unpacked values directly.  You
> would need two read statements, the original one for the packed data type,
> and a new one for the unpacked data type (fortran real).
>
> If something prevents you from changing the model code, then just convert
> the air temp file to packed using the NCL *pack_values* function.  See
> the general documentation for how to read and write a netcdf file.
>
>
> On Mon, Jan 11, 2021 at 12:00 PM Setareh Rahimi <setareh.rahimi at gmail.com>
> wrote:
>
>> Dear Dave,
>> Thank you for your reply. Yes I am sure, because the model I am running
>> only accept packed nc file with scale factor.
>> Please have a look at attached file showing the error I face while
>> running the model with mentioned nc file.
>> Best regard
>>
>> On Mon, Jan 11, 2021 at 22:20 Dave Allured - NOAA Affiliate <
>> dave.allured at noaa.gov> wrote:
>>
>>> That file is already compressed to 31% of its natural array size, using
>>> netcdf-4 internal compression.  Are you sure you want to do that?  If yes,
>>> then use the NCL function *pack_values*.
>>>
>>>
>>> On Mon, Jan 11, 2021 at 11:07 AM Setareh Rahimi via ncl-talk <
>>> ncl-talk at mailman.ucar.edu> wrote:
>>>
>>>> Dear all,
>>>> I need to convert an unpacked netCDF file to a packed one (
>>>> ftp://ftp2.psl.noaa.gov/Datasets/ncep.reanalysis2/pressure/air.2019.nc),
>>>> I wonder if this possible using NCL?
>>>> Many thanks in advance,
>>>> Best wishes,
>>>> --
>>>> S.Rahimi
>>>>
>>> _______________________________________________
> ncl-talk mailing list
> ncl-talk at mailman.ucar.edu
> List instructions, subscriber options, unsubscribe:
> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210111/03e379c1/attachment.html>