[ncl-talk] Convert unpacked netCDF file to packed one

[Dave Allured - NOAA Affiliate]

If you can't update the model, then I suggest method 3 to write a new
file.  This starts with Dennis's code to convert to packed in memory.
Untested:

    f = addfile ("air.2019.nc", "r")
    xFloat = f->air
    xShort = pack_values (xFloat, "short", False)
    fout = addfile ("new_air.2019.nc", "c")
    fout->air = xShort
    exit

This method 3 is not in the documentation, but it should be.  This only
works when all necessary metadata -- especially coordinate variables -- is
preserved or attached in memory before writing output. Fortunately in this
case, the *pack_values* function should preserve metadata.  Use
*printVarSummary* liberally to check intermediate results and verify
assumptions.


On Tue, Jan 12, 2021 at 10:18 AM Dennis Shea via ncl-talk <
ncl-talk at mailman.ucar.edu> wrote:

> As noted before, the BEST solution is to have the model code altered too
> accommodate the original values. IE: to not require packed values.
>
> ====
> The approach of created netCDF files containing variables with the
> scale_factor and add_offset attributes is cumbersome. You will have two
> files (original: netCDF4) and a (much larger) "packed" file.  That is for
> EVERY variable you need. It is best to fix the model source code.
>
> Also, deficiencies in the model code will have to be continuously
> addressed  by NCL thumb-in-the-dike solutions.
> =====
>
> Your netcdf new-hgt.2019 file has
>
> dimensions:
>
> time = UNLIMITED ; // (0 currently)
>
>
> I have no idea why there is no 'time' dimension associated with 'hgt',
> Is the source variable just (lat,lon)
>
> If so, delete
>
> ;===================================================================
> ; make time an UNLIMITED dimension; recommended  for most applications
> ;===================================================================
>   *filedimdef* <http://www.ncl.ucar.edu/Document/Functions/Built-in/filedimdef.shtml>(ncdf,"time",-1,True)
>
>
> On Tue, Jan 12, 2021 at 6:16 AM Setareh Rahimi <setareh.rahimi at gmail.com>
> wrote:
>
>> Dear Dennis,
>> Many thanks for your help. Once I follow the
>> http://www.ncl.ucar.edu/Applications/method_1.shtml method to save the
>> new nc file, I face some errors, in fact I am too new in writhing nc files.
>> That method just give me :
>>
>> netcdf new-hgt.2019 {
>>
>> dimensions:
>>
>> time = UNLIMITED ; // (0 currently)
>>
>>
>> // global attributes:
>>
>> :creation_date = "Tue Jan 12 16:33:58 +0330 2021" ;
>>
>> :Conventions = "CF-1.0 " ;
>>
>> :source_file = "NCEP/DOE AMIP-II Reanalysis (Reanalysis-2) Model" ;
>>
>> :title = "4x daily NCEP/DOE Reanalysis 2" ;
>>
>>
>> How can I have the correct nc file?
>>
>>
>> Best wishes
>>
>> On Mon, Jan 11, 2021 at 11:35 PM Dennis Shea <shea at ucar.edu> wrote:
>>
>>> I was just typing a response that contained the same information as
>>> Dave's.
>>>
>>> Likely, you have no control "the model I am running only accept packed
>>> nc file with scale factor."
>>> As noted by DaveA, the file you are using is netCDF-4
>>> [ Perhaps too much info but netCDF-4's compression is described *here ]*
>>> <https://www.unidata.ucar.edu/blogs/developer/entry/netcdf_compression>
>>> ===============
>>>
>>> %> *ncdump -k* air.2019.nc
>>> netCDF-4 classic model
>>>
>>> In fact, I believe *all* the netCDF files created by "NCEP-DOE AMIP-II
>>> Reanalysis" (and others) will be netCDF-4.
>>>
>>> Maybe you could have the model developers alter the code.
>>> Really, they should have checked if the 'scale_factor' and/or
>>> 'add_offset' attributes are present.
>>> If not, use the variable directly.
>>> ====================
>>>
>>>             f = addfile("air.2019.nc","r")
>>>             x = f->air
>>>             printVarSummary(x)                 ; full uncompressed values
>>>                                                              ; *** LOOK
>>> AT THE SIZE****
>>>                                                              ; It is
>>> MUCH LARGER than the ENTIRE netCDF file
>>>
>>>       xShort   = *pack_values*(xFloat, "short", False)
>>>       *printVarSummary* <http://www.ncl.ucar.edu/Document/Functions/Built-in/printVarSummary.shtml>(xShort)
>>>
>>>       xShort   = pack_values(x, "short", False)
>>>       printVarSummary(xShort)  ; *** LOOK AT THE SIZE****
>>>                                ; HALF the size of 'x'
>>>                                ; but much larger than the compressed value!
>>>
>>>  The use (say) the following to write 'xShort' a new (much larger) file that the original
>>>
>>> *http://www.ncl.ucar.edu/Applications/method_1.shtml* <http://www.ncl.ucar.edu/Applications/method_1.shtml>
>>>
>>> Then use that new file for the model.
>>>
>>> Likely,* EVERY *variable will have to be done like this for the "model you are using
>>>
>>>
>>> *===*
>>>
>>>
>>>
>>> *It is much easier for the model code to be altered.*
>>>
>>>
>>> *Good Luck*
>>>
>>>
>>> On Mon, Jan 11, 2021 at 12:46 PM Dave Allured - NOAA Affiliate via
>>> ncl-talk <ncl-talk at mailman.ucar.edu> wrote:
>>>
>>>> I see.  I suggest the best solution would be a simple upgrade to the
>>>> model code, so that it could read both packed and unpacked air temp files.
>>>> This will enable the model to use future data updates without extra
>>>> processing.
>>>>
>>>> In fortran, this would be an if statement to check whether there is a
>>>> scale_factor attribute.  If not, then skip reading scale_factor and
>>>> add_offset, also skip the unpacking statement or loop that converts packed
>>>> data values to unpacked.  Just read in the unpacked values directly.  You
>>>> would need two read statements, the original one for the packed data type,
>>>> and a new one for the unpacked data type (fortran real).
>>>>
>>>> If something prevents you from changing the model code, then just
>>>> convert the air temp file to packed using the NCL *pack_values* function.
>>>> See the general documentation for how to read and write a netcdf file.
>>>>
>>>>
>>>> On Mon, Jan 11, 2021 at 12:00 PM Setareh Rahimi <
>>>> setareh.rahimi at gmail.com> wrote:
>>>>
>>>>> Dear Dave,
>>>>> Thank you for your reply. Yes I am sure, because the model I am
>>>>> running only accept packed nc file with scale factor.
>>>>> Please have a look at attached file showing the error I face while
>>>>> running the model with mentioned nc file.
>>>>> Best regard
>>>>>
>>>>> On Mon, Jan 11, 2021 at 22:20 Dave Allured - NOAA Affiliate <
>>>>> dave.allured at noaa.gov> wrote:
>>>>>
>>>>>> That file is already compressed to 31% of its natural array size,
>>>>>> using netcdf-4 internal compression.  Are you sure you want to do that?  If
>>>>>> yes, then use the NCL function *pack_values*.
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 11, 2021 at 11:07 AM Setareh Rahimi via ncl-talk <
>>>>>> ncl-talk at mailman.ucar.edu> wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>> I need to convert an unpacked netCDF file to a packed one (
>>>>>>> ftp://ftp2.psl.noaa.gov/Datasets/ncep.reanalysis2/pressure/air.2019.nc),
>>>>>>> I wonder if this possible using NCL?
>>>>>>> Many thanks in advance,
>>>>>>> Best wishes,
>>>>>>> --
>>>>>>> S.Rahimi
>>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210112/46281e6c/attachment.html>