[ncl-talk] Convert unpacked netCDF file to packed one
Setareh Rahimi
setareh.rahimi at gmail.com
Tue Jan 12 23:56:07 MST 2021
Dear Dave,
Many thanks for your help and guide, my problem has been fixed now. Very
useful suggestions!
Best wishes,
On Tue, Jan 12, 2021 at 11:57 PM Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:
> If you can't update the model, then I suggest method 3 to write a new
> file. This starts with Dennis's code to convert to packed in memory.
> Untested:
>
> f = addfile ("air.2019.nc", "r")
> xFloat = f->air
> xShort = pack_values (xFloat, "short", False)
> fout = addfile ("new_air.2019.nc", "c")
> fout->air = xShort
> exit
>
> This method 3 is not in the documentation, but it should be. This only
> works when all necessary metadata -- especially coordinate variables -- is
> preserved or attached in memory before writing output. Fortunately in this
> case, the *pack_values* function should preserve metadata. Use
> *printVarSummary* liberally to check intermediate results and verify
> assumptions.
>
>
> On Tue, Jan 12, 2021 at 10:18 AM Dennis Shea via ncl-talk <
> ncl-talk at mailman.ucar.edu> wrote:
>
>> As noted before, the BEST solution is to have the model code altered too
>> accommodate the original values. IE: to not require packed values.
>>
>> ====
>> The approach of created netCDF files containing variables with the
>> scale_factor and add_offset attributes is cumbersome. You will have two
>> files (original: netCDF4) and a (much larger) "packed" file. That is for
>> EVERY variable you need. It is best to fix the model source code.
>>
>> Also, deficiencies in the model code will have to be continuously
>> addressed by NCL thumb-in-the-dike solutions.
>> =====
>>
>> Your netcdf new-hgt.2019 file has
>>
>> dimensions:
>>
>> time = UNLIMITED ; // (0 currently)
>>
>>
>> I have no idea why there is no 'time' dimension associated with 'hgt',
>> Is the source variable just (lat,lon)
>>
>> If so, delete
>>
>> ;===================================================================
>> ; make time an UNLIMITED dimension; recommended for most applications
>> ;===================================================================
>> *filedimdef* <http://www.ncl.ucar.edu/Document/Functions/Built-in/filedimdef.shtml>(ncdf,"time",-1,True)
>>
>>
>> On Tue, Jan 12, 2021 at 6:16 AM Setareh Rahimi <setareh.rahimi at gmail.com>
>> wrote:
>>
>>> Dear Dennis,
>>> Many thanks for your help. Once I follow the
>>> http://www.ncl.ucar.edu/Applications/method_1.shtml method to save the
>>> new nc file, I face some errors, in fact I am too new in writhing nc files.
>>> That method just give me :
>>>
>>> netcdf new-hgt.2019 {
>>>
>>> dimensions:
>>>
>>> time = UNLIMITED ; // (0 currently)
>>>
>>>
>>> // global attributes:
>>>
>>> :creation_date = "Tue Jan 12 16:33:58 +0330 2021" ;
>>>
>>> :Conventions = "CF-1.0 " ;
>>>
>>> :source_file = "NCEP/DOE AMIP-II Reanalysis (Reanalysis-2) Model" ;
>>>
>>> :title = "4x daily NCEP/DOE Reanalysis 2" ;
>>>
>>>
>>> How can I have the correct nc file?
>>>
>>>
>>> Best wishes
>>>
>>> On Mon, Jan 11, 2021 at 11:35 PM Dennis Shea <shea at ucar.edu> wrote:
>>>
>>>> I was just typing a response that contained the same information as
>>>> Dave's.
>>>>
>>>> Likely, you have no control "the model I am running only accept packed
>>>> nc file with scale factor."
>>>> As noted by DaveA, the file you are using is netCDF-4
>>>> [ Perhaps too much info but netCDF-4's compression is described *here
>>>> ]*
>>>> <https://www.unidata.ucar.edu/blogs/developer/entry/netcdf_compression>
>>>> ===============
>>>>
>>>> %> *ncdump -k* air.2019.nc
>>>> netCDF-4 classic model
>>>>
>>>> In fact, I believe *all* the netCDF files created by "NCEP-DOE AMIP-II
>>>> Reanalysis" (and others) will be netCDF-4.
>>>>
>>>> Maybe you could have the model developers alter the code.
>>>> Really, they should have checked if the 'scale_factor' and/or
>>>> 'add_offset' attributes are present.
>>>> If not, use the variable directly.
>>>> ====================
>>>>
>>>> f = addfile("air.2019.nc","r")
>>>> x = f->air
>>>> printVarSummary(x) ; full uncompressed
>>>> values
>>>> ; *** LOOK
>>>> AT THE SIZE****
>>>> ; It is
>>>> MUCH LARGER than the ENTIRE netCDF file
>>>>
>>>> xShort = *pack_values*(xFloat, "short", False)
>>>> *printVarSummary* <http://www.ncl.ucar.edu/Document/Functions/Built-in/printVarSummary.shtml>(xShort)
>>>>
>>>> xShort = pack_values(x, "short", False)
>>>> printVarSummary(xShort) ; *** LOOK AT THE SIZE****
>>>> ; HALF the size of 'x'
>>>> ; but much larger than the compressed value!
>>>>
>>>> The use (say) the following to write 'xShort' a new (much larger) file that the original
>>>>
>>>> *http://www.ncl.ucar.edu/Applications/method_1.shtml* <http://www.ncl.ucar.edu/Applications/method_1.shtml>
>>>>
>>>> Then use that new file for the model.
>>>>
>>>> Likely,* EVERY *variable will have to be done like this for the "model you are using
>>>>
>>>>
>>>> *===*
>>>>
>>>>
>>>>
>>>> *It is much easier for the model code to be altered.*
>>>>
>>>>
>>>> *Good Luck*
>>>>
>>>>
>>>> On Mon, Jan 11, 2021 at 12:46 PM Dave Allured - NOAA Affiliate via
>>>> ncl-talk <ncl-talk at mailman.ucar.edu> wrote:
>>>>
>>>>> I see. I suggest the best solution would be a simple upgrade to the
>>>>> model code, so that it could read both packed and unpacked air temp files.
>>>>> This will enable the model to use future data updates without extra
>>>>> processing.
>>>>>
>>>>> In fortran, this would be an if statement to check whether there is a
>>>>> scale_factor attribute. If not, then skip reading scale_factor and
>>>>> add_offset, also skip the unpacking statement or loop that converts packed
>>>>> data values to unpacked. Just read in the unpacked values directly. You
>>>>> would need two read statements, the original one for the packed data type,
>>>>> and a new one for the unpacked data type (fortran real).
>>>>>
>>>>> If something prevents you from changing the model code, then just
>>>>> convert the air temp file to packed using the NCL *pack_values* function.
>>>>> See the general documentation for how to read and write a netcdf file.
>>>>>
>>>>>
>>>>> On Mon, Jan 11, 2021 at 12:00 PM Setareh Rahimi <
>>>>> setareh.rahimi at gmail.com> wrote:
>>>>>
>>>>>> Dear Dave,
>>>>>> Thank you for your reply. Yes I am sure, because the model I am
>>>>>> running only accept packed nc file with scale factor.
>>>>>> Please have a look at attached file showing the error I face while
>>>>>> running the model with mentioned nc file.
>>>>>> Best regard
>>>>>>
>>>>>> On Mon, Jan 11, 2021 at 22:20 Dave Allured - NOAA Affiliate <
>>>>>> dave.allured at noaa.gov> wrote:
>>>>>>
>>>>>>> That file is already compressed to 31% of its natural array size,
>>>>>>> using netcdf-4 internal compression. Are you sure you want to do that? If
>>>>>>> yes, then use the NCL function *pack_values*.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jan 11, 2021 at 11:07 AM Setareh Rahimi via ncl-talk <
>>>>>>> ncl-talk at mailman.ucar.edu> wrote:
>>>>>>>
>>>>>>>> Dear all,
>>>>>>>> I need to convert an unpacked netCDF file to a packed one (
>>>>>>>> ftp://ftp2.psl.noaa.gov/Datasets/ncep.reanalysis2/pressure/air.2019.nc),
>>>>>>>> I wonder if this possible using NCL?
>>>>>>>> Many thanks in advance,
>>>>>>>> Best wishes,
>>>>>>>> --
>>>>>>>> S.Rahimi
>>>>>>>>
>>>>>>>
--
S.Rahimi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210113/0ea1751b/attachment.html>
More information about the ncl-talk
mailing list