[ncl-talk] Convert unpacked netCDF file to packed one
Dennis Shea
shea at ucar.edu
Tue Jan 12 10:18:08 MST 2021
As noted before, the BEST solution is to have the model code altered too
accommodate the original values. IE: to not require packed values.
====
The approach of created netCDF files containing variables with the
scale_factor and add_offset attributes is cumbersome. You will have two
files (original: netCDF4) and a (much larger) "packed" file. That is for
EVERY variable you need. It is best to fix the model source code.
Also, deficiencies in the model code will have to be continuously
addressed by NCL thumb-in-the-dike solutions.
=====
Your netcdf new-hgt.2019 file has
dimensions:
time = UNLIMITED ; // (0 currently)
I have no idea why there is no 'time' dimension associated with 'hgt',
Is the source variable just (lat,lon)
If so, delete
;===================================================================
; make time an UNLIMITED dimension; recommended for most applications
;===================================================================
*filedimdef* <http://www.ncl.ucar.edu/Document/Functions/Built-in/filedimdef.shtml>(ncdf,"time",-1,True)
On Tue, Jan 12, 2021 at 6:16 AM Setareh Rahimi <setareh.rahimi at gmail.com>
wrote:
> Dear Dennis,
> Many thanks for your help. Once I follow the
> http://www.ncl.ucar.edu/Applications/method_1.shtml method to save the
> new nc file, I face some errors, in fact I am too new in writhing nc files.
> That method just give me :
>
> netcdf new-hgt.2019 {
>
> dimensions:
>
> time = UNLIMITED ; // (0 currently)
>
>
> // global attributes:
>
> :creation_date = "Tue Jan 12 16:33:58 +0330 2021" ;
>
> :Conventions = "CF-1.0 " ;
>
> :source_file = "NCEP/DOE AMIP-II Reanalysis (Reanalysis-2) Model" ;
>
> :title = "4x daily NCEP/DOE Reanalysis 2" ;
>
>
> How can I have the correct nc file?
>
>
> Best wishes
>
> On Mon, Jan 11, 2021 at 11:35 PM Dennis Shea <shea at ucar.edu> wrote:
>
>> I was just typing a response that contained the same information as
>> Dave's.
>>
>> Likely, you have no control "the model I am running only accept packed nc
>> file with scale factor."
>> As noted by DaveA, the file you are using is netCDF-4
>> [ Perhaps too much info but netCDF-4's compression is described *here ]*
>> <https://www.unidata.ucar.edu/blogs/developer/entry/netcdf_compression>
>> ===============
>>
>> %> *ncdump -k* air.2019.nc
>> netCDF-4 classic model
>>
>> In fact, I believe *all* the netCDF files created by "NCEP-DOE AMIP-II
>> Reanalysis" (and others) will be netCDF-4.
>>
>> Maybe you could have the model developers alter the code.
>> Really, they should have checked if the 'scale_factor' and/or
>> 'add_offset' attributes are present.
>> If not, use the variable directly.
>> ====================
>>
>> f = addfile("air.2019.nc","r")
>> x = f->air
>> printVarSummary(x) ; full uncompressed values
>> ; *** LOOK
>> AT THE SIZE****
>> ; It is MUCH
>> LARGER than the ENTIRE netCDF file
>>
>> xShort = *pack_values*(xFloat, "short", False)
>> *printVarSummary* <http://www.ncl.ucar.edu/Document/Functions/Built-in/printVarSummary.shtml>(xShort)
>>
>> xShort = pack_values(x, "short", False)
>> printVarSummary(xShort) ; *** LOOK AT THE SIZE****
>> ; HALF the size of 'x'
>> ; but much larger than the compressed value!
>>
>> The use (say) the following to write 'xShort' a new (much larger) file that the original
>>
>> *http://www.ncl.ucar.edu/Applications/method_1.shtml* <http://www.ncl.ucar.edu/Applications/method_1.shtml>
>>
>> Then use that new file for the model.
>>
>> Likely,* EVERY *variable will have to be done like this for the "model you are using
>>
>>
>> *===*
>>
>>
>>
>> *It is much easier for the model code to be altered.*
>>
>>
>> *Good Luck*
>>
>>
>> On Mon, Jan 11, 2021 at 12:46 PM Dave Allured - NOAA Affiliate via
>> ncl-talk <ncl-talk at mailman.ucar.edu> wrote:
>>
>>> I see. I suggest the best solution would be a simple upgrade to the
>>> model code, so that it could read both packed and unpacked air temp files.
>>> This will enable the model to use future data updates without extra
>>> processing.
>>>
>>> In fortran, this would be an if statement to check whether there is a
>>> scale_factor attribute. If not, then skip reading scale_factor and
>>> add_offset, also skip the unpacking statement or loop that converts packed
>>> data values to unpacked. Just read in the unpacked values directly. You
>>> would need two read statements, the original one for the packed data type,
>>> and a new one for the unpacked data type (fortran real).
>>>
>>> If something prevents you from changing the model code, then just
>>> convert the air temp file to packed using the NCL *pack_values* function.
>>> See the general documentation for how to read and write a netcdf file.
>>>
>>>
>>> On Mon, Jan 11, 2021 at 12:00 PM Setareh Rahimi <
>>> setareh.rahimi at gmail.com> wrote:
>>>
>>>> Dear Dave,
>>>> Thank you for your reply. Yes I am sure, because the model I am running
>>>> only accept packed nc file with scale factor.
>>>> Please have a look at attached file showing the error I face while
>>>> running the model with mentioned nc file.
>>>> Best regard
>>>>
>>>> On Mon, Jan 11, 2021 at 22:20 Dave Allured - NOAA Affiliate <
>>>> dave.allured at noaa.gov> wrote:
>>>>
>>>>> That file is already compressed to 31% of its natural array size,
>>>>> using netcdf-4 internal compression. Are you sure you want to do that? If
>>>>> yes, then use the NCL function *pack_values*.
>>>>>
>>>>>
>>>>> On Mon, Jan 11, 2021 at 11:07 AM Setareh Rahimi via ncl-talk <
>>>>> ncl-talk at mailman.ucar.edu> wrote:
>>>>>
>>>>>> Dear all,
>>>>>> I need to convert an unpacked netCDF file to a packed one (
>>>>>> ftp://ftp2.psl.noaa.gov/Datasets/ncep.reanalysis2/pressure/air.2019.nc),
>>>>>> I wonder if this possible using NCL?
>>>>>> Many thanks in advance,
>>>>>> Best wishes,
>>>>>> --
>>>>>> S.Rahimi
>>>>>>
>>>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at mailman.ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
> --
> S.Rahimi
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210112/1268b85a/attachment.html>
More information about the ncl-talk
mailing list