[ncl-talk] Writing data to the existing record dimension of a netCDF
Dave Allured - NOAA Affiliate
dave.allured at noaa.gov
Thu Dec 1 14:29:21 MST 2016
Will and Dennis,
I tested one of Will's Netcdf-4 files that he sent me off line, with NCL
6.3.0. Bottom line, the file seems to be perfectly valid. The seg fault
tracks with NCL write access to the Netcdf-4 file format specifically. The
simple write test worked just fine when I converted the file to Netcdf-4
Classic format. This adds to Will's successful tests with Netcdf-3 format.
So Will, my suggested workaround is to convert this data set to Netcdf-4
Classic. This will retain all the benefits of Netcdf-4, in particular data
compression. Your original file does not use any advanced features that
would require the full Netcdf-4 format.
Dennis, you also suggested off line that this appears to be a bug in NCL
6.3.0. I agree.
Here is a small reproducer test file. It is full Netcdf-4 format with data
compression, 24 kbytes. This will fail in the same way when you try to
overwrite a single time coordinate value with NCL 6.3.0.
ftp://ftp.cdc.noaa.gov/Public/dallured/netcdf/t3.nc4.nc
--Dave
On Wed, Nov 30, 2016 at 5:42 PM, Will Hobbs <will.hobbs at utas.edu.au> wrote:
> Dave and Dennis
>
>
>
> Thanks for all your help. I took Dave’s suggestion of trying another
> netcdf file and it all worked fine.
>
>
>
> The problem files are all netcdf-4
>
> >ncdump –k <filename>
>
> returns ‘netCDF-4’
>
>
>
> All the other files I could find return the value ‘classic’ (is this
> netCDF3?)
>
>
>
> On a whim I converted one of the problem files to classic:
>
> >nccopy –k classic <problemfil.nc> ./tmp.nc
>
>
>
> I was able to modify the converted file in ncl as planned; so at least I
> now have a workaround.
>
>
>
> I still don’t know if this is a general netCDF-4 issue or specific to the
> files I have. The files were converted from binary format model output
> (COCO, in case anyone’s interested) but I didn’t do the conversion myself,
> so I can’t give too much info about that.
>
>
>
> Will
>
>
>
>
>
> *From: *Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Date: *Thursday, 1 December 2016 at 6:45 AM
> *To: *Will Hobbs <will.hobbs at utas.edu.au>
> *Cc: *"ncl-talk at ucar.edu" <ncl-talk at ucar.edu>
> *Subject: *Re: [ncl-talk] Writing data to the existing record dimension
> of a netCDF
>
>
>
> Will,
>
>
>
> Find a completely different netcdf-4 file, preferably a very small and
> simple one, and retry my simple test to overwrite a single time coordinate
> value. This will help differentiate between a file problem and an NCL
> problem.
>
>
>
> Also try one other thing. If by any chance your files are on a file
> server, then make local copies on your mac, and repeat the tests. My work
> group has recently experienced compatibility problems with NFS mounted file
> servers and netcdf-4 files, accessed from Mac.
>
>
>
> --Dave
>
>
>
>
>
> On Tue, Nov 29, 2016 at 9:16 PM, Will Hobbs <will.hobbs at utas.edu.au>
> wrote:
>
> Hi Dave
>
>
>
> I tried your test but I still get the same segmentation fault, so I guess
> the problem is with the files.
>
>
>
> Thanks anyway
>
>
>
> *From: *Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Date: *Wednesday, 30 November 2016 at 11:29 AM
> *To: *Will Hobbs <will.hobbs at utas.edu.au>
> *Cc: *"ncl-talk at ucar.edu" <ncl-talk at ucar.edu>
> *Subject: *Re: [ncl-talk] Writing data to the existing record dimension
> of a netCDF
>
>
>
> Will,
>
>
>
> Please keep messages on the discussion list. To debug your seg fault, try
> this simple test to overwrite a single junk time coordinate value:
>
> fi = addfile (fil, "w")
> fi->time(0) = (/ 1.0d /)
> print ("Single value written")
>
> exit
>
> If this test works, the problem is probably in your code. If this fails,
> there is probably something wrong with the file or with NCL, or else your
> time dimension is named something other than "time".
>
>
>
> On your side question, t3 would still be a record variable because it
> shares the dimension name "time" with your old time coordinate variable,
> which is also a record variable. I assumed that your actual time dimension
> name is "time". If the time dimension has a different name, please change
> "time" in these examples to the actual name of the time dimension.
>
>
>
> --Dave
>
>
>
>
>
> On Tue, Nov 29, 2016 at 5:03 PM, Will Hobbs <will.hobbs at utas.edu.au>
> wrote:
>
> Hi Dave
>
>
>
> I had a go trying what you suggested, (i.e. write a new time variable) and
> I still get the segmentation fault, so it’s possible there’s a problem with
> my file.
>
>
>
> As an aside, in your code snippet would the new file variables, t3, still
> be a record dimension? I presume not. This would become an issue with
> post-processing using CDO, NCO etc.
>
>
>
> Thanks
>
>
>
> Will
>
>
>
> *From: *Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Date: *Wednesday, 30 November 2016 at 10:38 AM
> *To: *Will Hobbs <will.hobbs at utas.edu.au>
> *Cc: *"ncl-talk at ucar.edu" <ncl-talk at ucar.edu>
> *Subject: *Re: [ncl-talk] Writing data to the existing record dimension
> of a netCDF
>
>
>
> Will,
>
>
>
> If the junk time array has the correct variable name and dimension name
> (probably both "time"), and a data type with sufficient precision, then the
> simplest method is to just overwrite the junk time coordinate values with
> correct time coordinates that you calculate in NCL. I have found that
> trying to swap a new coordinate variable in place of an old one is very
> troublesome.
>
>
>
> As a side note, I just ran very simple tests on NCL 6.3.0 on Mac. The
> following program easily added a new 1-D variable, to both netcdf-3 and
> netcdf-4 test files with unlimited time dimension. In short, I do not see
> why your program crashed.
>
>
>
> f = addfile ("test.nc", "w")
> t3 = f->time ; note both time and t3 are double in this case
> f->t3 = t3 ; both have single dimension "time"
> exit
>
> --Dave
>
>
>
> On Tue, Nov 29, 2016 at 3:45 PM, Will Hobbs <will.hobbs at utas.edu.au>
> wrote:
>
> Hi all
>
>
>
> I am working with some model output monthly files (one year for each file,
> with filename t.<year>.nc), which have a junk time array, and which I am
> trying to fix. I was hoping to create a new time array, add the values to
> each file’s time array, and update the variable attributes using NCO:
>
>
>
> >Begin
>
>
>
> yrst = 1979
>
> yren=2014
>
> vNam = “t”
>
>
>
>
>
> ;create new time array
>
>
>
> ydum = new(12, integer)
>
> mdum = ispan(1,12,1)
>
> ddum = conform_dims(12,15,-1)
>
> tdum = conform_dims(12,0,-1)
>
>
>
> opt = 1
>
> opt at calendar = "standard"
>
> tunits = "days after 1-1-1"
>
>
>
>
>
> ;loop through each year
>
> do yy = yrst, yren
>
>
>
> ;create time array
>
> ydum = yy
>
> ntime = cd_inv_calendar(ydum,mdum,ddum,tdum,tdum,tdum,tunits,opt)
>
>
>
>
>
> ;add time to file
>
> fil =systemfunc("ls "+vNam+"."+yy+".nc")
>
> fi = addfile(fil,"w")
>
>
>
> fi->time = (/ntime/)
>
>
>
> ;update attributes
>
> CMD = “ncatted –a time,units,o,c,”+tunits+” “+fil
>
> system(CMD)
>
>
>
> end do
>
>
>
>
>
> Unfortunately when I try and write the data to file (i.e. fi->time =
> (/ntime/) ) I get a segmentation fault:
>
>
>
> Segmentation fault: 11
>
>
>
> I’ve checked the file permissions and it doesn’t seem to be that. I
> ‘speculate’ that netCDF won’t allow me to update the record dimension – is
> this the case? If so, presumably I will need rewrite each file from scratch
> with new time arrays.
>
>
>
> Many thanks,
>
>
>
> Will
>
>
>
> University of Tasmania Electronic Communications Policy (December, 2014).
> This email is confidential, and is for the intended recipient only.
> Access, disclosure, copying, distribution, or reliance on any of it by
> anyone outside the intended recipient organisation is prohibited and may be
> a criminal offence. Please delete if obtained in error and email
> confirmation to the sender. The views expressed in this email are not
> necessarily the views of the University of Tasmania, unless clearly
> intended otherwise.
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20161201/bae38bcc/attachment.html
More information about the ncl-talk
mailing list