[ncl-talk] Writing data to the existing record dimension of a netCDF

[Will Hobbs]

Dave and Dennis

Thanks for all your help. I took Dave’s suggestion of trying another netcdf file and it all worked fine.

The problem files are all netcdf-4
>ncdump –k <filename>
returns ‘netCDF-4’

All the other files I could find return the value ‘classic’ (is this netCDF3?)

On a whim I converted one of the problem files to classic:
>nccopy –k classic <problemfil.nc> ./tmp.nc

I was able to modify  the converted file in ncl as planned; so at least I now have a workaround.

I still don’t know if this is a general netCDF-4 issue or specific to the files I have. The files were converted from binary format model output (COCO, in case anyone’s interested) but I didn’t do the conversion myself, so I can’t give too much info about that.

Will


From: Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
Date: Thursday, 1 December 2016 at 6:45 AM
To: Will Hobbs <will.hobbs at utas.edu.au>
Cc: "ncl-talk at ucar.edu" <ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Writing data to the existing record dimension of a netCDF

Will,

Find a completely different netcdf-4 file, preferably a very small and simple one, and retry my simple test to overwrite a single time coordinate value.  This will help differentiate between a file problem and an NCL problem.

Also try one other thing.  If by any chance your files are on a file server, then make local copies on your mac, and repeat the tests.  My work group has recently experienced compatibility problems with NFS mounted file servers and netcdf-4 files, accessed from Mac.

--Dave


On Tue, Nov 29, 2016 at 9:16 PM, Will Hobbs <will.hobbs at utas.edu.au<mailto:will.hobbs at utas.edu.au>> wrote:
Hi Dave

I tried your test but I still get the same segmentation fault, so I guess the problem is with the files.

Thanks anyway

From: Dave Allured - NOAA Affiliate <dave.allured at noaa.gov<mailto:dave.allured at noaa.gov>>
Date: Wednesday, 30 November 2016 at 11:29 AM
To: Will Hobbs <will.hobbs at utas.edu.au<mailto:will.hobbs at utas.edu.au>>
Cc: "ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>" <ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] Writing data to the existing record dimension of a netCDF

Will,

Please keep messages on the discussion list.  To debug your seg fault, try this simple test to overwrite a single junk time coordinate value:

    fi = addfile (fil, "w")
    fi->time(0) = (/ 1.0d /)
    print ("Single value written")
    exit
If this test works, the problem is probably in your code.  If this fails, there is probably something wrong with the file or with NCL, or else your time dimension is named something other than "time".

On your side question, t3 would still be a record variable because it shares the dimension name "time" with your old time coordinate variable, which is also a record variable.  I assumed that your actual time dimension name is "time".  If the time dimension has a different name, please change "time" in these examples to the actual name of the time dimension.

--Dave


On Tue, Nov 29, 2016 at 5:03 PM, Will Hobbs <will.hobbs at utas.edu.au<mailto:will.hobbs at utas.edu.au>> wrote:
Hi Dave

I had a go trying what you suggested, (i.e. write a new time variable) and I still get the segmentation fault, so it’s possible there’s a problem with my file.

As an aside, in your code snippet would the new file variables, t3, still be a record dimension? I presume not. This would become an issue with post-processing using CDO, NCO etc.

Thanks

Will

From: Dave Allured - NOAA Affiliate <dave.allured at noaa.gov<mailto:dave.allured at noaa.gov>>
Date: Wednesday, 30 November 2016 at 10:38 AM
To: Will Hobbs <will.hobbs at utas.edu.au<mailto:will.hobbs at utas.edu.au>>
Cc: "ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>" <ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] Writing data to the existing record dimension of a netCDF

Will,

If the junk time array has the correct variable name and dimension name (probably both "time"), and a data type with sufficient precision, then the simplest method is to just overwrite the junk time coordinate values with correct time coordinates that you calculate in NCL.  I have found that trying to swap a new coordinate variable in place of an old one is very troublesome.

As a side note, I just ran very simple tests on NCL 6.3.0 on Mac.  The following program easily added a new 1-D variable, to both netcdf-3 and netcdf-4 test files with unlimited time dimension.  In short, I do not see why your program crashed.

    f = addfile ("test.nc<http://test.nc>", "w")
    t3 = f->time      ; note both time and t3 are double in this case
    f->t3 = t3          ; both have single dimension "time"
    exit

--Dave

On Tue, Nov 29, 2016 at 3:45 PM, Will Hobbs <will.hobbs at utas.edu.au<mailto:will.hobbs at utas.edu.au>> wrote:
Hi all

I am working with some model output monthly files (one year for each file, with filename t.<year>.nc), which have a junk time array, and which I am trying to fix. I was hoping to create a new time array, add the values to each file’s time array, and update the variable attributes using NCO:

>Begin

yrst = 1979
yren=2014
vNam = “t”


;create new time array

ydum = new(12, integer)
mdum = ispan(1,12,1)
  ddum = conform_dims(12,15,-1)
 tdum = conform_dims(12,0,-1)

  opt          = 1
  opt at calendar = "standard"
  tunits        = "days after 1-1-1"


;loop through each year
  do yy = yrst, yren

    ;create time array
    ydum = yy
    ntime = cd_inv_calendar(ydum,mdum,ddum,tdum,tdum,tdum,tunits,opt)


    ;add time to file
    fil   =systemfunc("ls "+vNam+"."+yy+".nc")
    fi = addfile(fil,"w")

    fi->time = (/ntime/)

;update attributes
  CMD = “ncatted –a time,units,o,c,”+tunits+” “+fil
system(CMD)

end do


Unfortunately when I try and write the data to file (i.e.    fi->time = (/ntime/) ) I get a segmentation fault:

Segmentation fault: 11

I’ve checked the file permissions and it doesn’t seem to be that. I ‘speculate’ that netCDF won’t allow me to update the record dimension – is this the case? If so, presumably I will need rewrite each file from scratch with new time arrays.

Many thanks,

Will


University of Tasmania Electronic Communications Policy (December, 2014).
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20161201/9cc3fcbe/attachment.html