<div dir="ltr">Will and Dennis,<div><br></div><div>I tested one of Will's Netcdf-4 files that he sent me off line, with NCL 6.3.0. Bottom line, the file seems to be perfectly valid. The seg fault tracks with NCL write access to the Netcdf-4 file format specifically. The simple write test worked just fine when I converted the file to Netcdf-4 Classic format. This adds to Will's successful tests with Netcdf-3 format.</div><div><br></div><div>So Will, my suggested workaround is to convert this data set to Netcdf-4 Classic. This will retain all the benefits of Netcdf-4, in particular data compression. Your original file does not use any advanced features that would require the full Netcdf-4 format.<br><br>Dennis, you also suggested off line that this appears to be a bug in NCL 6.3.0. I agree.<br><br>Here is a small reproducer test file. It is full Netcdf-4 format with data compression, 24 kbytes. This will fail in the same way when you try to overwrite a single time coordinate value with NCL 6.3.0.<br><br> <a href="ftp://ftp.cdc.noaa.gov/Public/dallured/netcdf/t3.nc4.nc" target="_blank">ftp://ftp.cdc.noaa.gov/Public/<wbr>dallured/netcdf/t3.nc4.nc</a><br><br>--Dave</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 30, 2016 at 5:42 PM, Will Hobbs <span dir="ltr"><<a href="mailto:will.hobbs@utas.edu.au" target="_blank">will.hobbs@utas.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="m_-4767058699089186764m_-1183671210931165327WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Dave and Dennis<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks for all your help. I took Dave’s suggestion of trying another netcdf file and it all worked fine.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">The problem files are all netcdf-4
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">>ncdump –k <filename><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">returns ‘netCDF-4’<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">All the other files I could find return the value ‘classic’ (is this netCDF3?)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">On a whim I converted one of the problem files to classic:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">>nccopy –k classic <<a href="http://problemfil.nc" target="_blank">problemfil.nc</a>> ./<a href="http://tmp.nc" target="_blank">tmp.nc</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I was able to modify the converted file in ncl as planned; so at least I now have a workaround.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I still don’t know if this is a general netCDF-4 issue or specific to the files I have. The files were converted from binary format model output (COCO, in case anyone’s interested) but
I didn’t do the conversion myself, so I can’t give too much info about that.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Will<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">Dave Allured - NOAA Affiliate <<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>><br>
<b>Date: </b>Thursday, 1 December 2016 at 6:45 AM<br>
<b>To: </b>Will Hobbs <<a href="mailto:will.hobbs@utas.edu.au" target="_blank">will.hobbs@utas.edu.au</a>><br>
<b>Cc: </b>"<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>" <<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>><br>
<b>Subject: </b>Re: [ncl-talk] Writing data to the existing record dimension of a netCDF<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Will, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Find a completely different netcdf-4 file, preferably a very small and simple one, and retry my simple test to overwrite a single time coordinate value. This will help differentiate between a file problem and an NCL problem.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Also try one other thing. If by any chance your files are on a file server, then make local copies on your mac, and repeat the tests. My work group has recently experienced compatibility problems with NFS mounted file servers and netcdf-4
files, accessed from Mac.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">--Dave<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Tue, Nov 29, 2016 at 9:16 PM, Will Hobbs <<a href="mailto:will.hobbs@utas.edu.au" target="_blank">will.hobbs@utas.edu.au</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Hi Dave</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I tried your test but I still get the same segmentation fault, so I guess the problem is with the files.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks anyway</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From:
</span></b><span style="font-family:Calibri;color:black">Dave Allured - NOAA Affiliate <<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>><br>
<b>Date: </b>Wednesday, 30 November 2016 at 11:29 AM<br>
<b>To: </b>Will Hobbs <<a href="mailto:will.hobbs@utas.edu.au" target="_blank">will.hobbs@utas.edu.au</a>><br>
<b>Cc: </b>"<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>" <<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>><br>
<b>Subject: </b>Re: [ncl-talk] Writing data to the existing record dimension of a netCDF</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Will,
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Please keep messages on the discussion list. To debug your seg fault, try this simple test to overwrite a single junk time coordinate value:<br>
<br>
fi = addfile (fil, "w")<br>
fi->time(0) = (/ 1.0d /)<br>
print ("Single value written")<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> exit<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If this test works, the problem is probably in your code. If this fails, there is probably something wrong with the file or with NCL, or else your time dimension is named something
other than "time".<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">On your side question, t3 would still be a record variable because it shares the dimension name "time" with your old time coordinate variable, which is also a record variable.
I assumed that your actual time dimension name is "time". If the time dimension has a different name, please change "time" in these examples to the actual name of the time dimension.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">--Dave<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Tue, Nov 29, 2016 at 5:03 PM, Will Hobbs <<a href="mailto:will.hobbs@utas.edu.au" target="_blank">will.hobbs@utas.edu.au</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Hi Dave</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I had a go trying what you suggested, (i.e. write a new time variable) and I still get the segmentation fault, so it’s possible
there’s a problem with my file.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">As an aside, in your code snippet would the new file variables, t3, still be a record dimension? I presume not. This would become
an issue with post-processing using CDO, NCO etc.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Will</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From:
</span></b><span style="font-family:Calibri;color:black">Dave Allured - NOAA Affiliate <<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>><br>
<b>Date: </b>Wednesday, 30 November 2016 at 10:38 AM<br>
<b>To: </b>Will Hobbs <<a href="mailto:will.hobbs@utas.edu.au" target="_blank">will.hobbs@utas.edu.au</a>><br>
<b>Cc: </b>"<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>" <<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>><br>
<b>Subject: </b>Re: [ncl-talk] Writing data to the existing record dimension of a netCDF</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Will,
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If the junk time array has the correct variable name and dimension name (probably both "time"), and a data type with sufficient precision, then the simplest method is to just overwrite
the junk time coordinate values with correct time coordinates that you calculate in NCL. I have found that trying to swap a new coordinate variable in place of an old one is very troublesome.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">As a side note, I just ran very simple tests on NCL 6.3.0 on Mac. The following program easily added a new 1-D variable, to both netcdf-3 and netcdf-4 test files with unlimited
time dimension. In short, I do not see why your program crashed.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> f = addfile ("<a href="http://test.nc" target="_blank">test.nc</a>", "w")<br>
t3 = f->time ; note both time and t3 are double in this case<br>
f->t3 = t3 ; both have single dimension "time"<br>
exit<br>
<br>
--Dave<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Tue, Nov 29, 2016 at 3:45 PM, Will Hobbs <<a href="mailto:will.hobbs@utas.edu.au" target="_blank">will.hobbs@utas.edu.au</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi all</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am working with some model output monthly files (one year for each file, with filename t.<year>.nc), which have a junk time array, and which I am
trying to fix. I was hoping to create a new time array, add the values to each file’s time array, and update the variable attributes using NCO:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">>Begin</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">yrst = 1979</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">yren=2014</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">vNam = “t”</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">;create new time array</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ydum = new(12, integer)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">mdum = ispan(1,12,1)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ddum = conform_dims(12,15,-1)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> tdum = conform_dims(12,0,-1)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> opt = 1</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> opt@calendar = "standard"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> tunits = "days after 1-1-1"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">;loop through each year</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> do yy = yrst, yren</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ;create time array</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ydum = yy</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ntime = cd_inv_calendar(ydum,mdum,ddum<wbr>,tdum,tdum,tdum,tunits,opt)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ;add time to file</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> fil =systemfunc("ls "+vNam+"."+yy+".nc")</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> fi = addfile(fil,"w")</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> fi->time = (/ntime/)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">;update attributes</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CMD = “ncatted –a time,units,o,c,”+tunits+” “+fil</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">system(CMD)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">end do</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Unfortunately when I try and write the data to file (i.e. fi->time = (/ntime/) ) I get a segmentation fault:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Segmentation fault: 11</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’ve checked the file permissions and it doesn’t seem to be that. I ‘speculate’ that netCDF won’t allow me to update the record dimension – is this
the case? If so, presumably I will need rewrite each file from scratch with new time arrays.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Many thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Will</span><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<p style="line-height:10.0pt"><span style="font-size:10.0pt;font-family:Calibri"><br>
<br>
University of Tasmania Electronic Communications Policy (December, 2014). <br>
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained
in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.
<u></u><u></u></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>_________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/mailma<wbr>n/listinfo/ncl-talk</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<br>______________________________<wbr>_________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailma<wbr>n/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div></div>