<div dir="ltr">CORRECTION. I suggested using nccopy to add chunking to a large netcdf file. Current versions of nccopy do not work correctly in this case. If you want to convert large files from "contiguous" to chunked, use an NCO command like this:<br><br> ncks --deflate 0 --chunk_policy g3d --chunk_dimension time,1 \<br> --chunk_dimension lat,180 --chunk_dimension lon,180 \<br> <a href="http://tas_input.nc">tas_input.nc</a> <a href="http://tas_output.nc">tas_output.nc</a><br><br>However, as I said in #5 earlier, I do not think the lack of chunking is Tomoko's real problem. Large test files with "contiguous" have consistently performed well with the current NCL release on my Mac and Linux systems.<br><br>--Dave<div><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 18, 2017 at 10:14 PM, Dave Allured - NOAA Affiliate <span dir="ltr"><<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>1) I agree with Dennis, the use of "contiguous" for an array this large is suspicious and unusual. "Contiguous" means by definition that chunking is not in use. See Netcdf docs. However:</div><div><br></div><div>2) I made a 9 Gb "contiguous" test file with same dimensions, attributes, and coordinate vars as Tomoko's. Your small script worked <b>correctly</b> on this test file, on two different systems:</div><div><br></div><div> NCL 6.4.0, Mac OS 10.12.6, 32 Gb installed memory</div><div> NCL 6.4.0, Centos Linux, 256 Gb installed memory</div><div class="gmail_extra"><br></div><div class="gmail_extra">On Mac, read 9 Gb contiguous from <b>local solid state disk</b> in 7 seconds.</div><div class="gmail_extra">On Linux, read 9 Gb contiguous from <b>network file server</b> in 80 seconds.</div><div class="gmail_extra"><br></div><div class="gmail_extra">3. Speculation on possible problems:</div><div class="gmail_extra"><br></div><div class="gmail_extra">3a. Your system does not have enough installed memory, forcing virtual memory as mentioned by Guido. VM can be remarkably slow with large arrays.</div><div class="gmail_extra"><br></div><div class="gmail_extra">3b. Related to 3a, if you are submitting a batch script, you might need to explicitly increase the memory resource request for the batch job.</div><div class="gmail_extra"><br></div><div class="gmail_extra">3c. You said you are using NCL 6.3.0. There is a chance that this older version with its older internal libraries may not be correctly handling large NC4 contiguous arrays.</div><div class="gmail_extra"><br></div><div class="gmail_extra">3d. You might not be seeing the actual error in your batch job. You reported timeout, not an explicit error message. Try logging in interactively, and run this test in NCL command line mode.</div><div class="gmail_extra"><br></div><div class="gmail_extra">3e. Possible hidden structural problem inside this particular file.</div><div class="gmail_extra"><br></div><div class="gmail_extra">4. Possible solutions or work-arounds. Besides the things I suggested in 3 above:</div><div class="gmail_extra"><br></div><div class="gmail_extra">4a. Upgrade to the current NCL version 6.4.0.</div><div class="gmail_extra"><br></div><div class="gmail_extra">4b. Try the same test on a different computer with plenty of memory.</div><div class="gmail_extra"><br></div>4c. Add chunking with reasonable chunk sizes. Here is one way to do it, there are others. This Linux command also added compression, and it ran in less than 2.5 minutes on both of my systems:<br><br> nccopy -c time/1,i/180,j/180 -d1 <a href="http://tas_input.nc" target="_blank">tas_input.nc</a> <a href="http://tas_output.nc" target="_blank">tas_output.nc</a><div><br></div><div>* Note, this is imperfect because the time coordinate is left with chunk size 1. Better would be a different tool that could independently set a more appropriate time coordinate chunk size, something like 1000 to 100000.</div><div><br></div><div>5. I am not convinced that "contiguous" is the real problem here. But if this turns out to be the problem, then the best long term solution is to get the data provider to avoid large "contiguous" arrays, and add chunking for arrays larger than a few hundred Mbytes.</div><div><br></div><div>--Dave</div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 18, 2017 at 6:53 PM, Dennis Shea <span dir="ltr"><<a href="mailto:shea@ucar.edu" target="_blank">shea@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>FYI_1: <span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-m_855728194980678494gmail-breadcrumb"><br><br></span></div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-m_855728194980678494gmail-breadcrumb">re: </span><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-m_855728194980678494gmail-breadcrumb">tas:_Storage = "<b>contiguous</b>" </span><div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-m_855728194980678494gmail-breadcrumb"><br>2011 Unidata NetCDF Workshop > Chunking and Deflating Data with NetCDF-4 </span><br><a href="https://www.unidata.ucar.edu/software/netcdf/workshops/2011/nc4chunking/Contiguous.html" target="_blank">https://www.unidata.ucar.edu/s<wbr>oftware/netcdf/workshops/2011/<wbr>nc4chunking/Contiguous.html</a><br><br>
<ul><li><b>Only works for fixed-sized datasets (those without any unlimited
dimensions.)
</b></li><li><b>You can't use compression or other filters with contiguous data.</b>
</li><li><b>Use this for smallish variables</b>, like coordinate variables.
</li><li><b>This is the default storage for fixed-sized variables with no
filters.</b><br></li></ul></div><div>NOTE the recommendation: "<b>Use this for smallish variable</b>s"</div><div><br><b>9GB</b> is not "smallish" <br><br></div><div>I *SPECULATE* there is no chunking being used.<br></div><div><br>===<br></div><div>FYI_2: the file is not CF-conforming.<br><br><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-im"><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-m_-6806337471029029607Apple-tab-span" style="white-space:pre-wrap"> </span>int time(time) ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-m_-6806337471029029607Apple-tab-span" style="white-space:pre-wrap"> </span>string time:long_name = "time" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-m_-6806337471029029607Apple-tab-span" style="white-space:pre-wrap"> </span>string time:<b>units = "month"</b> ;</div></span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-m_-6806337471029029607Apple-tab-span" style="white-space:pre-wrap"> </span>time:_Storage = "contiguous" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_-4896383636914128345m_-7707691228060939448gmail-m_-8807388638885129764gmail-m_-6806337471029029607Apple-tab-span" style="white-space:pre-wrap"> </span>time:_Endianness = "little" ;<br><br></div><div>"units" of "months" is not CF-conformant.<br><br>===<span style="font-family:arial,sans-serif"></span><br>FYI_3: <span style="font-family:arial,sans-serif"><br>This IPSL-CM5A-LR was (likely) <b>not created by CNRS</b>. <br>It contains regridded data. [ ...<b><a href="http://rgrd.nc" target="_blank">rgrd.nc</a> ]</b></span><br><br>==============================<wbr>=============<br></div><div>I speculate that your system may be trying to allocate all 9GB of memory before reading values from the file.<br></div><div><br><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 18, 2017 at 5:50 PM, Tomoko Koyama <span dir="ltr"><<a href="mailto:Tomoko.Koyama@colorado.edu" target="_blank">Tomoko.Koyama@colorado.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div>Thank you very much, Dave.</div><div><br></div><div>I hope the following is showing enough information to nail down the cause.</div><div><br></div><div>[koyama@login03 IPSL-CM5A-LR]$ ncdump -hs <a href="http://tas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc" target="_blank">tas_day_IPSL-CM5A-LR_rcp85_r1i<wbr>1p1_20060101-22051231.rgrd.nc</a></div><span><div>netcdf tas_day_IPSL-CM5A-LR_rcp85_r1i<wbr>1p1_20060101-22051231.rgrd {</div><div>dimensions:</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>time = 73000 ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>i = 180 ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>j = 180 ;</div><div>variables:</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>int time(time) ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string time:long_name = "time" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string time:units = "month" ;</div></span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>time:_Storage = "contiguous" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>time:_Endianness = "little" ;</div><span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>float i(i) ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string i:units = "none" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string i:long_name = "i index" ;</div></span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>i:_Storage = "contiguous" ;</div><span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>float j(j) ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string j:units = "none" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string j:long_name = "j index" ;</div></span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>j:_Storage = "contiguous" ;</div><span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>float tas(time, i, j) ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:associated_files = "baseURL: <a href="http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation" target="_blank">http://cmip-pcmdi.llnl.gov/CMI<wbr>P5/dataLocation</a> gridspecFile: gridspec_atmos_fx_IPSL-CM5A-LR<wbr>_<a href="http://rcp85_r0i0p0.nc" target="_blank">rcp85_r0i0p0.nc</a> areacella: areacella_fx_IPSL-CM5A-LR_<a href="http://rcp85_r0i0p0.nc" target="_blank">rcp8<wbr>5_r0i0p0.nc</a>" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:coordinates = "height" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:history = "2011-08-16T22:13:26Z altered by CMOR: Treated scalar dimension: \'height\'. 2011-08-16T22:13:26Z altered by CMOR: replaced missing value flag (9.96921e+36) with standard missing value (1e+20). 2011-08-16T22:13:45Z altered by CMOR: Inverted axis: lat." ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:cell_measures = "area: areacella" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:cell_methods = "time: mean (interval: 30 minutes)" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:original_name = "t2m" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:units = "K" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:long_name = "Near-Surface Air Temperature" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:standard_name = "air_temperature" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string tas:remap = "remapped via ESMF_regrid_with_weights: Bilinear" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>tas:missing_value = 1.e+20f ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>tas:_FillValue = 1.e+20f ;</div></span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>tas:_Storage = "contiguous" ;</div><span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>float lat(j, i) ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string lat:long_name = "latitude" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string lat:units = "degrees_north" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>lat:_FillValue = -999.f ;</div></span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>lat:_Storage = "contiguous" ;</div><span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>float lon(j, i) ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string lon:long_name = "longitude" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>string lon:units = "degrees_east" ;</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>lon:_FillValue = -999.f ;</div></span><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>lon:_Storage = "contiguous" ;</div><div><br></div><div>// global attributes:</div><div><span class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-tab-span" style="white-space:pre-wrap"> </span>:_Format = "netCDF-4” ;</div><div><div class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705h5"><div>}</div><div><blockquote type="cite"><div>On Dec 18, 2017, at 4:05 PM, Dave Allured - NOAA Affiliate <<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>> wrote:</div><br class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995Apple-interchange-newline"><div><div dir="ltr">Tomoko,<div><br></div><div>Please add the "s" flag to Dennis's request. This will show chunk size parameters that may be relevant:</div><div><br></div><div>%> ncdump -hs <a href="http://secure-web.cisco.com/1RvOpL6YjTvAr2KOsZZoipbunMq6bhiktJdhz1n0TOzrW7Kca4nc59oM2g-2N-kBfsTvJq7EbmlS0Z0eO4xIOUUGW_31tXoipuew_3rVpiohC-518ODXnyFmFzR3smzNsHtNpBxiusoXkUpFilSVSwqsMZkdC-XQlFW8u11QwrHot_ne7RdeFPhoD5wROJB6k9CdtgjpxTb-_aez0Tad7dXeTlQS3e03cvhtyTppAhrh3JfDiMZ-wPPL3IxrC8RKfrObsJ1IJzS-axy1820aJai6zHy_wSWi3R-c1HLtu4HAI5n3tr2EPhcRoJP3pwWV5buWNtzKmLv8gwqUjknsHNOSZ3JQcuEjcH701UsyOhZQ6x3bS5Gim_Mb6Ez5ypNgivXHE-Baroa1_LD5JlLe0hvRxMkNAYoIm6NACLkfGmDh7xi3ImE5TglAUjnPxz6te/http%3A%2F%2Ftas_day_ipsl-cm5a-lr_rcp85_r1i1p1_20060101-22051231.rgrd.nc%2F" rel="noreferrer" target="_blank">tas_day_IPSL-CM5A-LR_rcp85<wbr>_r1i1p1_20060101-22051231.rgrd<wbr>.nc</a></div><div><br></div><div>There are known problems with Netcdf-4 chunk sizes that can dramatically slow down reading. In particular, chunk size exceeding chunk cache size, or chunk size exceeding available memory. If we know the structure details, we may be able to suggest a solution.</div><div><br></div><div>I recommend chunk sizes in the approximate range of 100 Kbytes to 4 Mbytes for large netcdf-4 files.</div><div><br></div><div>--Dave</div><div><br></div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 18, 2017 at 8:05 AM, Dennis Shea <span dir="ltr"><<a href="mailto:shea@ucar.edu" target="_blank">shea@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div>When you have a file issue, you should include some information:<br><br></div>(a) what what either of the fillowing show?<br><br></div>%> ncl_filedump <a href="http://secure-web.cisco.com/1OW_a3n1krqUftMdpsEFQLF9MnaCbIIUe4GLVf5ETSY_RBlx7nPiMsNeW7GTg_jFwVFYKnsxK2L_qXy_j6XTtzdM3YZ8QjDyik2JwmTcJQg3siC3KZrDZA2cO2Gv32zlVRdcGx4_lj1NbqlqF1YPNXoYRcBDbT47dqqLWWXTS21VFlWnmXqK7DraSHCwEtg6JBYE_A21VnfoUm9oWJiNW7wrZA5H6ruPd-80HJdDBmfejeC7DmM__iHAL3xtTwdevGM47ItGs9VoGK452GL3K5HZzOm00XzEB9aTMT75DkJ9LWEc6vFeibxgel2GRxCa4rIbcMJc72l-aWsXOxlsterNiDlaTwTO5UCTcAqGLgirzEPEIzEl3bmiJeKRefywuAz1IBVNpmwInKYF-i3vwAjBNmWMoleFEeRYXQo8RkYs7665o-yr9eBchBgNzFXkn/http%3A%2F%2Ftas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc" rel="noreferrer" target="_blank">tas_day_IPSL-CM5A-LR_rcp85_r1i<wbr>1p1_20060101-22051231.rgrd.nc</a><br><br></div>or <br><br></div>%> ncdump -h <a href="http://secure-web.cisco.com/1OW_a3n1krqUftMdpsEFQLF9MnaCbIIUe4GLVf5ETSY_RBlx7nPiMsNeW7GTg_jFwVFYKnsxK2L_qXy_j6XTtzdM3YZ8QjDyik2JwmTcJQg3siC3KZrDZA2cO2Gv32zlVRdcGx4_lj1NbqlqF1YPNXoYRcBDbT47dqqLWWXTS21VFlWnmXqK7DraSHCwEtg6JBYE_A21VnfoUm9oWJiNW7wrZA5H6ruPd-80HJdDBmfejeC7DmM__iHAL3xtTwdevGM47ItGs9VoGK452GL3K5HZzOm00XzEB9aTMT75DkJ9LWEc6vFeibxgel2GRxCa4rIbcMJc72l-aWsXOxlsterNiDlaTwTO5UCTcAqGLgirzEPEIzEl3bmiJeKRefywuAz1IBVNpmwInKYF-i3vwAjBNmWMoleFEeRYXQo8RkYs7665o-yr9eBchBgNzFXkn/http%3A%2F%2Ftas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc" rel="noreferrer" target="_blank">tas_day_IPSL-CM5A-LR_rcp85_r1i<wbr>1p1_20060101-22051231.rgrd.nc</a><br><br></div>(b) what version of NCL are you using><br><br></div>%> ncl -V<br><br></div>(c) your system info<br><br></div>%> uname -a<br><br>-----------<br><br>
fdir=“/root/dir4ncl/“<br>
fili="<a href="http://secure-web.cisco.com/1OW_a3n1krqUftMdpsEFQLF9MnaCbIIUe4GLVf5ETSY_RBlx7nPiMsNeW7GTg_jFwVFYKnsxK2L_qXy_j6XTtzdM3YZ8QjDyik2JwmTcJQg3siC3KZrDZA2cO2Gv32zlVRdcGx4_lj1NbqlqF1YPNXoYRcBDbT47dqqLWWXTS21VFlWnmXqK7DraSHCwEtg6JBYE_A21VnfoUm9oWJiNW7wrZA5H6ruPd-80HJdDBmfejeC7DmM__iHAL3xtTwdevGM47ItGs9VoGK452GL3K5HZzOm00XzEB9aTMT75DkJ9LWEc6vFeibxgel2GRxCa4rIbcMJc72l-aWsXOxlsterNiDlaTwTO5UCTcAqGLgirzEPEIzEl3bmiJeKRefywuAz1IBVNpmwInKYF-i3vwAjBNmWMoleFEeRYXQo8RkYs7665o-yr9eBchBgNzFXkn/http%3A%2F%2Ftas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc" rel="noreferrer" target="_blank">tas_day_IPSL-CM5A-LR_rc<wbr>p85_r1i1p1_20060101-22051231.r<wbr>grd.nc</a>"<br>
fn=addfile(fdir+fili,"r")<br>
buff=fn->tas<div><div><div><div> print(“Data is stored”)</div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 18, 2017 at 3:35 AM, Guido Cioni <span dir="ltr"><<a href="mailto:guidocioni@gmail.com" target="_blank">guidocioni@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word">Tomoko,<div>9 GB is anything but "large", although the concept of "large" is highly subjective :P </div><div><br></div><div>I've successfully read SINGLE netcdf files in NCL whose size was ~500GB so that shouldn't be the problem. For some reason a netcdf file of some size, say 400 GB, which has many timesteps is read more slowly than a file with the same size but with less timesteps; that was my impression.</div><div><br></div><div>You are setting a lot of options which I think are not needed. Did you just try to read the file with this line? </div><div><br></div><div><blockquote type="cite">fn=addfile(fdir+fili,"r")<br></blockquote><br></div><div>If it still takes a lot of time it could be system-dependent. When creating the variable NCL stores it into the RAM. If the system does not have enough RAM, some virtual memory will be created on your hard drive, which can slow down everything. But honestly I don't think you're even close to saturate your system's RAM. The problem may lie somewhere else...</div><div><br></div><div>Let us know.</div><div><br></div><div><div><div class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995gmail-m_9006037657592454840h5"><div><blockquote type="cite"><div>On 18. Dec 2017, at 07:17, Tomoko Koyama <<a href="mailto:Tomoko.Koyama@Colorado.EDU" target="_blank">Tomoko.Koyama@Colorado.EDU</a>> wrote:</div><br class="gmail-m_3960216588414308828gmail-m_1016393478977695646gmail-m_-2193091750609174705m_930108172738035995gmail-m_9006037657592454840m_6192207682118824426Apple-interchange-newline"><div><div>I’d like to extract some data from a large netcdf file, which size is about 9GB.<br><br>The following shows the partial script, but a submitted job was killed before “Data is stored” message appeared.<br>(I attempted several times with 3-hr max walltime )<br><br> setfileoption("nc", "FileStructure", "Advanced")<br> fdir=“/root/dir4ncl/“<br> fili="<a href="http://secure-web.cisco.com/1OW_a3n1krqUftMdpsEFQLF9MnaCbIIUe4GLVf5ETSY_RBlx7nPiMsNeW7GTg_jFwVFYKnsxK2L_qXy_j6XTtzdM3YZ8QjDyik2JwmTcJQg3siC3KZrDZA2cO2Gv32zlVRdcGx4_lj1NbqlqF1YPNXoYRcBDbT47dqqLWWXTS21VFlWnmXqK7DraSHCwEtg6JBYE_A21VnfoUm9oWJiNW7wrZA5H6ruPd-80HJdDBmfejeC7DmM__iHAL3xtTwdevGM47ItGs9VoGK452GL3K5HZzOm00XzEB9aTMT75DkJ9LWEc6vFeibxgel2GRxCa4rIbcMJc72l-aWsXOxlsterNiDlaTwTO5UCTcAqGLgirzEPEIzEl3bmiJeKRefywuAz1IBVNpmwInKYF-i3vwAjBNmWMoleFEeRYXQo8RkYs7665o-yr9eBchBgNzFXkn/http%3A%2F%2Ftas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc" target="_blank">tas_day_IPSL-CM5A-LR_rcp<wbr>85_r1i1p1_20060101-22051231.rg<wbr>rd.nc</a>"<br> fn=addfile(fdir+fili,"r")<br> setfileoption("nc","Format","N<wbr>etCDF4Classic")<br> buff=fn->tas<br> print(“Data is stored”)<br><br>Does it simply take a long time to read?<br>Is there anyway to speed up to read a large netcdf file?<br><br><br>Thank you,<br>Tomoko</div></div></blockquote></div></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></blockquote></div></div></div></div>