<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>What do you mean by combine each year together? If you can condense each year by computing intermediate results for each year in smaller arrays, then yes this is an excellent approach to a large file problem. If you mean read all daily grids in chunks, this is more difficult.</div><div><br></div><div>The code sample that I sent before should work as well as reading all daily grids in chunks. It will give helpful information about this problem, whether it works or fails. Can you please let us know what problem it generates, if any? Running too slow is a valid complaint in this case.</div><div><br></div><div>Please make two changes before running this again:</div><div><br></div><div>(1) There is a mistake on line 5 below. If you did not already fix this, please change to:</div><div><br></div><div> dims3(0) = nfiles</div><div><br></div><div>(2) For better diagnostics, change the print statement inside the loop to this:</div><div><br></div><div> print (systemfunc ("date") + " " + i)</div><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 23, 2019 at 8:57 AM Setareh Rahimi <<a href="mailto:setareh.rahimi@gmail.com">setareh.rahimi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear all,<div><br></div><div>Is this possible to combine each year together separately, and then combine all of those combined years? Does it make sense? Please kindly advice me in this regard.</div><div>Best wishes,</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 20, 2019 at 8:44 PM Dave Allured - NOAA Affiliate <<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">You are trying to create an array of about 35 Gbytes in memory. This is large and may be straining some limit inside the I/O system. This may be difficult to debug directly.<div><br></div><div>Instead try an alternate strategy, not using addfiles. Pre-allocate the large array in NCL. Then read files one at a time in a loop, and insert them into the large array. Something like this (not tested):</div><div><br></div><div> fils = systemfunc ("ls /*/*.nc")</div><div> f = addfile (fils(0), "r")</div><div> dims3 = getfilevardimsizes (f, "p")<br></div><div> nfiles = dimsizes (fils)</div><div> dims(0) = nfiles</div><div> print (dims3)</div><div><br></div><div> x = new (dims3, "float")</div><div> printVarSummary (x)</div><div><br></div><div> do i = 0, nfiles-1</div><div> print (i+"")</div><div> f = addfile (fils(i), "r")</div><div> x(i,:,:) = f->p</div><div> end do</div><div><br></div><div> printVarSummary (x)</div><div> printMinMax (x,0)</div><div><br></div><div>This will probably avoid memory problems in the I/O system, but you might run into an out of memory problem in the NCL core. This test program will help to localize the problem.</div><div><br></div><div>It is also time to check more version information. How was NCL on your system built and installed? Can you get the version numbers of the netcdf and HDF5 libraries that NCL was built with? There were hints in previous diagnostics that you may be using older library versions. Caution, the library versions linked with NCL may not be the same versions used in command line tools h5dump and ncdump.</div><div><br></div><div>Also what is the physical memory size (RAM) in your Mac?</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 20, 2019 at 8:00 AM Setareh Rahimi <<a href="mailto:setareh.rahimi@gmail.com" target="_blank">setareh.rahimi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear Dave,<div>Thank you for your suggestion. I think the problem comes from where I combine so many NetCDF files together. However, I could not remove the warnings.</div><div>Best wishes,</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 19, 2019 at 11:35 PM Dave Allured - NOAA Affiliate <<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>That is good information from the diagnostics. All file structure details look fine to me. However I notice that you are using addfiles to open about 12780 files simultaneously. I missed this the first time. You may be running into a system limit of number of simultaneous open files.</div><div><br></div><div>This is discussed in a paragraph about half way down the documentation page for the addfiles function. From that, I suggest adding this line before your addfiles command:</div><br> setfileoption ("nc", "SuppressClose", False)</div><div dir="ltr"><br></div><div>I am not familiar with this option. It is possible that you need to place this command after addfiles, and change the first argument from "nc" to just "f" without quotes. Try it both ways if necessary.</div><div><br></div><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 19, 2019 at 10:35 AM Setareh Rahimi <<a href="mailto:setareh.rahimi@gmail.com" target="_blank">setareh.rahimi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><font color="#000000">Dear all,</font><div><font color="#000000">Thank you for your suggestions. However, the problem still exists. I checked the files individually and found out nothing wrong with them. I run the script for each year separately and did not get the warnings, but once run the script for all the years together the warnings appear. I attached the output from tests that Dave has suggested. </font></div><div><font face="arial, sans-serif" color="#000000">NCL version:<span style="font-size:14px;font-variant-ligatures:no-common-ligatures"> 6.6.2 </span></font></div><div><font color="#000000" face="arial, sans-serif"><span style="font-size:14px;font-variant-ligatures:no-common-ligatures">Computer system: macOS Mojave, version 10.14</span></font></div><div><font color="#000000" face="arial, sans-serif"><span style="font-size:14px;font-variant-ligatures:no-common-ligatures">Best wishes,</span></font></div><div><font color="#000000" face="Menlo"><span style="font-size:14px;font-variant-ligatures:no-common-ligatures"><br></span></font></div>
</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 18, 2019 at 3:16 AM Dave Allured - NOAA Affiliate <<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Recently there are some known conditions that can cause unknown format and corrupted file messages for valid netcdf files. I recommend diagnosing individual files, not using NCL, before dismissing an entire file set as corrupted. Try this black magic first and see if NCL can then read the files:</div><div dir="ltr"><br></div> Bash: HDF5_USE_FILE_LOCKING=FALSE<br> C-shell: setenv HDF5_USE_FILE_LOCKING FALSE<div><br></div><div>If that does not work, then try these tests. The first two are guaranteed to work on all file types.</div><div><br></div><div> file <a href="http://data.nc" target="_blank">data.nc</a></div><div> od -c -N16 <a href="http://data.nc" target="_blank">data.nc</a><br></div><div><div> h5dump -BH <a href="http://data.nc" target="_blank">data.nc</a></div></div><div> ncdump -k <a href="http://data.nc" target="_blank">data.nc</a><br></div><div> ncdump -sh <a href="http://data.nc" target="_blank">data.nc</a></div><div dir="ltr"><br></div><div>If the problem has not become obvious, then post output from these tests to this mailing list. If more than 40 lines long, put all output into a text file with name ending in .txt, and send as a file attachment to your message. Please do not send any screen shots.</div><div><br></div><div>Also send your NCL version number and type of computer system.</div><div dir="ltr"><br></div><div>--Dave</div><div><br></div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 17, 2019 at 3:27 PM Dennis Shea via ncl-talk <<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>As the message states, the file is "<b>corrupted</b>". This is not an NCL issue. <br></div><div><br></div><div>[1] delete the file[s] and try reacquiring them</div><div>[2] Possibly, the source files are corrupted.</div><div>---<br></div><div>FYI: There are some Persiann examples.<br></div><div><a href="https://www.ncl.ucar.edu/Applications/HiResPrc.shtml" target="_blank"><b>https://www.ncl.ucar.edu/Applications/HiResPrc.shtml</b></a></div><div><br></div><div><br></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 17, 2019 at 12:15 PM Setareh Rahimi <<a href="mailto:setareh.rahimi@gmail.com" target="_blank">setareh.rahimi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear Adam, and Dennis,<div>Thanks for your help. hopefully I could combine those files together, but NCL tells there is something wrong with some files (attached image). In order to check what could be wrong, I redownload 1983 files again and got many warning for the second time. Any suggestion to remove those warning, please?</div><div>Best wishes,</div></div></blockquote></div></blockquote></div></div></div></div></div></div></blockquote></div></blockquote></div></div></div></blockquote></div></blockquote></div></div></blockquote></div>
</blockquote></div></div></div></div>