<div dir="ltr"><div class="gmail_default" style="font-size:small">Archana,</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I hope to look at this issue today and will try to answer your questions.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I'm hoping that you can remove the do loop altogether, but this can only happen if the grids are the same every time in the loop.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Thanks for providing the scripts and data files! Can you remove them from the <a href="http://ftp.cgd.ucar.edu">ftp.cgd.ucar.edu</a> site? I've already downloaded them.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">--Mary</div><div class="gmail_default" style="font-size:small"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 12, 2016 at 1:18 PM, Archana Dayalu <span dir="ltr"><<a href="mailto:adayalu@seas.harvard.edu" target="_blank">adayalu@seas.harvard.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sorry, a couple more things:<div>1) To answer your question, Mary: I am using the setfileoption command to change file type.</div><div>2) I just re-ran the whole script a different way (i.e., I broke it up by common days of a month rather than entire months ... so instead of 12 independent loops of 28-31 iterations, I did 31 independent loops of 4-12 iterations). So far it seems to be completing successfully which indicates that it was a problem with too many calls to the ESMF regrid with weights function? But I don't understand why that would be so (if it were the issue).</div><div>Regards</div><span class="HOEnZb"><font color="#888888"><div>Archana</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 12, 2016 at 3:04 PM, Archana Dayalu <span dir="ltr"><<a href="mailto:adayalu@seas.harvard.edu" target="_blank">adayalu@seas.harvard.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Mary,<div>I just uploaded my code + data to the ftp site. The files I have uploaded are:</div><div>1) 31 files of type "<a href="http://aux2vprm_d01_2006-01-dd_hrlyavg.nc" target="_blank">aux2vprm_d01_2006-01-dd_hrlyavg.nc</a>" (where dd=two digit day in january), 2.5MB each</div><div>2) 1 weights file "<a href="http://T2SWDOWN_to_MODIS_bilinear.faster.nc" target="_blank">T2SWDOWN_to_MODIS_bilinear.faster.nc</a>" (this is kind of big ... ~6GB)</div><div>3) 1 file with the script "regrid_using_weights_T2SWDOWN_MOD_noplots.ncl" (edit the header of file to specify directories; all else should not require editing).</div><div>Other requested information: </div><div>ncl -V = 6.2.1<br></div><div>uname -a = Linux <a href="http://rclogin11.rc.fas.harvard.edu" target="_blank">rclogin11.rc.fas.harvard.edu</a> 2.6.32-431.17.1.el6.x86_64 #1 SMP Wed May 7 23:32:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux</div><div>Thanks very much! Please let me know if you have any trouble with the code/files. </div><div>Regards</div><div>Archana</div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Tue, Jan 12, 2016 at 11:29 AM, Mary Haley <span dir="ltr"><<a href="mailto:haley@ucar.edu" target="_blank">haley@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-size:small">Are you by any chance using setfileoption to change the write type to "NetCDF4"? </div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I'm wondering if there's an issue with going into NetCDF4 mode, that may cause a problem if you stay in that mode when reading your weights file.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Can you provide us with the full code and all the files needed to run it? You can use our ftp if the files are not too large:<br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default"><a href="http://www.ncl.ucar.edu/report_bug.shtml#HowToFTP" target="_blank">http://www.ncl.ucar.edu/report_bug.shtml#HowToFTP</a><span><font color="#888888"><br></font></span></div><span><font color="#888888"><div class="gmail_default"><br></div><div class="gmail_default" style="font-size:small"> </div><div class="gmail_default" style="font-size:small">--Mary</div><div class="gmail_default" style="font-size:small"><br></div></font></span></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, Jan 11, 2016 at 2:09 PM, Archana Dayalu <span dir="ltr"><<a href="mailto:adayalu@seas.harvard.edu" target="_blank">adayalu@seas.harvard.edu</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi there,<div>I am using the ESMF tools to regrid a bunch of wrf-grid files to rectangular grid. I am running each month independently (running the script in 12 independent batches, where each batch corresponds to a month) but the days in a given month are done serially in that loop. My steps, as a quick summary, are:</div><div>(1) generate the weights once, valid for all variables for all files for the year. Then, for each of 365 daily temperature and radiation files:</div><div>(2) extract the daily T2 (Kelvin), SWDOWN variables which I previously defined as a 3D array (XDim,YDim,23)</div><div>(3) apply the weights to hourly T2, SWDOWN output for a year. This step involves a loop ... and the relevant part includes:</div><div>do hh=0,23</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>T2_regrid = ESMF_regrid_with_weights(T2,wgt_file_name,Opt)</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>SWDOWN_regrid = ESMF_regrid_with_weights(SWDOWN,wgt_file_name,Opt)</div><div>T2_array(:,:,hh) = T2_regrid</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>SWDOWN_array(:,:,hh) = SWDOWN_regrid</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>; I also have it print out min max for each variable for each hour regridded</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>end do</div></blockquote><div>(4) Write out the two regridded arrays to a daily netcdf file using method 1, Netcdf4, with file compression level set to 4</div><div><br></div><div>The problem: The script fails toward the end (e.g, around the 27th file of a month) with the following error message and begins to input nonsense Temperature and radiation values into the array and states the weights file cannot be opened (it does exist). When I re-run the failed time-period as a one-time regridding, it regrids just fine. Is ncl unhappy because of the high I/O? It doesn't fail at the netcdf generation step ... it seems to fail at the regridding step itself. Thanks for any help! The error output follows ... </div><div><br></div><div><div>Variable: hh</div><div>Type: integer</div><div>Total Size: 4 bytes</div><div> 1 values</div><div>Number of Dimensions: 1</div><div>Dimensions and sizes: [1]</div><div>Coordinates:</div><div>(0) 15</div><div>fatal:["NclNewHDF5.c":3787]:NclNewHDF5.c: Cannot open file: </n/regal/wofsy_lab/adayalu/<a href="http://T2SWDOWN_to_MODIS_bilinear.faster.nc" target="_blank">T2SWDOWN_to_MODIS_bilinear.faster.nc</a>></div><div>fatal:["NclAdvancedFile.c":2667]:_NclFileFillHLFS: Could not open (/n/regal/wofsy_lab/adayalu/<a href="http://T2SWDOWN_to_MODIS_bilinear.faster.nc" target="_blank">T2SWDOWN_to_MODIS_bilinear.faster.nc</a>)</div><div>(0) ESMF_regrid_with_weights: cannot open weight file '/n/regal/wofsy_lab/adayalu/<a href="http://T2SWDOWN_to_MODIS_bilinear.faster.nc" target="_blank">T2SWDOWN_to_MODIS_bilinear.faster.nc</a>'.</div><div>(0)</div><div>(0) TEMP at 2 M: min=239.284 max=299.333</div><div>(0)</div><div>(0) min=9.96921e+36 max=9.96921e+36</div><div>warning:VarVarWrite: rhs has no dimension name or coordinate variable, deleting name of lhs dimension number (0) and destroying coordinate var, use "(/../)" if this is not desired outcome</div><div>warning:VarVarWrite: rhs has no dimension name or coordinate variable, deleting name of lhs dimension number (1) and destroying coordinate var, use "(/../)" if this is not desired outcome</div><div>warning:["Execute.c":8578]:Execute: Error occurred at or near line 64 in file regrid_using_weights_T2SWDOWN_MOD_noplots.ncl</div><div><br></div><div>HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:</div></div><div><div> #000: H5D.c line 461 in H5Dget_space(): not a dataset</div><div> major: Invalid arguments to routine</div><div> minor: Inappropriate type</div></div><div>(...then a series of similar error messages are output .... and then it continues .... and nonsense is output):</div><div><br></div><div><div>Variable: hh</div><div>Type: integer</div><div>Total Size: 4 bytes</div><div> 1 values</div><div>Number of Dimensions: 1</div><div>Dimensions and sizes: [1]</div><div>Coordinates:</div><div>(0) 17</div><div>(0)</div><div>(0) TEMP at 2 M: min=0.0114805 max=299.333</div><div>(0)</div><div>(0) DOWNWARD SHORT WAVE FLUX AT GROUND SURFACE: min=0 max=2.90338e-31</div></div><div><br></div><div><div><div> </div>
</div></div></div>
<br></div></div><span>_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br>
<br></span></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div>____________________________________________________<br><b><font size="2"><span style="color:rgb(0,0,0)">Archana Dayalu</span></font></b><br><font style="color:rgb(0,0,0)" size="2">Graduate Student<br></font><font color="#888888"><font size="2"><span style="color:rgb(0,0,0)">Dept. of Earth and Planetary Sciences</span><br style="color:rgb(0,0,0)"><span style="color:rgb(0,0,0)">Harvard University</span><br style="color:rgb(0,0,0)"><span style="color:rgb(0,0,0)">24 Oxford Street #402</span><br style="color:rgb(0,0,0)"><span style="color:rgb(0,0,0)">Cambridge, MA 02138</span><br style="color:rgb(0,0,0)"><a style="color:rgb(0,0,153)" value="+16174966361">(617) 384-8206</a></font><br>
</font> </div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div>____________________________________________________<br><b><font size="2"><span style="color:rgb(0,0,0)">Archana Dayalu</span></font></b><br><font style="color:rgb(0,0,0)" size="2">Graduate Student<br></font><font color="#888888"><font size="2"><span style="color:rgb(0,0,0)">Dept. of Earth and Planetary Sciences</span><br style="color:rgb(0,0,0)"><span style="color:rgb(0,0,0)">Harvard University</span><br style="color:rgb(0,0,0)"><span style="color:rgb(0,0,0)">24 Oxford Street #402</span><br style="color:rgb(0,0,0)"><span style="color:rgb(0,0,0)">Cambridge, MA 02138</span><br style="color:rgb(0,0,0)"><a style="color:rgb(0,0,153)" value="+16174966361">(617) 384-8206</a></font><br>
</font> </div>
</div>
</div></div></blockquote></div><br></div>