<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-AU" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hello Bill,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> I had a similar problem with file handles (as you seem to have had too with your notes on setfileoption), so I suspect that this might be the problem (as I suspect that you do
too).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> But the difference here is that I suspect that the issue might be on the OPeNDAP server end?!?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> Have you been in contact with the owner of the OPeNDAP server? Can you/they inspect their logs?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> If indeed there is an issue with file handles on the server, then there may be some evidence in their logs...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> Is it possible that the remote nature of your customers is affecting file close commands coming back to the server? There is not enough evidence in the log you provided to determine
this.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> I admit that all this is all conjecture (but sometimes it gets you closer to the problem).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Griff.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Capehart, William J [mailto:William.Capehart@sdsmt.edu]
<br>
<b>Sent:</b> Wednesday, 20 September 2017 1:48 PM<br>
<b>To:</b> Griffith Young; ncl-talk@ucar.edu<br>
<b>Subject:</b> Re: Reopening an old trouble ticket: OPeNDAP failing when reading a large number of files [SEC=UNCLASSIFIED]<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Hi Griff. <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Correct. We get the classic 404 error. However, the file Is indeed there. As with Jon’s problem if you change the start date further down the street, the seemingly missing file will be read
and a later file will be the one where it stops. The script will read the first ~240 files it encounters and will then halt with an errant missing file error.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Bill<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="color:black">From: </span></b><span lang="EN-US" style="color:black">Griffith Young <<a href="mailto:griffith.young@bom.gov.au">griffith.young@bom.gov.au</a>><br>
<b>Date: </b>Tuesday, September 19, 2017 at 21:44 MDT<br>
<b>To: </b>William Capehart <<a href="mailto:William.Capehart@sdsmt.edu">William.Capehart@sdsmt.edu</a>>, "<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a>" <<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a>><br>
<b>Subject: </b>RE: Reopening an old trouble ticket: OPeNDAP failing when reading a large number of files [SEC=UNCLASSIFIED]<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">The file does not exist according to the log (<b>404</b>).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2017-09-20T02:52:12.951 +0000 [1841685452][ 123718] INFO - threddsServlet - Request Completed - 404 - -1 - 491915<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Additionally, I did not get a response when I clicked on your link.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">As to why that might be, I cannot answer that part.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Hope this helps.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Griff.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D"> </span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ncl-talk [<a href="mailto:ncl-talk-bounces@ucar.edu">mailto:ncl-talk-bounces@ucar.edu</a>]
<b>On Behalf Of </b>Capehart, William J<br>
<b>Sent:</b> Wednesday, 20 September 2017 1:31 PM<br>
<b>To:</b> <a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
<b>Subject:</b> [ncl-talk] Reopening an old trouble ticket: OPeNDAP failing when reading a large number of files</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">I am having the same issue that Jon Meyer had a few years ago. His original post is linked here.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><a href="http://mailman.ucar.edu/pipermail/ncl-talk/2015-June/thread.html#2853">http://mailman.ucar.edu/pipermail/ncl-talk/2015-June/thread.html#2853</a>
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">To repeat the problem, I have a large number of OPeNDAP files that we’re trying help my remote students access.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">After reading a large number of files (I don’t have a specific answer but it’s less than the magic 1024 value and seems to hover around 240 files) via the server I get the same error message.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">Cannot create cookie file</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">CURL Error: Couldn't resolve host name</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">curl error details: </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">fatal:Could not open (<a href="http://kyrill.ias.sdsmt.edu:8080/thredds/dodsC/BWW_GENS/2017/03/04/gens003_WRFRAP_ALL_2017-03-04_18.wrf_ci_turb.grib2">http://kyrill.ias.sdsmt.edu:8080/thredds/dodsC/BWW_GENS/2017/03/04/gens003_WRFRAP_ALL_2017-03-04_18.wrf_ci_turb.grib2</a>)</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">fatal:Either file (f) isn't defined or variable (Geopotential_height_isobaric_ens) is not a variable in the file</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Courier New"">fatal:["Execute.c":8640]:Execute: Error occurred at or near line 282 in file read_z500_ensembles_from_thredds.ncl</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">We are using the delete(f) to release the file units and also are using the
<i>setfileoption("grib2", "SuppressClose", False)</i> command</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">I also have the log files from our THREDDS server (linked below).</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><a href="https://gist.github.com/wjcapehart/46d3f594d223f9b8bb2c6e577a3ecaad">https://gist.github.com/wjcapehart/46d3f594d223f9b8bb2c6e577a3ecaad</a></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Hopefully, this will give the NCL or THREDDS or Tomcat people an idea of what is going on with this problem.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Cheers-n-Thanks</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">------------------------------------------------<br>
Bill Capehart <<a href="mailto:William.Capehart@sdsmt.edu">William.Capehart@sdsmt.edu</a>><br>
Atmospheric and Environmental Sciences Program Coordinator<br>
Civil and Environmental Engineering<br>
201 Mineral Industries Building<br>
South Dakota School of Mines and Technology<br>
501 East St Joseph Street<br>
Rapid City, SD 57701-3995<br>
Ph: +1-605-394-1994 Mobile: +1-605-484-5692</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</body>
</html>