[ncl-talk] Reopening an old trouble ticket: OPeNDAP failing when reading a large number of files [SEC=UNCLASSIFIED]

[Rick Brownrigg]

Hi William,

This is a stretch, but I'm at a loss. You might check your /tmp directory:
is it of very limited capacity, or are there tons of these "oc.*"
directories there?  They seem to be some sort of temporary cookie file.
Here's some discussion about them from the Unidata site:

https://github.com/Unidata/netcdf-c/issues/435

Rick

On Tue, Sep 19, 2017 at 10:10 PM, Griffith Young <griffith.young at bom.gov.au>
wrote:

> Hello Bill,
>
>                 I had a similar problem with file handles (as you seem to
> have had too with your notes on setfileoption), so I suspect that this
> might be the problem (as I suspect that you do too).
>
>
>
>                 But the difference here is that I suspect that the issue
> might be on the OPeNDAP server end?!?
>
>
>
>                 Have you been in contact with the owner of the OPeNDAP
> server?  Can you/they inspect their logs?
>
>
>
>                 If indeed there is an issue with file handles on the
> server, then there may be some evidence in their logs...
>
>
>
>                 Is it possible that the remote nature of your customers is
> affecting file close commands coming back to the server?  There is not
> enough evidence in the log you provided to determine this.
>
>
>
>                 I admit that all this is all conjecture (but sometimes it
> gets you closer to the problem).
>
>
>
> Griff.
>
>
>
>
>
> *From:* Capehart, William J [mailto:William.Capehart at sdsmt.edu]
> *Sent:* Wednesday, 20 September 2017 1:48 PM
> *To:* Griffith Young; ncl-talk at ucar.edu
> *Subject:* Re: Reopening an old trouble ticket: OPeNDAP failing when
> reading a large number of files [SEC=UNCLASSIFIED]
>
>
>
> Hi Griff.
>
>
>
> Correct.  We get the classic 404 error.  However, the file Is indeed
> there.  As with Jon’s problem if you change the start date further down the
> street, the seemingly missing file will be read and a later file will be
> the one where it stops.  The script will read the first ~240 files it
> encounters and will then halt with an errant missing file error.
>
>
>
> Bill
>
>
>
> *From: *Griffith Young <griffith.young at bom.gov.au>
> *Date: *Tuesday, September 19, 2017 at 21:44 MDT
> *To: *William Capehart <William.Capehart at sdsmt.edu>, "ncl-talk at ucar.edu" <
> ncl-talk at ucar.edu>
> *Subject: *RE: Reopening an old trouble ticket: OPeNDAP failing when
> reading a large number of files [SEC=UNCLASSIFIED]
>
>
>
> The file does not exist according to the log (*404*).
>
>
>
> 2017-09-20T02:52:12.951 +0000 [1841685452][ 123718] INFO - threddsServlet
> - Request Completed - 404 - -1 - 491915
>
>
>
> Additionally, I did not get a response when I clicked on your link.
>
>
>
> As to why that might be, I cannot answer that part.
>
>
>
> Hope this helps.
>
>
>
> Griff.
>
>
>
> *From:* ncl-talk [mailto:ncl-talk-bounces at ucar.edu
> <ncl-talk-bounces at ucar.edu>] *On Behalf Of *Capehart, William J
> *Sent:* Wednesday, 20 September 2017 1:31 PM
> *To:* ncl-talk at ucar.edu
> *Subject:* [ncl-talk] Reopening an old trouble ticket: OPeNDAP failing
> when reading a large number of files
>
>
>
>
>
> I am having the same issue that Jon Meyer had a few years ago.  His
> original post is linked here.
>
> http://mailman.ucar.edu/pipermail/ncl-talk/2015-June/thread.html#2853
>
>
>
>
>
> To repeat the problem, I have a large number of OPeNDAP files that we’re
> trying help my remote students access.
>
>
>
> After reading a large number of files (I don’t have a specific answer but
> it’s less than the magic 1024 value and seems to hover around 240 files)
> via the server I get the same error message.
>
>
>
> Cannot create cookie file
>
> CURL Error: Couldn't resolve host name
>
> curl error details:
>
> fatal:Could not open (http://kyrill.ias.sdsmt.edu:
> 8080/thredds/dodsC/BWW_GENS/2017/03/04/gens003_WRFRAP_ALL_
> 2017-03-04_18.wrf_ci_turb.grib2)
>
> fatal:Either file (f) isn't defined or variable
> (Geopotential_height_isobaric_ens) is not a variable in the file
>
> fatal:["Execute.c":8640]:Execute: Error occurred at or near line 282 in
> file read_z500_ensembles_from_thredds.ncl
>
>
>
>
>
> We are using the delete(f) to release the file units and also are using
> the *setfileoption("grib2", "SuppressClose", False)* command
>
>
>
> I also have the log files from our THREDDS server (linked below).
>
>
>
> https://gist.github.com/wjcapehart/46d3f594d223f9b8bb2c6e577a3ecaad
>
>
>
> Hopefully, this will give the NCL or THREDDS or Tomcat people an idea of
> what is going on with this problem.
>
>
>
> Cheers-n-Thanks
>
> ------------------------------------------------
> Bill Capehart <William.Capehart at sdsmt.edu>
> Atmospheric and Environmental Sciences Program Coordinator
> Civil and Environmental Engineering
> 201 Mineral Industries Building
> South Dakota School of Mines and Technology
> 501 East St Joseph Street
> Rapid City, SD 57701-3995
> Ph: +1-605-394-1994 <(605)%20394-1994>   Mobile: +1-605-484-5692
> <(605)%20484-5692>
>
>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170920/3820b601/attachment.html>