[ncl-talk] Reopening an old trouble ticket: OPeNDAP failing when reading a large number of files [SEC=UNCLASSIFIED]

[Griffith Young]

Hello Bill,
                I had a similar problem with file handles (as you seem to have had too with your notes on setfileoption), so I suspect that this might be the problem (as I suspect that you do too).

                But the difference here is that I suspect that the issue might be on the OPeNDAP server end?!?

                Have you been in contact with the owner of the OPeNDAP server?  Can you/they inspect their logs?

                If indeed there is an issue with file handles on the server, then there may be some evidence in their logs...

                Is it possible that the remote nature of your customers is affecting file close commands coming back to the server?  There is not enough evidence in the log you provided to determine this.

                I admit that all this is all conjecture (but sometimes it gets you closer to the problem).

Griff.


From: Capehart, William J [mailto:William.Capehart at sdsmt.edu]
Sent: Wednesday, 20 September 2017 1:48 PM
To: Griffith Young; ncl-talk at ucar.edu
Subject: Re: Reopening an old trouble ticket: OPeNDAP failing when reading a large number of files [SEC=UNCLASSIFIED]

Hi Griff.

Correct.  We get the classic 404 error.  However, the file Is indeed there.  As with Jon’s problem if you change the start date further down the street, the seemingly missing file will be read and a later file will be the one where it stops.  The script will read the first ~240 files it encounters and will then halt with an errant missing file error.

Bill

From: Griffith Young <griffith.young at bom.gov.au<mailto:griffith.young at bom.gov.au>>
Date: Tuesday, September 19, 2017 at 21:44 MDT
To: William Capehart <William.Capehart at sdsmt.edu<mailto:William.Capehart at sdsmt.edu>>, "ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>" <ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: RE: Reopening an old trouble ticket: OPeNDAP failing when reading a large number of files [SEC=UNCLASSIFIED]

The file does not exist according to the log (404).

2017-09-20T02:52:12.951 +0000 [1841685452][ 123718] INFO - threddsServlet - Request Completed - 404 - -1 - 491915

Additionally, I did not get a response when I clicked on your link.

As to why that might be, I cannot answer that part.

Hope this helps.

Griff.

From: ncl-talk [mailto:ncl-talk-bounces at ucar.edu] On Behalf Of Capehart, William J
Sent: Wednesday, 20 September 2017 1:31 PM
To: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: [ncl-talk] Reopening an old trouble ticket: OPeNDAP failing when reading a large number of files


I am having the same issue that Jon Meyer had a few years ago.  His original post is linked here.
http://mailman.ucar.edu/pipermail/ncl-talk/2015-June/thread.html#2853


To repeat the problem, I have a large number of OPeNDAP files that we’re trying help my remote students access.

After reading a large number of files (I don’t have a specific answer but it’s less than the magic 1024 value and seems to hover around 240 files) via the server I get the same error message.

Cannot create cookie file
CURL Error: Couldn't resolve host name
curl error details:
fatal:Could not open (http://kyrill.ias.sdsmt.edu:8080/thredds/dodsC/BWW_GENS/2017/03/04/gens003_WRFRAP_ALL_2017-03-04_18.wrf_ci_turb.grib2)
fatal:Either file (f) isn't defined or variable (Geopotential_height_isobaric_ens) is not a variable in the file
fatal:["Execute.c":8640]:Execute: Error occurred at or near line 282 in file read_z500_ensembles_from_thredds.ncl


We are using the delete(f) to release the file units and also are using the setfileoption("grib2", "SuppressClose", False) command

I also have the log files from our THREDDS server (linked below).

https://gist.github.com/wjcapehart/46d3f594d223f9b8bb2c6e577a3ecaad

Hopefully, this will give the NCL or THREDDS or Tomcat people an idea of what is going on with this problem.

Cheers-n-Thanks
------------------------------------------------
Bill Capehart <William.Capehart at sdsmt.edu<mailto:William.Capehart at sdsmt.edu>>
Atmospheric and Environmental Sciences Program Coordinator
Civil and Environmental Engineering
201 Mineral Industries Building
South Dakota School of Mines and Technology
501 East St Joseph Street
Rapid City, SD 57701-3995
Ph: +1-605-394-1994   Mobile: +1-605-484-5692


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170920/1e3c617a/attachment.html>