[ncl-talk] Reopening an old trouble ticket: OPeNDAP failing when reading a large number of files [SEC=UNCLASSIFIED]

Wed Sep 20 11:55:20 MDT 2017

One possible problem is the statement:
setfileoption("grib2", "SuppressClose", False)

Actually the "SuppressClose" option is for NetCDF files and not for
GRIB files. Since NCL's OPeNDAP interface is provided by the NetCDF
library, no matter what the original format of the data, the option
needs to be specified as a NetCDF option. So change the statement
abovel to:

setfileoption("nc", "SuppressClose", False)

This is probably more confusing than necessary. It should be
explicitly documented at any rate.
 -dave

On Wed, Sep 20, 2017 at 10:33 AM, Rick Brownrigg <brownrig at ucar.edu> wrote:
> Hi William,
>
> This is a stretch, but I'm at a loss. You might check your /tmp directory:
> is it of very limited capacity, or are there tons of these "oc.*"
> directories there?  They seem to be some sort of temporary cookie file.
> Here's some discussion about them from the Unidata site:
>
> https://github.com/Unidata/netcdf-c/issues/435
>
> Rick
>
> On Tue, Sep 19, 2017 at 10:10 PM, Griffith Young <griffith.young at bom.gov.au>
> wrote:
>>
>> Hello Bill,
>>
>>                 I had a similar problem with file handles (as you seem to
>> have had too with your notes on setfileoption), so I suspect that this might
>> be the problem (as I suspect that you do too).
>>
>>
>>
>>                 But the difference here is that I suspect that the issue
>> might be on the OPeNDAP server end?!?
>>
>>
>>
>>                 Have you been in contact with the owner of the OPeNDAP
>> server?  Can you/they inspect their logs?
>>
>>
>>
>>                 If indeed there is an issue with file handles on the
>> server, then there may be some evidence in their logs...
>>
>>
>>
>>                 Is it possible that the remote nature of your customers is
>> affecting file close commands coming back to the server?  There is not
>> enough evidence in the log you provided to determine this.
>>
>>
>>
>>                 I admit that all this is all conjecture (but sometimes it
>> gets you closer to the problem).
>>
>>
>>
>> Griff.
>>
>>
>>
>>
>>
>> From: Capehart, William J [mailto:William.Capehart at sdsmt.edu]
>> Sent: Wednesday, 20 September 2017 1:48 PM
>> To: Griffith Young; ncl-talk at ucar.edu
>> Subject: Re: Reopening an old trouble ticket: OPeNDAP failing when reading
>> a large number of files [SEC=UNCLASSIFIED]
>>
>>
>>
>> Hi Griff.
>>
>>
>>
>> Correct.  We get the classic 404 error.  However, the file Is indeed
>> there.  As with Jon’s problem if you change the start date further down the
>> street, the seemingly missing file will be read and a later file will be the
>> one where it stops.  The script will read the first ~240 files it encounters
>> and will then halt with an errant missing file error.
>>
>>
>>
>> Bill
>>
>>
>>
>> From: Griffith Young <griffith.young at bom.gov.au>
>> Date: Tuesday, September 19, 2017 at 21:44 MDT
>> To: William Capehart <William.Capehart at sdsmt.edu>, "ncl-talk at ucar.edu"
>> <ncl-talk at ucar.edu>
>> Subject: RE: Reopening an old trouble ticket: OPeNDAP failing when reading
>> a large number of files [SEC=UNCLASSIFIED]
>>
>>
>>
>> The file does not exist according to the log (404).
>>
>>
>>
>> 2017-09-20T02:52:12.951 +0000 [1841685452][ 123718] INFO - threddsServlet
>> - Request Completed - 404 - -1 - 491915
>>
>>
>>
>> Additionally, I did not get a response when I clicked on your link.
>>
>>
>>
>> As to why that might be, I cannot answer that part.
>>
>>
>>
>> Hope this helps.
>>
>>
>>
>> Griff.
>>
>>
>>
>> From: ncl-talk [mailto:ncl-talk-bounces at ucar.edu] On Behalf Of Capehart,
>> William J
>> Sent: Wednesday, 20 September 2017 1:31 PM
>> To: ncl-talk at ucar.edu
>> Subject: [ncl-talk] Reopening an old trouble ticket: OPeNDAP failing when
>> reading a large number of files
>>
>>
>>
>>
>>
>> I am having the same issue that Jon Meyer had a few years ago.  His
>> original post is linked here.
>>
>> http://mailman.ucar.edu/pipermail/ncl-talk/2015-June/thread.html#2853
>>
>>
>>
>>
>>
>> To repeat the problem, I have a large number of OPeNDAP files that we’re
>> trying help my remote students access.
>>
>>
>>
>> After reading a large number of files (I don’t have a specific answer but
>> it’s less than the magic 1024 value and seems to hover around 240 files) via
>> the server I get the same error message.
>>
>>
>>
>> Cannot create cookie file
>>
>> CURL Error: Couldn't resolve host name
>>
>> curl error details:
>>
>> fatal:Could not open
>> (http://kyrill.ias.sdsmt.edu:8080/thredds/dodsC/BWW_GENS/2017/03/04/gens003_WRFRAP_ALL_2017-03-04_18.wrf_ci_turb.grib2)
>>
>> fatal:Either file (f) isn't defined or variable
>> (Geopotential_height_isobaric_ens) is not a variable in the file
>>
>> fatal:["Execute.c":8640]:Execute: Error occurred at or near line 282 in
>> file read_z500_ensembles_from_thredds.ncl
>>
>>
>>
>>
>>
>> We are using the delete(f) to release the file units and also are using
>> the setfileoption("grib2", "SuppressClose", False) command
>>
>>
>>
>> I also have the log files from our THREDDS server (linked below).
>>
>>
>>
>> https://gist.github.com/wjcapehart/46d3f594d223f9b8bb2c6e577a3ecaad
>>
>>
>>
>> Hopefully, this will give the NCL or THREDDS or Tomcat people an idea of
>> what is going on with this problem.
>>
>>
>>
>> Cheers-n-Thanks
>>
>> ------------------------------------------------
>> Bill Capehart <William.Capehart at sdsmt.edu>
>> Atmospheric and Environmental Sciences Program Coordinator
>> Civil and Environmental Engineering
>> 201 Mineral Industries Building
>> South Dakota School of Mines and Technology
>> 501 East St Joseph Street
>> Rapid City, SD 57701-3995
>> Ph: +1-605-394-1994   Mobile: +1-605-484-5692
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>