[ncl-talk] isfilepresent checks on grib2 data from NOMADS

Fri Apr 6 13:35:22 MDT 2018

Jon and Rick,

There is a documentation problem with isfilepresent, addfile, and
addfiles.  The rules for the file_path argument are unclear in the function
descriptions.  File_path must follow the rules for NCL "supported formats",
which allow only two cases.  These are either (1) an absolute or relative
path to an actual file on the local file system, the so-called Unix path;
or (2) an OpenDAP URL.  General URL's to remote files on the internet, such
as FTP and HTTP paths, are not supported by these functions, except when
they are legal OpenDAP URL's properly hosted by remote OpenDAP servers.  I
would appreciate it if the documentation could be clarified up front about
this.

For example, here is the same argument description from the nc_open
function in the underlying netcdf library.  Ref. Unidata Netcdf website:

    path:  File name for netCDF dataset to be opened.
        When DAP support is enabled, then the path may be
        an OPeNDAP URL rather than a file path.

As Rick pointed out, it looks like you have one of these generic remote
file URL's.  I suggest that you first contact the data provider to see if
by any chance there is actual OpenDAP hosting on the newer NOMADS server,
for the files you need.  If available, this would be the most convenient
solution.

If that is not available, then I suggest something like curl or wget inside
NCL's systemfunc function, to download temporary files and test for
availability under NCL script control.  After downloading, you can now use
addfile to read the local temporary file if it is an NCL "supported" format
such as grib2.

--Dave

On Fri, Apr 6, 2018 at 4:30 AM, Jon Meyer <jonathan.meyer at aggiemail.usu.edu>
wrote:

> Hi Rick, thanks for the help on this.
>
> This server is not an OpenDap DODS server, but from my understanding, the
> isfilepresent function should successfully check any file type accepted by
> addfiles, which grib2 should be in my version of NCL. Correct me if I'm
> wrong on that, which may explain the problem now that my code is no longer
> looking at an opendap server.
>
> I'm open to alternate solutions to checking the presence of a file if
> these NCL functions are unable to check.
>
> On Thu, Apr 5, 2018 at 8:02 PM, Rick Brownrigg <brownrig at ucar.edu> wrote:
>
>> Hi John,
>>
>> I'm seeing the same result. Its not clear to me that the site you refer
>> to is an OpenDAP site?  Do you know it to be one?  I found some (possibly
>> stale) info that suggested a DODS server would response with headers
>> indicating it was a dods server:
>>
>> (https://www.opendap.org/support/faq/server/test-dods)
>>
>> When I run:
>>
>>   wget -S http://nomads.ncep.noaa.gov:80
>>
>> I get:
>>
>> -2018-04-05 19:52:17--  http://nomads.ncep.noaa.gov/
>> Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
>> Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:80...
>> connected.
>> HTTP request sent, awaiting response...
>>   HTTP/1.1 200 OK
>>   Date: Fri, 06 Apr 2018 01:54:09 GMT
>>   Server: Apache
>>   X-Frame-Options: SAMEORIGIN
>>   X-Content-Type-Options: nosniff
>>   X-XSS-Protection: 1; mode=block
>>   Accept-Ranges: bytes
>>   Cache-Control: max-age=14400
>>   Expires: Fri, 06 Apr 2018 05:54:09 GMT
>>   Vary: Accept-Encoding
>>   Keep-Alive: timeout=300, max=100
>>   Connection: Keep-Alive
>>   Transfer-Encoding: chunked
>>   Content-Type: text/html; charset=UTF-8
>> Length: unspecified [text/html]
>> Saving to: ‘index.html’
>>
>>     [ <=>
>> ] 55,810       275KB/s   in 0.2s
>>
>> 2018-04-05 19:52:18 (275 KB/s) - ‘index.html’ saved [55810]
>>
>> which sort of indicates its just a generic web server.
>>
>> I'm not an expert in this, just trying to help trouble shoot...
>> Rick
>>
>>
>> On Thu, Apr 5, 2018 at 1:45 PM, Jon Meyer <jonathan.meyer at aggiemail.usu.
>> edu> wrote:
>>
>>> Good day,
>>>
>>> For several years I've used ncl's opendap functionality to access
>>> current operational NWP output from NOMADS servers to produce custom
>>> forecasting charts. (currently running most recent NCLv6.4.0. I've
>>> discovered that the opendap server lags a bit behind in hosting the most
>>> recent data when compared to grib2 data hosted on a different NOMADS server
>>> and hence decided to switch to the faster available data.
>>>
>>> When modifying my ncl code to search the new server for grib2 data, I
>>> found the isfilepresent function is unable to successfully assess if these
>>> grib2 files are present. I've tried the fileexists function as well with
>>> similar results.
>>>
>>> This is odd because I can hardcode wget code to successfully download
>>> from the same URL that both isfilepresent and fileexist functions return
>>> "False" on. I've confirmed the URLs are correct but can't get these ncl
>>> checks to work. My code is automated and runs as a cron-job, so I need a
>>> way to identify when the most recent files are populated on the servers
>>> without hardcoding.
>>>
>>> My understanding is that grib2 is a supported file that isfilepresent
>>> should be capable of seeing--it worked great for the opendap servers.
>>> Perhaps there is something on the NOMADS side that prevents the file check,
>>> but given wget works without any certificate/user checks I would assume the
>>> issues lies on my end? I'm sure this is something minor that my tired eyes
>>> just aren't picking up on.
>>>
>>> Here's a code snipit from part of the function doing this. Notice I've
>>> hardcoded in a valid URL for testing. This same URL will work in wget and
>>> manually pasting into browser. Here it is directly for any testing on your
>>> end. (note for readers coming across this down the road: there is a 7 day
>>> buffer after which the link should become void as new data replaces it.)
>>>
>>> http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.2
>>> 0180405/nam.t00z.awphys00.tm00.grib2
>>>
>>> Any help is much appreciated!
>>>
>>> SNIP==============================================================
>>>
>>> ;----CHECK IF MOST RECENT TIME HAS OUTPUT AVAILABLE
>>> ;----IF NOT, WORK BACKWARDS TO IDENTIFY MOST RECENT AVAILABLE DATE
>>>   MostRecent_Julian = toint(cd_inv_calendar(toint(yy
>>> yy_current),mm_ind,toint(dd_current),toint(MostRecentInterva
>>> l),0,0,units,0))
>>>   MostRecent_Julian at units = units
>>>
>>>   ct = 0
>>>   do t = MostRecent_Julian, 0, 6
>>>      CheckTime = MostRecent_Julian - ct
>>>      CheckTime at units = units
>>>
>>>      MostRecent_UTC = cd_calendar(CheckTime,-3)
>>>
>>>      MostRecent_YYYYMMDD = str_get_cols(tostring(MostRecent_UTC), 0, 7)
>>>      MostRecent_file = "nam.t"+str_get_cols(tostring(MostRecent_UTC),
>>> 8,9) + "z.awphys00.tm00.grib2"
>>>      URL = dir + MostRecent_YYYYMMDD + "/" + MostRecent_file
>>>
>>> ;URL = "http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.
>>> 20180405/nam.t00z.awphys00.tm00.grib2"
>>>      if isfilepresent(URL)
>>>         print("Output is found: Most recent model is " + URL)
>>>         return(URL)
>>>         t = 0
>>>      else
>>>         print("Cant find file")
>>>         ct = ct + 6
>>>      end if
>>>   end do
>>>
>>> --
>>> Jonathan D.D. Meyer, Ph.D.
>>> Utah State University
>>> Department of Plants, Soils & Climate
>>> Utah Climate Center
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180406/ab9c8809/attachment.html>