[ncl-talk] isfilepresent checks on grib2 data from NOMADS
Rick Brownrigg
brownrig at ucar.edu
Fri Apr 6 21:58:00 MDT 2018
Hi Jon, Dave,
Thanks Dave for providing a clean explanation of the file_path rules. I'm
filing a ticket to tighten up the documentation, based upon your
suggestions.
Jon, as Dave suggests, the system/systemfunc functions in NCL can be used
in conjunction with wget/curl as a way to get what you need. I had devised
a somewhat elaborate way of emulating isfilepresent using wget's "--spider"
option (which probes for existence), but then realized that if you intend
to download the file *if* it exists, why not just outright attempt the
download and use simpler local means (like isfilepresent) of determining
success/failure.
RB
On Fri, Apr 6, 2018 at 1:35 PM, Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:
> Jon and Rick,
>
> There is a documentation problem with isfilepresent, addfile, and
> addfiles. The rules for the file_path argument are unclear in the function
> descriptions. File_path must follow the rules for NCL "supported formats",
> which allow only two cases. These are either (1) an absolute or relative
> path to an actual file on the local file system, the so-called Unix path;
> or (2) an OpenDAP URL. General URL's to remote files on the internet, such
> as FTP and HTTP paths, are not supported by these functions, except when
> they are legal OpenDAP URL's properly hosted by remote OpenDAP servers. I
> would appreciate it if the documentation could be clarified up front about
> this.
>
> For example, here is the same argument description from the nc_open
> function in the underlying netcdf library. Ref. Unidata Netcdf website:
>
> path: File name for netCDF dataset to be opened.
> When DAP support is enabled, then the path may be
> an OPeNDAP URL rather than a file path.
>
> As Rick pointed out, it looks like you have one of these generic remote
> file URL's. I suggest that you first contact the data provider to see if
> by any chance there is actual OpenDAP hosting on the newer NOMADS server,
> for the files you need. If available, this would be the most convenient
> solution.
>
> If that is not available, then I suggest something like curl or wget
> inside NCL's systemfunc function, to download temporary files and test for
> availability under NCL script control. After downloading, you can now use
> addfile to read the local temporary file if it is an NCL "supported" format
> such as grib2.
>
> --Dave
>
>
> On Fri, Apr 6, 2018 at 4:30 AM, Jon Meyer <jonathan.meyer at aggiemail.usu.
> edu> wrote:
>
>> Hi Rick, thanks for the help on this.
>>
>> This server is not an OpenDap DODS server, but from my understanding, the
>> isfilepresent function should successfully check any file type accepted by
>> addfiles, which grib2 should be in my version of NCL. Correct me if I'm
>> wrong on that, which may explain the problem now that my code is no longer
>> looking at an opendap server.
>>
>> I'm open to alternate solutions to checking the presence of a file if
>> these NCL functions are unable to check.
>>
>> On Thu, Apr 5, 2018 at 8:02 PM, Rick Brownrigg <brownrig at ucar.edu> wrote:
>>
>>> Hi John,
>>>
>>> I'm seeing the same result. Its not clear to me that the site you refer
>>> to is an OpenDAP site? Do you know it to be one? I found some (possibly
>>> stale) info that suggested a DODS server would response with headers
>>> indicating it was a dods server:
>>>
>>> (https://www.opendap.org/support/faq/server/test-dods)
>>>
>>> When I run:
>>>
>>> wget -S http://nomads.ncep.noaa.gov:80
>>>
>>> I get:
>>>
>>> -2018-04-05 19:52:17-- http://nomads.ncep.noaa.gov/
>>> Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
>>> Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:80...
>>> connected.
>>> HTTP request sent, awaiting response...
>>> HTTP/1.1 200 OK
>>> Date: Fri, 06 Apr 2018 01:54:09 GMT
>>> Server: Apache
>>> X-Frame-Options: SAMEORIGIN
>>> X-Content-Type-Options: nosniff
>>> X-XSS-Protection: 1; mode=block
>>> Accept-Ranges: bytes
>>> Cache-Control: max-age=14400
>>> Expires: Fri, 06 Apr 2018 05:54:09 GMT
>>> Vary: Accept-Encoding
>>> Keep-Alive: timeout=300, max=100
>>> Connection: Keep-Alive
>>> Transfer-Encoding: chunked
>>> Content-Type: text/html; charset=UTF-8
>>> Length: unspecified [text/html]
>>> Saving to: ‘index.html’
>>>
>>> [ <=>
>>> ] 55,810 275KB/s in 0.2s
>>>
>>> 2018-04-05 19:52:18 (275 KB/s) - ‘index.html’ saved [55810]
>>>
>>> which sort of indicates its just a generic web server.
>>>
>>> I'm not an expert in this, just trying to help trouble shoot...
>>> Rick
>>>
>>>
>>> On Thu, Apr 5, 2018 at 1:45 PM, Jon Meyer <jonathan.meyer at aggiemail.usu.
>>> edu> wrote:
>>>
>>>> Good day,
>>>>
>>>> For several years I've used ncl's opendap functionality to access
>>>> current operational NWP output from NOMADS servers to produce custom
>>>> forecasting charts. (currently running most recent NCLv6.4.0. I've
>>>> discovered that the opendap server lags a bit behind in hosting the most
>>>> recent data when compared to grib2 data hosted on a different NOMADS server
>>>> and hence decided to switch to the faster available data.
>>>>
>>>> When modifying my ncl code to search the new server for grib2 data, I
>>>> found the isfilepresent function is unable to successfully assess if these
>>>> grib2 files are present. I've tried the fileexists function as well with
>>>> similar results.
>>>>
>>>> This is odd because I can hardcode wget code to successfully download
>>>> from the same URL that both isfilepresent and fileexist functions return
>>>> "False" on. I've confirmed the URLs are correct but can't get these ncl
>>>> checks to work. My code is automated and runs as a cron-job, so I need a
>>>> way to identify when the most recent files are populated on the servers
>>>> without hardcoding.
>>>>
>>>> My understanding is that grib2 is a supported file that isfilepresent
>>>> should be capable of seeing--it worked great for the opendap servers.
>>>> Perhaps there is something on the NOMADS side that prevents the file check,
>>>> but given wget works without any certificate/user checks I would assume the
>>>> issues lies on my end? I'm sure this is something minor that my tired eyes
>>>> just aren't picking up on.
>>>>
>>>> Here's a code snipit from part of the function doing this. Notice I've
>>>> hardcoded in a valid URL for testing. This same URL will work in wget and
>>>> manually pasting into browser. Here it is directly for any testing on your
>>>> end. (note for readers coming across this down the road: there is a 7 day
>>>> buffer after which the link should become void as new data replaces it.)
>>>>
>>>> http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.2
>>>> 0180405/nam.t00z.awphys00.tm00.grib2
>>>>
>>>> Any help is much appreciated!
>>>>
>>>> SNIP==============================================================
>>>>
>>>> ;----CHECK IF MOST RECENT TIME HAS OUTPUT AVAILABLE
>>>> ;----IF NOT, WORK BACKWARDS TO IDENTIFY MOST RECENT AVAILABLE DATE
>>>> MostRecent_Julian = toint(cd_inv_calendar(toint(yy
>>>> yy_current),mm_ind,toint(dd_current),toint(MostRecentInterva
>>>> l),0,0,units,0))
>>>> MostRecent_Julian at units = units
>>>>
>>>> ct = 0
>>>> do t = MostRecent_Julian, 0, 6
>>>> CheckTime = MostRecent_Julian - ct
>>>> CheckTime at units = units
>>>>
>>>> MostRecent_UTC = cd_calendar(CheckTime,-3)
>>>>
>>>> MostRecent_YYYYMMDD = str_get_cols(tostring(MostRecent_UTC), 0, 7)
>>>> MostRecent_file = "nam.t"+str_get_cols(tostring(MostRecent_UTC),
>>>> 8,9) + "z.awphys00.tm00.grib2"
>>>> URL = dir + MostRecent_YYYYMMDD + "/" + MostRecent_file
>>>>
>>>> ;URL = "http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.
>>>> 20180405/nam.t00z.awphys00.tm00.grib2"
>>>> if isfilepresent(URL)
>>>> print("Output is found: Most recent model is " + URL)
>>>> return(URL)
>>>> t = 0
>>>> else
>>>> print("Cant find file")
>>>> ct = ct + 6
>>>> end if
>>>> end do
>>>>
>>>> --
>>>> Jonathan D.D. Meyer, Ph.D.
>>>> Utah State University
>>>> Department of Plants, Soils & Climate
>>>> Utah Climate Center
>>>>
>>>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180406/e8b76112/attachment.html>
More information about the ncl-talk
mailing list