[ncl-talk] isfilepresent checks on grib2 data from NOMADS

[Rick Brownrigg]

Hi John,

I'm seeing the same result. Its not clear to me that the site you refer to
is an OpenDAP site?  Do you know it to be one?  I found some (possibly
stale) info that suggested a DODS server would response with headers
indicating it was a dods server:

(https://www.opendap.org/support/faq/server/test-dods)

When I run:

  wget -S http://nomads.ncep.noaa.gov:80

I get:

-2018-04-05 19:52:17--  http://nomads.ncep.noaa.gov/
Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:80...
connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Fri, 06 Apr 2018 01:54:09 GMT
  Server: Apache
  X-Frame-Options: SAMEORIGIN
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
  Accept-Ranges: bytes
  Cache-Control: max-age=14400
  Expires: Fri, 06 Apr 2018 05:54:09 GMT
  Vary: Accept-Encoding
  Keep-Alive: timeout=300, max=100
  Connection: Keep-Alive
  Transfer-Encoding: chunked
  Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]
Saving to: ‘index.html’

    [
<=>
] 55,810       275KB/s   in 0.2s

2018-04-05 19:52:18 (275 KB/s) - ‘index.html’ saved [55810]

which sort of indicates its just a generic web server.

I'm not an expert in this, just trying to help trouble shoot...
Rick


On Thu, Apr 5, 2018 at 1:45 PM, Jon Meyer <jonathan.meyer at aggiemail.usu.edu>
wrote:

> Good day,
>
> For several years I've used ncl's opendap functionality to access current
> operational NWP output from NOMADS servers to produce custom forecasting
> charts. (currently running most recent NCLv6.4.0. I've discovered that the
> opendap server lags a bit behind in hosting the most recent data when
> compared to grib2 data hosted on a different NOMADS server and hence
> decided to switch to the faster available data.
>
> When modifying my ncl code to search the new server for grib2 data, I
> found the isfilepresent function is unable to successfully assess if these
> grib2 files are present. I've tried the fileexists function as well with
> similar results.
>
> This is odd because I can hardcode wget code to successfully download from
> the same URL that both isfilepresent and fileexist functions return "False"
> on. I've confirmed the URLs are correct but can't get these ncl checks to
> work. My code is automated and runs as a cron-job, so I need a way to
> identify when the most recent files are populated on the servers without
> hardcoding.
>
> My understanding is that grib2 is a supported file that isfilepresent
> should be capable of seeing--it worked great for the opendap servers.
> Perhaps there is something on the NOMADS side that prevents the file check,
> but given wget works without any certificate/user checks I would assume the
> issues lies on my end? I'm sure this is something minor that my tired eyes
> just aren't picking up on.
>
>
>
> Here's a code snipit from part of the function doing this. Notice I've
> hardcoded in a valid URL for testing. This same URL will work in wget and
> manually pasting into browser. Here it is directly for any testing on your
> end. (note for readers coming across this down the road: there is a 7 day
> buffer after which the link should become void as new data replaces it.)
>
> http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/
> nam.20180405/nam.t00z.awphys00.tm00.grib2
>
> Any help is much appreciated!
>
> SNIP==============================================================
>
>
> ;----CHECK IF MOST RECENT TIME HAS OUTPUT AVAILABLE
> ;----IF NOT, WORK BACKWARDS TO IDENTIFY MOST RECENT AVAILABLE DATE
>   MostRecent_Julian = toint(cd_inv_calendar(toint(
> yyyy_current),mm_ind,toint(dd_current),toint(
> MostRecentInterval),0,0,units,0))
>   MostRecent_Julian at units = units
>
>   ct = 0
>   do t = MostRecent_Julian, 0, 6
>      CheckTime = MostRecent_Julian - ct
>      CheckTime at units = units
>
>      MostRecent_UTC = cd_calendar(CheckTime,-3)
>
>      MostRecent_YYYYMMDD = str_get_cols(tostring(MostRecent_UTC), 0, 7)
>      MostRecent_file = "nam.t"+str_get_cols(tostring(MostRecent_UTC),
> 8,9) + "z.awphys00.tm00.grib2"
>      URL = dir + MostRecent_YYYYMMDD + "/" + MostRecent_file
>
> ;URL = "http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/
> nam.20180405/nam.t00z.awphys00.tm00.grib2"
>      if isfilepresent(URL)
>         print("Output is found: Most recent model is " + URL)
>         return(URL)
>         t = 0
>      else
>         print("Cant find file")
>         ct = ct + 6
>      end if
>   end do
>
>
> --
> Jonathan D.D. Meyer, Ph.D.
> Utah State University
> Department of Plants, Soils & Climate
> Utah Climate Center
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180405/9645722c/attachment.html>