[ncl-talk] isfilepresent checks on grib2 data from NOMADS

Fri Apr 6 04:30:10 MDT 2018

Hi Rick, thanks for the help on this.

This server is not an OpenDap DODS server, but from my understanding, the
isfilepresent function should successfully check any file type accepted by
addfiles, which grib2 should be in my version of NCL. Correct me if I'm
wrong on that, which may explain the problem now that my code is no longer
looking at an opendap server.

I'm open to alternate solutions to checking the presence of a file if these
NCL functions are unable to check.

On Thu, Apr 5, 2018 at 8:02 PM, Rick Brownrigg <brownrig at ucar.edu> wrote:

> Hi John,
>
> I'm seeing the same result. Its not clear to me that the site you refer to
> is an OpenDAP site?  Do you know it to be one?  I found some (possibly
> stale) info that suggested a DODS server would response with headers
> indicating it was a dods server:
>
> (https://www.opendap.org/support/faq/server/test-dods)
>
> When I run:
>
>   wget -S http://nomads.ncep.noaa.gov:80
>
> I get:
>
> -2018-04-05 19:52:17--  http://nomads.ncep.noaa.gov/
> Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
> Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:80...
> connected.
> HTTP request sent, awaiting response...
>   HTTP/1.1 200 OK
>   Date: Fri, 06 Apr 2018 01:54:09 GMT
>   Server: Apache
>   X-Frame-Options: SAMEORIGIN
>   X-Content-Type-Options: nosniff
>   X-XSS-Protection: 1; mode=block
>   Accept-Ranges: bytes
>   Cache-Control: max-age=14400
>   Expires: Fri, 06 Apr 2018 05:54:09 GMT
>   Vary: Accept-Encoding
>   Keep-Alive: timeout=300, max=100
>   Connection: Keep-Alive
>   Transfer-Encoding: chunked
>   Content-Type: text/html; charset=UTF-8
> Length: unspecified [text/html]
> Saving to: ‘index.html’
>
>     [ <=>
> ] 55,810       275KB/s   in 0.2s
>
> 2018-04-05 19:52:18 (275 KB/s) - ‘index.html’ saved [55810]
>
> which sort of indicates its just a generic web server.
>
> I'm not an expert in this, just trying to help trouble shoot...
> Rick
>
>
> On Thu, Apr 5, 2018 at 1:45 PM, Jon Meyer <jonathan.meyer at aggiemail.usu.
> edu> wrote:
>
>> Good day,
>>
>> For several years I've used ncl's opendap functionality to access current
>> operational NWP output from NOMADS servers to produce custom forecasting
>> charts. (currently running most recent NCLv6.4.0. I've discovered that the
>> opendap server lags a bit behind in hosting the most recent data when
>> compared to grib2 data hosted on a different NOMADS server and hence
>> decided to switch to the faster available data.
>>
>> When modifying my ncl code to search the new server for grib2 data, I
>> found the isfilepresent function is unable to successfully assess if these
>> grib2 files are present. I've tried the fileexists function as well with
>> similar results.
>>
>> This is odd because I can hardcode wget code to successfully download
>> from the same URL that both isfilepresent and fileexist functions return
>> "False" on. I've confirmed the URLs are correct but can't get these ncl
>> checks to work. My code is automated and runs as a cron-job, so I need a
>> way to identify when the most recent files are populated on the servers
>> without hardcoding.
>>
>> My understanding is that grib2 is a supported file that isfilepresent
>> should be capable of seeing--it worked great for the opendap servers.
>> Perhaps there is something on the NOMADS side that prevents the file check,
>> but given wget works without any certificate/user checks I would assume the
>> issues lies on my end? I'm sure this is something minor that my tired eyes
>> just aren't picking up on.
>>
>>
>>
>> Here's a code snipit from part of the function doing this. Notice I've
>> hardcoded in a valid URL for testing. This same URL will work in wget and
>> manually pasting into browser. Here it is directly for any testing on your
>> end. (note for readers coming across this down the road: there is a 7 day
>> buffer after which the link should become void as new data replaces it.)
>>
>> http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.
>> 20180405/nam.t00z.awphys00.tm00.grib2
>>
>> Any help is much appreciated!
>>
>> SNIP==============================================================
>>
>>
>> ;----CHECK IF MOST RECENT TIME HAS OUTPUT AVAILABLE
>> ;----IF NOT, WORK BACKWARDS TO IDENTIFY MOST RECENT AVAILABLE DATE
>>   MostRecent_Julian = toint(cd_inv_calendar(toint(yy
>> yy_current),mm_ind,toint(dd_current),toint(MostRecentInterva
>> l),0,0,units,0))
>>   MostRecent_Julian at units = units
>>
>>   ct = 0
>>   do t = MostRecent_Julian, 0, 6
>>      CheckTime = MostRecent_Julian - ct
>>      CheckTime at units = units
>>
>>      MostRecent_UTC = cd_calendar(CheckTime,-3)
>>
>>      MostRecent_YYYYMMDD = str_get_cols(tostring(MostRecent_UTC), 0, 7)
>>      MostRecent_file = "nam.t"+str_get_cols(tostring(MostRecent_UTC),
>> 8,9) + "z.awphys00.tm00.grib2"
>>      URL = dir + MostRecent_YYYYMMDD + "/" + MostRecent_file
>>
>> ;URL = "http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.
>> 20180405/nam.t00z.awphys00.tm00.grib2"
>>      if isfilepresent(URL)
>>         print("Output is found: Most recent model is " + URL)
>>         return(URL)
>>         t = 0
>>      else
>>         print("Cant find file")
>>         ct = ct + 6
>>      end if
>>   end do
>>
>>
>> --
>> Jonathan D.D. Meyer, Ph.D.
>> Utah State University
>> Department of Plants, Soils & Climate
>> Utah Climate Center
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>

-- 
Jonathan D.D. Meyer, Ph.D.
Utah State University
Department of Plants, Soils & Climate
Utah Climate Center
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180406/7f57f483/attachment.html>