[ncl-talk] isfilepresent checks on grib2 data from NOMADS

[Jon Meyer]

Good day,

For several years I've used ncl's opendap functionality to access current
operational NWP output from NOMADS servers to produce custom forecasting
charts. (currently running most recent NCLv6.4.0. I've discovered that the
opendap server lags a bit behind in hosting the most recent data when
compared to grib2 data hosted on a different NOMADS server and hence
decided to switch to the faster available data.

When modifying my ncl code to search the new server for grib2 data, I found
the isfilepresent function is unable to successfully assess if these grib2
files are present. I've tried the fileexists function as well with similar
results.

This is odd because I can hardcode wget code to successfully download from
the same URL that both isfilepresent and fileexist functions return "False"
on. I've confirmed the URLs are correct but can't get these ncl checks to
work. My code is automated and runs as a cron-job, so I need a way to
identify when the most recent files are populated on the servers without
hardcoding.

My understanding is that grib2 is a supported file that isfilepresent
should be capable of seeing--it worked great for the opendap servers.
Perhaps there is something on the NOMADS side that prevents the file check,
but given wget works without any certificate/user checks I would assume the
issues lies on my end? I'm sure this is something minor that my tired eyes
just aren't picking up on.



Here's a code snipit from part of the function doing this. Notice I've
hardcoded in a valid URL for testing. This same URL will work in wget and
manually pasting into browser. Here it is directly for any testing on your
end. (note for readers coming across this down the road: there is a 7 day
buffer after which the link should become void as new data replaces it.)

http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.20180405/nam.t00z.awphys00.tm00.grib2

Any help is much appreciated!

SNIP==============================================================


;----CHECK IF MOST RECENT TIME HAS OUTPUT AVAILABLE
;----IF NOT, WORK BACKWARDS TO IDENTIFY MOST RECENT AVAILABLE DATE
  MostRecent_Julian =
toint(cd_inv_calendar(toint(yyyy_current),mm_ind,toint(dd_current),toint(MostRecentInterval),0,0,units,0))
  MostRecent_Julian at units = units

  ct = 0
  do t = MostRecent_Julian, 0, 6
     CheckTime = MostRecent_Julian - ct
     CheckTime at units = units

     MostRecent_UTC = cd_calendar(CheckTime,-3)

     MostRecent_YYYYMMDD = str_get_cols(tostring(MostRecent_UTC), 0, 7)
     MostRecent_file = "nam.t"+str_get_cols(tostring(MostRecent_UTC), 8,9)
+ "z.awphys00.tm00.grib2"
     URL = dir + MostRecent_YYYYMMDD + "/" + MostRecent_file

;URL = "
http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.20180405/nam.t00z.awphys00.tm00.grib2
"
     if isfilepresent(URL)
        print("Output is found: Most recent model is " + URL)
        return(URL)
        t = 0
     else
        print("Cant find file")
        ct = ct + 6
     end if
  end do


-- 
Jonathan D.D. Meyer, Ph.D.
Utah State University
Department of Plants, Soils & Climate
Utah Climate Center
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180405/06a47670/attachment.html>