<div dir="ltr"><div>Hi Jon, Dave,<br><br></div><div>Thanks Dave for providing a clean explanation of the file_path rules. I'm filing a ticket to tighten up the documentation, based upon your suggestions.<br><br></div><div>Jon, as Dave suggests, the system/systemfunc functions in NCL can be used in conjunction with wget/curl as a way to get what you need. I had devised a somewhat elaborate way of emulating isfilepresent using wget's "--spider" option (which probes for existence), but then realized that if you intend to download the file *if* it exists, why not just outright attempt the download and use simpler local means (like isfilepresent) of determining success/failure.<br><br></div><div>RB<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 6, 2018 at 1:35 PM, Dave Allured - NOAA Affiliate <span dir="ltr"><<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Jon and Rick,<div><br></div><div>There is a documentation problem with isfilepresent, addfile, and addfiles. The rules for the file_path argument are unclear in the function descriptions. File_path must follow the rules for NCL "supported formats", which allow only two cases. These are either (1) an absolute or relative path to an actual file on the local file system, the so-called Unix path; or (2) an OpenDAP URL. General URL's to remote files on the internet, such as FTP and HTTP paths, are not supported by these functions, except when they are legal OpenDAP URL's properly hosted by remote OpenDAP servers. I would appreciate it if the documentation could be clarified up front about this.</div><div><br></div><div>For example, here is the same argument description from the nc_open function in the underlying netcdf library. Ref. Unidata Netcdf website:</div><br> path: File name for netCDF dataset to be opened.<div> When DAP support is enabled, then the path may be</div><div> an OPeNDAP URL rather than a file path.<div><br></div><div>As Rick pointed out, it looks like you have one of these generic remote file URL's. I suggest that you first contact the data provider to see if by any chance there is actual OpenDAP hosting on the newer NOMADS server, for the files you need. If available, this would be the most convenient solution.<br></div><div><br></div><div>If that is not available, then I suggest something like curl or wget inside NCL's systemfunc function, to download temporary files and test for availability under NCL script control. After downloading, you can now use addfile to read the local temporary file if it is an NCL "supported" format such as grib2.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>--Dave</div></font></span><div><div class="h5"><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 6, 2018 at 4:30 AM, Jon Meyer <span dir="ltr"><<a href="mailto:jonathan.meyer@aggiemail.usu.edu" target="_blank">jonathan.meyer@aggiemail.usu.<wbr>edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Hi Rick, thanks for the help on this.<br><br></div>This server is not an OpenDap DODS server, but from my understanding, the isfilepresent function should successfully check any file type accepted by addfiles, which grib2 should be in my version of NCL. Correct me if I'm wrong on that, which may explain the problem now that my code is no longer looking at an opendap server.<br><br></div>I'm open to alternate solutions to checking the presence of a file if these NCL functions are unable to check.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 5, 2018 at 8:02 PM, Rick Brownrigg <span dir="ltr"><<a href="mailto:brownrig@ucar.edu" target="_blank">brownrig@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Hi John,<br><br></div>I'm seeing the same result. Its not clear to me that the site you refer to is an OpenDAP site? Do you know it to be one? I found some (possibly stale) info that suggested a DODS server would response with headers indicating it was a dods server:<br><br>(<a href="https://www.opendap.org/support/faq/server/test-dods" target="_blank">https://www.opendap.org/suppo<wbr>rt/faq/server/test-dods</a>)<br><br></div>When I run:<br><br> wget -S <a href="http://nomads.ncep.noaa.gov:80" target="_blank">http://nomads.ncep.noaa.gov:80</a><br><br></div>I get:<br><br>-2018-04-05 19:52:17-- <a href="http://nomads.ncep.noaa.gov/" target="_blank">http://nomads.ncep.noaa.gov/</a><br>Resolving <a href="http://nomads.ncep.noaa.gov" target="_blank">nomads.ncep.noaa.gov</a> (<a href="http://nomads.ncep.noaa.gov" target="_blank">nomads.ncep.noaa.gov</a>)... 140.172.138.18<br>Connecting to <a href="http://nomads.ncep.noaa.gov" target="_blank">nomads.ncep.noaa.gov</a> (<a href="http://nomads.ncep.noaa.gov" target="_blank">nomads.ncep.noaa.gov</a>)|140.172<wbr>.138.18|:80... connected.<br>HTTP request sent, awaiting response...<br> HTTP/1.1 200 OK<br> Date: Fri, 06 Apr 2018 01:54:09 GMT<br> Server: Apache<br> X-Frame-Options: SAMEORIGIN<br> X-Content-Type-Options: nosniff<br> X-XSS-Protection: 1; mode=block<br> Accept-Ranges: bytes<br> Cache-Control: max-age=14400<br> Expires: Fri, 06 Apr 2018 05:54:09 GMT<br> Vary: Accept-Encoding<br> Keep-Alive: timeout=300, max=100<br> Connection: Keep-Alive<br> Transfer-Encoding: chunked<br> Content-Type: text/html; charset=UTF-8<br>Length: unspecified [text/html]<br>Saving to: ‘index.html’<br><br> [ <=> <wbr> <wbr> ] 55,810 275KB/s in 0.2s<br><br>2018-04-05 19:52:18 (275 KB/s) - ‘index.html’ saved [55810]<br><br></div>which sort of indicates its just a generic web server. <br><br></div>I'm not an expert in this, just trying to help trouble shoot...<br></div>Rick<br><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_5699023970737804865gmail-m_1107088727905830580h5">On Thu, Apr 5, 2018 at 1:45 PM, Jon Meyer <span dir="ltr"><<a href="mailto:jonathan.meyer@aggiemail.usu.edu" target="_blank">jonathan.meyer@aggiemail.usu.<wbr>edu</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div class="m_5699023970737804865gmail-m_1107088727905830580h5"><div dir="ltr"><div><div><div><div><div><div>Good day,<br><br></div>For several years I've used ncl's opendap functionality to access current operational NWP output from NOMADS servers to produce custom forecasting charts. (currently running most recent NCLv6.4.0. I've discovered that the opendap server lags a bit behind in hosting the most recent data when compared to grib2 data hosted on a different NOMADS server and hence decided to switch to the faster available data.<br><br></div>When modifying my ncl code to search the new server for grib2 data, I found the isfilepresent function is unable to successfully assess if these grib2 files are present. I've tried the fileexists function as well with similar results.<br><br></div>This is odd because I can hardcode wget code to successfully download from the same URL that both isfilepresent and fileexist functions return "False" on. I've confirmed the URLs are correct but can't get these ncl checks to work. My code is automated and runs as a cron-job, so I need a way to identify when the most recent files are populated on the servers without hardcoding.<br><br></div>My understanding is that grib2 is a supported file that isfilepresent should be capable of seeing--it worked great for the opendap servers. Perhaps there is something on the NOMADS side that prevents the file check, but given wget works without any certificate/user checks I would assume the issues lies on my end? I'm sure this is something minor that my tired eyes just aren't picking up on.<br><br></div>Here's a code snipit from part of the function doing this. Notice I've hardcoded in a valid URL for testing. This same URL will work in wget and manually pasting into browser. Here it is directly for any testing on your end. (note for readers coming across this down the road: there is a 7 day buffer after which the link should become void as new data replaces it.)<br><br>
<a href="http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.20180405/nam.t00z.awphys00.tm00.grib2" target="_blank">http://nomads.ncep.noaa.gov/pu<wbr>b/data/nccf/com/nam/prod/nam.2<wbr>0180405/nam.t00z.awphys00.tm00<wbr>.grib2</a>
<br><br></div>Any help is much appreciated!<br><div><br></div><div>SNIP==========================<wbr>==============================<wbr>======<br></div><div><br>;----CHECK IF MOST RECENT TIME HAS OUTPUT AVAILABLE<br>;----IF NOT, WORK BACKWARDS TO IDENTIFY MOST RECENT AVAILABLE DATE<br> MostRecent_Julian = toint(cd_inv_calendar(toint(yy<wbr>yy_current),mm_ind,toint(dd_cu<wbr>rrent),toint(MostRecentInterva<wbr>l),0,0,units,0))<br> MostRecent_Julian@units = units<br><br> ct = 0<br> do t = MostRecent_Julian, 0, 6<br> CheckTime = MostRecent_Julian - ct<br> CheckTime@units = units<br><br> MostRecent_UTC = cd_calendar(CheckTime,-3)<br><br> MostRecent_YYYYMMDD = str_get_cols(tostring(MostRece<wbr>nt_UTC), 0, 7)<br> MostRecent_file = "nam.t"+str_get_cols(tostring(<wbr>MostRecent_UTC), 8,9) + "z.awphys00.tm00.grib2"<br> URL = dir + MostRecent_YYYYMMDD + "/" + MostRecent_file<br><br>;URL = "<a href="http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.20180405/nam.t00z.awphys00.tm00.grib2" target="_blank">http://nomads.ncep.noaa.gov/p<wbr>ub/data/nccf/com/nam/prod/nam.<wbr>20180405/nam.t00z.awphys00.tm0<wbr>0.grib2</a>"<br> if isfilepresent(URL)<br> print("Output is found: Most recent model is " + URL)<br> return(URL)<br> t = 0<br> else<br> print("Cant find file")<br> ct = ct + 6<br> end if<br> end do<span class="m_5699023970737804865gmail-m_1107088727905830580m_1035589362528709602HOEnZb"><font color="#888888"><br><br><div><div><div><div><div><div>-- <br><div class="m_5699023970737804865gmail-m_1107088727905830580m_1035589362528709602m_-1118339514774875760gmail_signature"><div dir="ltr"><div><div dir="ltr">Jonathan D.D. Meyer, Ph.D.<div>Utah State University<br><div>Department of Plants, Soils & Climate</div><div>Utah Climate Center</div></div></div></div></div></div></div></div></div></div></div></div></font></span></div></div></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></div></div></div></div>
<br>______________________________<wbr>_________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>