[ncl-talk] Reading ECMWF THREDDS data using OPENDAP

Tue Sep 15 18:32:44 MDT 2015

Hi David,
I have looked into this issue a little bit. I realize now that Mary is
right. Thredds can serve data through several protocols. The URL you
specified

http://rda.ucar.edu/thredds/fileServer/aggregations/e/ds627.0/15/ERA_Interim_atmospheric_model_forecast_for_surface-198808/files/ei.oper.fc.sfc.regn128sc.198808010

does not use OPeNDAP to serve the file. This URL points to GRIB data
that is served using http. It cannot be subset the way it could be if
it OPeNDAP was serving it.

NCL cannot read data served using http directly even if you get around
the username/password issue. OPeNDAP does have a way to encode
username/password into the URL, although it is not very secure. And it
may be problematic with RDA user names that are actually email
addresses because the protocol uses the '@' symbol and would probably
be confused by its occurrence in the username.  But that is not
relevant here because NCL could not read data from this URL even if it
were not restricted.

I have no idea what could be causing the sftp problem, but it does not
sound like an NCL issue.
 -dave

On Fri, Sep 11, 2015 at 4:21 PM, David Brown <dbrown at ucar.edu> wrote:
> Hi David,
> It is likely that NCL will need some updates to work with
> access-restricted sites such as this one. I am going to look into it.
>  -dave
>
>
> On Fri, Sep 11, 2015 at 3:40 PM, Vollaro, David <dvollaro at albany.edu> wrote:
>> Hi Mary,
>>
>>
>>
>> No my issue was never resolved.  I contacted Dave Stepaniak about this and
>> he speculates that it is possible, however, trying to add my RDA
>> username/passwd to the NCL Opendap call is another issue.  I can set my
>> cookies using wget to remotely access an RDA dataset but doing this in NCL
>> is proving problematic.  I have decided to run a NCL script on geyser to
>> access the data on glade ans sftp it back to me.  However, I am running into
>> another issue.  If I transfer the data using NETCDF3 it works OK.  However
>> when I use the setfileoption to netcdf4 the file fills with missing values
>> on my local system(although it is Ok on glade).  I am assuming that it is
>> some issue with sftp.
>>
>>
>>
>> ;  setfileoption("nc","Format","NetCDF3")
>>
>> ;  setfileoption("nc","Format","NetCDF4")
>>
>>
>>
>>
>>
>> I have remotely access the RDA datsets
>>
>> From: Mary Haley <haley at ucar.edu>
>> Sent: Friday, September 11, 2015 5:14 PM
>> To: Vollaro, David
>> Cc: ncl-talk at ucar.edu
>> Subject: Re: [ncl-talk] Reading ECMWF THREDDS data using OPENDAP
>>
>>
>>
>> Dave,
>>
>>
>>
>> I didn't see any response to this. Was it ever resolved?
>>
>>
>>
>> The issue is that these files are not being served by an OPeNDAP server, and
>> hence you can't open them with an OPeNDAP-enabled client.
>>
>>
>>
>> --Mary
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 4, 2015 at 3:29 PM, Vollaro, David <dvollaro at albany.edu> wrote:
>>
>> Hi,
>>
>>
>>
>> I am trying to read an ECMWF dataset from the NCAR RDA THREDDS servver using
>> NCL.
>>
>> I am using the example OPENDAP script from NCL but am having trouble
>> accessing the data.
>>
>>
>>
>> The snippet of the NCL script I am using is below, as is the link to the
>> datasset I am trying to access.
>>
>> I have sucessfully accessed data using OPENDAP before, so I am hoping that
>> it is just an incorrecct
>>
>> link-id.  I am running 6.1.2...Thanks...
>>
>>
>>
>> Dave Vollaro
>>
>>
>>
>> --------------- code snippet --------------
>>
>>   url = "http://rda.ucar.edu/thredds/fileServer/aggregations/e/ds627.0/15/"
>> filename = url +
>> "ERA_Interim_atmospheric_model_forecast_for_surface-198808/files/ei.oper.fc.sfc.regn128sc.1988080100"
>> print(filename)
>> ;***********************************************
>> ;  DATA SOURCE
>> ;***********************************************
>>
>>   exists = isfilepresent(filename)
>>   if(.not.exists) then
>>     print("OPeNDAP isfilepresent test unsuccessful.")
>>     print("File doesn't exist or NCL doesn't have OPeNDAP ability on this
>> system")
>>   else
>>     print("OPeNDAP isfilepresent test successful.")
>>     file_nc = addfile(filename,"r")
>>   end if
>>   names = getfilevarnames(file_nc)          ;Get the variable names in the
>>   print(names)                               ;GRIB file and print them out.
>> end
>>
>>
>>
>> --------- end snippet-------
>>
>>   link:
>> http://rda.ucar.edu/thredds/catalog/aggregations/e/ds627.0/15/ERA_Interim_atmospheric_model_forecast_for_surface-198808/catalog.html?dataset=aggregations/e/ds627.0/15/ERA_Interim_atmospheric_model_forecast_for_surface-198808/files/ei.oper.fc.sfc.regn128sc.1988080100
>>
>>
>>
>>
>> error:
>>
>> (0)
>> http://rda.ucar.edu/thredds/fileServer/aggregations/e/ds627.0/15/ERA_Interim_atmospheric_model_forecast_for_surface-198808/files/ei.oper.fc.sfc.regn128sc.1988080100
>> syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or
>> SCAN_ERROR
>> context: <html^><head><title>Apache Tomcat/7.0.63 - Error
>> report</title><style><!--H1
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
>> H2
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
>> H3
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
>> BODY
>> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
>> P
>> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
>> {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
>> </head><body><h1>HTTP Status 401 - </h1><HR size="1"
>> noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b>
>> <u></u></p><p><b>description</b> <u>This request requires HTTP
>> authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache
>> Tomcat/7.0.63</h3></body></html>
>> ncopen: filename
>> "http://rda.ucar.edu/thredds/fileServer/aggregations/e/ds627.0/15/ERA_Interim_atmospheric_model_forecast_for_surface-198808/files/ei.oper.fc.sfc.regn128sc.1988080100":
>> NetCDF: Malformed or inaccessible DAP DDS
>> (0)     OPeNDAP isfilepresent test unsuccessful.
>> (0)     File doesn't exist or NCL doesn't have OPeNDAP ability on this
>> system
>>
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>