[ncl-talk] OPeNDAP retrieval of MODIS as .nc?

Tue Jul 9 16:01:51 MDT 2019

Hi Rick,

thanks for the ncl_convert2nc tip, will use it once I can get this OPeNDAP running properly.  For what it's worth, I had an older ncl script that worked fine with the following address for MOD10A1 v005:

http://nsidc.org:80/opendap/MOST/MOD10A1.005/

But that no longer works.

I had created my account at https://search.earthdata.nasa.gov<https://search.earthdata.nasa.gov/search>, if that helps.

Here is the full error message:

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ html><html lang="en" class="no-js"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> <title>Earthdata Login</title> <meta name="description" content="Earthdata Login"> <meta name="viewport" content="width=device-width, initial-scale=1.0">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push( {'gtm.start': new Date().getTime(),event:'gtm.js'} );var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-WNP7MLF');</script>  <link href="https://cdn.earthdata.nasa.gov/eui/1.1.3/stylesheets/application.css" rel="stylesheet" /> <link rel="stylesheet" media="all" href="/assets/application-e680a8e911075100e3c341cab7758e58.css" />  <link href="//netdna.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet"> <link href='https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,700' rel='stylesheet' type='text/css'> <meta name="csrf-param" content="authenticity_token" /><meta name="csrf-token" content="zPuPbAObiXdfaAgACDGo/5EsqWuhywneIjFSKk/fgiLEbaUl8ii5uZw6wqIhkS1jcRkBWYvUyeSZM/QZvwQ0Eg==" />   </head> <body class="oauth authorize" data-turbolinks-eval=false>  <noscript> <iframe src="https://www.googletagmanager.com/ns.html?id=GTM-WNP7MLF" height="0" width="0" style="display:none;visibility:hidden"></iframe> </noscript>  <header id="earthdata-tophat2" style="height: 32px;"></header>  <div class="container"> <header role="banner"> <div id="masthead-logo"> <h1><a class="ir" href="/">Earthdata Login</a></h1> <span class="eui-badge badge daac">Earthdata Login</span> </div> <a id="hamburger" href="#"><img title="Mobile Menu" alt="Three horizontal lines stacked" src="/assets/hamburger-20170f6d343cbafd81054e4bb47e5bf9.png" /></a> <nav role="navigation"> <div id="hide"> <ul> </ul> </div> </nav></header>  <section id="callout-login"> <div class="client-login">  <br> <h3 class="client-description"> NSIDC n5eil02u https services </h3> </div> <form id="login" action="/login" accept-charset="UTF-8" method="post"><input name="utf8" type="hidden" value="&#x2713;" /><input type="hidden" name="authenticity_token" value="QvFrxRaoxWajZWvCoKwJBw99RM/bN+YqGEEzNx47/fJKZ0GM5xv1qGA3oWCJDIyb70js/fEoJhCjQ5UE7uBLwg==" /> <p><label for="username">Username</label><i class="fa fa-question-circle fa-question-circle--blue fa-question-circle--blue user-name" title="Login using either your Username or Email Address"></i><input type="text" name="username" id="username" autofocus="autofocus" class="default" /></p> <p><label for="password">Password</label><br /><input type="password" name="password" id="password" /></p> <p><input type="hidden" name="client_id" id="client_id" value="PGVMJ5nUzSnQkI5o23gMxA" /></p> <p><input type="hidden" name="redirect_uri" id="redirect_uri" value="https://n5eil02u.ecs.nsidc.org/OPS/redirect" /></p> <p><input type="hidden" name="response_type" id="response_type" value="code" /></p> <p><input type="hidden" name="state" id="state" value="aHR0cHM6Ly9uNWVpbDAydS5lY3MubnNpZGMub3JnL29wZW5kYXAvTU9TVC9NT0QxMEExLjAwNi8yMDE5LjA2LjMwL01PRDEwQTEuQTIwMTkxODEuaDEydjA0LjAwNi4yMDE5MTgzMDI0MzA5LmhkZi5kZHM" /></p> <p><input type="checkbox" name="stay_in" id="stay_in" value="1" checked="checked" /> <label for="stay_in">Stay signed in (this is a private workstation)</label></p> <p class="button-with-notes"> <input type="submit" name="commit" value="Log in" class="eui-btn--round eui-btn--green" /> <a class="eui-btn--round eui-btn--blue" href="/users/new?client_id=PGVMJ5nUzSnQkI5o23gMxA&redirect_uri=https%3A%2F%2Fn5eil02u.ecs.nsidc.org%2FOPS%2Fredirect&response_type=code&state=aHR0cHM6Ly9uNWVpbDAydS5lY3MubnNpZGMub3JnL29wZW5kYXAvTU9TVC9NT0QxMEExLjAwNi8yMDE5LjA2LjMwL01PRDEwQTEuQTIwMTkxODEuaDEydjA0LjAwNi4yMDE5MTgzMDI0MzA5LmhkZi5kZHM">Register</a> </p> <p class="form-instructions"> <em class="icon-question-sign"></em> <a class="" href="/retrieve_info">I don’t remember my username</a> <br /><em class="icon-question-sign"></em> <a class="" href="/reset_passwords/new">I don’t remember my password</a> <br /> <em class="icon-question-sign"></em> <a href="javascript:feedback.showForm();" title = 'Need Help? Click on the Feedback button to request help'>Help</a> </p></form><aside class="govt-msg"> <h3></h3> <p><strong>Why must I register?</strong></p> <p> The Earthdata Login provides a single mechanism for user registration and profile management for all EOSDIS system components (DAACs, Tools, Services). Your Earthdata login also helps the EOSDIS program better understand the usage of EOSDIS services to improve user experience through customization of tools and improvement of services. EOSDIS data are openly available to all and free of charge except where governed by international agreements. </p></aside></section><section id="cta"> <h3>Get single sign-on access to all your favorite EOSDIS sites</h3> <a class="eui-btn--round eui-btn--blue" href="/users/new?client_id=PGVMJ5nUzSnQkI5o23gMxA&redirect_uri=https%3A%2F%2Fn5eil02u.ecs.nsidc.org%2FOPS%2Fredirect&response_type=code&state=aHR0cHM6Ly9uNWVpbDAydS5lY3MubnNpZGMub3JnL29wZW5kYXAvTU9TVC9NT0QxMEExLjAwNi8yMDE5LjA2LjMwL01PRDEwQTEuQTIwMTkxODEuaDEydjA0LjAwNi4yMDE5MTgzMDI0MzA5LmhkZi5kZHM">Register for a Profile</a></section><div class="govt-warning eui-info-box"> <div class="warning-desktop"> <p><strong>By accessing and using this information system, you acknowledge and consent to the following:</strong></p> You are accessing a U.S. Government information system, which includes: (1) this computer; (2) this computer network; (3) all computers connected to this network including end user systems; (4) all devices and storage media attached to this network or to any computer on this network; and (5) cloud and remote information services. This information system is provided for U.S. Government-authorized use only. You have no reasonable expectation of privacy regarding any communication transmitted through or data stored on this information system. At any time, and for any lawful purpose, the U.S. Government may monitor, intercept, search, and seize any communication or data transiting, stored on, or traveling to or from this information system. You are NOT authorized to process classified information on this information system. Unauthorized or improper use of this system may result in suspension or loss of access privileges, disciplinary action, and civil and/or criminal penalties. </div> <div class="warning-mobile"> <p><strong>By accessing and using this information system, you acknowledge and consent to the following:</strong></p> You are accessing a U.S. Government information system, which includes: (1) this computer; (2) this computer network; (3) all computers connected to this network including end user systems; (4) all devices and storage media attached to this network or to any computer on this network; and (5) cloud and remote information services. This information system is provided for U.S. Government-authorized use only. Unauthorized or improper use of this system may result in suspension or loss of access privileges, disciplinary action, and civil and/or criminal penalties. By using this information system, you acknowledge and consent to the terms and conditions established in NASA policy and regulatory guidance for NASA IT Systems. </div> <div class="warning-mobile-mini"> <strong> US Govt Property. Unauthorized use subject to prosecution. Use subject to monitoring per <a href="https://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPD&c=2810&s=1E">NPD2810</a>. </strong> </div></div> </div> <footer role="contentinfo"> <h3>For questions regarding the EOSDIS Earthdata Login, please contact <a href="javascript:feedback.showForm();" title="Earthdata Support form">Earthdata Support</a></h3> <ul> <li class="version badge eui-badge--md">V 4.98</li> <li><a href="/">Home</a></li> <li><a href="/users/new">Register</a></li> <li><a href="/documentation">Documentation</a></li> <li><a title="NASA Home" href="http://www.nasa.gov">NASA</a></li> </ul> <p>NASA Official: Stephen Berrick</p></footer> <script src="/assets/application-40ec588bf18a3e2af282eb511243edf3.js"></script> <script type="text/javascript" charset="utf-8"> $(window).scroll(function(e){ parallax(); }); function parallax(){ var scrolled = $(window).scrollTop(); $('#content').css('background-position', 'right ' + -(scrolled*0.25)+'px '); } </script> <script src="https://cdn.earthdata.nasa.gov/tophat2/tophat2.js" id="earthdata-tophat-script" data-show-fbm="true" data-show-status="true" data-status-api-url="https://status.earthdata.nasa.gov/api/v1/notifications"></script> <script type="text/javascript" src="https://fbm.earthdata.nasa.gov/for/URS4/feedback.js"></script> <script type="text/javascript"> feedback.init(); </script> <script type="text/javascript"> setTimeout(function() {var a=document.createElement("script"); var b=document.getElementsByTagName("script")[0]; a.src=document.location.protocol+"//dnn506yrbagrg.cloudfront.net/pages/scripts/0013/2090.js?"+Math.floor(new Date().getTime()/3600000); a.async=true;a.type="text/javascript";b.parentNode.insertBefore(a,b)} , 1); </script>   <script type="text/javascript" language="JavaScript" src="/ntpagetag.js"></script> <noscript> <img src="https://ws1.ems.eosdis.nasa.gov/images/ntpagetag.gif?js=0" height="1" width="1" border="0" hspace="0" vspace="0" alt=""> </noscript>   <script language="javascript" id="_fed_an_ua_tag" src="https://dap.digitalgov.gov/Universal-Federated-Analytics-Min.js?agency=NASA&subagency=GSFC&dclink=true"></script>   </body></html><script id="f5_cspm">(function(){var f5_cspm={f5_p:'NJAFPDAOGPCPEEHPFAPAHCKHMLNHNKCGKKBLPPHEKAEOMCOEOEPLPIGMKDFMAPDGOOLLBMJAPLOBFEGMBDGJFABCPPDANMFLAAHEOOIKNFIKCOLDAAHNPLJKPDJAOEHH',setCharAt:function(str,index,chr){if(index>str.length-1)return str;return str.substr(0,index)+chr+str.substr(index+1);},get_byte:function(str,i){var s=(i/16)|0;i=(i&15);s=s*32;return((str.charCodeAt(i+16+s)-65)<<4)|(str.charCodeAt(i+s)-65);},set_byte:function(str,i,b){var s=(i/16)|0;i=(i&15);s=s*32;str=f5_cspm.setCharAt(str,(i+16+s),String.fromCharCode((b>>4)+65));str=f5_cspm.setCharAt(str,(i+s),String.fromCharCode((b&15)+65));return str;},set_latency:function(str,latency){latency=latency&0xffff;str=f5_cspm.set_byte(str,48,(latency>>8));str=f5_cspm.set_byte(str,49,(latency&0xff));str=f5_cspm.set_byte(str,43,2);return str;},wait_perf_data:function(){try{var wp=window.performance.timing;if(wp.loadEventEnd>0){var res=wp.loadEventEnd-wp.navigationStart;if(res<60001){var cookie_val=f5_cspm.set_latency(f5_cspm.f5_p,res);window.document.cookie='f5avr1871570098aaaaaaaaaaaaaaaa='+encodeURIComponent(cookie_val)+';path=/';}return;}}catch(err){return;}setTimeout(f5_cspm.wait_perf_data,100);return;},go:function(){var chunk=window.document.cookie.split(/\s*;\s*/);for(var i=0;i<chunk.length;++i){var pair=chunk[i].split(/\s*=\s*/);if(pair[0]=='f5_cspm'&&pair[1]=='1234'){var d=new Date();d.setTime(d.getTime()-1000);window.document.cookie='f5_cspm=;expires='+d.toUTCString()+';path=/;';setTimeout(f5_cspm.wait_perf_data,100);}}}}f5_cspm.go();}());</script>
ncopen: filename "https://n5eil02u.ecs.nsidc.org/opendap/MOST/MOD10A1.006/2019.06.30/MOD10A1.A2019181.h12v04.006.2019183024309.hdf": NetCDF: Malformed or inaccessible DAP DDS

Dr. Elizabeth A Burakowski
She/Her/Hers
Research Assistant Professor
Earth Systems Research Center &
Institute for the Study of Earth, Oceans, and Space
University of New Hampshire
8 College Road Durham, NH 03824
603-862-1796
elizabeth.burakowski at unh.edu
________________________________
From: Rick Brownrigg <brownrig at ucar.edu>
Sent: Tuesday, July 9, 2019 5:53 PM
To: Burakowski, Elizabeth
Cc: ncl-talk at ucar.edu
Subject: Re: [ncl-talk] OPeNDAP retrieval of MODIS as .nc?

Caution - External Email
________________________________
Hi Elizabeth,

I unfortunately can not test this directly, as one must have an account on that OpenDAP site. However, if you can download the HDF version, you could use "ncl_convert2nc" to NetCDF.

Otherwise, seeing all the error messages might yield some insights.

Hope that helps...
Rick

On Tue, Jul 9, 2019 at 3:15 PM Burakowski, Elizabeth via ncl-talk <ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>> wrote:
Hello,

I am writing an NCL script to pull a subset of MOD10A1 Daily Snow Cover files for analysis. I'd like to pull the files as netCDF instead of hdf.

The UI page seems to suggest I can request as .nc:

https://n5eil02u.ecs.nsidc.org/opendap/MOST/MOD10A1.006/2000.03.10/MOD10A1.A2000070.h12v04.006.2016061222432.hdf.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__n5eil02u.ecs.nsidc.org_opendap_MOST_MOD10A1.006_2000.03.10_MOD10A1.A2000070.h12v04.006.2016061222432.hdf.html&d=DwMFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=JW2rHHzYnVYi6ZDBI-2JFMNSreDPLZO-wwOrczEnOlg&m=Mro58-h7ajrO1nUBMp5gNIPzvaLfG99-mvbM6anW2mE&s=Ahorg8HVDvw9p0wAfDWVHds6fBChcRfwQWHIqO_1rV4&e=>

But it's not clear how I would do this in NCL.  For hdf, I tried to use:

filename = "MOD10A1.A2019181.h12v04.006.2019183024309.hdf"
subdir = "2019.06.30"
url = "https://n5eil02u.ecs.nsidc.org/opendap/MOST/MOD10A1.006/<https://urldefense.proofpoint.com/v2/url?u=https-3A__n5eil02u.ecs.nsidc.org_opendap_MOST_MOD10A1.006_&d=DwMFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=JW2rHHzYnVYi6ZDBI-2JFMNSreDPLZO-wwOrczEnOlg&m=Mro58-h7ajrO1nUBMp5gNIPzvaLfG99-mvbM6anW2mE&s=RVUJnW6bUsZvHlJtDUaEZ-yfLZkVtiW2gfQGLbVui60&e=>"
exists = isfilepresent(url+sudir+"/"+filename)

However, I receive an error that starts with:

"syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR"

and ends with:

"ncopen: filename "https://n5eil02u.ecs.nsidc.org/opendap/MOST/MOD10A1.006/2019.06.30/MOD10A1.A2019181.h12v04.006.2019183024309.hdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__n5eil02u.ecs.nsidc.org_opendap_MOST_MOD10A1.006_2019.06.30_MOD10A1.A2019181.h12v04.006.2019183024309.hdf&d=DwMFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=JW2rHHzYnVYi6ZDBI-2JFMNSreDPLZO-wwOrczEnOlg&m=Mro58-h7ajrO1nUBMp5gNIPzvaLfG99-mvbM6anW2mE&s=HRC_y02lVRpB5bHAZ1AY5QzBOIg41Hn3bHQx11jaOGs&e=>": NetCDF: Malformed or inaccessible DAP DDS"

I suspect there is something wrong with the url, but this is the one provided by nsidc.

Any suggestions?

Thank you,

Liz.

Dr. Elizabeth A Burakowski
She/Her/Hers
Research Assistant Professor
Earth Systems Research Center &
Institute for the Study of Earth, Oceans, and Space
University of New Hampshire
8 College Road Durham, NH 03824
603-862-1796
elizabeth.burakowski at unh.edu<mailto:elizabeth.burakowski at unh.edu>
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk<https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ucar.edu_mailman_listinfo_ncl-2Dtalk&d=DwMFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=JW2rHHzYnVYi6ZDBI-2JFMNSreDPLZO-wwOrczEnOlg&m=Mro58-h7ajrO1nUBMp5gNIPzvaLfG99-mvbM6anW2mE&s=f-i07zLS-wjnh7A8FCEzQIqwg-eH-OlUoGIqnj1Fv7c&e=>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190709/cbe5bf4b/attachment.html>