<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Courier New";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-AU link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Hi Jennifer,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Firstly, a big "Thank you!"; I have been enjoying all of the information flow that your recent email messages have generated, because like you, I am also an end-user of the CMIP5 data, and like you, have also been wrangling with the incessantly slow web-page interface ... click to select another dataset parameter and wait an interminably long time for the list to update so you can select the next parameter and do some more waiting, ... and so I completely sympathise with your frustrations.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>However, I wanted to pick up on one point that you made about losing the "version" information. Now certainly, when you run the "wget" script you end up with (what I call) a long descriptive file name, e.g., "pr_Amon_HadCM3_historical_r1i1p1_195912-198411.nc", but what I do is this:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>The "wget" scripts contain one line of information for each of the files that is being downloaded, and those lines contain at least two things (some contain other things like MD5 checksums, etc.), but there is the long descriptive file name, but also there is the (usually) "thredds" path to the file that is to be downloaded by the "wget" command. I used that "thredds" server path to produce a directory path on our (NCI) computer and move the downloaded file into an (almost) identical path on our server so that, for example, the file mentioned above is stored at (something like):<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> /path/to/downloaded/cmip5data/cmip-dn.badc.rl.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/HadCM3/historical/mon/atmos/Amon/r1i1p1/v20110823/pr/pr_Amon_HadCM3_historical_r1i1p1_195912-198411.nc<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>We originally did this so that the ESG-NCI folk in Canberra could have some trace-ability as to what data was already downloaded, and from where that data had originated, so if/when an automated replication system eventuated they may be able to target their replication efforts.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>The next thing I do is generate a symbolic link to the newly added file, and the symbolic links use a simpler hierarchical directory path that is composed of the components found in the long descriptive file names, e.g.:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> /path/to/cmip5links/</span> <span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>historical/Amon/pr/HadCM3/r1i1p1/pr_Amon_HadCM3_historical_r1i1p1_195912-198411.nc<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> i.e., {Expt}/{Mrf}/{Var}/{Mdl}/{Rip}/{File}.nc<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> where:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> {Expt} ---> CMIP5 Experiment Identifier<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> e.g., 'historical', 'rcp45', 'rcp85', ...<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> {Mrf} ----> CMIP5 Modelling Realm and Data Frequency (abbreviation)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> e.g., '6hrLev', '6hrPlev', 'Amon', 'OImon', 'day', ...<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> {Var} ----> CMIP5 Short Variable Identifier<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> e.g., 'hur', 'psl', 'ta', 'ua', 'va', 'zg', ...<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> {Mdl} ----> CMIP5 Model Identifier<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> e.g., 'CNRM-CM5', 'CanESM2', 'HadGEM2-ES',<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> 'IPSL-CM5A-LR', 'bcc-csm1-1', 'inmcm4', ...<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> {Rip} ----> CMIP5 Run/Initialisation/Physics Ensemble Identifier,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> e.g., 'r1i1p1', 'r2i1p1', 'r3i1p1', ...<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> {File} ---> CMIP5 Descriptive Fila Name<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> e.g., 'hur_Amon_inmcm4_rcp45_r1i1p1_200601-201512.nc'<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> i.e., {Var}_{Mrf}_{Mdl}_{Expt}_{Rip}[_{date-Range}].nc<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>So in the end we do have the "version" number and other information that was delivered in the "thredds" server path, if needed, but we also have a much simpler path to the data files, which I find can simplify the handling of those files in the scripts that I, and other people, write to process the thousands of files we need to handle for our climate-related analyses.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>One other thing I have done here is to write a script that splits a multiple-file "wget-download.sh" script into separate single-file "wget" scripts, and then we launch these as batch jobs so many of these can be downloading in parallel. Australia is a rather remote country, compared to where many others in the CMIP5 data user community dwell (i.e., in or closer to the USA, or UK, or Europe, probably with very high-speed access to the CMIP5 data nodes), so I have been driven to try all sorts of things to help speed-up our data downloads. So far, over the last three months, I have managed to download 12 TB (terabyte) of CMIP5 data, and that is about 10% of what we want (initially).<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>I was _<i>very</i>_ pleased to read about the P2P functionality that is being developed, and like you, I will find this _<i>extremely</i>_ useful to be able to define a set of dataset describing parameters and construct the "wget" search request so I can simply obtain the associated "wget" script (instead of the current point, click, wait, wait, wait web-page affair that we have currently).<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Anyway, I have raved on for long enough :-).<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Again, many thanks for your messages describing your trials and tribulations in using the current tools. I thought I was the only one who was going (gone) grey, and tearing my hair out!<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>Lawson Hanson<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'>-------------<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> CAWCR (Centre for Australian Weather and Climate Research),<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> Climate Variability and Change group,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> Climate Change Science team,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> Bureau of Meteorology, 700 Collins Street,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'> Melbourne Docklands, VIC 3008<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> go-essp-tech-bounces@ucar.edu [mailto:go-essp-tech-bounces@ucar.edu] <b>On Behalf Of </b>Jennifer Adams<br><b>Sent:</b> Thursday, 15 December 2011 4:39 AM<br><b>To:</b> go-essp-tech@ucar.edu<br><b>Subject:</b> Re: [Go-essp-tech] Status of Gateway 2.0 (another use case)<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Well, after working from the client side to get CMIP3 and CMIP5 data, I can say that wget is a fine tool to rely on at the core of the workflow. Unfortunately, the step up in complexity from CMIP3 to CMIP5 and the switch from FTP to HTTP trashed the elegant use of wget. No amount of customized wrapper software, browser interfaces, or pre-packaged tools like DML fixes that problem. <o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>At the moment, the burden on the user is embarrassingly high. It's so easy to suggest that the user should "filter to remove what is not required" from a downloaded script, but the actual pratice of doing that in a timely and automated and distributed way is NOT simple! And if the solution to my problem of filling in the gaps in my incomplete collection is to go back to clicking in my browser and do the whole thing over again but make my filters smarter by looking for what's already been acquired or what has a new version number … this is unacceptable. The filtering must be a server-side responsibility and the interface must be accessible by automated scripts. Make it so! <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>By the way, the version number is a piece of metadata that is not in the downloaded files or the gateway's search criteria. It appears in the wget script as part of the path in the file's http location, but the path is not preserved after the wget is complete, so it is effectively lost after the download is done. I guess the file's date stamp would be the only way to know if the version number of the data file in question has been changed, but I'm not going to write that check into my filtering scripts. <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>--Jennifer<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><div><div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black'>--<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black'>Jennifer M. Adams<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black'>IGES/COLA<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black'>4041 Powder Mill Road, Suite 302<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black'>Calverton, MD 20705<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black'><a href="mailto:jma@cola.iges.org">jma@cola.iges.org</a><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black'><o:p> </o:p></span></p></div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black'><br><br></span><o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p></div></div></div></div></div></body></html>