<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<link href="chrome://translator/skin/floatingPanel.css"
type="text/css" rel="stylesheet">
</head>
<body bgcolor="#ffffff" text="#000000">
Dear colleagues<br>
<br>
I also devised a simple tool (actually a set of bash functions) which
do use wget and Thredds catalog
structure for providing a main versatile function "esgfiles" which doc
is reproduced below and another function "esgcheck" for monitoring new
CMIP5 data published; it is intended to fit the need of scientists in
querying and retrieving CMIP5 data in a scriptable way. It also handles
the file organizations that do not match the recommended DRS structure.
It uses non-documented aspects of Thredds catalog (and maybe
non-reliable ones) in a very crude way. Iit represents some 300 lines
of bash (including
100 lines of doc);<br>
<br>
Do you think it could be useful to scientists at that stage, given the
situation where an API is currently being developped, and taking into
account also Sébastien's recent offer ? <br>
<br>
Regards<br>
<br>
Stéphane<br>
<br>
<br>
<blockquote type="cite">esgfiles dn_pattern [ action [ base_url [
adn_pattern [ wgetargs [ nmax ]]]]]<br>
<br>
Performs ACTION for all dataset entries of the Thredds catalog hosted
at BASE_URL which do match DN_PATTERN<br>
<br>
DN_PATTERN should be a regular expression. It is matched against
dataset names (and not against file names), i.e. against strings like :
cmip5.output1.CNRM-CERFACS.CNRM-CM5.piControl.day.atmos.day.r1i1p1.v20110701.html.
<br>
Reference document for dataset names is
<a class="moz-txt-link-freetext"
href="http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf">http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf</a><br>
ACTION may be : <br>
list : Print short dataset names and number of files; this is
the default<br>
listlong : Print dataset names and number of files per dataset<br>
urls : Print atomic dataset files URLs<br>
check : in addition, checks that the atomic dataset files are
reachable (by wget --spider)<br>
get : in addition, downloads the files (with wget, which will
get argument WGETARGS as arguments)<br>
BASE_URL : either a part of a datanode name, ot the URL of a data node,
or 'all'<br>
'all' means all known datanodes (see list in code)<br>
defaults to $esg_base_url, which current value is
esg.cnrm-game-meteo.fr (tune in the code or by setting the environment
variable). <br>
ADN_PATTERN
is an optional regular expression acting as an additional
filter. It can be used to filter according to variable names, because
it is matched against atomic dataset names, i.e. against strings like :
cmip5.output1.CNRM-CERFACS.CNRM-CM5.historicalMisc.mon.landIce.LImon.r1i1p1.v20110722.sbl_LImon_CNRM-CM5_historicalMisc_r1i1p1_185001-189912.nc<br>
<br>
WGETARGS applies only in case ACTION == get, and accepts arguments to
wget. <br>
Use it to tune dowloaded files organization, e.g with '-r' -
see 'man wget')<br>
NMAX is the maximum number of data files to process. It does not apply
to actions : list and listlong. <br>
Default is : no limit<br>
<br>
For pre-requisites, type : esgdoc setup<br>
<br>
Examples : <br>
- esgfiles historicalMisc.fx list "" "" <br>
- esgfiles "amip.*6hrLev" check "" vesg.ipsl.fr 5<br>
<br>
</blockquote>
<br>
Example of esgfiles runs today (outputs are truncated)<br>
<blockquote type="cite"> > esgfiles historical\..*Amon listlong badc<br>
2/cmip5.output1.MOHC.HadGEM2-ES.historical.mon.atmos.Amon.r3i1p1.v20110418.html
-
0276 entries <br>
2/cmip5.output1.MOHC.HadGEM2-ES.historical.mon.atmos.Amon.r2i1p1.v20110418.html
-
0278 entries <br>
....<br>
2/cmip5.output1.MOHC.HadGEM2-ES.historicalGHG.mon.atmos.Amon.r1i1p1.v20110330.html
-
0276 entries </blockquote>
<br>
<blockquote type="cite">> esgfiles "historical\..*Amon.*r3" urls badc<br>
cmip-dn.badc.rl.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r3i1p1/v20110418/sci/sci_Amon_HadGEM2-ES_historical_r3i1p1_185912-188411.nc<br>
cmip-dn.badc.rl.ac.uk/thredds/fileServer/esg_dataroot/cmip5/output1/MOHC/HadGEM2-ES/historical/mon/atmos/Amon/r3i1p1/v20110418/sci/sci_Amon_HadGEM2-ES_historical_r3i1p1_188412-190911.nc
.........<br>
</blockquote>
<br>
<blockquote type="cite">> esgfiles "historical\..*Amon.*r3" urls cnrm<br>
esg.cnrm-game-meteo.fr/thredds/fileServer/esg_dataroot5/CMIP5/output/CNRM-CERFACS/CNRM-CM5/historical/mon/atmos/rsdt/r3i1p1/rsdt_Amon_CNRM-CM5_historical_r3i1p1_185001-189912.nc<br>
esg.cnrm-game-meteo.fr/thredds/fileServer/esg_dataroot5/CMIP5/output/CNRM-CERFACS/CNRM-CM5/historical/mon/atmos/rsdt/r3i1p1/rsdt_Amon_CNRM-CM5_historical_r3i1p1_190001-194912.nc</blockquote>
<br>
<br>
<br>
Sébastien Denvil wrote, On 12/09/2011 14:23:
<blockquote cite="mid:4E6DF9DA.6090405@ipsl.jussieu.fr" type="cite"> Jonathan,
<br>
<br>
we also felt the need to have a program to download files from the
CMIP5 archive in an easy way, for a list of variables and experiments.
At IPSL, we have developed a tool to help to do it. Its a first version
that will be progressively improved (in particular the "user guide").
The program will evolve together with the cmip5 archive backend
functionalities.
<br>
<br>
The user defines one or many templates. Each of them has a list of
variables, frequencies and experiments. The user also define a list of
models. Using these templates, the program explore the ESG grid and
dowload all the corresponding files that are available (and only for
the first ensemble member in the current version). The program may be
run regularly to download the possible new files. Typically each
template is associated with an analysis (cfmip template, downscaling
template and so on). Create as many TemplateName.txt as you want in the
user_selections folder (following the user_selections/default.txt
(trivial) syntax) and you are done.
<br>
<br>
Here is the procedure to install the CMIP5 data download program.
Except two dependencies (sqlite) it's a non root install:
<br>
<a class="moz-txt-link-freetext" href="http://dods.ipsl.jussieu.fr/jripsl/synchro_data/README">http://dods.ipsl.jussieu.fr/jripsl/synchro_data/README</a>
<br>
<br>
The program have the following features:
<br>
* support for myproxy-logon and myproxyclient
<br>
* simple data selection with model,experiment,realm and variable
<br>
* multi threaded downloads (8 tasks by default)
<br>
* manage datasets version following new drs
<br>
* incremental process (download only what's new)
<br>
* download history stored in a db
<br>
<br>
It has been tested with the following models: HadGEM2-ES, HadGEM2-A,
CanESM2, CNRM-CM5, NorESM1-M, CanCM4, CSIRO-Mk3-6-0. IPSL-CM5A-LR will
be added shortly ... :-)
<br>
<br>
Fill free to use it and to ask us if you have any questions,
difficulties or suggestions to improve the program.
<br>
<br>
Enjoy your analysis.
<br>
<br>
Cheers,
<br>
Sébastien
<br>
<br>
On 12/09/2011 13:12, Williams, Dean N. wrote:
<br>
<blockquote type="cite">Dear Jonathan and Stephen,
<br>
<br>
We are also working on other solutions to help alleviate the
problems
<br>
mentioned below, such as replicating the most of the archive at various
<br>
locations around the world. As Steven mentioned, we are aware of this
<br>
shortcomings and others and are working "quickly" to address them.
<br>
<br>
Thanks and best regards,
<br>
Dean
<br>
<br>
On 9/12/11 4:04 AM, <a class="moz-txt-link-rfc2396E" href="mailto:stephen.pascoe@stfc.ac.uk">"stephen.pascoe@stfc.ac.uk"</a>
<br>
<a class="moz-txt-link-rfc2396E" href="mailto:stephen.pascoe@stfc.ac.uk"><stephen.pascoe@stfc.ac.uk></a> wrote:
<br>
<br>
<blockquote type="cite">Dear Jonathan,
<br>
<br>
Thanks for taking the time to describe your concerns about the
usability
<br>
of the CMIP5 archive system. I am CC'ing this to <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>
<br>
as I think your feedback is particularly welcome and insightful and
<br>
deserves to be seen and discussed widely.
<br>
<br>
We are aware of many of the shortcomings you identify; improvements in
<br>
software and documentation are in progress that I hope will improve
your
<br>
experience. However, our progress has been slower than we'd hoped and
we
<br>
are now up against significant CMIP5 usage which will inevitably impede
<br>
rolling-out improvements. We would have hoped to have the system more
<br>
usable by now but we are pushing hard to improve the system as quickly
as
<br>
possible.
<br>
<br>
You identify several user interface and performance issues with the ESG
<br>
Gateway search system. Our colleagues at NCAR have been developing a
new
<br>
version of the Gateway with an improved search backend that I believe
<br>
solves many of your concerns. I've seen a test deployment at NCAR and
it
<br>
is a significant improvement. We at BADC will be deploying it for
<br>
testing in the next couple of days in the hope that it can be
rolled-out
<br>
quickly for end-users.
<br>
<br>
Another point in your feedback is scriptability of downloads and
checking
<br>
what is available. We had hoped that the wget script generation
feature
<br>
of the gateway would produce wget scripts that could be edited to
<br>
download different sorts of data by leveraging the Data Reference
Syntax
<br>
[1]. Unfortunately, although some download URLs contain DRS
information
<br>
that would help deducing alternative downloads, this isn't practical at
<br>
present. We are working to improve the DRS consistency of the archive
<br>
that we hope will improve download scriptability.
<br>
<br>
The other mechanism you could use to programmatically download data and
<br>
discover new data is reading the THREDDS catalogs. Every centre
serving
<br>
CMIP5 data is running a THREDDS Data Server [2] which lists all
download
<br>
URLs in a network of THREDDS XML catalogs. This is intended as an
<br>
internal interface so isn't well documented. However, I think it is no
<br>
secret that some users are doing this already. You can find the
THREDDS
<br>
source catalog of every dataset in the "History" tab of the Gateway's
<br>
dataset page or they can be deduced from download URLs and a little
<br>
knowledge of TDS.
<br>
<br>
I should add that downloading data directly from a TDS will only work
if
<br>
it is configured to use "tokenless" security. This is the case with
only
<br>
some datanodes at present but should be fixed in the near term.
<br>
<br>
In the medium-term ESGF are planning documented service APIs that would
<br>
allow users to query the system programmatically and there is a new P2P
<br>
architecture in the works with more focus on scalability [3]
<br>
<br>
Regards,
<br>
Stephen Pascoe.
<br>
<br>
[1] CMIP5 Data Reference Syntax:
<br>
<a class="moz-txt-link-freetext" href="http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf">http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf</a>
<br>
[2] THREDDS Data Server: <a class="moz-txt-link-freetext" href="http://www.unidata.ucar.edu/projects/THREDDS/">http://www.unidata.ucar.edu/projects/THREDDS/</a>
<br>
[3] ESGF P2P Architecture: <a class="moz-txt-link-freetext" href="http://esgf.org/wiki/ESGF_Index">http://esgf.org/wiki/ESGF_Index</a>
<br>
<br>
---
<br>
Stephen Pascoe +44 (0)1235 445980
<br>
Centre of Environmental Data Archival
<br>
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX,
UK
<br>
<br>
<br>
-----Original Message-----
<br>
From: Jonathan Gregory [<a class="moz-txt-link-freetext" href="mailto:j.m.gregory@reading.ac.uk">mailto:j.m.gregory@reading.ac.uk</a>]
<br>
Sent: 12 September 2011 11:12
<br>
To: <a class="moz-txt-link-abbreviated" href="mailto:esg-support@earthsystemgrid.org">esg-support@earthsystemgrid.org</a>
<br>
Subject: cmip5
<br>
<br>
Dear ESG
<br>
<br>
In preparation for working on the 1st draft of the AR5, I have begun to
<br>
try to
<br>
download CMIP5 data. I have to say I am discouraged by the experience.
<br>
Using
<br>
this web interface is slow and inconvenient, and I fear it will be an
<br>
obstacle
<br>
to the work required to be done. The biggest limitation, I would say,
is
<br>
that
<br>
there is *only* a web interface. For CMIP3, I used ftp to download the
<br>
data,
<br>
having written my own scripts. That minimised the manual effort
required,
<br>
and
<br>
most importantly I could use my script to fetch data I didn't already
<br>
have,
<br>
which it could easily identify. With a web interface, working out what
I
<br>
don't
<br>
already have will only be possible by manual comparison, which will
take
<br>
a lot
<br>
of time. Is the http protocol that the web interface uses something
that
<br>
could
<br>
be employed in a script? If so, could you document it? Even if the
<br>
protocol is
<br>
tricky, I would still much rather write a script than use a web
<br>
interface, as
<br>
in the end it will be more efficient.
<br>
<br>
However, the web interface could be improved in various ways, I think,
<br>
which
<br>
would make it more efficient. As it stands, I find the following
<br>
inconvenient:
<br>
<br>
* The PCMDI gateway is sometimes slow. This morning (UK time) it is
<br>
terribly
<br>
slow - unusable, in fact.
<br>
<br>
* It always searches when you change any of the criteria, so it
searches
<br>
all
<br>
of CMIP5 when you select the "Project", for instance. This wastes time.
<br>
<br>
* You have to select "all" in order to see the whole list again and
make a
<br>
new selection, again wasting time with unnecessary searching.
<br>
<br>
* There is no way to select more than one thing at a time e.g. more
than
<br>
one
<br>
experiment or more than one quantity.
<br>
<br>
* All the datasets have to be ticked individually to proceed to
download,
<br>
which is tedious.
<br>
<br>
* If there is more than one page, you can tick only one page at a time,
<br>
so you
<br>
have to start all over again to do the next page, by repeating the
whole
<br>
search laboriously.
<br>
<br>
* I can't (yet) get MRI or MIROC data, as it requires some further
<br>
authorisation that I have applied for. In fact I applied several days
ago,
<br>
and I have not yet been authorised. How can I chase this up?
<br>
<br>
* The search facility at the top seems flaky. The "loading, please
wait"
<br>
never
<br>
goes away and it crashes with an http error sometimes.
<br>
<br>
* Although I would have thought that many users said that CMIP3 would
<br>
have been
<br>
much more convenient if it had been possible to download annual data
<br>
rather
<br>
than monthly - I certainly made this comment - that facility has not
been
<br>
provided in the CMIP5 interface.
<br>
<br>
I am sure many people would be grateful if you could make some
<br>
improvements.
<br>
(And I expect I am not the first to make these suggestions!)
<br>
<br>
Best wishes
<br>
<br>
Jonathan Gregory
<br>
-- <br>
Scanned by iCritical.
<br>
</blockquote>
_______________________________________________
<br>
GO-ESSP-TECH mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
<br>
</blockquote>
<br>
<br>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Stéphane Sénési
Ingénieur - équipe Assemblage du Système Terre
Centre National de Recherches Météorologiques
Groupe de Météorologie à Grande Echelle et Climat
CNRM/GMGEC/ASTER
42 Av Coriolis
F-31057 Toulouse Cedex 1
+33.5.61.07.99.31 (Fax :....9610)</pre>
<div style="bottom: auto; left: 706px; right: auto; top: 105px;"
class="translator-theme-default" id="translator-floating-panel">
<div title="Cliquer pour traduire" id="translator-floating-panel-button"></div>
</div>
</body>
</html>