[Go-essp-tech] CMIP5 archive download program
Sébastien Denvil
sebastien.denvil at ipsl.jussieu.fr
Sat Oct 22 08:04:11 MDT 2011
Dear all,
I'm proud to announce/advertise the new version of the program that
enabled scripting access to the CMIP5 archive.
This program is already used by IPSL, KNMI, Met Office, Swiss ETH, AR5
chapters authors, and why not your organization? The program just work.
All we want is feedback!
*Description:*
This program can download files from the CMIP5 archive in an easy way,
for a list of variables, experiments and ensemble members. The program
configures itself using cmip5 thredds catalogues. It will evolve
together with the cmip5 archive backend functionalities.
The user defines one or many templates. Each of them have a list of
variables, frequencies experiments and ensemble members. The user
defines separately a list of models. Using these templates, the program
explore the ESGF grid and dowload all the corresponding available files.
The program may be run regularly to download the possible new files.
Typically each template is associated with an analysis (cfmip template,
downscaling template and so on). Create as many template_name.txt as
you want in the user_selections folder, following the
user_selections/default.txt (trivial) syntax and you are done.
*Installation and upgrade instructions:*
http://dods.ipsl.jussieu.fr/jripsl/synchro_data/README
*Quoting users:*
- "Your tool is really fantastic. It has the potential to be very useful
and will help us saving a lot of time."
- "Fantastic!"
- "Brilliant!"
- "Excellent! You are faster in releasing new versions than I'm with
writing feature requests. The update worked like a charm! I'm now
getting ensemble data."
- "Just to let you know that we are really happy with your tool!"
*Model:*
CCSM4 CNRM-CM5 CSIRO-Mk3-6-0 CanAM4 CanESM2 HadCM3 HadGEM2-A HadGEM2-ES IPSL-CM5A-LR MIROC4h MIROC5 MRI-CGCM3 MRI-AGCM3-2S MRI-AGCM3-2H MPI-ESM-LR NorESM1-M inmcm4
***Changelog:*
- Version 1.2 - 20111007
- improve "ensemble" support
- increase from 8 to 16 Wget threads
- add per model priority
- move main loop delay from 3 seconds to 6 seconds.
- support file ID with non-standard extension (".nc_0")
- add upgrade and archive option in script install.sh
- improve HTTP error handling in script get_data.sh
- Version 1.1 - 20110928
- improve datanode and model configuration
- improve HTTP error code handling
- add watchdog to check for frozen wget
- fix PCMDI datanode incorrect url
- fix incorrect configuration for models GISS-E2-H, GISS-E2-R and inmcm4
- add new models (HadCM3,IPSL-CM5A-LR,CanAM4,MIROC5,MIROC4h,CCSM4,MRI-CGCM3,MRI-AGCM3-2S,MRI-AGCM3-2H,MPI-ESM-LR)
- Version 1.0 - 20110909
- support for myproxy-logon and myproxyclient
- simple data selection with model,experiment,realm and variable
- multi threaded downloads (8 tasks by default)
- manage datasets version following new drs
- incremental process (download only what's new)
- download history stored in a db
**Regards.
Sébastien
--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111022/10b10186/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4172 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111022/10b10186/attachment-0001.bin
More information about the GO-ESSP-TECH
mailing list