[Go-essp-tech] CMIP5 archive download program

[Sébastien Denvil]

  Dear all,

I'm proud to announce/advertise the new version of the program that 
enabled scripting access to the CMIP5 archive.

This program is already used by IPSL, KNMI, Met Office, Swiss  ETH, AR5 
chapters authors, and why not your organization? The program just work.

All we want is feedback!

*Description:*
This program can download files from the CMIP5 archive in an easy way, 
for a list of variables, experiments and ensemble members. The program 
configures itself using cmip5 thredds catalogues. It will evolve 
together with the cmip5 archive backend functionalities.

The user defines one or many templates. Each of them have a list of 
variables, frequencies experiments and ensemble members. The user 
defines separately a list of models. Using these templates, the  program 
explore the ESGF grid and dowload all the corresponding available files. 
The program may be run regularly to download the possible new files. 
Typically each template is associated with an analysis (cfmip template, 
downscaling template and so on). Create as many  template_name.txt as 
you want in the user_selections folder, following the 
user_selections/default.txt (trivial) syntax and you are done.

*Installation and upgrade instructions:*
http://dods.ipsl.jussieu.fr/jripsl/synchro_data/README

*Quoting users:*
- "Your tool is really fantastic. It has the potential to be very useful 
and will help us saving a lot of time."
- "Fantastic!"
- "Brilliant!"
- "Excellent! You are faster in releasing new versions than I'm with 
writing feature requests. The update worked like a charm! I'm now 
getting ensemble data."
- "Just to let you know that we are really happy with your tool!"

*Model:*

CCSM4 CNRM-CM5 CSIRO-Mk3-6-0 CanAM4 CanESM2 HadCM3 HadGEM2-A HadGEM2-ES IPSL-CM5A-LR MIROC4h MIROC5 MRI-CGCM3 MRI-AGCM3-2S MRI-AGCM3-2H MPI-ESM-LR NorESM1-M inmcm4

***Changelog:*

- Version 1.2 - 20111007
	- improve "ensemble" support
	- increase from 8 to 16 Wget threads
	- add per model priority
	- move main loop delay from 3 seconds to 6 seconds.
	- support file ID with non-standard extension (".nc_0")
	- add upgrade and archive option in script install.sh
	- improve HTTP error handling in script get_data.sh
- Version 1.1 - 20110928
	- improve datanode and model configuration
	- improve HTTP error code handling
	- add watchdog to check for frozen wget
	- fix PCMDI datanode incorrect url
	- fix incorrect configuration for models GISS-E2-H, GISS-E2-R and inmcm4
	- add new models (HadCM3,IPSL-CM5A-LR,CanAM4,MIROC5,MIROC4h,CCSM4,MRI-CGCM3,MRI-AGCM3-2S,MRI-AGCM3-2H,MPI-ESM-LR)
- Version 1.0 - 20110909
	- support for myproxy-logon and myproxyclient
	- simple data selection with model,experiment,realm and variable
	- multi threaded downloads (8 tasks by default)
	- manage datasets version following new drs
	- incremental process (download only what's new)
	- download history stored in a db

**Regards.
Sébastien

-- 
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5

Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111022/10b10186/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4172 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111022/10b10186/attachment-0001.bin