[Go-essp-tech] CMIP5 data archive size estimate

[Bryan Lawrence]

Hi Frank

We discussed the number of files per atomic dataset at the 8th of December telco. You can see some notes at  http://proj.badc.rl.ac.uk/go-essp/wiki/CMIP5/Meetings/telco091208 ... or even listen to the audio.

Bottom line: 2 G ( bit/byte ... not sure, notes say bit, I think it might be byte)  limit in out of the box apache config is worth avoiding ...  o(tens) of files per atomic dataset on average (with some outliers).

Cheers
Bryan

On Wednesday 06 January 2010 15:12:25 Frank Toussaint wrote:
> Dear all:
> 
> As far as subatomic ;-) data handling is concerned:
> We presently seem to have no estimates of the number of files per atomic dataset. On the other hand we are asked by some of the data providers for recommended file sizes. 
> 
> Most of our atoms are in the order of tens to hundreds of GBytes. We tend to advise the providers not to exceed file sizes of few (1..10) GB if possible, which is 10 to 100 files per atom.
> 
> Do you have any recommendations? Do you know applications that are limited to files below a certain threshold? 
> 
> Best regards... frank
> 
> 
> 
> 



-- 
Bryan Lawrence
Director of Environmental Archival and Associated Research
(NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
STFC, Rutherford Appleton Laboratory
Phone +44 1235 445012; Fax ... 5848; 
Web: home.badc.rl.ac.uk/lawrence