[Go-essp-tech] +2Gb CMIP5 files

[V. Balaji]

If I understood correctly the most serious 2Gb problem is with apache!

Bentley, Philip writes:

> Hi Stephen,
>
> Yes, that's true, we did create a small number of test netCDF files in
> that size range. But this was because the CMOR library we used at the
> time didn't include functionality for chunking the output into smaller
> files. Plus we wanted to stress-test our pipeline!
>
> Two things have happened since then:
>
> 1. Jamie has been working with Charles at PCMDI to implement and test a
> solution whereby we can limit the size of the output netCDF files
> produced by CMOR.
>
> 2. We have made the local decision to limit our netCDF file sizes to 2
> GB (or thereabouts) as, logistically, that will cause us less headache
> moving these files around, and it should maximise the number of client
> applications in which the files can be read.
>
> IIRC, I think Balaji mentioned that the 64-bit offset format was
> required for output from the gridspec toolset. I could be wrong.
>
> Regards,
> Phil
>
>> -----Original Message-----
>> From: go-essp-tech-bounces at ucar.edu
>> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
>> stephen.pascoe at stfc.ac.uk
>> Sent: 14 May 2010 10:52
>> To: go-essp-tech at ucar.edu
>> Subject: [Go-essp-tech] +2Gb CMIP5 files
>>
>> The latest UKMO extraction for CMIP5 has produced some files
>> in the 30Gb range.  We had discussed previously the
>> assumption that all files would be <2Gb.  Do we feel it is
>> important to enforce a <2Gb limit or should this just be a
>> recommendation on modelling centres?
>>
>> To my knowledge there is two issues with +2Gb files:
>>
>>  1. +2GB NetCDF files will be in 64-bit offset format.
>> Therefore NetCDF libraries prior to v3.6 will not be able to
>> read them.
>>  2. Older file systems may have a 2Gb file limit. This will
>> mainly affect 32-bit systems that are a few years old. FAT32
>> has a 4Gb limit.
>>
>> These are end-user issues, is there any reason why the ESG
>> software might have problems with files over 2Gb?  If we do
>> want to ensure files are <2Gb do we want to mandate the
>> modelling centres deliver that or will the data centres need
>> to split files?
>>
>> Stephen.
>>
>> ---
>> Stephen Pascoe  +44 (0)1235 445980
>> British Atmospheric Data Centre
>> Rutherford Appleton Laboratory
>> --
>> Scanned by iCritical.
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>

-- 

V. Balaji                               Office:  +1-609-452-6516
Head, Modeling Systems Group, GFDL      Home:    +1-212-253-6662
Princeton University                    Email: v.balaji at noaa.gov