[Go-essp-tech] +2Gb CMIP5 files

[Serguei Nikonov]

That was on Apache version < 2.1. Now there is not such limit cause later versions handle files 
using 64-bit file offsets.

Sergey

V. Balaji wrote:
> If I understood correctly the most serious 2Gb problem is with apache!
> 
> Bentley, Philip writes:
> 
>> Hi Stephen,
>>
>> Yes, that's true, we did create a small number of test netCDF files in
>> that size range. But this was because the CMOR library we used at the
>> time didn't include functionality for chunking the output into smaller
>> files. Plus we wanted to stress-test our pipeline!
>>
>> Two things have happened since then:
>>
>> 1. Jamie has been working with Charles at PCMDI to implement and test a
>> solution whereby we can limit the size of the output netCDF files
>> produced by CMOR.
>>
>> 2. We have made the local decision to limit our netCDF file sizes to 2
>> GB (or thereabouts) as, logistically, that will cause us less headache
>> moving these files around, and it should maximise the number of client
>> applications in which the files can be read.
>>
>> IIRC, I think Balaji mentioned that the 64-bit offset format was
>> required for output from the gridspec toolset. I could be wrong.
>>
>> Regards,
>> Phil
>>
>>> -----Original Message-----
>>> From: go-essp-tech-bounces at ucar.edu
>>> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
>>> stephen.pascoe at stfc.ac.uk
>>> Sent: 14 May 2010 10:52
>>> To: go-essp-tech at ucar.edu
>>> Subject: [Go-essp-tech] +2Gb CMIP5 files
>>>
>>> The latest UKMO extraction for CMIP5 has produced some files
>>> in the 30Gb range.  We had discussed previously the
>>> assumption that all files would be <2Gb.  Do we feel it is
>>> important to enforce a <2Gb limit or should this just be a
>>> recommendation on modelling centres?
>>>
>>> To my knowledge there is two issues with +2Gb files:
>>>
>>>  1. +2GB NetCDF files will be in 64-bit offset format.
>>> Therefore NetCDF libraries prior to v3.6 will not be able to
>>> read them.
>>>  2. Older file systems may have a 2Gb file limit. This will
>>> mainly affect 32-bit systems that are a few years old. FAT32
>>> has a 4Gb limit.
>>>
>>> These are end-user issues, is there any reason why the ESG
>>> software might have problems with files over 2Gb?  If we do
>>> want to ensure files are <2Gb do we want to mandate the
>>> modelling centres deliver that or will the data centres need
>>> to split files?
>>>
>>> Stephen.
>>>
>>> ---
>>> Stephen Pascoe  +44 (0)1235 445980
>>> British Atmospheric Data Centre
>>> Rutherford Appleton Laboratory
>>> --
>>> Scanned by iCritical.
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>