[Go-essp-tech] +2Gb CMIP5 files

Gavin M Bell gavin at llnl.gov
Tue May 18 10:57:07 MDT 2010 

What self-respecting scientist would run windows....

Ooops did I say that out loud :-).

(just kidding... sort of) :-D

Nathan Wilhelmi wrote:
> Hi All,
>    
>     Here is a nice table summarizing the various Windows file system 
> limits. http://*www.*ntfs.com/ntfs_vs_fat.htm
> 
> -Nate*
> *
> stephen.pascoe at stfc.ac.uk wrote:
>> I've done some testing of these file limits this afternoon and I don't
>> think the filesystems will be a problem.
>>
>> >From Wikipedia it appears the FAT32 file system has a 4Gb limit
>> (http://*en.wikipedia.org/wiki/File_Allocation_Table).  That covers
>> Windows 95 onwards but my Windows XP box is NTFS and has no problem with
>> +4Gb files.  Similarly my 32-bit linux laptop (recent ubuntu) can handle
>> +4Gb files.
>>
>> Looks like anyone with a reasonably modern system will be able to handle
>> +4Gb files.  We may have more problems with old NetCDF library versions.
>>
>> S.
>>
>> ---
>> Stephen Pascoe  +44 (0)1235 445980
>> British Atmospheric Data Centre
>> Rutherford Appleton Laboratory
>>
>> -----Original Message-----
>> From: go-essp-tech-bounces at ucar.edu
>> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
>> ag.stephens at stfc.ac.uk
>> Sent: 18 May 2010 09:31
>> To: taylor13 at llnl.gov; go-essp-tech at ucar.edu
>> Cc: doutriaux1 at llnl.gov
>> Subject: Re: [Go-essp-tech] +2Gb CMIP5 files
>>
>> Dear Karl,
>>
>> Whether we think it's advisable or not, I'm sure that some of the wider
>> CMIP5 user community will be looking at the outputs on Windows. I think
>> it is sensible to set a 2GB file size limit.
>>
>> Regards,
>>
>> Ag 
>>
>> -----Original Message-----
>> From: go-essp-tech-bounces at ucar.edu
>> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Karl Taylor
>> Sent: 17 May 2010 18:45
>> To: go-essp-tech at ucar.edu
>> Cc: Doutriaux, Charles
>> Subject: Re: [Go-essp-tech] +2Gb CMIP5 files
>>
>> Dear all,
>>
>> CMOR has code already in place for checking whether a file exceeds 2 GB,
>> but it is currently turned off (it was turned on for CMIP3).  We thought
>> it was now unnecessary.  If the feeling is that there will be users
>> downloading CMIP5 files to windows machines using older operating
>> systems, I suppose that limiting CMIP5 files to whatever the limit is (2
>> GB or 4 GB -- does anyone know which it is?) might be wise.
>>
>> On the other hand, will anyone use a windows machine to look at netCDF
>> files?  If not, maybe this is a non-issue.
>>
>> Karl
>>
>> On 5/16/10 12:08 PM, stephen.pascoe at stfc.ac.uk wrote:
>>   
>>> I think I raised undue alarm here when suggesting we might be dealing
>>>     
>> with +2GB files.  Thanks Phil for clarifying that UKMO is still planning
>> to limit itself to<2GB files.
>>   
>>> I am wondering what the policy should be here?  My first thought is
>>>     
>> that modeling centres will mainly make the same decision as UKMO since
>> it is in their interest for their model output to be widely used.
>> However, enforcement could be difficult.  The logical place to enforce
>> the limit is in the level 1 QC but CMOR doesn't do this so it will be a
>> problem for people running datanodes.
>>   
>>> I suggest we make a strong recommendation to supply data in<2GB files
>>>     
>> and enforce it during level-2 QC before replicating.
>>   
>>> S.
>>>
>>> -----Original Message-----
>>> From: go-essp-tech-bounces at ucar.edu on behalf of Michael 
>>> Lautenschlager
>>> Sent: Sun 5/16/2010 1:35 PM
>>> To: V. Balaji
>>> Cc: go-essp-tech at ucar.edu
>>> Subject: Re: [Go-essp-tech] +2Gb CMIP5 files
>>>
>>> Hello *,
>>>
>>> we strongly support Phils decision for data files less than 2 GB. We 
>>> made decision in Hamburg for the same reasons because we cannot expect
>>>     
>>   
>>> that all users use 64 Bit systems. Most Windows environments are still
>>>     
>>   
>>> running with 32 Bits.
>>>
>>> Best wishes, Michael
>>>
>>> ---------------
>>> Dr. Michael Lautenschlager
>>>
>>> German Climate Computing Centre (DKRZ) World Data Center Climate
>>> (WDCC)
>>> ADDRESS: Bundesstrasse 45a, D-20146 Hamburg, Germany
>>> PHONE:   +4940-460094-118
>>> E-Mail:  lautenschlager at dkrz.de
>>>
>>> URL:    http://**www.**dkrz.de/
>>>           http://**www.**wdc-climate.de/
>>>
>>> V. Balaji schrieb:
>>>    
>>>     
>>>> If I understood correctly the most serious 2Gb problem is with
>>>>       
>> apache!
>>   
>>>> Bentley, Philip writes:
>>>>
>>>>      
>>>>       
>>>>> Hi Stephen,
>>>>>
>>>>> Yes, that's true, we did create a small number of test netCDF files 
>>>>> in that size range. But this was because the CMOR library we used at
>>>>>         
>>   
>>>>> the time didn't include functionality for chunking the output into 
>>>>> smaller files. Plus we wanted to stress-test our pipeline!
>>>>>
>>>>> Two things have happened since then:
>>>>>
>>>>> 1. Jamie has been working with Charles at PCMDI to implement and 
>>>>> test a solution whereby we can limit the size of the output netCDF 
>>>>> files produced by CMOR.
>>>>>
>>>>> 2. We have made the local decision to limit our netCDF file sizes to
>>>>>         
>>   
>>>>> 2 GB (or thereabouts) as, logistically, that will cause us less 
>>>>> headache moving these files around, and it should maximise the 
>>>>> number of client applications in which the files can be read.
>>>>>
>>>>> IIRC, I think Balaji mentioned that the 64-bit offset format was 
>>>>> required for output from the gridspec toolset. I could be wrong.
>>>>>
>>>>> Regards,
>>>>> Phil
>>>>>
>>>>>        
>>>>>         
>>>>>> -----Original Message-----
>>>>>> From: go-essp-tech-bounces at ucar.edu 
>>>>>> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of 
>>>>>> stephen.pascoe at stfc.ac.uk
>>>>>> Sent: 14 May 2010 10:52
>>>>>> To: go-essp-tech at ucar.edu
>>>>>> Subject: [Go-essp-tech] +2Gb CMIP5 files
>>>>>>
>>>>>> The latest UKMO extraction for CMIP5 has produced some files in the
>>>>>>           
>>   
>>>>>> 30Gb range.  We had discussed previously the assumption that all 
>>>>>> files would be<2Gb.  Do we feel it is important to enforce a<2Gb 
>>>>>> limit or should this just be a recommendation on modelling centres?
>>>>>>
>>>>>> To my knowledge there is two issues with +2Gb files:
>>>>>>
>>>>>>   1. +2GB NetCDF files will be in 64-bit offset format.
>>>>>> Therefore NetCDF libraries prior to v3.6 will not be able to read 
>>>>>> them.
>>>>>>   2. Older file systems may have a 2Gb file limit. This will mainly
>>>>>>           
>>   
>>>>>> affect 32-bit systems that are a few years old. FAT32 has a 4Gb 
>>>>>> limit.
>>>>>>
>>>>>> These are end-user issues, is there any reason why the ESG software
>>>>>>           
>>   
>>>>>> might have problems with files over 2Gb?  If we do want to ensure 
>>>>>> files are<2Gb do we want to mandate the modelling centres deliver 
>>>>>> that or will the data centres need to split files?
>>>>>>
>>>>>> Stephen.
>>>>>>
>>>>>> ---
>>>>>> Stephen Pascoe  +44 (0)1235 445980
>>>>>> British Atmospheric Data Centre
>>>>>> Rutherford Appleton Laboratory
>>>>>> --
>>>>>> Scanned by iCritical.
>>>>>> _______________________________________________
>>>>>> GO-ESSP-TECH mailing list
>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>> http://**mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>>
>>>>>>          
>>>>>>           
>>>>> _______________________________________________
>>>>> GO-ESSP-TECH mailing list
>>>>> GO-ESSP-TECH at ucar.edu
>>>>> http://**mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>
>>>>>        
>>>>>         
>>>>      
>>>>       
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://**mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>>>    
>>>     
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>> --
>> Scanned by iCritical.
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>   
> 
> 
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
> 
> 

-- 
Gavin M. Bell
Lawrence Livermore National Labs
--

 "Never mistake a clear view for a short distance."
       	       -Paul Saffo

(GPG Key - http://rainbow.llnl.gov/dist/keys/gavin.asc)

 A796 CE39 9C31 68A4 52A7  1F6B 66B7 B250 21D5 6D3E

More information about the GO-ESSP-TECH mailing list