[Go-essp-tech] CMIP5 parameter versions

Tue Sep 6 08:45:17 MDT 2011

That's great news Nathan.  We'll have plenty of examples to work with after next week so I'll be up for helping things along, creating issues, etc.

Stephen.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

-----Original Message-----
From: Nathan Wilhelmi [mailto:wilhelmi at ucar.edu] 
Sent: 06 September 2011 14:31
To: Pascoe, Stephen (STFC,RAL,RALSP)
Cc: abhipsl at ipsl.jussieu.fr; Stephen.Jeffrey at climatechange.qld.gov.au; Leon.Rotstayn at csiro.au; go-essp-tech at ucar.edu; Ben.Evans at anu.edu.au; Mark.Collier at csiro.au
Subject: Re: [Go-essp-tech] CMIP5 parameter versions

Hi Stephen,

It's pretty close in the gateway, as you note there is now a history 
tab. We hit a snag when the history was added that we need to resolve. 
We need to change how versions are identified in the database, so it's a 
bit more than a UI change.

Once this is in place it should be straight forward to finish out the 
UI. I will dig up the details with Nathan this week and we can take a 
look at what it would take.

Thanks!
-Nate

stephen.pascoe at stfc.ac.uk wrote:
> Hi Ashish,
> 
> I have recently checked Gateway 1.3.1 and it appears only the most recent version is downloadable.  Information on previous versions can be seen in the dataset's History tab, including the URL of the THREDDS catalog but the URL is not clickable.
> 
> The NCAR developers may be able to comment but my impression is that version support in the Gateway has been left at an early stage.  I saw mock-ups of full version support at a workshop many months ago but I expect they left implementation until the requirement was clearer.
> 
> BADC is also trying to get multi-version support working. Following the Gateway upgrade tomorrow I will be trying to expose as much version information as possible through our Gateway.  We currently have quite a few datasets with 2 versions, a few with 3 and some version upgrades we are waiting to process.
> 
> Cheers,
> Stephen.
> 
> ---
> Stephen Pascoe  +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
> 
> 
> -----Original Message-----
> From: Ashish Bhardwaj [mailto:abhipsl at ipsl.jussieu.fr] 
> Sent: 06 September 2011 10:53
> To: Pascoe, Stephen (STFC,RAL,RALSP)
> Cc: Mark.Collier at csiro.au; go-essp-tech at ucar.edu; Stephen.Jeffrey at climatechange.qld.gov.au; Leon.Rotstayn at csiro.au; Ben.Evans at anu.edu.au
> Subject: Re: [Go-essp-tech] CMIP5 parameter versions
> 
> Hi Stephen,
> 
> 
> stephen.pascoe at stfc.ac.uk wrote:
>> Hi Mark,
>>
>> 1) if the ESG system is designed to always supply the latest version  
>> (and hide away old versions)?
>>
>> Yes.  The ESG publisher keeps THREDDS catalogs of old versions but the Gateway will always display the most recent version.  Previous versions aren't directly accessible through a Gateway (at least not in the versions I've evaluated, 1.2.x and 1.3.x) but the existence of previous versions are visible through a Gateway.
>>
>> Of course this presumes you have separated the files of old and new versions in some way and correctly told ESG publisher which files constitute a new version.  See below.
>>
>>   
> 
> If I added a new version for a dataset containing new variables, is it 
> possible both versions to be accessible through gateway ?
> 
> Thanks.
> Ashish
> 
> 
> 
>>> 2) as the file system that is scanned by the ESG system is shared by  
>>> our analysts, as it stands old versions can still be seen and copied  
>>> and potentially end up in local archives, especially when distributed  
>>> by 3rd parties. We would like to try and avoid this from happening  
>>> _without_ messing up the ESG metadata.
>>>     
>> Are you using the DRS directory structure which includes a version directory "vYYYYMMDD"?  This is how we are separating the current version from previous versions.  The tool drslib[1] is designed to manage duplicate files across versions using symbolic links.
>>
>> [1] http://esgf.org/esgf-drslib-site 
>>
>> Cheers,
>> Stephen.
>>
>> ---
>> Stephen Pascoe  +44 (0)1235 445980
>> Centre of Environmental Data Archival
>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>>
>> -----Original Message-----
>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of mark collier
>> Sent: 06 September 2011 02:30
>> To: go-essp-tech at ucar.edu
>> Cc: Jeffrey Stephen; Aspendale) Rotstayn Leon (CMAR; Ben Evans
>> Subject: [Go-essp-tech] CMIP5 parameter versions
>>
>> Hi,
>> just a general question about versions of parameters.
>>
>> As we identify problems our list of new versions is growing, however,  
>> still manageable.
>>
>> What concerns us more is the possibility of outdated versions which  
>> may still be accessed. In terms of reconciling file (differences) it  
>> can be good to have all versions available, however, nightmarish if  
>> the wrong files get into analysts repositories and accidently (or  
>> unknowingly used if they aren't aware of them being outdated) used.
>>
>> I would like to know:
>>
>> 1) if the ESG system is designed to always supply the latest version  
>> (and hide away old versions)?
>>
>> 2) as the filesystem that is scanned by the ESG system is shared by  
>> our analysts, as it stands old versions can still be seen and copied  
>> and potentially end up in local archives, especially when distributed  
>> by 3rd parties. We would like to try and avoid this from happening  
>> _without_ messing up the ESG metadata.
>>
>> One solution for 2) is to ("in-situ") change the variable name from  
>> say pr to pr_error in the wrong/outdated file versions - the DRS  
>> structure (including filename) will stay the same but anyone trying to  
>> read the file will instantly be alerted to the problem.
>>
>> This is probably our biggest concern at the moment in terms of making  
>> data available to the CMIP5 community.
>>
>> Regards,
>> Mark Collier.
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>   
> 
> 
-- 
Scanned by iCritical.