[Go-essp-tech] CMIP5 parameter versions

[mark collier]

Hi Stephen,

On 06/09/2011, at 6:01 PM, Stephen.Pascoe at stfc.ac.uk wrote:

> Hi Mark,
>
> 1) if the ESG system is designed to always supply the latest version
> (and hide away old versions)?
>
> Yes.  The ESG publisher keeps THREDDS catalogs of old versions but  
> the Gateway will always display the most recent version.  Previous  
> versions aren't directly accessible through a Gateway (at least not  
> in the versions I've evaluated, 1.2.x and 1.3.x) but the existence  
> of previous versions are visible through a Gateway.
>
> Of course this presumes you have separated the files of old and new  
> versions in some way and correctly told ESG publisher which files  
> constitute a new version.  See below.
>
>> 2) as the file system that is scanned by the ESG system is shared by
>> our analysts, as it stands old versions can still be seen and copied
>> and potentially end up in local archives, especially when distributed
>> by 3rd parties. We would like to try and avoid this from happening
>> _without_ messing up the ESG metadata.
>
> Are you using the DRS directory structure which includes a version  
> directory "vYYYYMMDD"?  This is how we are separating the current  
> version from previous versions.  The tool drslib[1] is designed to  
> manage duplicate files across versions using symbolic links.
>
Yes we have used the versioning system, it was a necessity to get the  
quality control going. We found it quite useful as it was a convenient  
way of separating our processed to published stage, helping to ensure  
only wanted files end up being available via the ESG gateway. We also  
add a version global attribute to the file at the publishing phase,  
this is also a useful check that the version is known once removed  
away from the DRS structure (as might happen by end-users).

> [1] http://esgf.org/esgf-drslib-site
To be honest we knew of drslib but found it easier to develop our own  
systems where we knew exactly what was going on. However, hopefully we  
can utilise it for our next model submission later in the year. It is  
good to support local tools wherever possible.

The main thrust of my question was in regards to protecting analysts  
locally where there is a chance of the wrong version being used simply  
because they are all visible. Recommended solutions to this are sought.

We also will need to delete old versions in some cases to minimise  
resources used, we would like to know if there are problems if we  
delete data from the filesystem the ESG scans, or is it simply a case  
or rescanning everytime a chage is made?

Regards,

>
> Cheers,
> Stephen.
>
> ---
> Stephen Pascoe  +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11  
> 0QX, UK
>
> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu 
> ] On Behalf Of mark collier
> Sent: 06 September 2011 02:30
> To: go-essp-tech at ucar.edu
> Cc: Jeffrey Stephen; Aspendale) Rotstayn Leon (CMAR; Ben Evans
> Subject: [Go-essp-tech] CMIP5 parameter versions
>
> Hi,
> just a general question about versions of parameters.
>
> As we identify problems our list of new versions is growing, however,
> still manageable.
>
> What concerns us more is the possibility of outdated versions which
> may still be accessed. In terms of reconciling file (differences) it
> can be good to have all versions available, however, nightmarish if
> the wrong files get into analysts repositories and accidently (or
> unknowingly used if they aren't aware of them being outdated) used.
>
> I would like to know:
>
> 1) if the ESG system is designed to always supply the latest version
> (and hide away old versions)?
>
> 2) as the filesystem that is scanned by the ESG system is shared by
> our analysts, as it stands old versions can still be seen and copied
> and potentially end up in local archives, especially when distributed
> by 3rd parties. We would like to try and avoid this from happening
> _without_ messing up the ESG metadata.
>
> One solution for 2) is to ("in-situ") change the variable name from
> say pr to pr_error in the wrong/outdated file versions - the DRS
> structure (including filename) will stay the same but anyone trying to
> read the file will instantly be alerted to the problem.
>
> This is probably our biggest concern at the moment in terms of making
> data available to the CMIP5 community.
>
> Regards,
> Mark Collier.
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> -- 
> Scanned by iCritical.