[Go-essp-tech] Reasoning for the use of symbolic links in drslib

[V. Balaji]

So long as source and destination are on the same filesystem, you
can use hard links instead of symbolic ('ln' instead of 'ln -s'),
can't you? Hard links are transparent (or do I mean opaque... in
either case, I mean not a problem:-) for any data transfer protocol,
including gridftp, and do not cost anything in bandwidth or storage.

But hard links cannot cross filesystem partitions.

Karl Taylor writes:

> Hi Stephen,
>
> For replicating the latest version, I agree that your alternate structure 
> poses difficulties (but it seems like there must be a way to smartly 
> determine whether the file you already have a file and simply need to move 
> it, rather than bring it over again).
>
> I wanted to bring up another use case in which your alternative offers some 
> advantages (after a slight modification):  a user using gridftp wants to 
> retrieve the latest version of datasets distributed around several sites 
> (ignore replicated versions for simplicity).  If the user wants to write a 
> script to retrieve the data without having to look up what the latest version 
> number is for each dataset, wouldn't it be best to put the actual files in 
> "latest" subdirectory, with pointers going from the version identified 
> directory to "latest"?   All of these complications seem to be due to 
> gridftp's inability to follow links.
>
> Are there other ways to enable users to do the above without putting the 
> actual files in a directory named "latest"?
>
> regards,
> Karl
>
> On 9/17/11 1:16 PM, stephen.pascoe at stfc.ac.uk wrote:
>> 
>> Hi All,
>> 
>> I'm aware that many people are reluctant to use drslib because of it's use 
>> of symbolic links when constructing the DRS directory structure.  I 
>> completely understand caution when using symbolic links but I wanted to 
>> make the case for why I believe it is necessary to meet the goals of a 
>> consistent distributed and versioned archive.  Therefore I've prepared a 
>> wiki page that goes into the technical details:
>> 
>> http://esgf.org/wiki/DrsVersionLinking
>> 
>> Please read it carefully if you are currently considering how to implement 
>> the DRS directory structure.  I propose we discuss this at the next ESGF 
>> telco.
>> 
>> Thanks,
>> 
>> Stephen.
>> 
>> ---
>> 
>> Stephen Pascoe  +44 (0)1235 445980
>> 
>> Centre of Environmental Data Archival
>> 
>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>> 
>> 
>> -- 
>> Scanned by iCritical.
>> 
>> 
>

-- 

V. Balaji                               Office:  +1-609-452-6516
Head, Modeling Systems Group, GFDL      Home:    +1-212-253-6662
Princeton University                    Email: v.balaji at noaa.gov