[Go-essp-tech] Reasoning for the use of symbolic links in drslib
V. Balaji
V.Balaji at noaa.gov
Sun Sep 18 18:23:44 MDT 2011
So long as source and destination are on the same filesystem, you
can use hard links instead of symbolic ('ln' instead of 'ln -s'),
can't you? Hard links are transparent (or do I mean opaque... in
either case, I mean not a problem:-) for any data transfer protocol,
including gridftp, and do not cost anything in bandwidth or storage.
But hard links cannot cross filesystem partitions.
Karl Taylor writes:
> Hi Stephen,
>
> For replicating the latest version, I agree that your alternate structure
> poses difficulties (but it seems like there must be a way to smartly
> determine whether the file you already have a file and simply need to move
> it, rather than bring it over again).
>
> I wanted to bring up another use case in which your alternative offers some
> advantages (after a slight modification): a user using gridftp wants to
> retrieve the latest version of datasets distributed around several sites
> (ignore replicated versions for simplicity). If the user wants to write a
> script to retrieve the data without having to look up what the latest version
> number is for each dataset, wouldn't it be best to put the actual files in
> "latest" subdirectory, with pointers going from the version identified
> directory to "latest"? All of these complications seem to be due to
> gridftp's inability to follow links.
>
> Are there other ways to enable users to do the above without putting the
> actual files in a directory named "latest"?
>
> regards,
> Karl
>
> On 9/17/11 1:16 PM, stephen.pascoe at stfc.ac.uk wrote:
>>
>> Hi All,
>>
>> I'm aware that many people are reluctant to use drslib because of it's use
>> of symbolic links when constructing the DRS directory structure. I
>> completely understand caution when using symbolic links but I wanted to
>> make the case for why I believe it is necessary to meet the goals of a
>> consistent distributed and versioned archive. Therefore I've prepared a
>> wiki page that goes into the technical details:
>>
>> http://esgf.org/wiki/DrsVersionLinking
>>
>> Please read it carefully if you are currently considering how to implement
>> the DRS directory structure. I propose we discuss this at the next ESGF
>> telco.
>>
>> Thanks,
>>
>> Stephen.
>>
>> ---
>>
>> Stephen Pascoe +44 (0)1235 445980
>>
>> Centre of Environmental Data Archival
>>
>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>>
>>
>> --
>> Scanned by iCritical.
>>
>>
>
--
V. Balaji Office: +1-609-452-6516
Head, Modeling Systems Group, GFDL Home: +1-212-253-6662
Princeton University Email: v.balaji at noaa.gov
More information about the GO-ESSP-TECH
mailing list