[Go-essp-tech] Reasoning for the use of symbolic links in drslib

[Kettleborough, Jamie]

Hello Gavin,
 
thanks for this reply.  Can you (or anyone) be more specific on what
catalogue entry a data user is supposed to use to understand what files
have changed?  That would be really useful to know - it is already
hurting some users as they don't know how to get this information out
and so are downloading multiple copies of the same data.
 
Thanks,
 
Jamie


________________________________

	From: Gavin M. Bell [mailto:gavin at llnl.gov] 
	Sent: 19 September 2011 20:24
	To: Kettleborough, Jamie
	Cc: Taylor, Karl Taylor; go-essp-tech at ucar.edu;
esg-node-dev at lists.llnl.gov
	Subject: Re: [Go-essp-tech] Reasoning for the use of symbolic
links in drslib
	
	
	Gentle-people, 
	
	This is what the catalogs are for.
	There is too much filesystem talk.  The filesystem can only do
so much for us.
	There is semantic knowledge that is trying to be captured in a
relatively rigid tree structure.
	We must out 'outside of the box', I feel we are abusing the
filesystem and trying to do gymnastics with it that we don't need to do.
The answer is not with the filesystem.  The answer is the catalogs and
using them appropriately.  We need to stop hedging our bets against the
system we are building and actually use it to our benefit (the reason
why we made it).  The filesystem is a red herring.
	
	just my $0.02
	
	I should pull out the email I wrote in November of 2009
describing the catalog as the needed lingua fraca of this endeavor.
	
	-Cassandra.
	
	On 9/19/11 9:00 AM, Kettleborough, Jamie wrote: 

		Hello Karl, everyone,


			For replicating the latest version, I agree that
your alternate structure poses difficulties (but it seems like there
must be a way to smartly determine whether the file you already have a
file and simply need to move it, rather than bring it over again).   
			 

		Doesn't every user (not just the replication system)
have this problem: they want to know what files have changed (or not
changed) at a new publication version.  No one wants to be using band
width or storage space to fetch and store files they already have.  How
is a user expected to know what has really changed?  Estani mentions
check sums - OK, but I don't think all nodes expose them (is this
right?).  You may try to infer from modification dates (not sure, I
haven't look at them that closely).  You may try to infer from the
TRACKING_ID - but I'm not sure how reliable this is (I can imagine
scenarios where different files share the same TRACKING_ID - e.g. if
they have been modified with an nco tool).
		 
		Is there a recommended method for users to understand
what *files* have actually changed when a new publication version
appears?
		 
		Thanks,
		 
		Jamie


	-- 
	Gavin M. Bell
	Lawrence Livermore National Labs
	--
	
	 "Never mistake a clear view for a short distance."
	       	       -Paul Saffo
	
	(GPG Key - http://rainbow.llnl.gov/dist/keys/gavin.asc)
	
	 A796 CE39 9C31 68A4 52A7  1F6B 66B7 B250 21D5 6D3E

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110920/ef1c12b5/attachment.html