<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=us-ascii" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.19120"></HEAD>
<BODY bgColor=#ffffcc text=#000000>
<DIV dir=ltr align=left><SPAN class=065301910-21092011><FONT color=#0000ff
size=2 face=Arial>Hello Gavin,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=065301910-21092011><FONT color=#0000ff
size=2 face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=065301910-21092011><FONT color=#0000ff
size=2 face=Arial>so is that a consensus - every data node should record the
checksums for every file in the thredds catalogues?</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=065301910-21092011><FONT color=#0000ff
size=2 face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=065301910-21092011><FONT color=#0000ff
size=2 face=Arial>(Urm... not sure its really my role to say this - so sorry if
I've stepped out of line).</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=065301910-21092011><FONT color=#0000ff
size=2 face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=065301910-21092011><FONT color=#0000ff
size=2 face=Arial>Jamie</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=065301910-21092011><FONT color=#0000ff
size=2 face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=065301910-21092011><FONT color=#0000ff
size=2 face=Arial>p.s. do I have friends? I thought I was just
making enemies</FONT></SPAN></DIV><BR>
<BLOCKQUOTE
style="BORDER-LEFT: #0000ff 2px solid; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px"
dir=ltr>
<DIV dir=ltr lang=en-us class=OutlookMessageHeader align=left>
<HR tabIndex=-1>
<FONT size=2 face=Tahoma><B>From:</B> Gavin M. Bell [mailto:gavin@llnl.gov]
<BR><B>Sent:</B> 20 September 2011 17:26<BR><B>To:</B> Kettleborough,
Jamie<BR><B>Cc:</B> V. Balaji; go-essp-tech@ucar.edu;
esg-node-dev@lists.llnl.gov<BR><B>Subject:</B> Re: [Go-essp-tech] Reasoning
for the use of symbolic links in drslib<BR></FONT><BR></DIV>
<DIV></DIV>Jamie and friends.<BR><BR>You've answered your own questions :-)...
<BR>It is the catalog where these checksums (and other features) are
recorded.<BR>And thus using the catalog you can see what has changed.<BR>There
is a new catalog for every version of a dataset. Given that...<BR>you can
quickly and easily inspect catalog_v1 and catalog_v2 to find what the changes
are.<BR>This all answers the question of "WHAT" (to download)... the other
question of "HOW" is a different, but related story.<BR>The trick is to not
conflate the two issues which is what filesystem discussions do. When
talking about filesystems you are stipulating the what but implicitly
conflating the HOW because you are implicitly designing for tools that
intrinsically use the filesystem. It is a muddying of the waters that
doesn't separate the two concerns. We need to deal with these two
concepts independently in a way that does not limit the system or cause
undo burden on institutions by requiring a rigid structure.<BR><BR>As I
mentioned... it's not the filesystem we need to look at... it's the
catalogs.<BR><BR>just my $0.02 - I'll stop flogging this particular horse...
but I hope I have done a better job expressing the issues and where the
solution lies (IMHO).<BR><BR>On 9/20/11 8:14 AM, Kettleborough, Jamie wrote:
<BLOCKQUOTE
cite=mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F703@EXXMAIL02.desktop.frd.metoffice.com
type="cite"><PRE wrap="">Hello Balaji,
I agree - getting all nodes to make the checksums available would be a
good thing. It gives you both the data integrity check on download, and
the ability to see what files really have changed from one publication
version to the next.
I don't know how hard it is to do this, particularly for data that is
already published.
Jamie
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">-----Original Message-----
From: V. Balaji [<A class=moz-txt-link-freetext href="mailto:V.Balaji@noaa.gov">mailto:V.Balaji@noaa.gov</A>]
Sent: 20 September 2011 16:01
To: Kettleborough, Jamie
Cc: Karl Taylor; <A class=moz-txt-link-abbreviated href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</A>; <A class=moz-txt-link-abbreviated href="mailto:esg-node-dev@lists.llnl.gov">esg-node-dev@lists.llnl.gov</A>
Subject: Re: [Go-essp-tech] Reasoning for the use of symbolic
links in drslib
If nodes can currently choose to record checksums or not, I'd
strongly recommend this be a non-optional requirement.. how
could anyone download any data with confidence without being
able to checksum?
You can of course check timestamps and filesizes and so on,
but you have to consider those optimizations... a fast option
for the less paranoid to avoid the sum computation, which has
to be the gold standard.
"Trust but checksum".
Kettleborough, Jamie writes:
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">Hello Karl, everyone,
For replicating the latest version, I agree that your alternate
structure poses difficulties (but it seems like there must
</PRE></BLOCKQUOTE><PRE wrap="">be a way to
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">smartly determine whether the file you already have a file
</PRE></BLOCKQUOTE><PRE wrap="">and simply
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">need to move it, rather than bring it over again).
Doesn't every user (not just the replication system) have
</PRE></BLOCKQUOTE><PRE wrap="">this problem:
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">they want to know what files have changed (or not changed) at a new
publication version. No one wants to be using band width
</PRE></BLOCKQUOTE><PRE wrap="">or storage
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">space to fetch and store files they already have. How is a user
expected to know what has really changed? Estani mentions
</PRE></BLOCKQUOTE><PRE wrap="">check sums
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">- OK, but I don't think all nodes expose them (is this
</PRE></BLOCKQUOTE><PRE wrap="">right?). You
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">may try to infer from modification dates (not sure, I
</PRE></BLOCKQUOTE><PRE wrap="">haven't look at
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">them that closely). You may try to infer from the
</PRE></BLOCKQUOTE><PRE wrap="">TRACKING_ID - but
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">I'm not sure how reliable this is (I can imagine scenarios where
different files share the same TRACKING_ID - e.g. if they have been
modified with an nco tool).
Is there a recommended method for users to understand what *files*
have actually changed when a new publication version appears?
Thanks,
Jamie
</PRE></BLOCKQUOTE><PRE wrap="">--
V. Balaji Office: +1-609-452-6516
Head, Modeling Systems Group, GFDL Home: +1-212-253-6662
Princeton University Email: <A class=moz-txt-link-abbreviated href="mailto:v.balaji@noaa.gov">v.balaji@noaa.gov</A>
</PRE></BLOCKQUOTE><PRE wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<A class=moz-txt-link-abbreviated href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</A>
<A class=moz-txt-link-freetext href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</A>
</PRE></BLOCKQUOTE><BR><PRE class=moz-signature cols="72">--
Gavin M. Bell
--
"Never mistake a clear view for a short distance."
-Paul Saffo
</PRE></BLOCKQUOTE></BODY></HTML>