<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffcc" text="#000000">
Jamie and friends.<br>
<br>
You've answered your own questions :-)... <br>
It is the catalog where these checksums (and other features) are
recorded.<br>
And thus using the catalog you can see what has changed.<br>
There is a new catalog for every version of a dataset. Given that...<br>
you can quickly and easily inspect catalog_v1 and catalog_v2 to find
what the changes are.<br>
This all answers the question of "WHAT" (to download)... the other
question of "HOW" is a different, but related story.<br>
The trick is to not conflate the two issues which is what filesystem
discussions do. When talking about filesystems you are stipulating
the what but implicitly conflating the HOW because you are
implicitly designing for tools that intrinsically use the
filesystem. It is a muddying of the waters that doesn't separate
the two concerns. We need to deal with these two concepts
independently in a way that does not limit the system or cause undo
burden on institutions by requiring a rigid structure.<br>
<br>
As I mentioned... it's not the filesystem we need to look at... it's
the catalogs.<br>
<br>
just my $0.02 - I'll stop flogging this particular horse... but I
hope I have done a better job expressing the issues and where the
solution lies (IMHO).<br>
<br>
On 9/20/11 8:14 AM, Kettleborough, Jamie wrote:
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F703@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<pre wrap="">Hello Balaji,
I agree - getting all nodes to make the checksums available would be a
good thing. It gives you both the data integrity check on download, and
the ability to see what files really have changed from one publication
version to the next.
I don't know how hard it is to do this, particularly for data that is
already published.
Jamie
</pre>
<blockquote type="cite">
<pre wrap="">-----Original Message-----
From: V. Balaji [<a class="moz-txt-link-freetext" href="mailto:V.Balaji@noaa.gov">mailto:V.Balaji@noaa.gov</a>]
Sent: 20 September 2011 16:01
To: Kettleborough, Jamie
Cc: Karl Taylor; <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>; <a class="moz-txt-link-abbreviated" href="mailto:esg-node-dev@lists.llnl.gov">esg-node-dev@lists.llnl.gov</a>
Subject: Re: [Go-essp-tech] Reasoning for the use of symbolic
links in drslib
If nodes can currently choose to record checksums or not, I'd
strongly recommend this be a non-optional requirement.. how
could anyone download any data with confidence without being
able to checksum?
You can of course check timestamps and filesizes and so on,
but you have to consider those optimizations... a fast option
for the less paranoid to avoid the sum computation, which has
to be the gold standard.
"Trust but checksum".
Kettleborough, Jamie writes:
</pre>
<blockquote type="cite">
<pre wrap="">Hello Karl, everyone,
For replicating the latest version, I agree that your alternate
structure poses difficulties (but it seems like there must
</pre>
</blockquote>
<pre wrap="">be a way to
</pre>
<blockquote type="cite">
<pre wrap="">smartly determine whether the file you already have a file
</pre>
</blockquote>
<pre wrap="">and simply
</pre>
<blockquote type="cite">
<pre wrap="">need to move it, rather than bring it over again).
Doesn't every user (not just the replication system) have
</pre>
</blockquote>
<pre wrap="">this problem:
</pre>
<blockquote type="cite">
<pre wrap="">they want to know what files have changed (or not changed) at a new
publication version. No one wants to be using band width
</pre>
</blockquote>
<pre wrap="">or storage
</pre>
<blockquote type="cite">
<pre wrap="">space to fetch and store files they already have. How is a user
expected to know what has really changed? Estani mentions
</pre>
</blockquote>
<pre wrap="">check sums
</pre>
<blockquote type="cite">
<pre wrap="">- OK, but I don't think all nodes expose them (is this
</pre>
</blockquote>
<pre wrap="">right?). You
</pre>
<blockquote type="cite">
<pre wrap="">may try to infer from modification dates (not sure, I
</pre>
</blockquote>
<pre wrap="">haven't look at
</pre>
<blockquote type="cite">
<pre wrap="">them that closely). You may try to infer from the
</pre>
</blockquote>
<pre wrap="">TRACKING_ID - but
</pre>
<blockquote type="cite">
<pre wrap="">I'm not sure how reliable this is (I can imagine scenarios where
different files share the same TRACKING_ID - e.g. if they have been
modified with an nco tool).
Is there a recommended method for users to understand what *files*
have actually changed when a new publication version appears?
Thanks,
Jamie
</pre>
</blockquote>
<pre wrap="">
--
V. Balaji Office: +1-609-452-6516
Head, Modeling Systems Group, GFDL Home: +1-212-253-6662
Princeton University Email: <a class="moz-txt-link-abbreviated" href="mailto:v.balaji@noaa.gov">v.balaji@noaa.gov</a>
</pre>
</blockquote>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Gavin M. Bell
--
"Never mistake a clear view for a short distance."
-Paul Saffo
</pre>
</body>
</html>