<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi<div><br></div><div>esgNode.mandatory(checksums + PKI) = a better night's sleep.</div><div><br></div><div>Mark</div><div><br></div><div><br><div><div>On 20 Sep 2011, at 17:14, Kettleborough, Jamie wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hello Balaji,<br><br>I agree - getting all nodes to make the checksums available would be a<br>good thing. It gives you both the data integrity check on download, and<br>the ability to see what files really have changed from one publication<br>version to the next.<br><br>I don't know how hard it is to do this, particularly for data that is<br>already published.<br><br>Jamie <br><br><blockquote type="cite">-----Original Message-----<br></blockquote><blockquote type="cite">From: V. Balaji [mailto:V.Balaji@noaa.gov] <br></blockquote><blockquote type="cite">Sent: 20 September 2011 16:01<br></blockquote><blockquote type="cite">To: Kettleborough, Jamie<br></blockquote><blockquote type="cite">Cc: Karl Taylor; <a href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>; <a href="mailto:esg-node-dev@lists.llnl.gov">esg-node-dev@lists.llnl.gov</a><br></blockquote><blockquote type="cite">Subject: Re: [Go-essp-tech] Reasoning for the use of symbolic <br></blockquote><blockquote type="cite">links in drslib<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">If nodes can currently choose to record checksums or not, I'd <br></blockquote><blockquote type="cite">strongly recommend this be a non-optional requirement.. how <br></blockquote><blockquote type="cite">could anyone download any data with confidence without being <br></blockquote><blockquote type="cite">able to checksum?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">You can of course check timestamps and filesizes and so on, <br></blockquote><blockquote type="cite">but you have to consider those optimizations... a fast option <br></blockquote><blockquote type="cite">for the less paranoid to avoid the sum computation, which has <br></blockquote><blockquote type="cite">to be the gold standard.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">"Trust but checksum".<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Kettleborough, Jamie writes:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">Hello Karl, everyone,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span class="Apple-tab-span" style="white-space:pre"> </span>For replicating the latest version, I agree that your alternate <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">structure poses difficulties (but it seems like there must <br></blockquote></blockquote><blockquote type="cite">be a way to <br></blockquote><blockquote type="cite"><blockquote type="cite">smartly determine whether the file you already have a file <br></blockquote></blockquote><blockquote type="cite">and simply <br></blockquote><blockquote type="cite"><blockquote type="cite">need to move it, rather than bring it over again).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Doesn't every user (not just the replication system) have <br></blockquote></blockquote><blockquote type="cite">this problem:<br></blockquote><blockquote type="cite"><blockquote type="cite">they want to know what files have changed (or not changed) at a new <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">publication version. No one wants to be using band width <br></blockquote></blockquote><blockquote type="cite">or storage <br></blockquote><blockquote type="cite"><blockquote type="cite">space to fetch and store files they already have. How is a user <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">expected to know what has really changed? Estani mentions <br></blockquote></blockquote><blockquote type="cite">check sums <br></blockquote><blockquote type="cite"><blockquote type="cite">- OK, but I don't think all nodes expose them (is this <br></blockquote></blockquote><blockquote type="cite">right?). You <br></blockquote><blockquote type="cite"><blockquote type="cite">may try to infer from modification dates (not sure, I <br></blockquote></blockquote><blockquote type="cite">haven't look at <br></blockquote><blockquote type="cite"><blockquote type="cite">them that closely). You may try to infer from the <br></blockquote></blockquote><blockquote type="cite">TRACKING_ID - but <br></blockquote><blockquote type="cite"><blockquote type="cite">I'm not sure how reliable this is (I can imagine scenarios where <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">different files share the same TRACKING_ID - e.g. if they have been <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">modified with an nco tool).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Is there a recommended method for users to understand what *files* <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">have actually changed when a new publication version appears?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Thanks,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Jamie<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-- <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">V. Balaji Office: +1-609-452-6516<br></blockquote><blockquote type="cite">Head, Modeling Systems Group, GFDL Home: +1-212-253-6662<br></blockquote><blockquote type="cite">Princeton University Email: <a href="mailto:v.balaji@noaa.gov">v.balaji@noaa.gov</a><br></blockquote><blockquote type="cite"><br></blockquote>_______________________________________________<br>GO-ESSP-TECH mailing list<br><a href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>http://mailman.ucar.edu/mailman/listinfo/go-essp-tech<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div>---------------------------------------------------</div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">Mark Morgan</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">Software Architect / Engineer</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">Institut Pierre Simon Laplace (IPSL),</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">Université Pierre Marie Curie,</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">4 Place Jussieu,</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">Tour 45-55, Salle #207,</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">Paris 75005</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">France.</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">Tel : +33 (0) 1 44 27 49 10</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">Email: <a href="mailto:momipsl@ipsl.jussieu.fr">momipsl@ipsl.jussieu.fr</a></span></div><div>---------------------------------------------------</div><div><br></div></span><br class="Apple-interchange-newline">
</div>
<br></div></body></html>