<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hello Jamie,<br>
<br>
Yes, you are right ... it would be better to rely on the catalogs
(I'll take your word for it that THREDDS is the appropriate catalog
to consult, but someone should confirm), but as you note I don't see
tools being put in place quickly (in the next couple of weeks) to
make it easy for casual users to access and interpret the
information in those catalogs.<br>
<br>
I've made some quick comments below:<br>
<br>
On 9/23/11 3:10 AM, Kettleborough, Jamie wrote:
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F72C@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<meta name="GENERATOR" content="MSHTML 8.00.6001.19120">
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">Hello Karl,</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"></span> </div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">thanks for responding
on this and making the user view much more explicit. And
thanks for the note on the checksum - its good to know this
is close to being 'required'.</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"></span> </div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">I also agree that the
risk to CMIP5 (and the model contribution to IPCC AR5) due
to data access problems is sufficiently high that simple
(non-general) solutions that can be delivered quickly are
needed. I think that many of the early CMIP5/working group
1 users are happy to take some of the responsibility for
filtering which data they need, q.c., etc on themselves. So
this user base can toleterate simpler solutions. This may
not apply to working groups 2, 3... I don't know. Later
studies of working group 1 may need richer model meta-data.</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"></span> </div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">Some questions
/comments on your proposal:</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"></span> </div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">1. I think the list
files are derivable from the thredds catalogue entries for
the publication version dataset (if they all contained the
checksums) - I think you suspect this. In a sense (I think)
they are a reformatting of the thredds catalogues into a
form more parsable by users. If it can be achieved in time
then I think its safer to get the checksums in the thredds
cataloges and derive any other format from there.</font></span></div>
</blockquote>
See earlier comment, and I agree getting the checksums in the
thredds catalogs is indeed a good idea, even if they can be found is
the "listing file"<br>
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F72C@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">2. do you think you
would expose these list files through http? You mention
gridFTP but how soon do you think gridFTP will be available
for the users that need it...</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2"> </font></span><span
class="604260509-23092011"><font color="#0000ff" face="Arial"
size="2">a. I'm not sure how many have data available
through gridFTP (<a moz-do-not-send="true"
href="http://esgf.org/wiki/Cmip5Status/ArchiveView">http://esgf.org/wiki/Cmip5Status/ArchiveView</a> suggest
not many?)</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2"> b. I'm not sure how
many users will have gridFTP clients (or maybe you can use a
standard ftp client?)</font></span></div>
</blockquote>
Yes, definitely http should be able to see the listing files. Users
should have options, and I agree the gridFTP option may not be in
place soon enough.<br>
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F72C@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">3. Do you need to
capture this idea of 'latest' in the user view, or can the
user work this out based on the version number?</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2"> a. including
'latest' makes is easier for users as it takes one bit of
responsibility away from them</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2"> b. but you may be
introducing an inconsistency between the thredds interface
(which doesn't really expose this idea of latest), and the
more file based interface).</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2"> c. this exposure of
'latest' may be a minor point, (but its ringing alarm bells
with me).</font></span></div>
</blockquote>
My impression is that the implementation for CMIP5 of thredds may
not easily handle versioning (but I could be completely wrong about
this). If it can, and simply doesn't recognize "latest", I don't
see that as a big problem. The catalog structure we're talking
about wouldn't have to include branches labeled "latest" if the
software written to access the data were made smart enough to read
the names of all the version subdirectories (or extract from thredds
all the version numbers), and then it could determine latest. <br>
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F72C@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">4. I don't think you
need time sample do you - isn't that in the file name?</font></span></div>
</blockquote>
Well, I don't want to entirely trust the file name to accurately
reflect this information, and it also doesn't include the units of
time, which might be necessary for a fuller understanding (not sure
about this though. The main motivation was to make it easier for
the user ... slightly easier to read the information from the
listing file table than to parse the file name, I think. Open to
omitting time though.<br>
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F72C@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">5. what is the full
path to the file - the one visible through gridFTP, or
through the thredds file server or what?</font></span></div>
</blockquote>
Someone will have to advise me on this.<br>
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F72C@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">6. an addition - but
in the same vain of simplicity, can we have an easy to parse
list of servers that hold CMIP5 data available via http. In
the first instance this could be populated by hand. It
could be as simple as a csv file - server,pki_status.</font></span></div>
</blockquote>
Good idea, I think.<br>
<br>
7. Another issue: How would authentication work for users
accessing data with http or gridftp using the "listing file" or
thredds catalog?<br>
<br>
best regards,<br>
Karl<br>
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F72C@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<div dir="ltr" align="left"><span class="604260509-23092011"></span> </div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">I'm afraid I haven't
had time to think about all the issues around hard links,
soft links and tape storage - and there may be more major
issues there.</font></span></div>
<div dir="ltr" align="left"><span class="604260509-23092011"></span> </div>
<div dir="ltr" align="left"><span class="604260509-23092011"><font
color="#0000ff" face="Arial" size="2">Jamie</font></span></div>
<br>
<blockquote style="border-left: 2px solid rgb(0, 0, 255);
padding-left: 5px; margin-left: 5px; margin-right: 0px;"
dir="ltr">
<div dir="ltr" class="OutlookMessageHeader" align="left"
lang="en-us">
<hr tabindex="-1"> <font face="Tahoma" size="2"><b>From:</b>
Karl Taylor [<a class="moz-txt-link-freetext" href="mailto:taylor13@llnl.gov">mailto:taylor13@llnl.gov</a>] <br>
<b>Sent:</b> 21 September 2011 23:19<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a><br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:gavin@llnl.gov">gavin@llnl.gov</a>; Kettleborough, Jamie;
<a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>; <a class="moz-txt-link-abbreviated" href="mailto:esg-node-dev@lists.llnl.gov">esg-node-dev@lists.llnl.gov</a><br>
<b>Subject:</b> Re: [Go-essp-tech] Reasoning for the use of
symbolic links in drslib<br>
</font><br>
</div>
<font face="Times New Roman">Hi Stephen and all,<br>
<br>
I would add another requirement (or is this part of 4?):<br>
<br>
5. A user (as opposed to a data provider or a "replicator" or
a data center data manager) should be able to determine
(through an automated scripted process) whether a file
previously downloaded is in the current (i.e., "latest")
version of a dataset, or has been withdrawn or replaced.<br>
<br>
To meet all the requirements in a practical way in the next
few weeks, I'll suggest an alternative approach: We could use
drsLib to create the DRS directory structure, but populate the
lowest level (where the files would normally be found) with a
single text file (referred to subsequently as the "listing
file") containing the following information:<br>
<br>
the publication-level dataset version THREDDS id, which is:
<activity>.<product>.<institute>.<model>.<experiment>.<frequency>.<modeling
realm>.<MIP table>.<ensemble
member>.<version number><br>
plus the <variable name><br>
followed by a table with:<br>
filename time units time of 1st time sample
time of last time sample full path to file </font><font
face="Times New Roman">tracking_id checksum </font><br>
<font face="Times New Roman">---------- ------------
----------------------------
----------------------------- -------------------
------------- ----------<br>
file1<br>
file2 <br>
. <br>
.<br>
.<br>
fileN<br>
<br>
The "listing file" would be stored twice for the latest
version of each dataset: once under the numbered version
subdirectory and *also* under the generically labeled "latest"
directory. [This is so a user interested in the latest
version can find it without knowing its actual number.] By
the way the time information included in the list might not be
absolutely essential, but it could be helpful for those only
wanting to download specific time-segments of an integration.<br>
<br>
I realize this is not a particularly elegant approach, but if
users were given access to the drs directory structure (say,
through gridftp), they could run a script that navigated
directly to a variable of interest (based on the DRS directory
structure specifications) and download the "listing file"
stored there. Then, the "latest" listing file could be
compared to the older "listing file" (previously downloaded by
the user) to determine whether a new version was available (by
simply comparing the <version numbers> stored in the
THREDDS ID). If the user didn't have the most recent version,
he could then compare the two "listing files" (old and new) to
determine which files were new and which (if any) had been
eliminated.<br>
<br>
At that point, the user could generate a local copy of the
latest version by moving/deleting files not found in the
latest "listing file" and by downloading (using, for example,
gridftp) only the new files.<br>
<br>
I bet that in a single day</font><font face="Times New Roman">
Stephen could </font><font face="Times New Roman">enhance
drslib to produce these list files, rather than creating the
symbolic links to the actual file locations as it currently
does. Note that if the actual files were moved into new
directories sometime in the future, a utility would have to be
written to modify all the "list files" to point to the new
file locations (but that's also true of the symbolic links, I
think)<br>
<br>
Also note that creation of a new version would *not* require
changing any of the existing "list files" (except the list
file in the "latest" directory would be removed). A new
version subdirectory would have to be created and for each
variable in the dataset, the new "list file" for that version
would have to be generated (and copied also to "latest").<br>
<br>
I'll be interested in your response to this idea and trust
that any time spent thinking about it is warranted (i.e., that
this is not a completely stupid suggestion). Will it meet all
of Stephen's needs? Are there any other solutions to the data
users' troubles in obtaining data, which we can implement in
the next few weeks (since that should be our goal here).<br>
<br>
My primary interest is in making CMIP5 data easily obtainable
by users (which appears not to be the case at present), and to
allow users to write scripts to troll for new data they are
interested in and discover any new versions of data that
should replace the old. This is not meant to be a general
solution to all of the possible ESG applications. Also, I'm
guessing that a similar approach could be followed where
instead of reading the "list files", one read the catalogs,
but I doubt that this would be as easy for the typical user to
do.<br>
<br>
Best regards,<br>
Karl<br>
<br>
</font><font face="Times New Roman">P.S. to weigh in on another
issue, I think it *will* be essential to require, as part of
ESG publication that the check-sum be recorded (in the THREDDS
catalog, </font><font face="Times New Roman">if I'm not
mistaken</font><font face="Times New Roman">). We haven't
asked groups to republish data conforming to this new
requirement because I want to make sure that any other
required alterations in the configuration of the publisher are
also communicated, so we only have to ask groups to republish
once.</font> Note also that if my "alternative" approach
outlined above is adopted, the checksums could either be gotten
from the catalog (if they were computed and stored there) or be
calculated by drslib itself; there would be no need to republish
data to ESG<br>
<font face="Times New Roman"><br>
</font><br>
On 9/20/11 2:35 PM, <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>
wrote:
<blockquote
cite="mid:4C353E6E4A08AE4792B350DAA392B5210EC79DDE@EXCHMBX01.fed.cclrc.ac.uk"
type="cite">
<meta name="Generator" content="Microsoft Word 12 (filtered
medium)">
<style>@font-face {
font-family: Cambria Math;
}
@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@font-face {
font-family: Consolas;
}
@page WordSection1 {size: 612.0pt 792.0pt; margin: 72.0pt 72.0pt 72.0pt 72.0pt; }
P.MsoNormal {
MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: black; FONT-SIZE: 12pt
}
LI.MsoNormal {
MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: black; FONT-SIZE: 12pt
}
DIV.MsoNormal {
MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: black; FONT-SIZE: 12pt
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
PRE {
MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Courier New"; COLOR: black; FONT-SIZE: 10pt; mso-style-priority: 99; mso-style-link: "HTML Preformatted Char"
}
SPAN.HTMLPreformattedChar {
FONT-FAMILY: Consolas; COLOR: black; mso-style-priority: 99; mso-style-link: "HTML Preformatted"; mso-style-name: "HTML Preformatted Char"
}
SPAN.EmailStyle19 {
FONT-FAMILY: "Calibri","sans-serif"; COLOR: #1f497d; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.WordSection1 {
page: WordSection1
}
</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">Hi All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">Lots of good discussion here and sorry
I've been keeping quiet. I want to remind ourselves of
the requirements I laid out in the wiki page<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">1. It should allow data from multiple
versions to be kept on disk simultaneously.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">2. It should avoid storing multiple
copies of files that are present in more than one
version.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">3. It should be straightforward to
copy dataset changes (i.e. differences between versions)
between nodes to allow efficient replication.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">4. It should rely only on the
filesystem so that generic tools like FTP could be used
to expose the structure if necessary.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">In my view we should address these
directly. Are they needed? Which are the most
important?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">Gavin said about catalogs<o:p></o:p></span></p>
<p class="MsoNormal">> you can quickly and easily inspect
catalog_v1 and catalog_v2 to find what the changes are.<br>
> This all answers the question of "WHAT" (to
download)... the other question of "HOW" is a different,
but related story.<br>
> The trick is to not conflate the two issues which is
what filesystem discussions do. . <span
style="font-family: 'Calibri','sans-serif'; color:
rgb(31, 73, 125); font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">But THREDDS conflates the two as
well! A THREDDS catalog contains descriptions of
service endpoints that are not independent of the node
serving the data (the "HOW"). Maybe we should have
developed a true catalog format but that is not where we
are now. The replication client use THREDDS catalogs in
this way but when I last looked it was completely
unaware of versions -- i.e. it won't help with #3. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">I don't see how Gavin's point
addresses any of the requirements above. Even if we
ditch #4, which I expect Gavin would argue for, it
doesn't directly solve the problem for #1-#3 either.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">Briefly on some other points that have
been made...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">Balaji, some archive tools maybe can
detect 2 paths pointing to the same filesystem inode but
both Estani and I have enquired with our backup people
and they say hard links must be avoided. I am happy to
include a hard-linking option in drslib though. I've
created a bugzilla ticket for it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">Karl, I think putting real files in
"latest" is equivalent to putting real files in the
latest "vYYYYMMDD" directory. The directories can be
renamed trivially on upgrade but you still have the same
problems as the wiki page says.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">I'm sure there were other points but
I've lost track. Checksums will have to wait for
another email.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;">Stephen.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family: Consolas;
color: rgb(31, 73, 125); font-size: 10.5pt;">---<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family: Consolas;
color: rgb(31, 73, 125); font-size: 10.5pt;">Stephen
Pascoe +44 (0)1235 445980<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family: Consolas;
color: rgb(31, 73, 125); font-size: 10.5pt;">Centre of
Environmental Data Archival<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family: Consolas;
color: rgb(31, 73, 125); font-size: 10.5pt;">STFC
Rutherford Appleton Laboratory, Harwell Oxford, Didcot
OX11 0QX, UK<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:
'Calibri','sans-serif'; color: rgb(31, 73, 125);
font-size: 11pt;"><o:p></o:p></span></p>
<div>
<div style="border-width: 1pt medium medium; border-style:
solid none none; border-color: rgb(181, 196, 223)
-moz-use-text-color -moz-use-text-color; padding: 3pt
0cm 0cm;">
<p class="MsoNormal"><b><span style="font-family:
'Tahoma','sans-serif'; color: windowtext;
font-size: 10pt;" lang="EN-US">From:</span></b><span
style="font-family: 'Tahoma','sans-serif'; color:
windowtext; font-size: 10pt;" lang="EN-US"> <a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>
[<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>]
<b>On Behalf Of </b>Gavin M. Bell<br>
<b>Sent:</b> 20 September 2011 17:26<br>
<b>To:</b> Kettleborough, Jamie<br>
<b>Cc:</b> <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>;
<a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:esg-node-dev@lists.llnl.gov">esg-node-dev@lists.llnl.gov</a><br>
<b>Subject:</b> Re: [Go-essp-tech] Reasoning for the
use of symbolic links in drslib<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal">Jamie and friends.<br>
<br>
You've answered your own questions :-)... <br>
It is the catalog where these checksums (and other
features) are recorded.<br>
And thus using the catalog you can see what has changed.<br>
There is a new catalog for every version of a dataset.
Given that...<br>
you can quickly and easily inspect catalog_v1 and
catalog_v2 to find what the changes are.<br>
This all answers the question of "WHAT" (to download)...
the other question of "HOW" is a different, but related
story.<br>
The trick is to not conflate the two issues which is what
filesystem discussions do. When talking about filesystems
you are stipulating the what but implicitly conflating the
HOW because you are implicitly designing for tools that
intrinsically use the filesystem. It is a muddying of the
waters that doesn't separate the two concerns. We need to
deal with these two concepts independently in a way that
does not limit the system or cause undo burden on
institutions by requiring a rigid structure.<br>
<br>
As I mentioned... it's not the filesystem we need to look
at... it's the catalogs.<br>
<br>
just my $0.02 - I'll stop flogging this particular
horse... but I hope I have done a better job expressing
the issues and where the solution lies (IMHO).<br>
<br>
On 9/20/11 8:14 AM, Kettleborough, Jamie wrote: <o:p></o:p></p>
<pre>Hello Balaji,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I agree - getting all nodes to make the checksums available would be a<o:p></o:p></pre>
<pre>good thing. It gives you both the data integrity check on download, and<o:p></o:p></pre>
<pre>the ability to see what files really have changed from one publication<o:p></o:p></pre>
<pre>version to the next.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I don't know how hard it is to do this, particularly for data that is<o:p></o:p></pre>
<pre>already published.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Jamie <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>-----Original Message-----<o:p></o:p></pre>
<pre>From: V. Balaji [<a href="mailto:V.Balaji@noaa.gov" moz-do-not-send="true">mailto:V.Balaji@noaa.gov</a>] <o:p></o:p></pre>
<pre>Sent: 20 September 2011 16:01<o:p></o:p></pre>
<pre>To: Kettleborough, Jamie<o:p></o:p></pre>
<pre>Cc: Karl Taylor; <a href="mailto:go-essp-tech@ucar.edu" moz-do-not-send="true">go-essp-tech@ucar.edu</a>; <a href="mailto:esg-node-dev@lists.llnl.gov" moz-do-not-send="true">esg-node-dev@lists.llnl.gov</a><o:p></o:p></pre>
<pre>Subject: Re: [Go-essp-tech] Reasoning for the use of symbolic <o:p></o:p></pre>
<pre>links in drslib<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>If nodes can currently choose to record checksums or not, I'd <o:p></o:p></pre>
<pre>strongly recommend this be a non-optional requirement.. how <o:p></o:p></pre>
<pre>could anyone download any data with confidence without being <o:p></o:p></pre>
<pre>able to checksum?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>You can of course check timestamps and filesizes and so on, <o:p></o:p></pre>
<pre>but you have to consider those optimizations... a fast option <o:p></o:p></pre>
<pre>for the less paranoid to avoid the sum computation, which has <o:p></o:p></pre>
<pre>to be the gold standard.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>"Trust but checksum".<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Kettleborough, Jamie writes:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>Hello Karl, everyone,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre> For replicating the latest version, I agree that your alternate <o:p></o:p></pre>
<pre>structure poses difficulties (but it seems like there must <o:p></o:p></pre>
</blockquote>
<pre>be a way to <o:p></o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>smartly determine whether the file you already have a file <o:p></o:p></pre>
</blockquote>
<pre>and simply <o:p></o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>need to move it, rather than bring it over again).<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Doesn't every user (not just the replication system) have <o:p></o:p></pre>
</blockquote>
<pre>this problem:<o:p></o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>they want to know what files have changed (or not changed) at a new <o:p></o:p></pre>
<pre>publication version. No one wants to be using band width <o:p></o:p></pre>
</blockquote>
<pre>or storage <o:p></o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>space to fetch and store files they already have. How is a user <o:p></o:p></pre>
<pre>expected to know what has really changed? Estani mentions <o:p></o:p></pre>
</blockquote>
<pre>check sums <o:p></o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>- OK, but I don't think all nodes expose them (is this <o:p></o:p></pre>
</blockquote>
<pre>right?). You <o:p></o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>may try to infer from modification dates (not sure, I <o:p></o:p></pre>
</blockquote>
<pre>haven't look at <o:p></o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>them that closely). You may try to infer from the <o:p></o:p></pre>
</blockquote>
<pre>TRACKING_ID - but <o:p></o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>I'm not sure how reliable this is (I can imagine scenarios where <o:p></o:p></pre>
<pre>different files share the same TRACKING_ID - e.g. if they have been <o:p></o:p></pre>
<pre>modified with an nco tool).<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Is there a recommended method for users to understand what *files* <o:p></o:p></pre>
<pre>have actually changed when a new publication version appears?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Jamie<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>-- <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>V. Balaji Office: +1-609-452-6516<o:p></o:p></pre>
<pre>Head, Modeling Systems Group, GFDL Home: +1-212-253-6662<o:p></o:p></pre>
<pre>Princeton University Email: <a href="mailto:v.balaji@noaa.gov" moz-do-not-send="true">v.balaji@noaa.gov</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>GO-ESSP-TECH mailing list<o:p></o:p></pre>
<pre><a href="mailto:GO-ESSP-TECH@ucar.edu" moz-do-not-send="true">GO-ESSP-TECH@ucar.edu</a><o:p></o:p></pre>
<pre><a href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech" moz-do-not-send="true">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><o:p></o:p></pre>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Gavin M. Bell<o:p></o:p></pre>
<pre>--<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> "Never mistake a clear view for a short distance."<o:p></o:p></pre>
<pre> -Paul Saffo<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</div>
<br>
<p>-- <br>
Scanned by iCritical. </p>
<br>
</blockquote>
</blockquote>
</blockquote>
</body>
</html>