[Go-essp-tech] Thoughts Relating to your phone conference on replication....

[Gavin M Bell]

Luca, et. al.

The idea here is to have a simple representation of the data.  Right now
it is the catalog posted by the data node.  But this representation does
not preclude any other represenation.  The main point is that each
portion of the system can interface with any other portion of the system
if we agree on the representation of the data, more specifically the
transport representation of the data.

So, in the current case, yes, I am talking about catalogs and using
catalogs to capture data is what we are doing.  What I am saying is that
 regardless of what the initial representation is it should be
maintained or at least able to be dealt with at the different levels of
the system since the initial representation contains the the full field
of information that is needed.  In the current case catalogs.

- Data node knows how to produce the 'representation'

- Gateways knows how to read and parse and glean information from the
represenation for what it does with it (index / store it).

- Replication knows how to read and parse and glean information from the
representation for what it does with it (diff and move files)

- FOO BAR system knows how to read and parse and glean information from
the representation for what it does with it (fooing and barring)....

etc...

Much like the UNIX philosophy of having blocks be strung together to
create a larger big of functionality... I am saying the catalog (in our
current case) or whatever the most informed representation is should be
used at each block of the system.  Perhaps it may have more information
that necessary for some parts, but it will always have the necessary
information.  As different representations come on line they too will
need to be able to be handled.  Then the next step would probably be to
create a system level transport mechanism that wraps the payload data.
Precisely building our own system equivalent of an IP stack.  Where the
catalog and whatever else becomes the data payload.  There really is
nothing new under the sun.

But before we extrapolate ad infinitum the issue of the day, the
representation of the moment is the catalog and all I am saying is that
it should be the representation understood by each block of the system.
 All that means specifically is that each block of the system be able to
parse the representation to take what it needs from it to to its job.
when the schema changes for the xml, well everyone will get a new
package of readers, and the delivery of new reader/parser code, can be
as imaginative as you want, but the only thing that would be the only
thing that changes for the whole rest of the system.  (Ref JINI... they
did a great job with pushing active code for protocol handling).

Luca Cinquini wrote:
> Hi Gavin,
>     the bottom line is that the THREDDS catalogs are not the currency of
> the system... it's simply one of many possible envisioned ways in which
> a data node can organize its data and send it to the gateway. The way
> the gateway is architected, the currency of the system is the metadata
> stored in the relational database. There are other ways to ingest data
> in the system: one is to parse a local directory, another (future) one
> is to parse an FTP server of a GridFTP server. You can also imagine
> parsing XML catalogs that are in a different schema, and in fact this is
> what likely will happen to interoperate with other systems that already
> have their data management system. The idea was to make the data node
> flexible so to not absolutely require that it has a TDS installation
> (although this will happen for all CMIP5 data nodes).
> 
> thanks, luca
> 
> On Nov 3, 2009, at 12:41 PM, Gavin M Bell wrote:
> 
>> Hello Gentle-people,
>>
>> I just wanted to say a few words.  I should have said something at the
>> end of our phone conference but I thought it wiser to collect my
>> thoughts a bit for a more cogent and durable presentation.
>>
>> Okay, please be patient and read through the entire email, the complete
>> thought.  Please send feed back so we can hash this out.... Bare with me.
>>
>> The main idea I want to get across is for us to have a *catalog-centric*
>> view of they system.  It is the catalog that is the primary currency of
>> the system.
>>
>> - The catalog gets generated and published from the data via the
>> data-node/publisher to the gateway.
>>
>> - The gateway is simply, in the context of this model, a searchable
>> index over a collection of catalogs.
>>
>> - Changes to catalogs are what is versioned.
>>
>> - Changes to catalogs are what trigger notifications
>>
>> - Replication should be about replicating catalogs, where files
>> transfers are the necessary side-effect of proper catalog replication.
>>
>> It is the catalog that is the central 'document' that we are interested
>> in.  It is the single entity that contains the necessary information
>> used in all levels of this system.
>>
>> The very good point that was brought up on the call was, what is the
>> interface between parts of the system?  It has become clear to me that
>> if each part of the system understood the catalog then they could
>> operate quite well, gleaning the information out of catalogs.
>>
>> The topic today was replication:
>> So... In a catalog centric model, the question of replication becomes
>> simply, what datasets have changed?  This is equivalent to asking, what
>> catalogs have changed?  This can be answered by the gateway which, in
>> this context, is essentially a catalog store. The gateway knows when a
>> catalog changes because a new catalog would usurp an older one - this
>> can be detected at publication time and versioned appropriately and
>> announced (via notification).  The replication agent is interested in
>> these notifications thus should be defacto subscribed to getting such
>> notification messages.  When the replication agent is notified it would
>> look on it's system and see if the notification is something in it's
>> list to have a replica of, it's "replication list".  If so it can pull
>> down the catalog or some subset (diff) of that catalog, or simply the
>> necessary tuple to find the location(s) of holders of the newest
>> catalog.  The catalog will always have in it its authoritative source
>> (dataset name and gateway).  This can be resolved to the actual data
>> node that has the new version of that catalog (and any other replicas
>> that are up-to-date).  Then it is the job of the replication agent that
>> wants to be updated to contact the authoritative data-node or any
>> up-to-date replica holder and basically sync catalogs.  Syncing catalogs
>> means grabbing the latest catalog, from the authoritative source or an
>> updated data-node replica, and diffing it with the stale catalog it
>> currently has... the result of the diff is the set files and such that
>> need to be transfered in order to make the state of the stale node
>> equivalent to the state of the latest catalog.  It is the catalog that
>> contains the 'inventory' and all other necessary information.  Once
>> files are transfered integrity checking can be accomplished at a few
>> levels.  First is to have the stale, node generate it's own catalog and
>> then check it against the reference (up-to-date) catalog it got from the
>> source. If replication has been done successfully they should be
>> identical!  The catalog should have a 'header' portion that contains the
>> checksum of the immutable portion 'body' of the catalog.  The first
>> level integrity check would be to see if what is generated and the
>> reference are the same, if not a second level check that required
>> walking the catalog's (xml) tree and compare the two trees.  it is in
>> the latter check where individual files entries are checked to detect
>> what files may need to be fetched again.  Also if the connection goes
>> down or fails in some way, generating a catalog over the partial set of
>> files that have already been downloaded, and comparing it with the
>> source catalog will tell the replication agent where to puck up from.
>> The source catalog could be cached on the replication catalog and then
>> purged after replication is done.  Or to be more up-to-date, can refetch
>> a catalog from any in the list of already up-to-date replica holders.
>>
>> The model is consistent.  Perhaps what needs to happen is for every part
>> of this system to be able to parse and glean information from the
>> catalog.
>>
>> There are system tweaks and optimizations that can be made (Ex:
>> subscribing to be notified for specific entities or doing a general
>> subscription blast.  Refetching latest catalog from source or up-to-date
>> replicas vs holding on to the source you already have - a question of
>> freshness, etc...).  But the model of being catalog centric is
>> consistent and complete.  I think this is the direction we should go in
>> if we want this system to be scalable and provide clean interfacing of
>> the different parts.  Furthermore testing becomes easier because
>> essentially all you would need is a bag-of-catalogs to ingest into your
>> piece.
>>
>>
>> Thanks for putting up with my stream of consciousness. :-)
>> I hope I was cogent.  Feel free to contact me if any additional
>> clarification is required.
>>
>> Gavin.
>>
>> -- 
>> Gavin M. Bell
>> Lawrence Livermore National Labs
>> -- 
>>
>> "Never mistake a clear view for a short distance."
>>                  -Paul Saffo
>>
> 

-- 
Gavin M. Bell
Lawrence Livermore National Labs
--

 "Never mistake a clear view for a short distance."
       	       -Paul Saffo