[Go-essp-tech] network status and why replication matters

[Dean N. Williams]

Hi Bryan,

	I agree with what you stating below. (We have not deviated from our  
target.) Our network folks (i.e., Jeff Long and many others) are try  
to make this happen from our end, but as you know this is no small  
order. The other part to this that you did not mention is the possible  
cost for this light path. (Yet to be determined.) Replication is next  
up on the development path once the initial release is done, so I  
think we are all still on the same page as to the importance of  
replication.

	I'll talk to you and others later this morning.

Best regards,
	Dean

On Mar 23, 2010, at 2:33 AM, Bryan Lawrence wrote:

> Hi Dean
> cc: all
>
> Just to let you know we haven't dropped our efforts on light paths  
> and to
> address the replication issue.
>
> We had most of a draft for a bid for light paths put together, but I
> believe it would be pointless to make a case for light paths until we
> understand the existing bandwidth bottlenecks etc, so that's what  
> we've
> got underway.
>
> I'm given to understand that an es.net engineer believes part of the
> bottle neck between you and us is in the Geant network, and is
> investigating. Additionally, we have found some small issues with our
> local 10G network, which have negligible impact locally, but with long
> latency (ie. to you), cause major problems with bandwidth. We're on  
> the
> case with that issue.
>
> You will note that part of our justification for these links is that  
> to
> maintain synchronisation of a 1 PB archive at 1% daily, requires 10 TB
> per day traffic, which is doable with a 1 Gbit/s light path ... but
> probabably not on production networks day in day out (yet to be
> deterimined). Unlike Karl, I think we do *need* our core archives to  
> be
> synchronised to the highest level possible, otherwise all the users  
> will
> gravitate to one place and overload both servers and international
> network links.
>
> To me, that makes replication a big deal, and also suggests that at
> least for DKRZ, BADC and PCMDI, we need to have software systems in
> place that
> a) understand what the core data that shoudl be the same is, and
> b) attempt to keep it synchronised.
> I suspect that other players may feel the same.
Absolutely...

>
> I have no faith that a manual  "we all choose what data to move to our
> data centres" would achieve anything other than a disjointed service  
> to
> users. Yes, the catalogs can show where everything is, but users will
> not want to generate multiple wget scripts at multiple sites, they'll
> want one script to execute for each download task. I think the
> complexity of getting wget to know which site is "closest" and
> generating cross-site downloads will be too hard to manage.

>
> Cheers
> Bryan
>
> -- 
> Bryan Lawrence
> Director of Environmental Archival and Associated Research
> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
> STFC, Rutherford Appleton Laboratory
> Phone +44 1235 445012; Fax ... 5848;
> Web: home.badc.rl.ac.uk/lawrence