[Go-essp-tech] network status and why replication matters
Bryan Lawrence
bryan.lawrence at stfc.ac.uk
Tue Mar 23 03:33:52 MDT 2010
Hi Dean
cc: all
Just to let you know we haven't dropped our efforts on light paths and to
address the replication issue.
We had most of a draft for a bid for light paths put together, but I
believe it would be pointless to make a case for light paths until we
understand the existing bandwidth bottlenecks etc, so that's what we've
got underway.
I'm given to understand that an es.net engineer believes part of the
bottle neck between you and us is in the Geant network, and is
investigating. Additionally, we have found some small issues with our
local 10G network, which have negligible impact locally, but with long
latency (ie. to you), cause major problems with bandwidth. We're on the
case with that issue.
You will note that part of our justification for these links is that to
maintain synchronisation of a 1 PB archive at 1% daily, requires 10 TB
per day traffic, which is doable with a 1 Gbit/s light path ... but
probabably not on production networks day in day out (yet to be
deterimined). Unlike Karl, I think we do *need* our core archives to be
synchronised to the highest level possible, otherwise all the users will
gravitate to one place and overload both servers and international
network links.
To me, that makes replication a big deal, and also suggests that at
least for DKRZ, BADC and PCMDI, we need to have software systems in
place that
a) understand what the core data that shoudl be the same is, and
b) attempt to keep it synchronised.
I suspect that other players may feel the same.
I have no faith that a manual "we all choose what data to move to our
data centres" would achieve anything other than a disjointed service to
users. Yes, the catalogs can show where everything is, but users will
not want to generate multiple wget scripts at multiple sites, they'll
want one script to execute for each download task. I think the
complexity of getting wget to know which site is "closest" and
generating cross-site downloads will be too hard to manage.
Cheers
Bryan
--
Bryan Lawrence
Director of Environmental Archival and Associated Research
(NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
STFC, Rutherford Appleton Laboratory
Phone +44 1235 445012; Fax ... 5848;
Web: home.badc.rl.ac.uk/lawrence
More information about the GO-ESSP-TECH
mailing list