[Go-essp-tech] network status and why replication matters

[Michael Lautenschlager]

Hi all,

I support Bryan's replication arguments in all points and would like to add the political dimension. At least for the European core data archives BADC and WDCC it is essential to be visible from begin on of the CMIP5 data archive.

In order to keep technical problems solvable we should define the first order replicated subset of data as those variables which are most relevant for the IPCC process. Another boundary condition will be the available network bandwidth for replication.

Best wishes, Michael

PS. I am sorry but I wiil be not at the call today becaus I am sitting in a train back to Hamburg.

Michael Lautenschlager (DKRZ/WDCC). Mail sent by PDA.

----- Ursprüngliche Nachricht -----
Von: Bryan Lawrence <bryan.lawrence at stfc.ac.uk>
Gesendet: Dienstag, 23. März 2010 10:33
An: Dean Williams <williams13 at llnl.gov>
Cc: go-essp-tech at ucar.edu
Betreff: [Go-essp-tech] network status and why replication matters

Hi Dean
cc: all

Just to let you know we haven't dropped our efforts on light paths and to 
address the replication issue.

We had most of a draft for a bid for light paths put together, but I 
believe it would be pointless to make a case for light paths until we 
understand the existing bandwidth bottlenecks etc, so that's what we've 
got underway.

I'm given to understand that an es.net engineer believes part of the 
bottle neck between you and us is in the Geant network, and is 
investigating. Additionally, we have found some small issues with our 
local 10G network, which have negligible impact locally, but with long 
latency (ie. to you), cause major problems with bandwidth. We're on the 
case with that issue.

You will note that part of our justification for these links is that to 
maintain synchronisation of a 1 PB archive at 1% daily, requires 10 TB 
per day traffic, which is doable with a 1 Gbit/s light path ... but 
probabably not on production networks day in day out (yet to be 
deterimined). Unlike Karl, I think we do *need* our core archives to be 
synchronised to the highest level possible, otherwise all the users will 
gravitate to one place and overload both servers and international 
network links. 

To me, that makes replication a big deal, and also suggests that at 
least for DKRZ, BADC and PCMDI, we need to have software systems in 
place that 
a) understand what the core data that shoudl be the same is, and
b) attempt to keep it synchronised.
I suspect that other players may feel the same.

I have no faith that a manual  "we all choose what data to move to our 
data centres" would achieve anything other than a disjointed service to 
users. Yes, the catalogs can show where everything is, but users will 
not want to generate multiple wget scripts at multiple sites, they'll 
want one script to execute for each download task. I think the 
complexity of getting wget to know which site is "closest" and 
generating cross-site downloads will be too hard to manage.

Cheers
Bryan

-- 
Bryan Lawrence
Director of Environmental Archival and Associated Research
(NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
STFC, Rutherford Appleton Laboratory
Phone +44 1235 445012; Fax ... 5848; 
Web: home.badc.rl.ac.uk/lawrence
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech