[Go-essp-tech] Bulk data moving and the UNIDATA LDM

[martin.juckes at stfc.ac.uk]

Thanks Don, that would be interesting to see. The TIGGE data transfers
are the same scale as we will have to deal with, nso the figures will be
very relevant.

I was interested in LDM because it appears to offer a lot of support for
data management which might significantly reduce the amount of coding
and design we need to do ourselves -- relative to what would be
necessary with GridFTP. 

Regards,
Martin

> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
> bounces at ucar.edu] On Behalf Of Don Middleton
> Sent: 04 June 2010 16:29
> To: Lawrence, Bryan (STFC,RAL,SSTD)
> Cc: go-essp-tech at ucar.edu
> Subject: Re: [Go-essp-tech] Bulk data moving and the UNIDATA LDM
> 
> We're using LDM for TIGGE, and replicating a couple of terabytes of
> forecast data a week, around the world. I'll inquire about filesizes
> and rates.
> 
> don
> 
> 
> On Jun 4, 2010, at 9:15 AM, Bryan Lawrence wrote:
> 
> >
> >
> > My understanding is that it doesn't compare on a file by file basis:
> > GridFTP is clearly optomised to move big files fast. However, if we
> > are
> > moving more than dozens of files (as we are), then as I understand
> it,
> > LDM would open multiple file transfer streams, so GridFTP's
advantage
> > will boil down to the (not inconsiderable) negotiated window size.
> >
> > Someone else ought to be able to give much better information than
> > that
> > :-)
> >
> > Cheers
> > Bryan
> >
> >
> > On Friday 04 Jun 2010 15:58:33 Alex Sim wrote:
> >> Can you tell us  about the wide-area transfer performance with
> >> Unidata LDM compared to GridFTP server based transfers?
> >>
> >>
> >> -- Alex
> >>
> >> On 6/4/10 3:39 AM, martin.juckes at stfc.ac.uk wrote:
> >>
> >> Hello all,
> >>
> >>
> >>
> >> There may be a simple answer to this, but is there a reason why we
> >> shouldn't use the Unidata Local Data Manager (LDM) for bulk data
> >> movement within the CMIP5 distributed archive? It appears to have a
> >> lot of good features built in to verify success of data transfer
> >> between sites, and runs successfully at many operational sites.
This
> >> would simplify the work flow, since all the complexity of the site
> >> to site transfers would be dealt with by a tried and tested system,
> >>
> >>
> >>
> >> Cheers,
> >>
> >> Martin
> >>
> >
> > --
> > Bryan Lawrence
> > Director of Environmental Archival and Associated Research
> > (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
> > STFC, Rutherford Appleton Laboratory
> > Phone +44 1235 445012; Fax ... 5848;
> > Web: home.badc.rl.ac.uk/lawrence
> > _______________________________________________
> > GO-ESSP-TECH mailing list
> > GO-ESSP-TECH at ucar.edu
> > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> 
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
-- 
Scanned by iCritical.