[Go-essp-tech] Bulk data moving and the UNIDATA LDM

Mon Jun 7 11:04:44 MDT 2010

Hello Don, Pauline,

I had a quick look at globus.org -- it looks good but is only available
as a beta release, whereas LDM is on version 6.8. Having a tried and
tested bit of software to handle a key part of our data management would
make life a lot easier.

Form Don's message, it appears that LDM does not have GridFTP's ability
to deal with large files. So, we have a choice between LDM which
supports a "fire-and-forget" approach but needs small files or GridFTP,
which will deal with small files but requires additional software to
determine whether transfer succeeded.

It looks to me as though LDM has a big advantage in terms of maturity. I
think the overhead of having to split and rejoin files is likely to be
significantly less problematic than getting reliable site to site
communication about thousands of files being transferred.

Cheers,
Martin

> -----Original Message-----
> From: Don Middleton [mailto:don at ucar.edu]
> Sent: 07 June 2010 13:11
> To: Juckes, Martin (STFC,RAL,SSTD)
> Cc: Don Middleton; Lawrence, Bryan (STFC,RAL,SSTD); go-essp-
> tech at ucar.edu
> Subject: Re: [Go-essp-tech] Bulk data moving and the UNIDATA LDM
> 
> I don't have figures, but some general information. In the TIGGE
> context, LDM is dealing with pretty small packages: 2D grids, which
> can be reassembled at the receiving end into individual 3D fields for
> each timestep and forecast. Cost of transmission failure for large
> files is reported to be high, but LDM does have features to support
> retry until success. I don't know how difficult it might be to replace
> LDM's transport layer with GridFTP, or even if it makes any sense to
> think about that, particularly given Pauline's message. LDM does
> appear to be working quite well in an operational context.
> 
> don
> 
> 
> On Jun 7, 2010, at 1:50 AM, <martin.juckes at stfc.ac.uk>
> <martin.juckes at stfc.ac.uk
>  > wrote:
> 
> > Thanks Don, that would be interesting to see. The TIGGE data
> transfers
> > are the same scale as we will have to deal with, nso the figures
> > will be
> > very relevant.
> >
> > I was interested in LDM because it appears to offer a lot of support
> > for
> > data management which might significantly reduce the amount of
coding
> > and design we need to do ourselves -- relative to what would be
> > necessary with GridFTP.
> >
> > Regards,
> > Martin
> >
> >> -----Original Message-----
> >> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
> >> bounces at ucar.edu] On Behalf Of Don Middleton
> >> Sent: 04 June 2010 16:29
> >> To: Lawrence, Bryan (STFC,RAL,SSTD)
> >> Cc: go-essp-tech at ucar.edu
> >> Subject: Re: [Go-essp-tech] Bulk data moving and the UNIDATA LDM
> >>
> >> We're using LDM for TIGGE, and replicating a couple of terabytes of
> >> forecast data a week, around the world. I'll inquire about
filesizes
> >> and rates.
> >>
> >> don
> >>
> >>
> >> On Jun 4, 2010, at 9:15 AM, Bryan Lawrence wrote:
> >>
> >>>
> >>>
> >>> My understanding is that it doesn't compare on a file by file
> basis:
> >>> GridFTP is clearly optomised to move big files fast. However, if
we
> >>> are
> >>> moving more than dozens of files (as we are), then as I understand
> >> it,
> >>> LDM would open multiple file transfer streams, so GridFTP's
> > advantage
> >>> will boil down to the (not inconsiderable) negotiated window size.
> >>>
> >>> Someone else ought to be able to give much better information than
> >>> that
> >>> :-)
> >>>
> >>> Cheers
> >>> Bryan
> >>>
> >>>
> >>> On Friday 04 Jun 2010 15:58:33 Alex Sim wrote:
> >>>> Can you tell us  about the wide-area transfer performance with
> >>>> Unidata LDM compared to GridFTP server based transfers?
> >>>>
> >>>>
> >>>> -- Alex
> >>>>
> >>>> On 6/4/10 3:39 AM, martin.juckes at stfc.ac.uk wrote:
> >>>>
> >>>> Hello all,
> >>>>
> >>>>
> >>>>
> >>>> There may be a simple answer to this, but is there a reason why
we
> >>>> shouldn't use the Unidata Local Data Manager (LDM) for bulk data
> >>>> movement within the CMIP5 distributed archive? It appears to have
> a
> >>>> lot of good features built in to verify success of data transfer
> >>>> between sites, and runs successfully at many operational sites.
> > This
> >>>> would simplify the work flow, since all the complexity of the
site
> >>>> to site transfers would be dealt with by a tried and tested
> system,
> >>>>
> >>>>
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Martin
> >>>>
> >>>
> >>> --
> >>> Bryan Lawrence
> >>> Director of Environmental Archival and Associated Research
> >>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
> >>> STFC, Rutherford Appleton Laboratory
> >>> Phone +44 1235 445012; Fax ... 5848;
> >>> Web: home.badc.rl.ac.uk/lawrence
> >>> _______________________________________________
> >>> GO-ESSP-TECH mailing list
> >>> GO-ESSP-TECH at ucar.edu
> >>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >>
> >> _______________________________________________
> >> GO-ESSP-TECH mailing list
> >> GO-ESSP-TECH at ucar.edu
> >> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> > --
> > Scanned by iCritical.

-- 
Scanned by iCritical.