[Go-essp-tech] Bulk data moving and the UNIDATA LDM

[Don Middleton]

I don't have figures, but some general information. In the TIGGE  
context, LDM is dealing with pretty small packages: 2D grids, which  
can be reassembled at the receiving end into individual 3D fields for  
each timestep and forecast. Cost of transmission failure for large  
files is reported to be high, but LDM does have features to support  
retry until success. I don't know how difficult it might be to replace  
LDM's transport layer with GridFTP, or even if it makes any sense to  
think about that, particularly given Pauline's message. LDM does  
appear to be working quite well in an operational context.

don


On Jun 7, 2010, at 1:50 AM, <martin.juckes at stfc.ac.uk> <martin.juckes at stfc.ac.uk 
 > wrote:

> Thanks Don, that would be interesting to see. The TIGGE data transfers
> are the same scale as we will have to deal with, nso the figures  
> will be
> very relevant.
>
> I was interested in LDM because it appears to offer a lot of support  
> for
> data management which might significantly reduce the amount of coding
> and design we need to do ourselves -- relative to what would be
> necessary with GridFTP.
>
> Regards,
> Martin
>
>> -----Original Message-----
>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
>> bounces at ucar.edu] On Behalf Of Don Middleton
>> Sent: 04 June 2010 16:29
>> To: Lawrence, Bryan (STFC,RAL,SSTD)
>> Cc: go-essp-tech at ucar.edu
>> Subject: Re: [Go-essp-tech] Bulk data moving and the UNIDATA LDM
>>
>> We're using LDM for TIGGE, and replicating a couple of terabytes of
>> forecast data a week, around the world. I'll inquire about filesizes
>> and rates.
>>
>> don
>>
>>
>> On Jun 4, 2010, at 9:15 AM, Bryan Lawrence wrote:
>>
>>>
>>>
>>> My understanding is that it doesn't compare on a file by file basis:
>>> GridFTP is clearly optomised to move big files fast. However, if we
>>> are
>>> moving more than dozens of files (as we are), then as I understand
>> it,
>>> LDM would open multiple file transfer streams, so GridFTP's
> advantage
>>> will boil down to the (not inconsiderable) negotiated window size.
>>>
>>> Someone else ought to be able to give much better information than
>>> that
>>> :-)
>>>
>>> Cheers
>>> Bryan
>>>
>>>
>>> On Friday 04 Jun 2010 15:58:33 Alex Sim wrote:
>>>> Can you tell us  about the wide-area transfer performance with
>>>> Unidata LDM compared to GridFTP server based transfers?
>>>>
>>>>
>>>> -- Alex
>>>>
>>>> On 6/4/10 3:39 AM, martin.juckes at stfc.ac.uk wrote:
>>>>
>>>> Hello all,
>>>>
>>>>
>>>>
>>>> There may be a simple answer to this, but is there a reason why we
>>>> shouldn't use the Unidata Local Data Manager (LDM) for bulk data
>>>> movement within the CMIP5 distributed archive? It appears to have a
>>>> lot of good features built in to verify success of data transfer
>>>> between sites, and runs successfully at many operational sites.
> This
>>>> would simplify the work flow, since all the complexity of the site
>>>> to site transfers would be dealt with by a tried and tested system,
>>>>
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Martin
>>>>
>>>
>>> --
>>> Bryan Lawrence
>>> Director of Environmental Archival and Associated Research
>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>> STFC, Rutherford Appleton Laboratory
>>> Phone +44 1235 445012; Fax ... 5848;
>>> Web: home.badc.rl.ac.uk/lawrence
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> --
> Scanned by iCritical.