[Go-essp-tech] Bulk data moving and the UNIDATA LDM

[Don Middleton]

We're using LDM for TIGGE, and replicating a couple of terabytes of  
forecast data a week, around the world. I'll inquire about filesizes  
and rates.

don


On Jun 4, 2010, at 9:15 AM, Bryan Lawrence wrote:

>
>
> My understanding is that it doesn't compare on a file by file basis:
> GridFTP is clearly optomised to move big files fast. However, if we  
> are
> moving more than dozens of files (as we are), then as I understand it,
> LDM would open multiple file transfer streams, so GridFTP's advantage
> will boil down to the (not inconsiderable) negotiated window size.
>
> Someone else ought to be able to give much better information than  
> that
> :-)
>
> Cheers
> Bryan
>
>
> On Friday 04 Jun 2010 15:58:33 Alex Sim wrote:
>> Can you tell us  about the wide-area transfer performance with
>> Unidata LDM compared to GridFTP server based transfers?
>>
>>
>> -- Alex
>>
>> On 6/4/10 3:39 AM, martin.juckes at stfc.ac.uk wrote:
>>
>> Hello all,
>>
>>
>>
>> There may be a simple answer to this, but is there a reason why we
>> shouldn't use the Unidata Local Data Manager (LDM) for bulk data
>> movement within the CMIP5 distributed archive? It appears to have a
>> lot of good features built in to verify success of data transfer
>> between sites, and runs successfully at many operational sites. This
>> would simplify the work flow, since all the complexity of the site
>> to site transfers would be dealt with by a tried and tested system,
>>
>>
>>
>> Cheers,
>>
>> Martin
>>
>
> -- 
> Bryan Lawrence
> Director of Environmental Archival and Associated Research
> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
> STFC, Rutherford Appleton Laboratory
> Phone +44 1235 445012; Fax ... 5848;
> Web: home.badc.rl.ac.uk/lawrence
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech