[Go-essp-tech] Bulk data moving and the UNIDATA LDM

[Alex Sim]

GridFTP works well with many small files too.  ESG/Globus had
Supercomputing bandwidth challenge demo last year with CMIP-3 data.

-- Alex


On 6/4/10 8:15 AM, Bryan Lawrence wrote:
>
> My understanding is that it doesn't compare on a file by file basis: 
> GridFTP is clearly optomised to move big files fast. However, if we are 
> moving more than dozens of files (as we are), then as I understand it, 
> LDM would open multiple file transfer streams, so GridFTP's advantage 
> will boil down to the (not inconsiderable) negotiated window size.
>
> Someone else ought to be able to give much better information than that 
> :-)
>
> Cheers
> Bryan
>
>
> On Friday 04 Jun 2010 15:58:33 Alex Sim wrote:
>   
>> Can you tell us  about the wide-area transfer performance with
>>  Unidata LDM compared to GridFTP server based transfers?
>>
>>
>> -- Alex
>>
>> On 6/4/10 3:39 AM, martin.juckes at stfc.ac.uk wrote:
>>
>> Hello all,
>>
>>
>>
>> There may be a simple answer to this, but is there a reason why we
>>  shouldn't use the Unidata Local Data Manager (LDM) for bulk data
>>  movement within the CMIP5 distributed archive? It appears to have a
>>  lot of good features built in to verify success of data transfer
>>  between sites, and runs successfully at many operational sites. This
>>  would simplify the work flow, since all the complexity of the site
>>  to site transfers would be dealt with by a tried and tested system,
>>
>>
>>
>> Cheers,
>>
>> Martin
>>
>>     
>