[Go-essp-tech] Bulk data moving and the UNIDATA LDM
Alex Sim
asim at lbl.gov
Fri Jun 4 09:25:00 MDT 2010
GridFTP works well with many small files too. ESG/Globus had
Supercomputing bandwidth challenge demo last year with CMIP-3 data.
-- Alex
On 6/4/10 8:15 AM, Bryan Lawrence wrote:
>
> My understanding is that it doesn't compare on a file by file basis:
> GridFTP is clearly optomised to move big files fast. However, if we are
> moving more than dozens of files (as we are), then as I understand it,
> LDM would open multiple file transfer streams, so GridFTP's advantage
> will boil down to the (not inconsiderable) negotiated window size.
>
> Someone else ought to be able to give much better information than that
> :-)
>
> Cheers
> Bryan
>
>
> On Friday 04 Jun 2010 15:58:33 Alex Sim wrote:
>
>> Can you tell us about the wide-area transfer performance with
>> Unidata LDM compared to GridFTP server based transfers?
>>
>>
>> -- Alex
>>
>> On 6/4/10 3:39 AM, martin.juckes at stfc.ac.uk wrote:
>>
>> Hello all,
>>
>>
>>
>> There may be a simple answer to this, but is there a reason why we
>> shouldn't use the Unidata Local Data Manager (LDM) for bulk data
>> movement within the CMIP5 distributed archive? It appears to have a
>> lot of good features built in to verify success of data transfer
>> between sites, and runs successfully at many operational sites. This
>> would simplify the work flow, since all the complexity of the site
>> to site transfers would be dealt with by a tried and tested system,
>>
>>
>>
>> Cheers,
>>
>> Martin
>>
>>
>
More information about the GO-ESSP-TECH
mailing list