[Go-essp-tech] Bulk data moving and the UNIDATA LDM

[Pauline Mak]

Hi all,

I'm not sure what the requirements for the transfers are, but we've been 
doing a bit of work with globus lately:

We've been experimenting on the latest version of the GridFTP server 
(version 5.0.2 or something like that).  The new client offers new 
functionality like globus-url-sync that syncs destination and source 
directories.

We've also had a look at the globus.org transfer system too.  The good 
thing about globus.org is that it's more than just transfers.  I think 
(but don't quote me!) it will resume/retransmit failed transfers.  One 
can also monitor progress of files in transit.  It is currently a hosted 
server at globus, but I think there are plans for other users to install 
the management server.

If you have any questions, I'm more than happy to pass this onto our 
GridFTP expert (although, he's away this week), or can get him to join 
the discussion on the list.

Cheers,

-Pauline.

martin.juckes at stfc.ac.uk wrote:
> Thanks Don, that would be interesting to see. The TIGGE data transfers
> are the same scale as we will have to deal with, nso the figures will be
> very relevant.
> 
> I was interested in LDM because it appears to offer a lot of support for
> data management which might significantly reduce the amount of coding
> and design we need to do ourselves -- relative to what would be
> necessary with GridFTP. 
> 
> Regards,
> Martin
> 
>> -----Original Message-----
>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
>> bounces at ucar.edu] On Behalf Of Don Middleton
>> Sent: 04 June 2010 16:29
>> To: Lawrence, Bryan (STFC,RAL,SSTD)
>> Cc: go-essp-tech at ucar.edu
>> Subject: Re: [Go-essp-tech] Bulk data moving and the UNIDATA LDM
>>
>> We're using LDM for TIGGE, and replicating a couple of terabytes of
>> forecast data a week, around the world. I'll inquire about filesizes
>> and rates.
>>
>> don
>>
>>
>> On Jun 4, 2010, at 9:15 AM, Bryan Lawrence wrote:
>>
>>>
>>> My understanding is that it doesn't compare on a file by file basis:
>>> GridFTP is clearly optomised to move big files fast. However, if we
>>> are
>>> moving more than dozens of files (as we are), then as I understand
>> it,
>>> LDM would open multiple file transfer streams, so GridFTP's
> advantage
>>> will boil down to the (not inconsiderable) negotiated window size.
>>>
>>> Someone else ought to be able to give much better information than
>>> that
>>> :-)
>>>
>>> Cheers
>>> Bryan
>>>
>>>
>>> On Friday 04 Jun 2010 15:58:33 Alex Sim wrote:
>>>> Can you tell us  about the wide-area transfer performance with
>>>> Unidata LDM compared to GridFTP server based transfers?
>>>>
>>>>
>>>> -- Alex
>>>>
>>>> On 6/4/10 3:39 AM, martin.juckes at stfc.ac.uk wrote:
>>>>
>>>> Hello all,
>>>>
>>>>
>>>>
>>>> There may be a simple answer to this, but is there a reason why we
>>>> shouldn't use the Unidata Local Data Manager (LDM) for bulk data
>>>> movement within the CMIP5 distributed archive? It appears to have a
>>>> lot of good features built in to verify success of data transfer
>>>> between sites, and runs successfully at many operational sites.
> This
>>>> would simplify the work flow, since all the complexity of the site
>>>> to site transfers would be dealt with by a tried and tested system,
>>>>
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Martin
>>>>
>>> --
>>> Bryan Lawrence
>>> Director of Environmental Archival and Associated Research
>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>> STFC, Rutherford Appleton Laboratory
>>> Phone +44 1235 445012; Fax ... 5848;
>>> Web: home.badc.rl.ac.uk/lawrence
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

-- 
Pauline Mak

Assistant Manager, ARCS Data Services
Ph:  +61 3 6226 7518
Mob: +61 411 638 196
Email: pauline.mak at arcs.org.au
Jabber: pauline.mak at arcs.org.au
Calendar: http://tinyurl.com/pmak-arcs-calendar
http://www.arcs.org.au/

TPAC
Email: pauline.mak at utas.edu.au
http://www.tpac.org.au/