[Go-essp-tech] Bulk data moving and the UNIDATA LDM

Thu Jun 10 01:50:50 MDT 2010

Hello Rachana,

Thanks for those useful examples. My main interest with LDM was the fire-and-forget feature, which simplifies management of a data-stream.  The phrase I used, "verify success", was not really accurate -- what I should of said was guarantee success. 

It sounds as though I also got a mis-leading view of the maturity of globus.org from the web sites I found on the topic -- my apologies for that.

On a slightly different topic, a quick look at the links you sent appears to reflect similar problems with long range transfers to ones we are encountering. We have also been doing trials with GridFTP looking at transfer rates between here (southern England) and France, Germany and the US. Transfer rates to the US are substantially slower, reflecting high package loss rates. It looks as though there is an important distinction to be made between wide area networks within Europe or within the US, for which very high transfer rates can be achieved, and global networks, for which package loss appears to seriously throttle transfer rates. This problem is reflected in the first link you sent, with high resend rates to Australia compared with very small number on trials with larger files in the US.

This is an important issue for us because we need to move a lot of data around the world and the current transfer rates we are getting are not likely to be sufficient. Do you know of any work on assessing and alleviating package loss on intercontinental transfers? (I'm not expecting LDM to provide any improvement in this area -- but it would be a very welcome surprise if it did).

regards,
Martin

-----Original Message-----
From: Rachana Ananthakrishnan [mailto:ranantha at mcs.anl.gov]
Sent: Thu 10/06/2010 03:46
To: Juckes, Martin (STFC,RAL,SSTD)
Cc: don at ucar.edu; go-essp-tech at ucar.edu
Subject: Re: [Go-essp-tech] Bulk data moving and the UNIDATA LDM

Hi,

I wanted to address some points raised on this thread, and also  
provide pointers to some relevant material.

On the GridFTP transfer with small files, there has been significant  
work to improve performance with small files. Here is a reference that  
explains this work: http://www.mcs.anl.gov/~kettimut/publications/Pipelining.pdf 
. Is the concern of using GridFTP for small files a performance issue  
or a reliability issue?

It is hard to compare performance without hard numbers, but taking  
this thread back to our GridFTP team, here are some questions that  
were raised:

  - Checksums at end of transfer is supported by the GridFTP client  
libraries. What additional features are covered by the statement "LDM  
appears to have a lot of good features built in to verify success of  
data transfer between sites".
- GridFTP supports both concurrent transfer of multiple files and  
concurrent transfer of multiple chunks within a file. This combination  
can outperform just concurrent transfer of multiple files on high  
latency links. Is there comparable feature with LDM?
- What security model does LDM support?

Regarding globus.org, it is built around GridFTP server and client  
technology to provide the fire and forget mode for file transfers.  
Like Pauline mentions, it at the minimum, provides reliability, and  
restarts with the transfers. It is provided as a Software as a Service  
solution around GridFTP software that is both mature and robust, and  
will be provided as a hosted solution operated by us (Argonne National  
Lab in the US) for user communities. We are working with individual  
communities to do performance and reliability testing, and to gather  
specific requirements from them. Integrity check is one of the  
features in the pipeline. Some references to performance numbers thus  
far:

Transfer managed from a site in US to Australia: http://www.mcs.anl.gov/~childers/VPAC/
Transfer test for tutorials: http://www.mcs.anl.gov/~childers/GlobusWorld2010/results.html

Alex referenced a Super Computing Bandwidth Challenge work that used  
globus.org to replicate 10TB of CMIP3 data between two sites in the  
US: http://www.mcs.anl.gov/~kettimut/publications/HPDC10_BWC_Final.pdf.

If there is of interest in this group, like Pauline suggested, we  
could have someone from our team present about the service to this  
group. We are certainly interested to hear experiences with LDM, and  
experiences on using both these solutions.

Thanks,
Rachana

On Jun 7, 2010, at 12:04 PM, <martin.juckes at stfc.ac.uk> <martin.juckes at stfc.ac.uk 
 > wrote:

> Hello Don, Pauline,
>
> I had a quick look at globus.org -- it looks good but is only  
> available
> as a beta release, whereas LDM is on version 6.8. Having a tried and
> tested bit of software to handle a key part of our data management  
> would
> make life a lot easier.
>
> Form Don's message, it appears that LDM does not have GridFTP's  
> ability
> to deal with large files. So, we have a choice between LDM which
> supports a "fire-and-forget" approach but needs small files or  
> GridFTP,
> which will deal with small files but requires additional software to
> determine whether transfer succeeded.
>
> It looks to me as though LDM has a big advantage in terms of  
> maturity. I
> think the overhead of having to split and rejoin files is likely to be
> significantly less problematic than getting reliable site to site
> communication about thousands of files being transferred.
>
> Cheers,
> Martin
>
>> -----Original Message-----
>> From: Don Middleton [mailto:don at ucar.edu]
>> Sent: 07 June 2010 13:11
>> To: Juckes, Martin (STFC,RAL,SSTD)
>> Cc: Don Middleton; Lawrence, Bryan (STFC,RAL,SSTD); go-essp-
>> tech at ucar.edu
>> Subject: Re: [Go-essp-tech] Bulk data moving and the UNIDATA LDM
>>
>> I don't have figures, but some general information. In the TIGGE
>> context, LDM is dealing with pretty small packages: 2D grids, which
>> can be reassembled at the receiving end into individual 3D fields for
>> each timestep and forecast. Cost of transmission failure for large
>> files is reported to be high, but LDM does have features to support
>> retry until success. I don't know how difficult it might be to  
>> replace
>> LDM's transport layer with GridFTP, or even if it makes any sense to
>> think about that, particularly given Pauline's message. LDM does
>> appear to be working quite well in an operational context.
>>
>> don
>>
>>
>> On Jun 7, 2010, at 1:50 AM, <martin.juckes at stfc.ac.uk>
>> <martin.juckes at stfc.ac.uk
>>> wrote:
>>
>>> Thanks Don, that would be interesting to see. The TIGGE data
>> transfers
>>> are the same scale as we will have to deal with, nso the figures
>>> will be
>>> very relevant.
>>>
>>> I was interested in LDM because it appears to offer a lot of support
>>> for
>>> data management which might significantly reduce the amount of
> coding
>>> and design we need to do ourselves -- relative to what would be
>>> necessary with GridFTP.
>>>
>>> Regards,
>>> Martin
>>>
>>>> -----Original Message-----
>>>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
>>>> bounces at ucar.edu] On Behalf Of Don Middleton
>>>> Sent: 04 June 2010 16:29
>>>> To: Lawrence, Bryan (STFC,RAL,SSTD)
>>>> Cc: go-essp-tech at ucar.edu
>>>> Subject: Re: [Go-essp-tech] Bulk data moving and the UNIDATA LDM
>>>>
>>>> We're using LDM for TIGGE, and replicating a couple of terabytes of
>>>> forecast data a week, around the world. I'll inquire about
> filesizes
>>>> and rates.
>>>>
>>>> don
>>>>
>>>>
>>>> On Jun 4, 2010, at 9:15 AM, Bryan Lawrence wrote:
>>>>
>>>>>
>>>>>
>>>>> My understanding is that it doesn't compare on a file by file
>> basis:
>>>>> GridFTP is clearly optomised to move big files fast. However, if
> we
>>>>> are
>>>>> moving more than dozens of files (as we are), then as I understand
>>>> it,
>>>>> LDM would open multiple file transfer streams, so GridFTP's
>>> advantage
>>>>> will boil down to the (not inconsiderable) negotiated window size.
>>>>>
>>>>> Someone else ought to be able to give much better information than
>>>>> that
>>>>> :-)
>>>>>
>>>>> Cheers
>>>>> Bryan
>>>>>
>>>>>
>>>>> On Friday 04 Jun 2010 15:58:33 Alex Sim wrote:
>>>>>> Can you tell us  about the wide-area transfer performance with
>>>>>> Unidata LDM compared to GridFTP server based transfers?
>>>>>>
>>>>>>
>>>>>> -- Alex
>>>>>>
>>>>>> On 6/4/10 3:39 AM, martin.juckes at stfc.ac.uk wrote:
>>>>>>
>>>>>> Hello all,
>>>>>>
>>>>>>
>>>>>>
>>>>>> There may be a simple answer to this, but is there a reason why
> we
>>>>>> shouldn't use the Unidata Local Data Manager (LDM) for bulk data
>>>>>> movement within the CMIP5 distributed archive? It appears to have
>> a
>>>>>> lot of good features built in to verify success of data transfer
>>>>>> between sites, and runs successfully at many operational sites.
>>> This
>>>>>> would simplify the work flow, since all the complexity of the
> site
>>>>>> to site transfers would be dealt with by a tried and tested
>> system,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>
>>>>> --
>>>>> Bryan Lawrence
>>>>> Director of Environmental Archival and Associated Research
>>>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>>>> STFC, Rutherford Appleton Laboratory
>>>>> Phone +44 1235 445012; Fax ... 5848;
>>>>> Web: home.badc.rl.ac.uk/lawrence
>>>>> _______________________________________________
>>>>> GO-ESSP-TECH mailing list
>>>>> GO-ESSP-TECH at ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>> --
>>> Scanned by iCritical.
>
> -- 
> Scanned by iCritical.
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

Rachana Ananthakrishnan
Argonne National Lab | University of Chicago

-- 
Scanned by iCritical.