[Rda-users] NCAR Research Data Archive Blog

[NCAR Research Data Archive Blog]

NCAR Research Data Archive Blog

///////////////////////////////////////////
Transferring RDA Data with Globus

Posted: 11 Jun 2015 01:17 PM PDT
http://feedproxy.google.com/~r/NcarResearchDataArchiveBlog/~3/rDJuX3uaNq8/transferring-rda-data-with-globus.html?utm_source=feedburner&utm_medium=email

The RDA supports many different data access pathways for our users, and it  
should come as no surprise that the most popular method is web downloads  
via HTTP.  With HTTP, our users can download data files directly from our  
website, or via cURL and wget commands.

The popularity of web downloads from the RDA has been growing exponentially  
in recent years.  In 2014 alone, we served more than 1.1 petabytes of data  
to over 11,000 users. We fulfilled more than 45,000 customized data  
requests for our users, the majority (if not all) of which were downloaded  
via HTTP.

In order to keep up with this fast growth in data usage, the RDA has  
developed the capability for its users to transfer data via the Globus data  
transfer service.

What is Globus?Globus (globus.org) can be described in the simplest terms  
as a third-party data transfer service, and provides a fast, secure, and  
reliable method for transferring large data volumes.  Data transfers are  
carried out using the GridFTP protocol and are facilitated by a Globus  
Connect server running on an IBM cluster housed at NCAR.

Step-by-step instructions on how to create a Globus account and transfer  
files are available on the Globus website.  To initiate a transfer, a user  
defines two endpoints—one origination endpoint and one receiving  
endpoint—and then submits the transfer task to Globus.  Once a data  
transfer job has been started, Globus takes care of the rest, including  
processing the transfer and monitoring its performance and progress.  After  
the transfer completes, Globus sends you an e-mail notification with a  
report detailing the transfer statistics and performance.

Users can initiate and manage Globus data transfers from their web browser;  
more savvy users who wish to integrate their data transfers into their  
automated workflows can use the Globus command line interface (CLI) or  
Python/Java API client.

How to transfer RDA data using GlobusStarting from any RDA dataset  
description page (see, for example, the 20th Century Reanalysis dataset  
page), select the Data Access tab, then select the link labeled 'Request  
Globus Invitation'.  Globus transfers can also be requested for customized  
data requests by selecting the 'Globus Download' button on the data  
download page.

Example Data Access matrix from a dataset page on the RDA website.   
Selecting the link labeled 'Request Globus Invitation' initiates a Globus  
data share for the dataset.
At this point, a pop-up window will appear and you will be asked to confirm  
that you are requesting a Globus invitation to transfer the data from the  
dataset.  Once you have submitted the request, Globus will send you an  
e-mail invitation containing a unique URL to accept the data share  
invitation.  After accepting the invitation, you may then begin  
transferring data to your receiving endpoint via the Globus website.

The data transfer interface on the Globus website.  The RDA shared endpoint  
is shown on the left and the user's receiving endpoint is displayed on the  
right.
Alternate Identity loginTo use the Globus service, users must register for  
a (free) Globus account.  If you do not have one, Globus will prompt you to  
register for an account prior to accepting the data share and transferring  
data. After you have a Globus account, you may then link your RDA and  
Globus accounts under your Globus user settings (after logging in, select  
Manage Identities –> Add linked identity –> Add single sign-on identity).   
Doing so will allow you to log into Globus using your RDA user e-mail and  
password, thus requiring you to remember only one username/password  
combination.


The Globus alternate identity provider login interface.  RDA users may log  
into their Globus account using their RDA user e-mail and password by  
selecting the 'NCAR RDA' identity provider.Data endpoint managementAll  
Globus data shares created for our users are listed under each  
user's 'dashboard' (select 'Dashboard' at the top of the RDA home page).   
Here, users can view and manage all active Globus shares assigned to their  
account.  Data shares can be deleted; or, if a user has misplaced the  
Globus e-mail invitation for a data share, a new one can be sent.
A listing of Globus data shares, displayed in the RDA user's dashboard.



--
You are subscribed to email updates from "NCAR Research Data Archive Blog."
To stop receiving these emails, you may unsubscribe now:  
https://feedburner.google.com/fb/a/mailunsubscribe?k=J0HJCpZ3cfcfvRzzDcG9piIXHIs

Email delivery powered by Google.
Google Inc., 1600 Amphitheatre Parkway, Mountain View, CA 94043, United  
States

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/rda-users/attachments/20150612/4be6cb8a/attachment.html