[Go-essp-tech] Status of Gateway 2.0

[Cinquini, Luca (3880)]

Hi Jennifer,
thanks for the feedback.... I can answer your questions about the P2P system, see below, and thanks for giving it a try:

On Dec 7, 2011, at 7:14 AM, Jennifer Adams wrote:

Dear Colleagues,
Here is more feedback on the prototype of Gateway 2.0. My current use case is searching with a fixed experiment, realm, frequency, and a selected subset of variables. I'm delighted that you can select more than one variable and it will keep adding them to the list of selected results. THANK YOU for listing the variables alphabetically by short name ("Variable Name"); the listing by long name ("CF Variable") is not useful to me, it is not alphabetized, and some names are so long they wrap so it looks like there are three variables called "Flux in the Atmosphere", "Due to Natural", and "Sources". Oops, my bad, it's actually "Surface Carbon Mass Flux into the Atmosphere Due to Natural Sources". Heh. QED.

When I'm done clicking on all my desired constraints, I get a URL that looks like this :
http://search-esg.prototype.ucar.edu/search?Type=Dataset&Experiment=decadal1960&Frequency=mon&Realm=atmos&Variable+Name=clt&Variable+Name=prw
That is great, because it means I can take a short cut from all the mouse clicking by building a URL that will create my desired collection.

I am happy with the search results, so I click on the "Download Selected" button, and get a wget script called "script.wget".

1. It would be great if you could give the script a name that reflects the search constraints used to build that script -- something like
    wget.cmip5.decadal1960.mon.atmos.clt+prw.sh
Every time I download a new wget script, I have to rename it to something meaningful (such as the above) because the very next search result will overwrite script.wget, or give me "script(1).wget" instead, and I'll have no simple way to know what data set that wget script is configured to bring back. Can you PLEASE give the wget script a relevant name for me so I can skip this time-consuming step?

We already talked about this, and Estani (the author of the wget script) was looking for a good model to name the file... Maybe you and I can decide on a good naming convention ? The query parameters are also embedded in the script itself.

2. Is there a way to use our favorite tool (wget) or curl to download the wget script so I can bypass the Gateway 2.0 browser interface altogether? My guess is the answer is no, because of the elaborate login process with OpenID and username/password, but it's worth asking…

-----------

As an alternative, I have looked carefully at the wiki page (and its embedded links) on automated generation of wget scripts, http://www.esgf.org/wiki/ESGF_scripting. This is very promising, and Luca assures me that the P2P service will be working in a matter of weeks.

I think the syntax of the URL for the same search as my example above would be:
http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt,prw

1. Is the comma-delimited list of desired variables legitimate syntax?

actually, you would use variable=clt&variable=prw to get the OR of the two constraints (this is a new feature, documented in http://www.esgf.org/wiki/ESGF_Search_Service)
2. Will it work the way the Gateway 2.0 does to deliver my subset of variables?

Not quite sure what the gateway does, but as the documentation explains, facets with different names are combined in AND, facets with the same name in OR. So my guess is yes :)

3. Will "http://esg-datanode.jpl.nasa.gov<http://esg-datanode.jpl.nasa.gov/>" be the only location providing this service?

No, you could start from any of the P2P nodes and obtain exactly the same result
4. Will the delivered wget script use certificates and checksums?

Yes, all thanks to Estani - assuming the datasets are published with checksums at the TDSs..

thanks again,

Luca


Since this isn't fully operational at the moment, I can't say for sure whether this will be the better route for reaching my goal of acquiring CMIP5 data using only scripts and no browser.

Respectfully submitted,
Jennifer




--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma at cola.iges.org<mailto:jma at cola.iges.org>



_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111207/ef8f717d/attachment.html