[Go-essp-tech] CMIP5 publishing to Gateway and P2P nodes

[Estanislao Gonzalez]

Hi Serguei,

we are doing a very similar move, so perhaps it's worth sharing it.

Since the Gateway cannot publish replicas, we had to "unpublish them" 
form it, though we still have the catalogs in place (meaning --thredds 
but no --publish for the esgpublish tool)
So I installed a data node at other machine, cmip3 (I'm using devel, 
since thought there are a few issues, it's stable and is more closely to 
what will end up in "master" soon enough).
I've dumped the complete esgcet DB (database inclusive), drop the one 
installed at cmip3 and ingested the original one.
Then I copied all catalogs to cmip3 and updated the host name in all of 
them via sed (that's because we have gridFTP endpoint with the hostname 
encoded in each catalog)
I removed out own catalogs, but that's not something you'll be doing I 
guess. If you need to remove multiple datasets tell me because I can 
give you a hint on how to do that faster.

So now we have a duplicate of the original node, from which I will 
remove all replicas afterwards to gain more memory space. cmip3 has all 
the metrics from the original node and I might have to replace some 
tables that have the hostname encoded in them (not sure, haven't found 
any, but I guess the metrics do have the complete URL... have to check)

The bad is that as it is, the node is not working with people not having 
the CMIP5 membership already... but it wasn't working before either 
(previously you would have seen a 403, now you get a 500 because of some 
issues we are working on).

After that you can publish to whatever you want, but if you have a 
second node, I'll suggest you publish to pcmdi9. You don't need any idp 
nor index, though as Luca said, the complete installation is the most 
tested one, so you migh want to install a complete stack, though I 
recommend you to care for the data part only.

I'll definitely suggest you to have to nodes running until everything is 
setup. It will lower the pressure of having it on-line if anything goes 
wrong (though it shouldn't).

Well, hope that's useful to you.

Thanks,
Estani




Am 10.02.2012 19:07, schrieb Serguei Nikonov:
> Hi Bob, Gavin.
>
> We are going to upgrade data node to P2P version here in GFDL and before that we
> need to clear up some questions:
> - first of all, what the robust and working version is and when it will be
>     released if it's not yet?
> - after installation, should our publications be indexed on our own node or
>     do we still "publish to PCMDI"?
> - Is it possible to install it without republishing datasets we have?
> - For avoiding time when system will be out of operation we would like to run
>     2 systems (old Data Node and P2P) in parallel for some period of time till
>     all tweaks and tuning will be done and new system will be stable. Is it
>     possible to have to data nodes providing the same data in Federation?
>
>
> Other question is about status of pcmdi3 - was it down on Feb 8? I could not
> access to any dataset that day - neither GFDL's, nor others centers. Also after
> that day all GFDL datasets are not downlodable from thredds catalog
> (e.g.https://esgdata.gfdl.noaa.gov/thredds/esgcet/1/cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.day.atmos.day.r1i1p1.v20110601.html?dataset=cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.day.atmos.day.r1i1p1.v20110601.pr_day_GFDL-CM3_historical_r1i1p1_18650101-18691231.nc).
> But at the same time they are  healthy on pcmdi - searchable, downloadable. I
> recall we had similar situation 2 months ago which you fixed tweaking something
> on pcmdi side.
>
> Thanks,
> Sergey,
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech


-- 
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  gonzalez at dkrz.de