[Go-essp-tech] Replica support in Gateway 2.0

Thu Nov 10 09:25:50 MST 2011

Hi Estani,

You missed the start -- the bit which is not achievable is publishing a replica to the same gateway used for the original publication of that data. E.g. IPSL data published to BADC,

Cheers,
Martin

> >-----Original Message-----
> >From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
> >bounces at ucar.edu] On Behalf Of Estanislao Gonzalez
> >Sent: 10 November 2011 16:20
> >To: go-essp-tech at ucar.edu; esg-gateway-dev at earthsystemgrid.org
> >Subject: Re: [Go-essp-tech] Replica support in Gateway 2.0
> >
> >Hi,
> >
> >this analogy seams perfect. Now regarding to the options we have at
> >the
> >moment:
> >1) How "unique" is the dataset id in a Gateway? Federation wide, local
> >gAteway or project unique?
> >2) Depending on one, the procedure could involve "moving" the
> >published
> >data to some other project, Gateway, Federation :-)
> >
> >I think this could be achievable:
> >1) Data gets replicated to some Gateway (redundancy enforced)
> >2) The originated Gateway, if it's also replicating, should replicate
> >(just data, no publication yet) from the QC checked replica.
> >3) The "pre-print" gets removed (which either mean move to a different
> >project, Gateway, etc or really completely delete it from the Gateway)
> >4) The replica gets published.
> >
> >I might be omitting something, but it seams achievable right now.
> >
> >My 2c,
> >Estani
> >
> >Am 10.11.2011 07:15, schrieb Bryan Lawrence:
> >> Martin has been quite vociferous (quite rightly) in personal email
> >to me that as far as QC goes, the dataset which gets through QC2 will
> >*not* be the original dataset - we have no control over the original
> >dataset's permanence and/or immutability.
> >>
> >> This raises some interesting issues about the role of ESGF ... and
> >it's interaction with the data owner and the publication process which
> >is governed by DKRZ as the Publisher (and in the future probably
> >multiple publication processes and multiple Publishers). The correct
> >analogy here, as I said on an earlier email today, is to consider the
> >original dataset as a preprint, of a Published dataset (at QC level
> >3).
> >>
> >> Incidentally, this disctinction might offer us a possible (distinct)
> >future for two different types of gateways into ESGF: the Published
> >datasets view (which makes pre-eminent the QC'd copy) and the
> >published view (which makes pre-eminenent whatever someone sticks on a
> >data node).
> >>
> >> But meanwhile, I think we can live with what you proposed, as long
> >as the QC status of the replicas is clearly visible - and the DOI
> >points to a landing page that somehow prioritises those versions,
> >which would be trivial if your page was organised in the same way
> >(prioritising the replicants of QC level 3, then replicants of QC
> >level 2, and then originals).
> >>
> >> Cheers
> >> Bryan
> >>
> >>
> >>> Hi Stephen,
> >>>
> >>> On 11/10/2011 05:23 AM, stephen.pascoe at stfc.ac.uk wrote:
> >>>> Hi Eric,
> >>>>
> >>>> Replicas are beginning to show up in CMIP5 and this is exposing
> >some
> >>>> gaps in what Gateway 1.x can do. I know you are reimplementing
> >replica
> >>>> support in Gateway 2.0 so I'd like to raise these issues now.
> >>>>
> >>>> We need to be able to publish a replica to the same Gateway that
> >hosts
> >>>> the original. I can't imagine this being possible with Gateway 1.x
> >since
> >>>> the URL http://<GATEWAY>/dataset/<dataset-id>.html only points to
> >one
> >>>> dataset on that Gateway. Either that page needs to link to the
> >original
> >>>> and all replicas for that dataset or we need separate URLs for
> >each
> >>>> replica/original, or both.
> >>> The current direction for the implementation would be to have a 1
> >page
> >>> for the original dataset and have that page list where replicas are
> >>> located.
> >>>
> >>> If there are use cases for the other options we should get those
> >identified.
> >>>
> >>> Thanks!
> >>> -Nate
> >>>
> >>>
> >>>> Is this part of your design for Gateway 2.0's replica support?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Stephen.
> >>>>
> >>>> ---
> >>>>
> >>>> Stephen Pascoe +44 (0)1235 445980
> >>>>
> >>>> Centre of Environmental Data Archival
> >>>>
> >>>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11
> >0QX, UK
> >>>>
> >>>>
> >>>> --
> >>>> Scanned by iCritical.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> GO-ESSP-TECH mailing list
> >>>> GO-ESSP-TECH at ucar.edu
> >>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >>> _______________________________________________
> >>> GO-ESSP-TECH mailing list
> >>> GO-ESSP-TECH at ucar.edu
> >>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >>>
> >> --
> >> Bryan Lawrence
> >> University of Reading:  Professor of Weather and Climate Computing.
> >> National Centre for Atmospheric Science: Director of Models and
> >Data.
> >> STFC: Director of the Centre for Environmental Data Archival.
> >> Ph: +44 118 3786507 or 1235 445012; Web:home.badc.rl.ac.uk/lawrence
> >> _______________________________________________
> >> GO-ESSP-TECH mailing list
> >> GO-ESSP-TECH at ucar.edu
> >> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >
> >
> >--
> >Estanislao Gonzalez
> >
> >Max-Planck-Institut für Meteorologie (MPI-M)
> >Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
> >Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
> >
> >Phone:   +49 (40) 46 00 94-126
> >E-Mail:  gonzalez at dkrz.de
> >
> >_______________________________________________
> >GO-ESSP-TECH mailing list
> >GO-ESSP-TECH at ucar.edu
> >http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
-- 
Scanned by iCritical.