[Go-essp-tech] Non-DRS File structure at data nodes

Wed Aug 31 03:55:40 MDT 2011

Hi Estani,

> Karl said he thought the replica was attached at a semantic level.

I'd be very surprised if the Gateway pays any attention to the semantics of the URL path -- i.e. I'd expect different url paths to work.  Everything I've seen so far suggests it takes all metadata from the THREDDS properties.

Re. the rest of your email.  We don't have tools for doing this right now.  I have so far kept drslib separate from the esgcet database but to add these features we should probably create hooks in drslib to access that database.  It already has a record of dataset versions, files and file checksums.

Help to implement these features would be appreciated but we should spec. out what's needed and I won't be working on it until after the BADC system upgrade next week.  I think we could tackle it effectively in the IS-ENES coding sprint Sept 21-23rd.

Cheers,
Stephen.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

-----Original Message-----
From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Estanislao Gonzalez
Sent: 31 August 2011 10:23
To: Juckes, Martin (STFC,RAL,RALSP)
Cc: stockhause at dkrz.de; go-essp-tech at ucar.edu
Subject: Re: [Go-essp-tech] Non-DRS File structure at data nodes

Hi Martin,

Are you planning to publish that data as a new instance or as a replica?
If I recall it right, Karl said he thought the replica was attached at a 
semantic level. But I have my doubts and haven't got any feed back on 
this. does anyone know if the gateway can handle a replica with a 
different url path? (dataset and version "should" be the same, although 
keeping the same version will be difficult, because no tool can handle 
this AFAIK, i.e. replicating or publishing multiple datasets with 
different versions)

And regarding replication (independently from the previous question), 
how are you going to cope with new versions? Do you already have tools 
for harvesting the TDS and building a list of which files do need to be 
replicated, regarding from what you already have?

The catalog will just publish a dataset and version along with a bunch 
of files, you would need to keep a DB with the fies you've already 
downloaded, and compare with the catalog to realize what should be done 
next. This information is what drslib should use to create the next 
version. Is that what will happen? If not, how will you be solving this?

Thanks,
Estani

Am 31.08.2011 10:54, schrieb martin.juckes at stfc.ac.uk:
> Hello Martina,
>
> For BADC, I don't think we are considering storing data in anything other than the DRS structure -- we just don't have the time to build systems around multiple structures. This means that data that comes from a node with a different directory structure will have to be re-mapped. Verification of file identities will rely on check-sums, as it always will when dealing with files from archives from which we have no curation guarantees,
>
> cheers,
> Martin
>
> ________________________________
> From: go-essp-tech-bounces at ucar.edu [go-essp-tech-bounces at ucar.edu] on behalf of Martina Stockhause [stockhause at dkrz.de]
> Sent: 31 August 2011 09:44
> To: go-essp-tech at ucar.edu
> Subject: [Go-essp-tech] Non-DRS File structure at data nodes
>
> Hi everyone,
>
> we promised to describe the problems regarding the non-DRS file structures at the data nodes.
> Estani has already started the discussion on the replication/user download problems (see attached email and document).
>
> Implications for the QC:
> - In the QCDB we need DRS syntax. The DOI process, creation of CIM documents, and identification of the data the QC results are connected to rely on that.
> - The QC needs to know the version of the data checked. The DOI at the end of the QC process is assigned to a specific not-changable data version. At least at DKRZ we have to guarantee that the data is not changed after assignment of the DOI, therefore we store a data copy in our archive.
> - The QC checker tool runs on files in a given directory structure and creates results in a copy of this structure. The QC wrapper can deal with recombinations of path parts. So, if the directory structure includes all parts of the DRS syntax, the wrapper can create the DRS syntax before insert in the QCDB. But we deal with structures at the data nodes, where some information is missing in the directory path, i.e. version and MIP table. Therefore an additional information would be needed for that mapping.
>
> Possible solutions to map the given file structure to the DRS directory structure before insert in the QCDB:
>
> 1. The publication on the data nodes of the three gateways who store replicas (PCMDI, BADC, DKRZ) publish data in the DRS directory structure. Then the QC run is possible without mapping. Replication problems?
>
> 2. The directory structures of the data nodes are replicated as they are. We store the data under a certain version. How? Are there implications for the replication from the data nodes? The individual file structures down to the chunk level are stored together with its DRS identification in a repository and a service is created to access the DRS id for the given file in the given file structure. The QC and maybe other user data services use this service for mapping. That will slow down the QC insert process. Before each insert of a chunk name, a qc result for a specific variable, and the qc result on the experiment level that service has to be called. And who can set-up and maintain such a repository? DKRZ has not the man power to do that in the next months.
>
> Cheers,
> Martina
>
>
>
> -------- Original-Nachricht --------
> Betreff:        RE: ESG discussion
> Datum:  Wed, 10 Aug 2011 15:35:04 +0100
> Von:    Kettleborough, Jamie<jamie.kettleborough at metoffice.gov.uk><mailto:jamie.kettleborough at metoffice.gov.uk>
> An:     Karl Taylor<taylor13 at llnl.gov><mailto:taylor13 at llnl.gov>, Wood, Richard<richard.wood at metoffice.gov.uk><mailto:richard.wood at metoffice.gov.uk>
> CC:     Carter, Mick<mick.carter at metoffice.gov.uk><mailto:mick.carter at metoffice.gov.uk>, Elkington, Mark<mark.elkington at metoffice.gov.uk><mailto:mark.elkington at metoffice.gov.uk>, Bentley, Philip<philip.bentley at metoffice.gov.uk><mailto:philip.bentley at metoffice.gov.uk>, Senior, Cath<cath.senior at metoffice.gov.uk><mailto:cath.senior at metoffice.gov.uk>, Hines, Adrian<adrian.hines at metoffice.gov.uk><mailto:adrian.hines at metoffice.gov.uk>, Dean N. Williams<williams13 at llnl.gov><mailto:williams13 at llnl.gov>, Estanislao Gonzalez<gonzalez at dkrz.de><mailto:gonzalez at dkrz.de>,<martin.juckes at stfc.ac.uk><mailto:martin.juckes at stfc.ac.uk>, Kettleborough, Jamie<jamie.kettleborough at metoffice.gov.uk><mailto:jamie.kettleborough at metoffice.gov.uk>
>
>
> Hello Karl, Dean,
>
> Thanks for you reply on this, and the fact you are taking our concerns seriously. You are right to challenge us for the specific issues, rather than us just highlighting the things that don't meet our (possibly idealised) expectations of how the system should look.  As a result, we have had a thorough review of our key issues. I think some of them are issues that make if harder for us to do things now; other issues are maybe more concerns of problems being stored up.
> This document has been prepared with the help Estani Gonzalez.  We would like to have Martin Juckes input on this too - but he is currently away on holiday.  I hope he can add to this when he returns - he has spent a lot of time thinking about the implications of data node directory structure on versioning.
> I hope this helps clarify issues, if not please let use know,
> Thanks,
> Jamie
>
> ________________________________
> From: Karl Taylor [mailto:taylor13 at llnl.gov]
> Sent: 09 August 2011 01:48
> To: Wood, Richard
> Cc: Carter, Mick; Kettleborough, Jamie; Elkington, Mark; Bentley, Philip; Senior, Cath; Hines, Adrian; Dean N. Williams
> Subject: Re: ESG discussion
>
> Dear all,
>
> Thanks for taking the time to bring to my attention the ESG issues that I hope can be addressed reasonably soon.  I think we're in general agreement that the user's experience should be improved.
>
> I've discussed this briefly with Dean.  I plan to meet with him and others here, and, drawing on your suggestions, we'll attempt to find solutions and methods of communication that might improve matters.  Before doing this, it would help if you could briefly answer the following questions:
>
> 1.  Is the main issue that it is currently difficult to script downloads from all the nodes because only some support PKI?  What other uniformity among nodes is required for you to be able to do what you want to do (i.e., what do you specifically want to do that is difficult to do now)?  [nb. all data nodes are scheduled to be operating with PKI authentication by September 1.]
>
> 2.  Is there anything from the perspective of a data *provider*  that needs to be done (other than make things easier for data users)?
>
> 3.  Currently ESG and CMIP5 do not dictate the directory structure found at each data node (although most nodes are adhering to the recommendations of the DRS).   The gateway software and catalog make it possible to get to the data regardless of directory structure.  It is possible that "versioning" might impose additional constraints on the directory structure, but I'm not sure about this.  (By the way, I'm not sure what the "versioning" issue is since currently I think it's impossible for users to know about more than one version; is that the issue?)  From a user's or provider's perspective, is there any essential reason that the directory structure should be the same at each node?
>
> 4.  ESG allows considerable flexibility in publishing data, and CMIP5 has suggested "best practices" to reduce differences.  Only some of the "best practices" are currently requirements.  A certain amount of flexibility is essential since different data providers have resources to support the potential capabilities of ESG (e.g., not all can support server-side calculations, which will be put in place at some nodes).   Likewise a provider can currently turn off the "checksum", if this is deemed to slow publication too much (although we could insist that checksums be stored in the thredds catalogue).  Nevertheless, it is unlikely that every data node will be identically configured for all options.    What are the *essential* ways that the data nodes should respond identically (we may not be able to insist on uniformity that isn't essential for serving our users)?
>
> Thanks again for your input, and I look forward to your further help with this.
>
> Best regards,
> Karl
>
>
> On 8/5/11 10:43 AM, Wood, Richard wrote:
>
> Dear Karl,
>    Following on from our phone call I had a discussion with technical
> colleagues here (Mick Carter, Jamie Kettleborough, Mark Elkington, also
> earlier with Phil Bentley), and with Adrian Hines who is coordinating
> our CMIP5 analysis work, about ideas for future development of the ESG.
> Our observations are from the user perspective, and based on what we can
> gather from mailing lists and our own experience. Coming out of our
> discussion we have a couple of suggestions that could help with
> visibility for data providers and users:
>
> - Some areas need agreement among the data nodes as to the technical
> solution, and then implementation across all the nodes, while others
> need a specific solution to be developed in one place and rolled out.
> The group teleconferences that Dean organises appear to be a good forum
> for airing specific technical ideas and solutions. However, in our
> experience it can be  difficult in that kind of forum to discuss
> planning and prioritisation questions. From our perspective we don't
> have visibility of the more project-related issues such as key technical
> decisions, prioritisation and timelines, or of whether issues that have
> arisen in the mailing list discussions are being followed up. We guess
> these may be discussed in separate project teleconferences involving the
> technical leads from the data nodes. As users we would not necessarily
> expect to be involved in those discussions, but as data providers and
> dowloaders it would be very helpful for our planning to see the outcomes
> of the discussions. The sort of thing we had in mind would be a simple
> web page showing the priority development areas, agreed solutions and
> estimated dates for completion/release. Some solutions will need to be
> implemented separately across all the participating data nodes, and in
> these cases it would be useful to see the estimated timeframe for
> implementation at each node.
> This would not be intended as a 'big stick' to the partners, but simply
> as a planning aid so that everyone can see what's available when and the
> project can identify any potential bottlenecks or issues in advance.
> Also the intention is not to generate a lot of extra work. Hopefully
> providing this information would be pretty light on people's time.
>
> - From where we sit it appears that some nodes are quite successful in
> following best practice and implementing the federation policies as far
> as they are aware of them. Could what these nodes do be made helpful to
> all the data nodes (e.g. by using identical software)?  We realise there
> may be real differences between some data nodes - but where possible we
> think that what is similar could be enforced or made explicitly the same
> through sharing the software components and tools.
>
> To set the discussion on priorities rolling, Jamie has prepared, in
> consultation with others here, a short document showing the Met Office
> view of current priority issues (attached). If you could update us on
> the status of work on these issues, that would be very useful (ideally
> via the web pages proposed above, which we think would be of interest to
> many users, or via email in the interim). Many thanks for the update on
> tokenless authentication, which is very good news.
>
>    Once again, our thanks to you, Dean and the team for all the hard work
> we know is going into this. Please let us know what you think of the
> above ideas and the attachment, and if there is anything we can do to
> help.
>
>        Best wishes,
>         Richard
>
>
> --------------
> Richard Wood
> Met Office Fellow and Head (Oceans, Cryosphere and Dangerous Climate
> Change)
> Met Office Hadley Centre
> FitzRoy Road, Exeter EX1 3PB, UK
> Phone +44 (0)1392 886641  Fax +44 (0)1392 885681
> Email richard.wood at metoffice.gov.uk<mailto:richard.wood at metoffice.gov.uk>   http://www.metoffice.gov.uk
> Personal web page
> http://www.metoffice.gov.uk/research/scientists/cryosphere-oceans/richar
> d-wood
>
> *** Please note I also work as Theme Leader (Climate System) for the
> Natural Environment Research Council ***
> *** Where possible please send emails on NERC matters to rwtl at nerc.ac.uk<mailto:rwtl at nerc.ac.uk>
> ***
>

-- 
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  gonzalez at dkrz.de

_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
-- 
Scanned by iCritical.