[Go-essp-tech] Replication: requested and output DRS products.
stephen.pascoe at stfc.ac.uk
stephen.pascoe at stfc.ac.uk
Tue Jul 13 04:41:07 MDT 2010
Hi,
As soon as I started writing code to manage the DRS I worked out that it refers to 2 different "variable components": variable_name and variable_name+table. In the directory path you just use variable_name but the filename contains the table. I agree it's messy but isenes.drslib has been designed to handle it -- quite a lot of the code is there to work around the issue. I can see it came about due to backward compatibility of CMOR which is reasonable.
So I'm pro Martin's option 1).
I have never thought of tas_6hrPlev and tas_6hrLev as different variables but I have never really thought about them at all :-). I presume they are tas on different grids. How will this cause problems? The only possibility I can think of is if we write tools that assume all files in a directory can be aggregated into 1 but I'm pretty sure esgpublish can work out what can be aggregated and what not. Provided we are aware a directory can contain multiple grids I don't see a major issue.
Cheers,
Stephen.
---
Stephen Pascoe +44 (0)1235 445980
British Atmospheric Data Centre
Rutherford Appleton Laboratory
-----Original Message-----
From: Juckes, Martin (STFC,RAL,SSTD)
Sent: 12 July 2010 23:19
To: Karl Taylor; Pascoe, Stephen (STFC,RAL,SSTD)
Cc: Lawrence, Bryan (STFC,RAL,SSTD); go-essp-tech at ucar.edu
Subject: RE: Replication: requested and output DRS products.
Hello Karl,
Perhaps Stephen can comment on this. One consideration is that the files will have to be moved at some point in order to put them in the right version sub-directory. This gives us, I think, two options:
(1) Retain MIP variable names, "tas" etc as directory names: in this case I think the DRS document should be updated to define both MIP variable names and MIP table names as DRS components -- so that all directory names are DRS components. This leaves, in a some cases, two variables in one directory which I think is messy, but I can't think of a game breaking problem.
(2) Make the software which moves files into the correct version subdirectory also create "tas_6hrLev" etc directories, so that we only have one variable per directory at the lowest level.
cheers,
Martin
-----Original Message-----
From: Karl Taylor [mailto:taylor13 at llnl.gov]
Sent: Mon 12/07/2010 22:32
To: Juckes, Martin (STFC,RAL,SSTD)
Cc: Pascoe, Stephen (STFC,RAL,SSTD); Lawrence, Bryan (STFC,RAL,SSTD); go-essp-tech at ucar.edu
Subject: Re: Replication: requested and output DRS products.
Hi Martin,
6
[A detail that doesn't affect your argument: it is 6hrPlev and 6hrLev
(not 3hr).]
You are right that the files from both tables will put put in a single
directory (6hr). The the <MIP table> will be different in the filenames
and distinguish the two. You say this will add unnecessary complexity
to the data management problems. What do you suggest we do about that?
[Given that folks are already using CMOR to write model output.]
thanks,
Karl
On 7/12/10 2:22 PM, martin.juckes at stfc.ac.uk wrote:
> Hello Karl,
>
> If I've understood your response correctly, your suggestion would imply that both
> tas_3hrPlev_HADCM3_historical_r1i1p1_185001-200512.nc
> and
> tas_3hrLev_HADCM3_historical_r1i1p1_185001-200512.nc
> would sit under the same directory:
> /CMIP5/output/UKMO/HADCM3/historical/3hr/atmos/v1/tas/r3i2p1/?
>
> This is possible, but, I think, undesirable, as having multiple variables in one directory will add unnecessary complexity to the data management problems. We have also said in the DRS that directory names will be DRS components, and, in the current wording, "tas" is not a DRS component. So we have to change the wording somewhere. Given Stephen's comments, I suggest we define the "variable identifier" component to be<MIP name>_<MIP table>, and make an explicit exception to the general rule of not having "_" in DRS components.
>
> cheers,
> Martin
>
> -----Original Message-----
> From: Karl Taylor [mailto:taylor13 at llnl.gov]
> Sent: Mon 12/07/2010 20:22
> To: Pascoe, Stephen (STFC,RAL,SSTD)
> Cc: Juckes, Martin (STFC,RAL,SSTD); Lawrence, Bryan (STFC,RAL,SSTD); go-essp-tech at ucar.edu
> Subject: Re: Replication: requested and output DRS products.
>
> Hi all,
>
> Yes, in filenames, CMOR separates the two parts of the "variable
> identifier" with an underscore, but we don't mention this in the DRS
> description (although the examples are written this way).
>
> The original design of all this was probably not as clean as it could be
> (partly because of backward compatibility issues).
>
> I think either "variable name" and "time frequency" can serve to
> uniquely identify each variable, or, alternatively, "variable name" and
> "variable table" (which together are referred to in the DRS as the
> "variable identifier") can serve to uniquely identify each variable.
> The directory structure follows the first option, whereas the filename
> follows the second option. [again, probably not ideal, but I think
> we'll have to live with it.] The "variable table" serves to group
> variables that "go together", like, for example, the cf3hr data, which
> is distinguishable from the 3hr data. The frequency alone can't be used
> for this purpose.
>
> In the DRS, I suppose the two parts of the variable identifier could be
> separated, so in place of "variable identifier" we would have "variable
> name" and "variable table", but to my mind "table" doesn't quite have
> the same significance as the other components of the DRS.
>
> regards,
> Karl
>
>
> On 7/11/10 9:43 AM, stephen.pascoe at stfc.ac.uk wrote:
>
>> Hi Martin,
>>
>> This is an excellent and timely summary. Thanks for putting in the effort to write this stuff down, it will make developer's jobs much easier!
>>
>> I have a couple of comments that could be used to update the document:
>>
>> 1. It is not the case that each MIP table describes only 1 realm. For instance Table CMIP5_3hr contains variables in realm atmos, land and ocean. It is true that each table contains mainly variables from 1 realm. It would definitely be simpler if each table did describe only 1 realm but I expect it's too late to enforce this now.
>>
>> 2. On Section 4.2 "The DRS Variable Identifier". What ever the DRS document says in reality CMOR separates the variable short name and table name with an underscore in file names. Logic to read/write these filenames are also implemented in isenes.drslib. In isenes.drslib I treat the MIP table name as a separate DRS component that is expressed in filenames but not directory structures.
>>
>> My take-home message from your document is that we need product=replicated and EPU's with product=replicated aren't a subset of EPU's with product=requested. You leave open whether we really need product=requested at all.
>>
>> Cheers,
>> Stephen.
>>
>> -----Original Message-----
>> From: Juckes, Martin (STFC,RAL,SSTD)
>> Sent: Sun 7/11/2010 4:15 PM
>> To: Karl Taylor; Pascoe, Stephen (STFC,RAL,SSTD); Lawrence, Bryan (STFC,RAL,SSTD)
>> Cc: go-essp-tech at ucar.edu
>> Subject: Replication: requested and output DRS products.
>>
>> Hello Karl,
>>
>> Attached is a note on the implementation of the DRS and what I think are some contradictions between the planned replication procedure and the way in which the replicated subset if identified in your archive_size spreadsheet. The way to resolve this is, I think, to slightly modify the definition of the two products in the DRS.
>>
>> There are a couple of other issues in the document, but the main one is to get clarification of how we handle the decision about what should be replicated and represent it in a code-friendly form exploiting the DRS.
>>
>> cheers,
>> Martin
>>
>> -----Original Message-----
>> From: is-enes-sa2-jra4-bounces at lists.enes.org on behalf of Karl Taylor
>> Sent: Sat 10/07/2010 01:55
>> To: Pascoe, Stephen (STFC,RAL,SSTD)
>> Cc: go-essp-tech at ucar.edu; drach at llnl.gov; Drach,Bob; gavin at llnl.gov; Doutriaux, Charles; is-enes-sa2-jra4 at lists.enes.org
>> Subject: Re: [is-enes-sa2-jra4] [Go-essp-tech] Replication: requested andoutput DRS products.
>>
>> Dear all,
>>
>> I'm not sure whether or not this matters at all, but I just realized
>> that "product" is a unique "component" of the DRS in that a file may be
>> designated to be included in two different "products" at the same time.
>> This is the only "component" for which this is true. For all others
>> (e.g., institute, model, experiment), each file belongs to one and only
>> one of the options. Is this why "product" is causing headaches? One
>> consequence is that datasets can overlap (i.e., contain common files).
>>
>> Disregard this if it seems irrelevant.
>>
>> regards,
>> Karl
>>
>> On 7/9/10 2:27 AM, stephen.pascoe at stfc.ac.uk wrote:
>>
>>
>>> Hi Gavin,
>>>
>>> This sounds remarkably like what we are planning to do. In particular the "DRS naming scheme for requested datasets" is surely the product component of the DRS! All requested datasets would be published with DRS.product=requested and all full datasets published with DRS.product=output.
>>>
>>> Where we differ is that I think you are suggesting this approach means we don't put data in a DRS structure on disk. Well, as I've said before I'd be happy with this if all download URLs published in THREDDS catalogues look like the DRS. Data centres could have a choice as to whether to use symbolic links or to do the indirection in software (or even to take the disk-space hit and store the files twice).
>>>
>>> A catalogue comment follows in a separate thread.
>>>
>>> Cheers,
>>> Stephen
>>>
>>>
>>> ---
>>> Stephen Pascoe +44 (0)1235 445980
>>> British Atmospheric Data Centre
>>> Rutherford Appleton Laboratory
>>>
>>> -----Original Message-----
>>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Gavin M Bell
>>> Sent: 08 July 2010 23:25
>>> To: Bob Drach
>>> Cc: go-essp-tech at ucar.edu; Drach,Bob; is-enes-sa2-jra4 at lists.enes.org; Doutriaux, Charles
>>> Subject: Re: [Go-essp-tech] Replication: requested and output DRS products.
>>>
>>> Hello Gentle-Folks,
>>>
>>> May I suggest the following to potential remedy this issue of requested...
>>>
>>> How about we generate a separate catalog for the requested files in a dataset. Have that separate catalog look, smell and taste like our standard catalog, except it only has the selected files present. Then we can specify a DRS naming scheme for this "requested" dataset. The additional wrinkle would be associating the "requested" dataset
>>> (catalog) with the superset dataset (catalog) it comes from / belongs to. This can perhaps be easily solved via DRS naming convention.
>>> During replication, since the files in the requested catalog points the same files in the 'regular' catalog, there would be no additional work done to fetch/replicat the constituent files if you've already replicated the superset 'regular' catalog since part of the replication process is to compare what you have with what you want. You would have two catalogs pointing to the same files... perfectly legal. Keeping the requested catalog in step with the originating catalog would be rather straightforward, especially in our single-author versioning universe.
>>>
>>> I think this is pretty much the same thing that Bryan mentioned earlier in this thread... I am just putting it in catalog-ese.
>>>
>>> Again, if our linqua franca is the catalog (not files) then things boil down to rather straightforward solutions
>>>
>>> The replicated catalogs can be published to the search engine database (like any other catalog) and show up as a first class search result (subject to result filtering shenanigans at the end-user's discretion).
>>>
>>> The powers that be that determine what is in "requested" would only have to create "requested" catalogs and publish them.
>>>
>>> P.S.
>>> Yes, we need checksums (or id tantamount to them) everywhere!
>>>
>>> On 7/8/10 12:28 PM, Bob Drach wrote:
>>>
>>>
>>>
>>>> Hi Karl,
>>>>
>>>> As I recall we discussed whether to build into the publisher (probably
>>>> within the CMIP5 / IPCC5 handler) the definition of 'requested
>>>> datasets'. The publisher could then associate a 'requested' value for
>>>> some property (product?) to aid the identification. This hasn't been
>>>> done yet - and it's not obvious how straightforward it would be - but
>>>> could be if the definition is sufficiently well defined at this point.
>>>>
>>>> Bob
>>>>
>>>>
>>>> On Jul 7, 2010, at 5:59 PM, Karl Taylor wrote:
>>>>
>>>>
>>>>
>>>>
>>>>> Dear all,
>>>>>
>>>>> I'm not sure scientists care much about the distinction between the
>>>>> "output", "requested", and "replicated" categories. Martin indicates
>>>>> their might be mild interest in being able to search only over the
>>>>> "requested" category, since outside that category, there may be
>>>>> little uniformity in what is available from the different models.
>>>>> There will be little interest in being able to distinguish between
>>>>> "requested" and "replicated" unless there is a difference in the
>>>>> quality control tests that have been applied to these two categories
>>>>> (and then only if a noticeable amount of data in the "requested"
>>>>> category wouldn't pass the "replicated" tests). Will this likely be
>>>>> the case?
>>>>>
>>>>> Clearly the ESG federation must be able to decide which files to
>>>>> replicate, so unlike the scientists there is real interest to some of
>>>>> you on this list that we be able to distinguish that subset. I'm not
>>>>> sure this information has to be part of the DRS though. Couldn't we
>>>>> just have some database that lists the criteria for selecting data to
>>>>> be replicated? The database coupled with coding to access that
>>>>> information could be used to decide whether each file in the "output"
>>>>> category needs to be replicated or not. This is why that although the
>>>>> current DRS document allows "product" to be either "output" or
>>>>> "requested", an all caps note appears stating: "[WILL POSSIBLY MODIFY
>>>>> THE ABOVE IF WE DON'T NEED TO KNOW ABOUT "REQUESTED"]."
>>>>>
>>>>> Bob Drach and I had some extended discussions about this some time
>>>>> ago, but I can't recall if he decided to include some capability
>>>>> along these lines in the publisher (i.e., enables the publisher to
>>>>> determine whether files are in the "requested" category or not), or
>>>>> if we've left that for completely independent external coding. Bob
>>>>> returns from vacation later this week, so I suggest we wait for some
>>>>> input from him.
>>>>>
>>>>> Best regards,
>>>>> Karl
>>>>>
>>>>> On 7/7/10 2:48 AM, martin.juckes at stfc.ac.uk wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Hello Estani,
>>>>>>
>>>>>> The reference to CMIP5_archive_size.xls was not very useful,
>>>>>> apologies for referencing a file that isn't publicly available -- it
>>>>>> is attached.
>>>>>>
>>>>>> According to the DRS document, everything should be found under the
>>>>>> "output" branch, and the "requested" branch will be a subset of the
>>>>>> "output".
>>>>>>
>>>>>> An end user may want a homogeneous dataset, and so may opt to
>>>>>> restrict attention to the "requested" data where he is likely to
>>>>>> find the same variables from a large range of models. He may, on the
>>>>>> other hand, want all available data for a given set of experiments,
>>>>>> in which case he should go to the "output" branch. He will then find
>>>>>> additional (low priority) variables and extended time coverage from
>>>>>> a small number of models.
>>>>>>
>>>>>> I'll see what can be done about a "DRS:requested" and
>>>>>> "ESGF:replicated" document (or wiki page),
>>>>>>
>>>>>> cheers,
>>>>>> Martin
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Estanislao Gonzalez [mailto:estanislao.gonzalez at zmaw.de]
>>>>>> Sent: Wed 07/07/2010 08:54
>>>>>> To: Juckes, Martin (STFC,RAL,SSTD)
>>>>>> Cc: Pascoe, Stephen (STFC,RAL,SSTD); gavin at llnl.gov;
>>>>>> drach1 at llnl.gov; go-essp-tech at ucar.edu;
>>>>>> is-enes-sa2-jra4 at lists.enes.org ; doutriaux1 at llnl.gov
>>>>>> Subject: Re: Replication: requested and output DRS products.
>>>>>>
>>>>>> Hi Martin,
>>>>>>
>>>>>> I couldn't find the file you mentioned (CMIP5_archive_size.xls),
>>>>>> could you please provide a link to it?
>>>>>>
>>>>>> I'm aware now that output> requested> replicated. But the
>>>>>> distinction between the later ones is not clear to me. I totally
>>>>>> agree that it would be great if someone could sum that up.
>>>>>>
>>>>>> And one question from the "monster" thread that still remains is:
>>>>>> It is clear that requested is a subset of output. Does this imply
>>>>>> that all data under .../requested/... should also be found under the
>>>>>> .../output/... DRS sub-structure?
>>>>>>
>>>>>> I think not... but then again, why would the end user need to know
>>>>>> about this separation?
>>>>>>
>>>>>> Thanks,
>>>>>> Estnai
>>>>>>
>>>>>>
>>>>>> martin.juckes at stfc.ac.uk wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hello again,
>>>>>>>
>>>>>>> The decision as to what is to be replicated is, I think embedded in
>>>>>>> "CMIP5_archive_size.xls", and its implementation through the DRS is
>>>>>>> based on the separation between "requested" and "output"
>>>>>>> products. It would be useful to have a brief document outlining
>>>>>>> these decisions and some code to implement them. I'm not sure of
>>>>>>> the latest status on these two points, perhaps Stephen can add
>>>>>>> something.
>>>>>>>
>>>>>>> cheers,
>>>>>>> Martin
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Estanislao Gonzalez [mailto:estanislao.gonzalez at zmaw.de]
>>>>>>> Sent: Tue 06/07/2010 17:15
>>>>>>> To: V. Balaji
>>>>>>> Cc: Juckes, Martin (STFC,RAL,SSTD); Pascoe, Stephen
>>>>>>> (STFC,RAL,SSTD); gavin at llnl.gov; drach1 at llnl.gov;
>>>>>>> go-essp-tech at ucar.edu ; is-enes-sa2-jra4 at lists.enes.org; doutriaux1 at llnl.gov; taylor13 at llnl.gov
>>>>>>> Subject: Re: [Go-essp-tech] [is-enes-sa2-jra4] Example of
>>>>>>> configuringadatanode to serve CMIP3-DRS
>>>>>>>
>>>>>>> Hi Balaji,
>>>>>>>
>>>>>>> To put things in context once more: (I think there's no such thing
>>>>>>> as over-clarification :-)
>>>>>>>
>>>>>>> DRS file and directory structure will be assured. The problem is if
>>>>>>> for some reason we have two different directories, e.g. A and B,
>>>>>>> and we want to publish data in DRS from both directories. So we
>>>>>>> have A/<DRS
>>>>>>> structure> and B/<DRS structure>.
>>>>>>> We'd like both of them to be mapped to a central URL, e.g.
>>>>>>> http://*****www.*****server.de/thredds/fileserver/<DRS structure> so
>>>>>>> that the user requires absolutely no knowledge about this
>>>>>>> separation.
>>>>>>>
>>>>>>> The remaining question is: why on earth would someone want to have
>>>>>>> A and B?! :-) Well some reasons are:
>>>>>>> 1) simplified management. We don't have a mega-mix of millions of
>>>>>>> files from which some have to me replicated, some are held only at
>>>>>>> our institution, some are "temporarily" held as being cached from
>>>>>>> tape.
>>>>>>> Telling these all apart might not be an easy task.
>>>>>>> 2) Safety. In such a context a simple error might be disastrous
>>>>>>> (e.g.
>>>>>>> someone tries to remove the replicated files to re-deploy them
>>>>>>> without being aware that they share the directory with other
>>>>>>> files...)
>>>>>>> 3) Backup. If we (ok, somebody else, we will have everything on
>>>>>>> tape, I
>>>>>>> think...) want to backup a portion of the data, this won't be
>>>>>>> easily achieved (the replicated data is already redundant, but the
>>>>>>> other
>>>>>>> isn't)
>>>>>>> 4) Storage. We might get more disks, but we will certainly won't be
>>>>>>> able to "merge" all of them into a single storage (well, that's
>>>>>>> because they will arrive way after we start publishing things, so
>>>>>>> the first disks will already have some data). In any case, for
>>>>>>> political (e.g.
>>>>>>> institutional), technical (e.g. disk speed) or philosophical (e.g
>>>>>>> ...uh....) reasons it might be desirable to keep different storages.
>>>>>>>
>>>>>>> And as I said we have to cope with that, somehow.
>>>>>>> The starter question was: can this be achieved with the publisher?
>>>>>>> And
>>>>>>> the answer was "no".
>>>>>>>
>>>>>>> And I totally agree with you regarding AR5. I must have a very good
>>>>>>> reason for not attaining to a default, even defacto ones. But the
>>>>>>> decision behind the storage in AR5 is a political one that, AFAIK,
>>>>>>> isn't taken yet.
>>>>>>>
>>>>>>> Well, I hope this helped to clarify things a bit.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Estani
>>>>>>>
>>>>>>> V. Balaji wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> There are undoubtedly parts of this I'm not following too well, so
>>>>>>>> I apologize in advance for any misunderstandings. This is all from
>>>>>>>> the perspective of a modeling center.
>>>>>>>>
>>>>>>>> I do not understand the logic for _not_ wishing to lay data out in
>>>>>>>> DRS-compliant fashion on the public data server. I know you can do
>>>>>>>> it, but I don't understand why you'd want to. One thing I'd like
>>>>>>>> to make sure is captured as a requirement is that 'wget -r' should
>>>>>>>> deliver data laid out per DRS directory structure.
>>>>>>>>
>>>>>>>> The second issue is that, again from the modeling centre
>>>>>>>> perspective, I fervently hope that whatever's done for CMIP5
>>>>>>>> becomes a de-facto standard for other projects requiring
>>>>>>>> coordinated model data output. We (modeling centres) cannot build
>>>>>>>> one-off solutions for each project. We have with some success made
>>>>>>>> CMOR1/AR4 a template which was forked off for other projects
>>>>>>>> (ENSEMBLES, CHFP, HTAP), because there's no way we can repeatedly
>>>>>>>> undertake the task of integrating multiple inconsistent CMORs and
>>>>>>>> DRSes into our data processing workflow. This is in ref to
>>>>>>>> Martin's question about "non-CMIP5 data".
>>>>>>>>
>>>>>>>> martin.juckes at stfc.ac.uk writes:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hello Estanislao, Gavin,
>>>>>>>>>
>>>>>>>>> There is a key part of your problem I don't understand -- what do
>>>>>>>>> you mean by "non CMIP5 data"?
>>>>>>>>>
>>>>>>>>> Before going into the ESGF CMIP5 archive, all files will be CMOR2
>>>>>>>>> compliant. This means that they fit in the "requested" or
>>>>>>>>> "product"
>>>>>>>>> categories of the DRS. The data to be replicated will be a subset
>>>>>>>>> of the "ESG published units" (also known as realm level datasets)
>>>>>>>>> in the "requested" category.
>>>>>>>>>
>>>>>>>>> There has been an agreement that the ESGF CMIP5 archive would be
>>>>>>>>> run on disk, and so it is not surprising that the infrastructure
>>>>>>>>> does not support tape storage. I can see that something along the
>>>>>>>>> lines Gavin describes would resolve the problems with tape
>>>>>>>>> storage, but we need to get the disk based system working as the
>>>>>>>>> first priority.
>>>>>>>>>
>>>>>>>>> Stephen raises the issue of replication and this is relevant,
>>>>>>>>> since straight disk to disk copies (i.e. to an external hard
>>>>>>>>> drive which can be posted) is a vital aspect of the replication
>>>>>>>>> plan. For the time being, this requires people to stick to the
>>>>>>>>> DRS directory structure.
>>>>>>>>>
>>>>>>>>> Within CMIP5 the data from different institutions is clearly
>>>>>>>>> separated at the institution directory level, I can't see why
>>>>>>>>> there should be any confusion here.
>>>>>>>>>
>>>>>>>>> For non-CMIP5 data -- why would you want to describe it with the
>>>>>>>>> CMIP5 DRS?
>>>>>>>>>
>>>>>>>>> cheers,
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: is-enes-sa2-jra4-bounces at lists.enes.org on behalf of
>>>>>>>>> stephen.pascoe at stfc.ac.uk
>>>>>>>>> Sent: Tue 06/07/2010 12:13
>>>>>>>>> To: estanislao.gonzalez at zmaw.de; gavin at llnl.gov
>>>>>>>>> Cc: drach1 at llnl.gov; go-essp-tech at ucar.edu;
>>>>>>>>> is-enes-sa2-jra4 at lists.enes.org; doutriaux1 at llnl.gov
>>>>>>>>> Subject: Re: [is-enes-sa2-jra4] [Go-essp-tech] Example of
>>>>>>>>> configuringadatanode to serve CMIP3-DRS
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Estanislao,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> * The only true problem is to differentiate between core and
>>>>>>>>>> non-core data (which as far as I node is a file issue instead of
>>>>>>>>>> a dataset one, i.e. some datasets contain core and non core
>>>>>>>>>> data)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I'm not sure you were involved then but we had lengthy
>>>>>>>>> discussions last year on how we would deal with the separation of
>>>>>>>>> requested and non-requested data (Karl discourages the term
>>>>>>>>> "core"). There is a fundamental problem that the DRS
>>>>>>>>> vocabularies don't cleanly map onto what is requested and not
>>>>>>>>> requested. The outcome was to introduce the DRS component
>>>>>>>>> "product" to divide the two. If you are interested take a look
>>>>>>>>> at the following threads:
>>>>>>>>>
>>>>>>>>> http://*****mailman.ucar.edu/pipermail/go-essp-tech/2010-January/
>>>>>>>>> 000335.html
>>>>>>>>> http://*****mailman.ucar.edu/pipermail/go-essp-tech/2009-December/
>>>>>>>>> 000255.html
>>>>>>>>>
>>>>>>>>> There hasn't been much discussion of how we identify and manage
>>>>>>>>> requested data since then and the nitty-gritty details still
>>>>>>>>> aren't fixed. This is going to be a challenge when we come to
>>>>>>>>> replicate.
>>>>>>>>>
>>>>>>>>> S.
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Stephen Pascoe +44 (0)1235 445980 British Atmospheric Data
>>>>>>>>> Centre Rutherford Appleton Laboratory
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: is-enes-sa2-jra4-bounces at lists.enes.org
>>>>>>>>> [mailto:is-enes-sa2-jra4-bounces at lists.enes.org] On Behalf Of
>>>>>>>>> Estanislao Gonzalez
>>>>>>>>> Sent: 06 July 2010 11:18
>>>>>>>>> To: Gavin M Bell
>>>>>>>>> Cc: drach1 at llnl.gov; go-essp-tech at ucar.edu;
>>>>>>>>> is-enes-sa2-jra4 at lists.enes.org; doutriaux1 at llnl.gov
>>>>>>>>> Subject: Re: [is-enes-sa2-jra4] [Go-essp-tech] Example of
>>>>>>>>> configuring adatanode to serve CMIP3-DRS
>>>>>>>>>
>>>>>>>>> Hi people,
>>>>>>>>>
>>>>>>>>> well I think we do require something like this (at least at the
>>>>>>>>> major data nodes where data will get replicated). Managing all
>>>>>>>>> data mixed up under one single directory is not a very neat
>>>>>>>>> solution for the data administrator. In our particular case we
>>>>>>>>> will be publishing many (much?
>>>>>>>>> :-) data from different institutions and even types (not only
>>>>>>>>> CMIP5).
>>>>>>>>> And we shouldn't forget about the replicated data (is that ===
>>>>>>>>> core?), how can we tell which data requires being replicated? by
>>>>>>>>> maintaining a second "catalog" in a DB? I think by maintaining a
>>>>>>>>> separate filesystem a simple rsynch will do the job (after the
>>>>>>>>> very first replication, of course).
>>>>>>>>> In any case the fact that we at DKRZ cannot hold all CMIP5 data
>>>>>>>>> on disk (yes, the core one we can :-) implies that we will have
>>>>>>>>> to maintain a cache somewhere, and mixing this cache with the
>>>>>>>>> core data is something we should probably avoid.
>>>>>>>>>
>>>>>>>>> Gavin's solution, if I got it right, has a major problem. The
>>>>>>>>> catalogs will be created pointing to the real files (e.g.
>>>>>>>>> .../core/CMIP5), so that the filter can alter the request from
>>>>>>>>> the DRS query
>>>>>>>>> (../CMIP5/<core_data>) to the real one, and thus allow the TDS to
>>>>>>>>> work as usual. This leaves the catalogs unaltered and thereby the
>>>>>>>>> harvest data which will have no reference to the mapped DRS
>>>>>>>>> structure but to the real one. OR did I miss something here?
>>>>>>>>>
>>>>>>>>> I have already tried several possible solutions without any
>>>>>>>>> success at all:
>>>>>>>>> 1) Setting multiple datasetRoot entries is not allowed
>>>>>>>>> 2) Altering the TDS to accept multiple datasetRoot entries and
>>>>>>>>> look in all of them one after the other after something matches
>>>>>>>>> is almost impossible (for the time we have ahead, the mere
>>>>>>>>> architecture of the TDS is, in my opinion, a mess).
>>>>>>>>> 3) In general altering the TDS is not a "nice" solution.
>>>>>>>>> 4) Filtering the request breaks the coherence between the
>>>>>>>>> catalogs and the DRS "virtual" structure (the catalogs have no
>>>>>>>>> information whatsoever that a second link to the files exists.
>>>>>>>>>
>>>>>>>>> The only viable solution I can think of (and it is still to see
>>>>>>>>> if it's really viable) is to maintain the files somewhere else
>>>>>>>>> and link them to the "central" DRS filesystem before being
>>>>>>>>> published.
>>>>>>>>>
>>>>>>>>> After discussing this with Stephan we come up with something I'd
>>>>>>>>> like to sum up here:
>>>>>>>>> * All non CMIP5 data can be mapped to a DRS structure "not"
>>>>>>>>> starting
>>>>>>>>> with CMIP5 so it can be easily mapped to somewhere else (TDS
>>>>>>>>> allows
>>>>>>>>> that)
>>>>>>>>> * The only true problem is to differentiate between core and non-
>>>>>>>>> core data (which as far as I node is a file issue instead of a
>>>>>>>>> dataset one, i.e. some datasets contain core and non core data)
>>>>>>>>> * The replication can rely on external sources for
>>>>>>>>> differentiating this, e.g. a DB.
>>>>>>>>> * The cached non-core data can co-live, in the worst case
>>>>>>>>> scenario, with the core data by removing the write permits of the
>>>>>>>>> later (beside the security that it implies, this will be used as
>>>>>>>>> a flag in case the server is restarted. All non-flagged (write
>>>>>>>>> enabled) files will be treated as left overs from the stopped
>>>>>>>>> cache and will be further
>>>>>>>>> served)
>>>>>>>>>
>>>>>>>>> So we might get out with it without performing any major changes.
>>>>>>>>> But this is something we should definitely discuss before next
>>>>>>>>> iteration :-)
>>>>>>>>>
>>>>>>>>> I hope this brings some light into the matter... sorry for the
>>>>>>>>> lengthy mail...
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Estani
>>>>>>>>>
>>>>>>>>> Gavin M Bell wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Martin,
>>>>>>>>>>
>>>>>>>>>> The savings is that the data provider / data-node admin doesn't
>>>>>>>>>> have to any additional work, whether it be provide any
>>>>>>>>>> filesystem<-
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> drs
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> mapping or (re)arranging their file system. In the current
>>>>>>>>>> state of things all the salient information is already in the
>>>>>>>>>> database created as a result of the publisher [software] scan.
>>>>>>>>>> I think it would be prudent to use that information to the
>>>>>>>>>> benefit of our end users instead of imposing a DRS directory
>>>>>>>>>> structure requirement for esg participation.
>>>>>>>>>>
>>>>>>>>>> You said:
>>>>>>>>>> "Remember that not having to configure the file system is only a
>>>>>>>>>> real saving if the alternative (configuring the file system to
>>>>>>>>>> URL
>>>>>>>>>> mapping)
>>>>>>>>>> is actually easier than configuring the file system."
>>>>>>>>>>
>>>>>>>>>> I am saying:
>>>>>>>>>> The 'alternative' you describe, does not exist. Because there
>>>>>>>>>> is no "configuring the file system to URL mapping" necessary...
>>>>>>>>>> unless the
>>>>>>>>>> end-user wants there to be. In which case we, as dutiful
>>>>>>>>>> programmers, provide that opportunity. This is what my code
>>>>>>>>>> sketch was illustrating with the property
>>>>>>>>>> "drs.resolve.strategy", and the use of a factory and strategy
>>>>>>>>>> pattern - of which we will set a default that requires them to
>>>>>>>>>> do *no additional work*. The data-node admin won't have to do
>>>>>>>>>> any actual setup outside of running an "esg-node -- update".
>>>>>>>>>> The upgrade/update process (determined by the esg-node install
>>>>>>>>>> script)
>>>>>>>>>> will install the filter, without them having to do anything
>>>>>>>>>> additional.
>>>>>>>>>>
>>>>>>>>>> Indeed, the code I posted was a quick and dirty filter code
>>>>>>>>>> sketch demonstrating that putting a filter in place is easy.
>>>>>>>>>> Yes, the resolution work would be done in the code that I only
>>>>>>>>>> alluded to, the "DRSResolver". Current duties preclude me from
>>>>>>>>>> actually implementing this issue outright, today, for this email
>>>>>>>>>> conversation.
>>>>>>>>>> However, if
>>>>>>>>>> we all conclude that it is worthwhile then I or someone else
>>>>>>>>>> could make it happen.
>>>>>>>>>>
>>>>>>>>>> I hope I have done a better job of making more clear my point;
>>>>>>>>>> that we can free our end-users of this DRS directory structure
>>>>>>>>>> requirement while allowing the DRS itself to be more flexible
>>>>>>>>>> with it's representation.
>>>>>>>>>> Also that the mechanism I described does not preclude anyone
>>>>>>>>>> from setting up their filesystem to follow the DRS structure, we
>>>>>>>>>> get that for free! :-)
>>>>>>>>>>
>>>>>>>>>> I am glad that we do indeed agree that the effort to bring this
>>>>>>>>>> to fruition can and should be done in a way that does not impede
>>>>>>>>>> or distract the current deliverable path.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> martin.juckes at stfc.ac.uk wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Er... the attachment you sent didn't actually do any mapping.
>>>>>>>>>>> But I'm
>>>>>>>>>>> sure it could be done. The extra work I'm talking about is the
>>>>>>>>>>> same as the extra work you talk about at the end of your mail,
>>>>>>>>>>> so I'm going to ignore your suggestion at the start of your
>>>>>>>>>>> email that there isn't any,
>>>>>>>>>>>
>>>>>>>>>>> cheers,
>>>>>>>>>>> Martin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Gavin M Bell [mailto:gavin at llnl.gov]
>>>>>>>>>>> Sent: Mon 05/07/2010 21:37
>>>>>>>>>>> To: Juckes, Martin (STFC,RAL,SSTD)
>>>>>>>>>>> Cc: drach1 at llnl.gov; go-essp-tech at ucar.edu;
>>>>>>>>>>> is-enes-sa2-jra4 at lists.enes.org; doutriaux1 at llnl.gov
>>>>>>>>>>> Subject: Re: [Go-essp-tech] [is-enes-sa2-jra4] Example of
>>>>>>>>>>> configuring adatanode to serve CMIP3-DRS
>>>>>>>>>>>
>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>
>>>>>>>>>>> With regards to the savings... One, perhaps default, setup is
>>>>>>>>>>> not having the data provider do anything additional at all with
>>>>>>>>>>> respect to configuration or setup. They simply use the
>>>>>>>>>>> publisher to scan their files into the system, something that
>>>>>>>>>>> must be done in all cases... (so we can normalize that out).
>>>>>>>>>>> With that said, they would not have to do
>>>>>>>>>>> *any* additional work. No work is easier than some work,
>>>>>>>>>>> regardless of how easy ;-).
>>>>>>>>>>>
>>>>>>>>>>> I have attached the filter code that would almost do it. The
>>>>>>>>>>> real intelligence would be in the "DRSResolver" object to do
>>>>>>>>>>> the resolution.
>>>>>>>>>>> I would have sketched out that class as well but that would be
>>>>>>>>>>> tantamount to completing this task... and to finish it off I
>>>>>>>>>>> would have to confer with Bob on the publisher database. And
>>>>>>>>>>> have us all settled on the DRS query syntax.
>>>>>>>>>>> With a DRS URL query scheme we could wrap this up quite
>>>>>>>>>>> directly.
>>>>>>>>>>>
>>>>>>>>>>> The DRSResolver would:
>>>>>>>>>>> -parse the request url (the query) and pull out the salient
>>>>>>>>>>> parts.
>>>>>>>>>>> -fashion those parts into a SQL query against the publisher
>>>>>>>>>>> database -Return the thredds' root based url to the rest of the
>>>>>>>>>>> processing stream. If it is not able to be resolved, punt and
>>>>>>>>>>> return the same input string as the output and let some other
>>>>>>>>>>> part of the process stream regurgitate an error.
>>>>>>>>>>>
>>>>>>>>>>> Because all the metadata is pulled out in the publisher's scan,
>>>>>>>>>>> file system placement of the scanned files is moot.
>>>>>>>>>>>
>>>>>>>>>>> In the code I attached, I leave room for the data-node user to
>>>>>>>>>>> select their own implementation of the resolver following a
>>>>>>>>>>> factory/ strategy pattern. At that point indeed we allow end
>>>>>>>>>>> users to do 'work'
>>>>>>>>>>> with
>>>>>>>>>>> doing their own mappings. Perhaps we integrate a few canned
>>>>>>>>>>> mapping schemes etc... We can be arbitrarily cleaver with these
>>>>>>>>>>> kinds of things of course. :-)
>>>>>>>>>>>
>>>>>>>>>>> P.S.
>>>>>>>>>>> The DRSResolver logic would/should be ported to all ingress
>>>>>>>>>>> request streams. Also the published catalogs would be
>>>>>>>>>>> published with the DRS query syntax scheme as the canonical
>>>>>>>>>>> name of the resource - something the search facility would use
>>>>>>>>>>> to identify the resource.
>>>>>>>>>>>
>>>>>>>>>>> done.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> martin.juckes at stfc.ac.uk wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Hi Gavin,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not convinced about the connection to Estanislao's email,
>>>>>>>>>>>> but the idea of thinking about the next step while
>>>>>>>>>>>> implementing the current system is certainly a good one.
>>>>>>>>>>>> Remember that not having to configure the file system is only
>>>>>>>>>>>> a real saving if the alternative (configuring the file system
>>>>>>>>>>>> to URL mapping) is actually easier than configuring the file
>>>>>>>>>>>> system. Setting up the DRS is not difficult,
>>>>>>>>>>>>
>>>>>>>>>>>> cheers,
>>>>>>>>>>>> Martin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Gavin M Bell [mailto:gavin at llnl.gov]
>>>>>>>>>>>> Sent: Mon 05/07/2010 19:45
>>>>>>>>>>>> To: Juckes, Martin (STFC,RAL,SSTD)
>>>>>>>>>>>> Cc: drach1 at llnl.gov; go-essp-tech at ucar.edu;
>>>>>>>>>>>> is-enes-sa2-jra4 at lists.enes.org; doutriaux1 at llnl.gov
>>>>>>>>>>>> Subject: Re: [Go-essp-tech] [is-enes-sa2-jra4] Example of
>>>>>>>>>>>> configuring adatanode to serve CMIP3-DRS
>>>>>>>>>>>>
>>>>>>>>>>>> Martin and friends,
>>>>>>>>>>>>
>>>>>>>>>>>> This is false economy. Two things. First implementing this
>>>>>>>>>>>> is not hard. Secondly implementing this will resolve the
>>>>>>>>>>>> issues r.w.t. the incongruence between DRS and the filesystem
>>>>>>>>>>>> that Estanislao's email illuminated. So it seems to me that
>>>>>>>>>>>> the alternative is keep fitting this square DRS peg in to the
>>>>>>>>>>>> round file system hole. That would mean having to do a whole
>>>>>>>>>>>> other set of gymnastics to get the DRS<-> file system beast
>>>>>>>>>>>> tamed. There is work to be done either way because things are
>>>>>>>>>>>> not ready to go as it stands. I suggest we fix the problem at
>>>>>>>>>>>> the root, now, not "later". Essentially the current course
>>>>>>>>>>>> requires the data providers to jump through file system layout
>>>>>>>>>>>> hoops. I am of the opinion that we should "require" as little
>>>>>>>>>>>> as possible from our users, especially something like this...
>>>>>>>>>>>> it hurts adoption IMHO.
>>>>>>>>>>>>
>>>>>>>>>>>> Actually, let me frame this differently. How about we fork
>>>>>>>>>>>> efforts, and have some folks think about what the *query* URL
>>>>>>>>>>>> should be for the functionality I suggested, while others
>>>>>>>>>>>> continue the current path. When the former development is
>>>>>>>>>>>> ripe I update the install script and have it installed upon
>>>>>>>>>>>> the clients' next install automagically, no slowdown for
>>>>>>>>>>>> anyone. The null transform would be equivalent to what we
>>>>>>>>>>>> have now so we would be backward compatible for folks whole
>>>>>>>>>>>> have done the task of making their file systems congruent to
>>>>>>>>>>>> DRS. Fair enough?
>>>>>>>>>>>>
>>>>>>>>>>>> Sound good?
>>>>>>>>>>>>
>>>>>>>>>>>> martin.juckes at stfc.ac.uk wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Gavin, Bob,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree that this is a good idea in principle, but I think it
>>>>>>>>>>>>> is a bad idea now. The thing about "now" is that we want to
>>>>>>>>>>>>> deploy and test the system we have agreed on. We want to do
>>>>>>>>>>>>> it now because modelling centres have supercomputers running
>>>>>>>>>>>>> and churning out vast volumes of data, there are thousands of
>>>>>>>>>>>>> scientists waiting to get at it and we have the job of
>>>>>>>>>>>>> installing a system to distribute it.
>>>>>>>>>>>>> It is, I think, I bad time to start implementing changes in
>>>>>>>>>>>>> the system design. Sorry if this sounds a bit harsh, but
>>>>>>>>>>>>> impending deadlines make me nervous,
>>>>>>>>>>>>>
>>>>>>>>>>>>> cheers,
>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: go-essp-tech-bounces at ucar.edu on behalf of Bob Drach
>>>>>>>>>>>>> Sent: Mon 05/07/2010 19:18
>>>>>>>>>>>>> To: Gavin M Bell
>>>>>>>>>>>>> Cc: go-essp-tech at ucar.edu; is-enes-sa2-jra4 at lists.enes.org;
>>>>>>>>>>>>> Charles Doutriaux
>>>>>>>>>>>>> Subject: Re: [Go-essp-tech] [is-enes-sa2-jra4] Example of
>>>>>>>>>>>>> configuring adatanode to serve CMIP3-DRS
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Gavin,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree completely. Having a regularized DRS syntax is a very
>>>>>>>>>>>>> good idea, but to implement it we will need to introduce a
>>>>>>>>>>>>> level of indirection between the DRS URL (your 'query') and
>>>>>>>>>>>>> the underlying filesystem. Separating these two concerns will
>>>>>>>>>>>>> have a very important
>>>>>>>>>>>>> benefit: it will allow the data node managers to organize
>>>>>>>>>>>>> their filesystems as they see fit.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Bob
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 5, 2010, at 11:10 AM, Gavin M Bell wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello gentle-people,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here is my two cents on this whole DRS business. I think
>>>>>>>>>>>>>> that the fundamental issue to all of this is the ability to
>>>>>>>>>>>>>> do resource resolution (lookup). The issue of having urls
>>>>>>>>>>>>>> match a DRS structure that matches the filesystem is a red
>>>>>>>>>>>>>> herring (IMHO).
>>>>>>>>>>>>>> The basic issue is to be able to issue a query to the system
>>>>>>>>>>>>>> such that you find what you are looking for. This query
>>>>>>>>>>>>>> mechanism should be separate mechanism than filesystem
>>>>>>>>>>>>>> correspondence. The driving issue behind the file system
>>>>>>>>>>>>>> correspondence push is so that people and/or applications
>>>>>>>>>>>>>> can infer the location of resources in some regimented way.
>>>>>>>>>>>>>> The true heart of the issue is not with the file system.
>>>>>>>>>>>>>> The heart of the issue is to perform a query such that you
>>>>>>>>>>>>>> provide resource resolution. The file system is a familiar
>>>>>>>>>>>>>> mechanism but it isn't the only one. The file system takes
>>>>>>>>>>>>>> a query (the file system path) and returns the resource to
>>>>>>>>>>>>>> us (the bits sitting at an inode location somewhere that is
>>>>>>>>>>>>>> memory mapped to some physical platter and spindle location,
>>>>>>>>>>>>>> that is mapped to the file system path). We are overloading
>>>>>>>>>>>>>> the file system query mechanism when it is not necessary.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I propose the following: We create a *filter* and a small
>>>>>>>>>>>>>> database (the latter we already have in the publisher). We
>>>>>>>>>>>>>> send a
>>>>>>>>>>>>>> *query* to the web server the web server *filter* intercepts
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> *query* and resolves it, using the database to the actual
>>>>>>>>>>>>>> resource location and returns the resource you want.
>>>>>>>>>>>>>> Implementing this in a filter divorces the query structure
>>>>>>>>>>>>>> from the file system structure. The use of the database
>>>>>>>>>>>>>> (that is generated by the publisher when it scans) provides
>>>>>>>>>>>>>> the resolution.
>>>>>>>>>>>>>> With this mechanism in place, WGET, as well as any other URL
>>>>>>>>>>>>>> based tool will be able to fetch the data as intended.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BTW: The "query" is whatever we make it up to be... (not a
>>>>>>>>>>>>>> reference to SQL query).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This gives the data-node admin the ability to put their
>>>>>>>>>>>>>> files wherever they want. If they move files around and so
>>>>>>>>>>>>>> on, they just have to rescan with the publisher. The issues
>>>>>>>>>>>>>> around design and efficiency can be address with varying
>>>>>>>>>>>>>> degrees of cleverness.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I welcome any thoughts on this issue... Please talk me down
>>>>>>>>>>>>>> :-). I think it is about time we put this DRS issue to bed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Estanislao Gonzalez wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Bob,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I guess you must be on vacations now. Anyway, here's the
>>>>>>>>>>>>>>> question, maybe someone else can answer it:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The very first idea I had was almost what you proposed.
>>>>>>>>>>>>>>> Your proposal though leaves URLs of the form:
>>>>>>>>>>>>>>> http://*********myserver/thredds/fileserver/CMIP5_replicas/
>>>>>>>>>>>>>>> output/...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>> (almost) DRS Structure ----------->
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Which has no valid DRS structure (CMIP5_replicas nor
>>>>>>>>>>>>>>> CMIP5_core are in the DRS vocabulary).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My proposal has a very similar flaw:
>>>>>>>>>>>>>>> http://*********myserver/thredds/fileserver/replicated/CMIP5/
>>>>>>>>>>>>>>> output/...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <--- full DRS Structure -----------> The DRS structure is
>>>>>>>>>>>>>>> preserved, but you cannot easily infer the correct URL from
>>>>>>>>>>>>>>> any dataset. I think the Idea is: if you know the prefix
>>>>>>>>>>>>>>> (http.../fileserver/) and the dataset DRS name you can
>>>>>>>>>>>>>>> always get the file without even browising the TDS:
>>>>>>>>>>>>>>> prefix + DRS = URL to file
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> AFAIK the URL structure used by the TDS will never be 100%
>>>>>>>>>>>>>>> DRS conform (according to the DRS version 0.27) This one
>>>>>>>>>>>>>>> has the
>>>>>>>>>>>>>>> form:
>>>>>>>>>>>>>>> http://*********<hostname>/<activity>/<product>/<institute>/
>>>>>>>>>>>>>>> <model>/
>>>>>>>>>>>>>>> <experiment>/<frequency>/<modeling
>>>>>>>>>>>>>>> realm>/<variable identifier>/<ensemble member>/<version>/
>>>>>>>>>>>>>>> [<endpoint>],
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> where the TDS one has the endpoint moved to the front (the
>>>>>>>>>>>>>>> thredds/fileserver, thredds/dodsC, etc parts).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To sum things up:
>>>>>>>>>>>>>>> Is it possible to publish files from different directory
>>>>>>>>>>>>>>> structures into an unified URL structure so that it is
>>>>>>>>>>>>>>> completely transparent to the user?
>>>>>>>>>>>>>>> Am I the only one addressing this problem? Are all other
>>>>>>>>>>>>>>> institutions planning to publish all files from only one
>>>>>>>>>>>>>>> directory?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The only viable solution I can think of is to rely on
>>>>>>>>>>>>>>> Stephen's versioning concept and maintaining a single true
>>>>>>>>>>>>>>> DRS structure with links to files kept in other more
>>>>>>>>>>>>>>> manageable directory structures (This will probably involve
>>>>>>>>>>>>>>> adapting Stephen's tool).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Estani
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Bob Drach wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Estani,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It should be possible to do what you want without running
>>>>>>>>>>>>>>>> multiple data nodes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The purpose of the THREDDS dataset roots is to hide the
>>>>>>>>>>>>>>>> directory structure from the end user, and to limit what
>>>>>>>>>>>>>>>> the TDS can access.
>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>> THREDDS can certainly have multiple dataset roots.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In your example below, you should associate different
>>>>>>>>>>>>>>>> paths with the locations, for example:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> <datasetRoot path="CMIP5_replicas"
>>>>>>>>>>>>>>>>> location="/replicated/CMIP5"/> <datasetRoot
>>>>>>>>>>>>>>>>> path="CMIP5_core"
>>>>>>>>>>>>>>>>> location="/core/CMIP5"/>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also be aware that in the publisher configuration:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - the directory_format can have multiple values, separated
>>>>>>>>>>>>>>>> by vertical bars (|). The publisher will use the first
>>>>>>>>>>>>>>>> format that matches the directory structure being scanned.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - a useful strategy is to create different project
>>>>>>>>>>>>>>>> sections for various groups of directives. You could
>>>>>>>>>>>>>>>> define a cmip5_replica project, a cmip5_core project, etc.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Bob
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Jul 1, 2010, at 5:42 AM, Estanislao Gonzalez wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Bryan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thanks for your answer!
>>>>>>>>>>>>>>>>> Running multiple ESG data nodes is always a possibility,
>>>>>>>>>>>>>>>>> but it seems an overkill to us as we may have several
>>>>>>>>>>>>>>>>> different "data repositories".
>>>>>>>>>>>>>>>>> We would like to separate: core-replicated,
>>>>>>>>>>>>>>>>> core-non-replicated, non-core, non-core-on-hpss, as well
>>>>>>>>>>>>>>>>> as other non-cmip5 data.
>>>>>>>>>>>>>>>>> Having 5+
>>>>>>>>>>>>>>>>> ESG data nodes is not viable in our scenario.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The TDS allows the separation of access URL from the
>>>>>>>>>>>>>>>>> underlying file structure so that it might be possible.
>>>>>>>>>>>>>>>>> AFAIK the publisher does not provide a simple way of
>>>>>>>>>>>>>>>>> doing this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Setting thredds_dataset_roots to different values while
>>>>>>>>>>>>>>>>> publishing doesn't appear to work as those are mapped to
>>>>>>>>>>>>>>>>> a map-entry at the catalog root:
>>>>>>>>>>>>>>>>> <datasetRoot path="CMIP5" location="/replicated/CMIP5"/>
>>>>>>>>>>>>>>>>> <datasetRoot path="CMIP5" location="/core/CMIP5"/> ..
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> which is clearly non bijective and can't therefore be
>>>>>>>>>>>>>>>>> reversed to locate the file from a given URL.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> While publishing all referred data will be held on a
>>>>>>>>>>>>>>>>> known location.
>>>>>>>>>>>>>>>>> Is it possible to use somehow this information to setup a
>>>>>>>>>>>>>>>>> proper catalog configuration so that the URL can be
>>>>>>>>>>>>>>>>> properly mapped? At least on a dataset level?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The whole HPSS staging procedure should be completely
>>>>>>>>>>>>>>>>> transparent to the user, as well as the location of the
>>>>>>>>>>>>>>>>> files.
>>>>>>>>>>>>>>>>> I was just looking at other options in case we cannot
>>>>>>>>>>>>>>>>> publish them the way we want...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Estani
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Bryan Lawrence wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> sorry.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the first sentence should have read
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Just to note that *our* approach to the local versus
>>>>>>>>>>>>>>>>>> replication issue will be ...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>> Bryan
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thursday 01 Jul 2010 11:25:37 Bryan Lawrence wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Estani
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Just to note that your approach to the local versus
>>>>>>>>>>>>>>>>>>> replication will be to run two different ESG nodes ...
>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>> is in fact the desired outcome so as to get the right
>>>>>>>>>>>>>>>>>>> things in the catalogues at the right time (vis- a-viz
>>>>>>>>>>>>>>>>>>> qc etc).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The issue with respect to cache, I'm not so sure about,
>>>>>>>>>>>>>>>>>>> in what way do you want to expose that into ESG?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Bryan
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wednesday 30 Jun 2010 17:05:57 Estanislao Gonzalez
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Stephen,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> the page contains really helpful information, thanks a
>>>>>>>>>>>>>>>>>>>> lot!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'm also interested in some variables of the DEFAULT
>>>>>>>>>>>>>>>>>>>> section from the esg.ini configuration file. More
>>>>>>>>>>>>>>>>>>>> specifically:
>>>>>>>>>>>>>>>>>>>> thredds_dataset_roots (and maybe
>>>>>>>>>>>>>>>>>>>> thredds_aggregation_services or any other which was
>>>>>>>>>>>>>>>>>>>> changed or you think it might be important)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The main question here is: how can different local
>>>>>>>>>>>>>>>>>>>> directory structures be published to the same DRS
>>>>>>>>>>>>>>>>>>>> structure?
>>>>>>>>>>>>>>>>>>>> The example scenario in our case will be:
>>>>>>>>>>>>>>>>>>>> /replicated/<DRS structure> - for replicated data
>>>>>>>>>>>>>>>>>>>> /local/<DRS structure> - for non replicated data hold
>>>>>>>>>>>>>>>>>>>> on disk /cache/<DRS structure> - for data staged from
>>>>>>>>>>>>>>>>>>>> a HPSS system
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The only solution I can think of is to extend the URL
>>>>>>>>>>>>>>>>>>>> before the DRS structure starts (the URL won't be 100%
>>>>>>>>>>>>>>>>>>>> DRS conform anyway). So
>>>>>>>>>>>>>>>>>>>> http://**********server/thredds/fileserver/<DRS
>>>>>>>>>>>>>>>>>>>> structure> will
>>>>>>>>>>>>>>>>>>>> turn into
>>>>>>>>>>>>>>>>>>>> http://**********server/thredds/fileserver/replicated/
>>>>>>>>>>>>>>>>>>>> <DRS
>>>>>>>>>>>>>>>>>>>> structure>
>>>>>>>>>>>>>>>>>>>> http://**********server/thredds/fileserver/local/<DRS
>>>>>>>>>>>>>>>>>>>> structure>
>>>>>>>>>>>>>>>>>>>> http://**********server/thredds/fileserver/cache/<DRS
>>>>>>>>>>>>>>>>>>>> structure>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Is that viable? Are there any other options?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Estani
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> stephen.pascoe at stfc.ac.uk wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> To illustrate how the ESG datanode can be configured
>>>>>>>>>>>>>>>>>>>>> to serve data for CMIP5 we have deployed a datanode
>>>>>>>>>>>>>>>>>>>>> containing a subset of
>>>>>>>>>>>>>>>>>>>>> CMIP3 in the Data Reference Syntax. Some key features
>>>>>>>>>>>>>>>>>>>>> of this deployment are:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * The underlying directory structure is based on
>>>>>>>>>>>>>>>>>>>>> the Data
>>>>>>>>>>>>>>>>>>>>> Reference Syntax.
>>>>>>>>>>>>>>>>>>>>> * Datasets published at the realm level.
>>>>>>>>>>>>>>>>>>>>> * The token-based security filter is replaced by the
>>>>>>>>>>>>>>>>>>>>> OpenidRelyingParty security filter.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Further notes can be found at
>>>>>>>>>>>>>>>>>>>>> http://**********proj.badc.rl.ac.uk/go-essp/wiki/
>>>>>>>>>>>>>>>>>>>>> CMIP3_Datanode
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This test deployment should be of interest to anyone
>>>>>>>>>>>>>>>>>>>>> wanting to know how DRS identifiers could be exposed
>>>>>>>>>>>>>>>>>>>>> in THREDDS catalogues and the TDS HTML interface.
>>>>>>>>>>>>>>>>>>>>> You can also try downloading files with OpenID
>>>>>>>>>>>>>>>>>>>>> authentication or via wget with SSL-client
>>>>>>>>>>>>>>>>>>>>> certificate authentication. See the link above for
>>>>>>>>>>>>>>>>>>>>> details.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Stephen.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>> Stephen Pascoe +44 (0)1235 445980 British
>>>>>>>>>>>>>>>>>>>>> Atmospheric Data Centre Rutherford Appleton
>>>>>>>>>>>>>>>>>>>>> Laboratory
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>>>>>>>>>>> ------
>>>>>>>>>>>>>>>>>>>>> ------
>>>>>>>>>>>>>>>>>>>>> -- -----
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>> GO-ESSP-TECH mailing list GO-ESSP-TECH at ucar.edu
>>>>>>>>>>>>>>>>>>>>> http://**********mailman.ucar.edu/mailman/listinfo/go-
>>>>>>>>>>>>>>>>>>>>> essp-tech
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Estanislao Gonzalez
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Max-Planck-Institut für Meteorologie (MPI-M) Deutsches
>>>>>>>>>>>>>>>>> Klimarechenzentrum (DKRZ) - German Climate Computing
>>>>>>>>>>>>>>>>> Centre Room 108 - Bundesstrasse 45a, D-20146 Hamburg,
>>>>>>>>>>>>>>>>> Germany
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Phone: +49 (40) 46 00 94-126
>>>>>>>>>>>>>>>>> E-Mail: estanislao.gonzalez at zmaw.de
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> GO-ESSP-TECH mailing list GO-ESSP-TECH at ucar.edu
>>>>>>>>>>>>>>>>> http://**********mailman.ucar.edu/mailman/listinfo/go-essp-
>>>>>>>>>>>>>>>>> tech
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Gavin M. Bell
>>>>>>>>>>>>>> Lawrence Livermore National Labs
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Never mistake a clear view for a short distance."
>>>>>>>>>>>>>> -Paul Saffo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (GPG Key - http://********rainbow.llnl.gov/dist/keys/gavin.asc)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> A796 CE39 9C31 68A4 52A7 1F6B 66B7 B250 21D5 6D3E
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> GO-ESSP-TECH mailing list
>>>>>>>>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>>>>>>>>> http://********mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Estanislao Gonzalez
>>>>>>>>>
>>>>>>>>> Max-Planck-Institut für Meteorologie (MPI-M) Deutsches
>>>>>>>>> Klimarechenzentrum (DKRZ) - German Climate Computing Centre Room
>>>>>>>>> 108
>>>>>>>>> - Bundesstrasse 45a, D-20146 Hamburg, Germany
>>>>>>>>>
>>>>>>>>> Phone: +49 (40) 46 00 94-126
>>>>>>>>> E-Mail: estanislao.gonzalez at zmaw.de
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> is-enes-sa2-jra4 mailing list
>>>>>>>>> is-enes-sa2-jra4 at lists.enes.org
>>>>>>>>> https://*****lists.enes.org/mailman/listinfo/is-enes-sa2-jra4
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Estanislao Gonzalez
>>>>>>
>>>>>> Max-Planck-Institut für Meteorologie (MPI-M) Deutsches
>>>>>> Klimarechenzentrum (DKRZ) - German Climate Computing Centre Room 108
>>>>>> - Bundesstrasse 45a, D-20146 Hamburg, Germany
>>>>>>
>>>>>> Phone: +49 (40) 46 00 94-126
>>>>>> E-Mail: estanislao.gonzalez at zmaw.de
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Scanned by iCritical.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://****mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Gavin M. Bell
>>> Lawrence Livermore National Labs
>>> --
>>>
>>> "Never mistake a clear view for a short distance."
>>> -Paul Saffo
>>>
>>> (GPG Key - http://***rainbow.llnl.gov/dist/keys/gavin.asc)
>>>
>>> A796 CE39 9C31 68A4 52A7 1F6B 66B7 B250 21D5 6D3E _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://***mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>> --
>>> Scanned by iCritical.
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://***mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>>>
>>>
>>>
>> _______________________________________________
>> is-enes-sa2-jra4 mailing list
>> is-enes-sa2-jra4 at lists.enes.org
>> https://**lists.enes.org/mailman/listinfo/is-enes-sa2-jra4
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Scanned by iCritical.
>>
>>
>>
>
> --
> Scanned by iCritical.
>
>
--
Scanned by iCritical.
More information about the GO-ESSP-TECH
mailing list