<html>
<head>
<meta content="text/html; charset=GB2312" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="Times New Roman">Hi all,<br>
<br>
thanks for the good discussion. Some good arguments have been
made for keeping all versions. I'll not make a policy decision
immediately, but am tending toward strong encouragement to keep
all versions. I'll distribute a draft statement about this for
your input and comment before posting. I'll, of course, also
consult directly with other IPCC DDC folks too.<br>
<br>
Best regards,<br>
Karl<br>
</font><br>
On 1/10/12 5:31 AM, Estanislao Gonzalez wrote:
<blockquote cite="mid:4F0C3DB7.6060508@dkrz.de" type="cite">
<meta content="text/html; charset=GB2312"
http-equiv="Content-Type">
Hi Jamie,<br>
<br>
Indeed, DOIs are not going to solve everything. The DOI is
analogous to the ISBN of a book, citing the whole book is not
always what you want in any case. To continue the analogy, users
are indeed working with pre-prints which get corrected all the
time (i.e. the CMIP5 archive is in flux). People are writing
papers citing a pre-print. Of course this makes no sense, but they
are not doing so willingly, they have to as the dead line
approaches but the computing groups are not ready.<br>
<br>
So what do we have now? Some archives with a strong commitment for
preserving data.<br>
If the DRS were honored, the URL would be enough for citing any
file as it has the version in it. Indeed citing +1000 Urls is not
practical, but a redirection could be added so that the scientist
cites one URL in which all files URLs are listed (There's no
implementation for this AFAIK). But at least the URL of DRS
committed sites could be safely cited, and if the checksum is
attached to the citation, it is sure that the correct file is
always being cited (and it could even be found if moved).<br>
<br>
I don't know how citations are being done now, nor do I know how
they were done before when everyone was citing data that it was
almost impossible to get. DOIs are the very first step in the
right direction, not the last one. <br>
IMHO the community should come up with some best practices to
overcome the problem we are facing: how to cite something that's
permanently changing. Sharing this will certainly help everyone.<br>
Before jumping away from this subject I'd also like to add that I
don't see any proper communication mechanism in the community. All
(or at least most) questions regarding CMIP5 are AFAICT directed
to the help-desk, so mostly developers are trying to help the
community instead of the community trying to help itself. I think
we might be missing some kind of platform for doing this. We don't
have the means to support the growing community (and new
communities which we are now serving), we need them to help with
the "helping". Just a thought....<br>
<br>
And last, and probably least, the only way to get the latest
version of any dataset is by re-issuing the search. Especially
since multiple datasets are referred to in a wget script, finding
the latest versions of each of them "by hand" will be more
time-consuming than issuing the search query again.<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 10.01.2012 13:00, schrieb Kettleborough, Jamie:
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90FB7F9F3@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<meta content="text/html; charset=GB2312"
http-equiv="Content-Type">
<meta name="GENERATOR" content="MSHTML 8.00.6001.19154">
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012">Hello,</span></font></div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012"></span></font> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012">I'm not sure how
to say this: but I'm not sure its just down to DOI's to
determine whether a data set should always be visible. I
think data needs to be visible where its sufficiently
important that a user might want to download it. e.g they
want to check or extend someone elses study (and I think
there are other reasons). Its not clear to me that all
data of this kind will have a DOI - for instance how many
of the datasets referenced in papers being written now for
the summer deadline of AR5 have (or will have in
time) DOIs?</span></font></div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012"></span></font> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012">I know its
tempting to say - any dataset referenced in a paper should
have a DOI. But I</span></font><font color="#0000ff"
face="Arial" size="2"><span class="078303811-10012012"> think
you need to be realistic about the prospects of this
happening on the right timescales.</span></font></div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012"></span></font> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012">If the DOI is used
as the determinent of whether data is always visible then
should users be made aware of the risk they are carrying
now? For instance, so they know to have local backups of
data that is really important to them. (With the possible
implication too that they may need to be prepared to
'reshare' this data with others.)</span></font></div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012"></span></font> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012">For what its worth
my personal preference is with the BADC/DKRZ (and I'm sure
others) philosophy of keeping all versions - though I
realise there are costs in doing this, like getting DRSlib
sufficiently bug free and getting it to work in all the
contexts it needs to (hard links/soft links), getting it
deployed, getting the update mechanism in place for when
new bugs are found etc. If you used DRSlib doesn't
Estanis use case that caused user grief become easier too
- the wget scripts do not need regenerating, you should
instead be able to replace the version strings in the url
(though I may be assuming things about load balancing etc
in saying this).</span></font></div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012"></span></font> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="078303811-10012012">Jamie</span></font></div>
<br>
<blockquote style="BORDER-LEFT: #0000ff 2px solid; PADDING-LEFT:
5px; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px" dir="ltr">
<div dir="ltr" class="OutlookMessageHeader" align="left"
lang="en-us">
<hr tabindex="-1"> <font face="Tahoma" size="2"><b>From:</b>
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>
[<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>]
<b>On Behalf Of </b>Estanislao Gonzalez<br>
<b>Sent:</b> 10 January 2012 10:21<br>
<b>To:</b> Karl Taylor<br>
<b>Cc:</b> Drach, Bob; <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>;
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:serguei.nikonov@noaa.gov">serguei.nikonov@noaa.gov</a><br>
<b>Subject:</b> Re: [Go-essp-tech] Fwd: Re: Publishing
dataset with option --update<br>
</font><br>
</div>
Well to be honest I do agree this is a decision each
institution has to make, but for us I'd prefer offering
everything we have and let the systems decide what to do with
this information. I.e. I've used it to generate some comments
(I might have already show you this), just go here: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://ipcc-ar5.dkrz.de/dataset/cmip5.output1.NCC.NorESM1-M.sstClim.mon.land.Lmon.r1i1p1.html">http://ipcc-ar5.dkrz.de/dataset/cmip5.output1.NCC.NorESM1-M.sstClim.mon.land.Lmon.r1i1p1.html</a>
and click on history. <br>
That information could be generated only because we store the
metadata to the previous version. <br>
<br>
By the way, The only way of inhibiting the user from getting
an older version, if that's what it's wanted, is by either
removing the files from the TDS served directory, or changing
the access restriction at the Gateway. Because of a well-known
TDS bug (or feature) files present at that directory and not
found in any catalog are served without any restriction (AFAIK
no certificate is required for this). So, normally the wget
script would work even if the files where unpublished.<br>
<br>
It really depends on the use-case... but e.g. I had to explain
all this to a couple of people in the help-desk since the wget
script they've downloaded wasn't working anymore (files were
removed). They weren't thrilled to know they had to re issue
the search again (there's no workaround for this) and they
wanted to know what was changed in the new version, and
there's where we can't help our users any more since we don't
have that information...<br>
<br>
I don't know what our users prefer, but I think they have more
important problems to cope with at this time... if they could
reliably get one version they could start worrying about
others. From my perspective as a data manager, it's worth the
tiny additional effort, if there's any.<br>
<br>
Cheers,<br>
Estani<br>
<br>
Am 09.01.2012 20:05, schrieb Karl Taylor:
<blockquote cite="mid:4F0B3A87.7080803@llnl.gov" type="cite"><font
face="Times New Roman"> Hi Estani,<br>
<br>
I agree that a new version number should (I'd say must) be
assigned when any changes are made. However, except for
DOI datasets, most groups will not want older versions to
be visible or downloadable.<br>
<br>
Do you agree?<br>
<br>
cheers,<br>
Karl<br>
</font><br>
On 1/9/12 10:37 AM, Estanislao Gonzalez wrote:
<blockquote cite="mid:4F0B3407.70009@dkrz.de" type="cite">Hi
Karl,<br>
<br>
It is indeed a good point, but I must add that we are not
talking about preserving a version (although we do it here
at DKRZ) but of signaling that a version has been changed.
So the version is a key to find a specific dataset which
changes in time.<br>
<br>
Even before a DOI assignment I'd encourage all to create a
new version every time the dataset changes in any way.
Institutions have the right to preserve whatever version
they want (they may even delete DOI-assigned versions, on
the other hand archives can't, that's why archives are
for).<br>
But altering the dataset preserving the version just bring
chaos for the users and for us at the help-desk as we have
to explain why something has changed (or rather answer
that we don't know why...). It means that the same key now
points to a different dataset.<br>
<br>
The only benefits I can see for preserving the same
version is that publishing using the same version seems to
be easier to some (for our workflow it's not, it's exactly
the same) and that if only new files are added this seems
to work fine for publication at both the data-node and the
gateway as it's properly supported.<br>
If anything else changes, this does not work as expected
(wrong checksums, ghost files at the gateway, etc). And
changing a version contents makes no sense to the user
IMHO (e.g. it's as if you might sometimes get more files
from a tarred file... how often should you extract it to
be sure you got "all of them")<br>
<br>
If old versions were preserved (which take almost no
resources if using hardlinks), a simple comparison would
tell that the only changes were the addition of some
specific files.<br>
<br>
Basically, reusing the version ends in a non-recoverable
loss of information. That's why I discourage it.<br>
<br>
My 2c,<br>
Estani<br>
<br>
Am 09.01.2012 17:25, schrieb Karl Taylor:
<blockquote cite="mid:4F0B14EC.2020600@llnl.gov"
type="cite"><font face="Times New Roman">Dear all,<br>
<br>
I do not have time to read this thoroughly, so perhaps
what I'll mention here is irrelevant. There may be
some miscommunication about what is meant by
"version". There are two cases to consider:<br>
<br>
1. Before a dataset has become official (i.e.,
assigned a DOI), a group may choose to remove all
record of it from the database and publish a
replacement version.<br>
<br>
2. Alternatively, if a group wants to preserve a
previous version (as is required after a DOI has been
assigned), then the new version will not "replace" the
previous version, but simply be added to the archive.<br>
<br>
It is possible that different publication procedures
will apply in these different cases.<br>
<br>
best,<br>
Karl<br>
</font><br>
On 1/9/12 4:26 AM, Estanislao Gonzalez wrote:
<blockquote cite="mid:4F0ADCF0.1090602@dkrz.de"
type="cite">
<pre wrap="">Just to mentioned that we do the same thing. We use directly
--new-version and a map file containing all files for the new version,
but we do create hard-links to the files being reused, so they are
indeed all "new" as their paths always differ from those of previous
versions. (In any case for the publisher they are the same and thus
encode them with the nc_0 name if I recall correctly)
Thanks,
Estani
Am 09.01.2012 12:15, schrieb <a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk:" moz-do-not-send="true">stephen.pascoe@stfc.ac.uk:</a>
</pre>
<blockquote type="cite">
<pre wrap="">Hi Bob,
This "unpublish first" requirement is news to me. We've been publishing new versions without doing this for some time. Now, we have come across difficulties with a few datasets but it's generally worked.
We don't use the --update option though. Each time we publish a new version we provide a mapfile of all files in the dataset(s). I'd recommend Sergey try doing this before removing a previous version.
If you unpublish from the Gateway first you'll loose the information in the "History" tab. For instance <a class="moz-txt-link-freetext" href="http://cmip-gw.badc.rl.ac.uk/dataset/cmip5.output2.MOHC.HadGEM2-ES.rcp85.mon.aerosol.aero.r1i1p1.html" moz-do-not-send="true">http://cmip-gw.badc.rl.ac.uk/dataset/cmip5.output2.MOHC.HadGEM2-ES.rcp85.mon.aerosol.aero.r1i1p1.html</a> shows 2 versions.
Stephen.
---
Stephen Pascoe +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
-----Original Message-----
From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu" moz-do-not-send="true">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-freetext" href="mailto:go-essp-tech-bounces@ucar.edu" moz-do-not-send="true">mailto:go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Drach, Bob
Sent: 06 January 2012 20:53
To: Serguei Nikonov; Eric Nienhouse
Cc: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu" moz-do-not-send="true">go-essp-tech@ucar.edu</a>
Subject: Re: [Go-essp-tech] Fwd: Re: Publishing dataset with option --update
Hi Sergey,
When updating a dataset, it's also important to unpublish it before publishing the new version. E.g, first run
esgunpublish<dataset_id>
The reason is that, when you publish to the gateway, the gateway software tries to *add* the new information to the existing dataset entry, rather that replace it.
--Bob
________________________________________
From: Serguei Nikonov [<a class="moz-txt-link-abbreviated" href="mailto:serguei.nikonov@noaa.gov" moz-do-not-send="true">serguei.nikonov@noaa.gov</a>]
Sent: Friday, January 06, 2012 10:45 AM
To: Eric Nienhouse
Cc: Bob Drach; <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu" moz-do-not-send="true">go-essp-tech@ucar.edu</a>
Subject: Re: [Go-essp-tech] Fwd: Re: Publishing dataset with option --update
Hi Eric,
thanks for you help. I have no any objections about any adopted versioning
policy. What I need is to know how to apply it. The ways I used did not work for
me. Hopefully, the reasons is bad things in thredds and database you pointed
put. I am cleaning them right now, then will see...
Just for clarification, if I need to update dataset (with changing version) I
create map file containing full set of files (old and new ones) and then use
this map file in esgpublish script with option --update, is it correct? Will it
be enough for creating dataset of new version? BTW, there is nothing about
version for option 'update' in esgpublish help.
Thanks,
Sergey
On 01/04/2012 04:27 PM, Eric Nienhouse wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Serguei,
Following are a few more suggestions to diagnose this publishing issue. I agree
with others on this thread that adding new files (or changing existing ones)
should always trigger a new dataset version.
It does not appear you are receiving a final "SUCCESS" or failure message when
publishing to the Gateway (with esgpublish --publish). Please try increasing
your "polling" levels in your $ESGINI file. Eg:
hessian_service_polling_delay = 10
hessian_service_polling_iterations = 500
You should see a final "SUCCESS" or "ERROR" with Java trace output at the
termination of the command.
I've reviewed the Thredds catalog for the dataset you note below:
<a class="moz-txt-link-freetext" href="http://esgdata.gfdl.noaa.gov/thredds/esgcet/1/cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1.v2.xml" moz-do-not-send="true">http://esgdata.gfdl.noaa.gov/thredds/esgcet/1/cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1.v2.xml</a>
There appear to be multiple instances of certain files within the catalog which
is a problem. The Gateway publish will fail if a particular file (URL) is
referenced multiple times with differing metadata. An example is:
*/gfdl_dataroot/NOAA-GFDL/GFDL-CM3/historical/mon/atmos/Amon/r1i1p1/v20110601/rtmt/rtmt_Amon_GFDL-CM3_historical_r1i1p1_186001-186412.nc
This file appears as two separate file versions in the Thredds catalog (one with
id ending in ".nc" and another with ".nc_0"). There should be only one reference
to this file URL in the catalog.
The previous version of the dataset in the publisher/node database may be
leading to this issue. You may need to add "--database-delete" to your
esgunpublish command to clean things up. Bob can advise on this. Note that the
original esgpublish command shown in this email thread included "--keep-version".
After publishing to the Gateway successfully, you can check the dataset details
by URL with the published dataset identifier. For example:
<a class="moz-txt-link-freetext" href="http://pcmdi3.llnl.gov/esgcet/dataset/cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1.html" moz-do-not-send="true">http://pcmdi3.llnl.gov/esgcet/dataset/cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1.html</a>
I hope this helps.
Regards,
-Eric
Serguei Nikonov wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Bob,
I still can not do anything about updating datasets. The commands you
suggested executed successfully but datasets did not appear on gateway. I
tried it several times for different datasets but result is the same.
Do you have any idea what to undertake in such situation.
Here it is some details about what I tried.
I needed to add file to dataset
cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1.
As you advised I unpublished it (esgunpublish
cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1) and then
created full mapfile (with additional file) and then publised it:
esgpublish --read-files --map new_mapfile --project cmip5 --thredd --publish
As I told there were no any errors. Dataset is in database and in thredds but
not in gateway.
The second way I tried is using mapfile containing only files to update. I
needed to substitute several existing files in dataset for new ones. I created
mapfile with only files needed to substitute:
esgscan_directory --read-files --project cmip5 -o mapfile.txt
/data/CMIP5/output1/NOAA-GFDL/GFDL-ESM2M/historical/mon/ocean/Omon/r1i1p1/v20111206
and then published it with update option:
esgpublish --update --map mapfile.txt --project cmip5 --thredd --publish.
The result is the same as in a previous case - all things are fine locally but
nothing happened on gateway.
Thanks,
Sergey
-------- Original Message --------
Subject: Re: [Go-essp-tech] Publishing dataset with option --update
Date: Thu, 29 Dec 2011 11:02:05 -0500
From: Serguei Nikonov<a class="moz-txt-link-rfc2396E" href="mailto:Serguei.Nikonov@noaa.gov" moz-do-not-send="true"><Serguei.Nikonov@noaa.gov></a>
Organization: GFDL
To: Drach, Bob<a class="moz-txt-link-rfc2396E" href="mailto:drach1@llnl.gov" moz-do-not-send="true"><drach1@llnl.gov></a>
CC: Nathan Wilhelmi<a class="moz-txt-link-rfc2396E" href="mailto:wilhelmi@ucar.edu" moz-do-not-send="true"><wilhelmi@ucar.edu></a>, "Ganzberger, Michael"
<a class="moz-txt-link-rfc2396E" href="mailto:Ganzberger1@llnl.gov" moz-do-not-send="true"><Ganzberger1@llnl.gov></a>, <a class="moz-txt-link-rfc2396E" href="mailto:go-essp-tech@ucar.edu" moz-do-not-send="true">"go-essp-tech@ucar.edu"</a><a class="moz-txt-link-rfc2396E" href="mailto:go-essp-tech@ucar.edu" moz-do-not-send="true"><go-essp-tech@ucar.edu></a>
Hi Bob,
I tried the 1st way you suggested and it worked partially - the dataset was
created om datanode with version 2 but it was not popped up on gateway. To make
sure that it's not occasional result I repeated it with another datasets with
the same result.
Now I have 2 datasets on datanode (visible in thredds server) but they are
absent on gateway:
cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1.v2
cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r2i1p1.v2.
Does it make sense to repeat esgpublish with 'publish' option?
Thanks and Happy New Year,
Sergey
On 12/21/2011 08:41 PM, Drach, Bob wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Sergey,
The way I would recommend adding new files to an existing dataset is as
follows:
- Unpublish the previous dataset from the gateway and thredds
% esgunpublish
cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1
- Add the new files to the existing mapfile for the dataset they are being
added to.
- Republish with the expanded mapfile:
% esgpublish --read-files --map newmap.txt --project cmip5 --thredds
--publish
The publisher will:
- not rescan existing files, only the new files
- create a new version to reflect the additional files
Alternatively you can create a mapfile with *only* the new files (Using
esgscan_directory), then republish using the --update command.
--Bob
On 12/21/11 8:40 AM, "Serguei Nikonov"<a class="moz-txt-link-rfc2396E" href="mailto:serguei.nikonov@noaa.gov" moz-do-not-send="true"><serguei.nikonov@noaa.gov></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Nate,
unfortunately this is not the only dataset I have a problem - there are at
least
5 more. Should I unpublish them locally (db, thredds) and than create new
version containing full set of files? What is the official way to update
dataset?
Thanks,
Sergey
On 12/20/2011 07:06 PM, Nathan Wilhelmi wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Bob/Mike,
I believe the problem is that when files were added the timestamp on the
dataset
wasn't updated.
The triple store will only harvest datasets that have files and an updated
timestamp after the last harvest.
So what likely happened is the dataset was created without files, so it
wasn't
initially harvested. Files were subsequently added, but the timestamp wasn't
updated, so it was still not a candidate for harvesting.
Can you update the date_updated timestamp for the dataset in question and
then
trigger the RDF harvesting, I believe the dataset will show up then.
Thanks!
-Nate
On 12/20/2011 11:49 AM, Serguei Nikonov wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Mike,
I am a member of data publishers group. I have been publishing considerable
amount of data without such kind of troubles but this one occurred only when
I
tried to add some files to existing dataset. Publishing from scratch works
fine
for me.
Thanks,
Sergey
On 12/20/2011 01:29 PM, Ganzberger, Michael wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Serguei,
That task is on a scheduler and will re-run every 10 minutes. If your data
does not appear after that time then perhaps there is another issue. One
issue could be that publishing to the gateway requires that you have the
role
of "Data Publisher";
"check that the account is member of the proper group and has the special
role of Data Publisher"
<a class="moz-txt-link-freetext" href="http://esgf.org/wiki/ESGFNode/FAQ" moz-do-not-send="true">http://esgf.org/wiki/ESGFNode/FAQ</a>
Mike
-----Original Message-----
From: Serguei Nikonov [<a class="moz-txt-link-freetext" href="mailto:serguei.nikonov@noaa.gov" moz-do-not-send="true">mailto:serguei.nikonov@noaa.gov</a>]
Sent: Tuesday, December 20, 2011 10:12 AM
To: Ganzberger, Michael
Cc: St§ªphane Senesi; Drach, Bob; <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu" moz-do-not-send="true">go-essp-tech@ucar.edu</a>
Subject: Re: [Go-essp-tech] Publishing dataset with option --update
Hi Mike,
thansk for suggestion but I don't have any privileges to do anything on
gateway.
I am just publishing data on GFDL data node.
Regards,
Sergey
On 12/20/2011 01:05 PM, Ganzberger, Michael wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Serguei,
I'd like to suggest this that may help you from
<a class="moz-txt-link-freetext" href="http://esgf.org/wiki/Cmip5Gateway/FAQ" moz-do-not-send="true">http://esgf.org/wiki/Cmip5Gateway/FAQ</a>
"The search does not reflect the latest DB changes I've made
You have to manually trigger the 3store harvesting. Logging as root and go
to Admin->"Gateway Scheduled Tasks"->"Run tasks" and restart the job named
RDFSynchronizationJobDetail"
Mike Ganzberger
-----Original Message-----
From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu" moz-do-not-send="true">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-freetext" href="mailto:go-essp-tech-bounces@ucar.edu" moz-do-not-send="true">mailto:go-essp-tech-bounces@ucar.edu</a>]
On Behalf Of St§ªphane Senesi
Sent: Tuesday, December 20, 2011 9:42 AM
To: Serguei Nikonov
Cc: Drach, Bob; <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu" moz-do-not-send="true">go-essp-tech@ucar.edu</a>
Subject: Re: [Go-essp-tech] Publishing dataset with option --update
Serguei
We have for some time now experienced similar problems when publishing
to the PCMDI gateway, i.e. not getting a "SUCCESS" message when
publishing . Sometimes, files are actually published (or at least
accessible through the gateway, their status being actually
"START_PUBLISHING", after esg_list_datasets report) , sometimes not. An
hypothesis is that the PCMDI Gateway load do generate the problem. We
havn't yet got a confirmation by Bob.
In contrast to your case, this happens when publishing a dataset from
scratch (I mean, not an update)
Best regards (do not expect any feeback from me since early january, yet)
S
Serguei Nikonov wrote, On 20/12/2011 18:11:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Bob,
I needed to add some missed variables to existing dataset and I found in
esgpublish command an option --update. When I tried it I've got normal
message like
INFO 2011-12-20 11:21:00,893 Publishing:
cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1, parent
=
pcmdi.GFDL
INFO 2011-12-20 11:21:07,564 Result: PROCESSING
INFO 2011-12-20 11:21:11,209 Result: PROCESSING
....
but nothing happened on gateway - new variables are not there. The files
corresponding to these variables are in database and in THREDDS catalog
but
apparently were not published on gateway.
I used command line
esgpublish --update --keep-version --map<map_file> --project cmip5
--noscan
--publish.
Should map file be of some specific format to make it works in mode I
need?
Thanks,
Sergey Nikonov
GFDL
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu" moz-do-not-send="true">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech" moz-do-not-send="true">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu" moz-do-not-send="true">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech" moz-do-not-send="true">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu" moz-do-not-send="true">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech" moz-do-not-send="true">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</blockquote>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu" moz-do-not-send="true">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech" moz-do-not-send="true">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<pre wrap="">--
Estanislao Gonzalez
Max-Planck-Institut f¨¹r Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de" moz-do-not-send="true">gonzalez@dkrz.de</a>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu" moz-do-not-send="true">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech" moz-do-not-send="true">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut f¨¹r Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de" moz-do-not-send="true">gonzalez@dkrz.de</a> </pre>
</blockquote>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut f¨¹r Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> </pre>
</blockquote>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut f¨¹r Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> </pre>
</blockquote>
</body>
</html>