[Go-essp-tech] http 403 errors

Thu Oct 6 03:32:16 MDT 2011

Hi Jamie,

Under your point 2., I was just distinguishing between load problems at the data-node and load problems at the authorisation service. You can also get a 403 error, I think, if you submit a second request while the first is still being negotiated between the data-node and the authorisation service. I guess it doesn't matter much to you where the load problem is, except that if it is a data-node problem it should, generally, not affect all data-nodes at the same time. If it is an authorisation problem, the whole archive will be affected.

Cheers,
Martin

> >-----Original Message-----
> >From: Kettleborough, Jamie
> >[mailto:jamie.kettleborough at metoffice.gov.uk]
> >Sent: 06 October 2011 10:26
> >To: Juckes, Martin (STFC,RAL,RALSP); bryan.lawrence at ncas.ac.uk
> >Cc: go-essp-tech at ucar.edu; Pascoe, Stephen (STFC,RAL,RALSP);
> >Kettleborough, Jamie
> >Subject: RE: [Go-essp-tech] http 403 errors
> >
> >Hello Martin,
> >
> >I'm not sure I understand your email well enough to get my reply right
> >-
> >but I *think* it's a terminology thing.  As I understood it there are
> >2
> >(main?) reasons we can get 403's back from the nodes
> >
> >1. because we are trying to get a dataset that we've seen in the
> >thredds
> >catalogue, but which is not in the gateways (either because 1a. it
> >hasn't been published or 1b. because it got into a funny state during
> >publishing).
> >
> >2. because the authorization is timing out (which I thought seemed to
> >happen at times of higher loads - so I've been calling this a load
> >problem, but maybe that's not the right term).
> >
> >I can live with 1a. - I'm just taking this as a risk in the way we are
> >accessing the data. We can sort of live with 2 up to a point - we just
> >keep retrying - though it is a pain and I'd really rather not have to
> >(or at least get a different response so I know this is more like a
> >service unavailable than a 403).  1b. May be the hardest to detect and
> >deal with.  We've had several cases of this.
> >
> >Jamie
> >
> >> -----Original Message-----
> >> From: martin.juckes at stfc.ac.uk [mailto:martin.juckes at stfc.ac.uk]
> >> Sent: 06 October 2011 09:44
> >> To: Kettleborough, Jamie; bryan.lawrence at ncas.ac.uk
> >> Cc: go-essp-tech at ucar.edu; stephen.pascoe at stfc.ac.uk
> >> Subject: RE: [Go-essp-tech] http 403 errors
> >>
> >> Hi Jamie,
> >>
> >> Prioritisation is essential, but that doesn't have to mean
> >> not doing other stuff.
> >>
> >> On the 403 issue: are you sure this is a data-node load
> >> problem, rather than an authorisation service problem? I've
> >> increased the delay between submission of wget scripts from
> >> 15 seconds to 30 in the last couple of weeks, because it
> >> appears that the time taken for a request to get security
> >> clearance, and hence the likelihood of interference with
> >> subsequent requests, has increased. There have been some
> >> changes at STFC which may be contributing to this increased
> >> security clearance time, on tip of any
> >> data-node/authorisation service congestion,
> >>
> >> Cheers,
> >> Martin
> >>
> >> > >-----Original Message-----
> >> > >From: Kettleborough, Jamie
> >> > >[mailto:jamie.kettleborough at metoffice.gov.uk]
> >> > >Sent: 05 October 2011 16:47
> >> > >To: Juckes, Martin (STFC,RAL,RALSP); bryan.lawrence at ncas.ac.uk
> >> > >Cc: go-essp-tech at ucar.edu; Pascoe, Stephen (STFC,RAL,RALSP);
> >> > >Kettleborough, Jamie
> >> > >Subject: RE: [Go-essp-tech] Incorrect dataset names
> >> > >
> >> > >Hello Martin,
> >> > >
> >> > >Sorry if I've been unclear.  I don't 'not want to fix' the
> >> problem, I
> >> > >was trying to highlight the fact you have to prioritise, and
> >> > >somethings you'd like to do better with you have to
> >> compromise on.
> >> > >From my point of view PKI, checksums, understanding and
> >> elimintating
> >> > >those 403's we keep getting when the data nodes are under load
> >are
> >> > >much higher priority issues.  I'm not even sure if for other
> >users
> >> > >fixing netcdf base time and forcing problems aren't higher
> >> priority -
> >> > >I *suspect* these are MUCH higher used meta-data elements
> >> than this
> >> > >project.  (Yes I know we may have different views on the
> >> risks posed
> >> > >by these).
> >> > >
> >> > >But as I've said before I do not contribute/manage any
> >> real effort to
> >> > >this so you are very entitled to (down) weight my opinions
> >> based on
> >> > >this.
> >> > >
> >> > >OK I'll shut up on this issue - I think I've more than had my
> >say.
> >> > >
> >> > >(Hope this didn't come across more forceful or grumpy than
> >> I meant -
> >> > >oh email can be such a pain)
> >> > >
> >> > >Jamie
> >> > >
> >> > >> -----Original Message-----
> >> > >> From: martin.juckes at stfc.ac.uk
> >[mailto:martin.juckes at stfc.ac.uk]
> >> > >> Sent: 05 October 2011 15:54
> >> > >> To: Kettleborough, Jamie; bryan.lawrence at ncas.ac.uk
> >> > >> Cc: go-essp-tech at ucar.edu; stephen.pascoe at stfc.ac.uk
> >> > >> Subject: RE: [Go-essp-tech] Incorrect dataset names
> >> > >>
> >> > >> The difficulty is that putting information which we known to be
> >> > >> wrong in a central catalog is sure to lead to confusion.
> >> > >> That doesn't mean we know where or when the confusion
> >> will arise,
> >> > >> but it is certain to cause problems. The longer we leave it,
> >the
> >> > >> harder it will be to fix. Rather than trying to image all the
> >> > >> places it might cause problems and trying to path up the
> >> symptoms,
> >> > >> we need to correct the catalog.
> >> > >>
> >> > >> I'm not sure why you want this not to be fixed -- are
> >> you thinking
> >> > >> that it is a big diversion of effort?
> >> > >>
> >> > >> cheers,
> >> > >> Martin
> >> > >> ________________________________________
> >> > >> From: Kettleborough, Jamie
> >[jamie.kettleborough at metoffice.gov.uk]
> >> > >> Sent: 05 October 2011 15:48
> >> > >> To: Bryan Lawrence
> >> > >> Cc: go-essp-tech at ucar.edu; Juckes, Martin
> >> (STFC,RAL,RALSP); Pascoe,
> >> > >> Stephen (STFC,RAL,RALSP); Kettleborough, Jamie
> >> > >> Subject: RE: [Go-essp-tech] Incorrect dataset names
> >> > >>
> >> > >> Hello Bryan,
> >> > >>
> >> > >> If the problem with this project name is *only* within
> >> the context
> >> > >> of citation then I think you may be able to manage the
> >> situation by
> >> > >> modifying the guidelines on CMIP5 Data Citation
> >> > >> (http://cmip-pcmdi.llnl.gov/cmip5/citation.html?submenuheader=
> >> > >> 3).  I don't see that the guidelines suggest using the
> >> project name
> >> > >> as a term to use in citation, but even if they did you could
> >add
> >> > >> some text to the web page to help minimise the risks you
> >> mention.
> >> > >> In this case modifying this web page is probably better than
> >the
> >> > >> errata page I suggested.
> >> > >>
> >> > >> As you say you can still add it to a list somewhere to fix when
> >> > >> it's higher up the priority list.
> >> > >>
> >> > >> Thanks,
> >> > >>
> >> > >> Jamie
> >> > >>
> >> > >> > -----Original Message-----
> >> > >> > From: Bryan Lawrence [mailto:bryan.lawrence at ncas.ac.uk]
> >> > >> > Sent: 05 October 2011 13:36
> >> > >> > To: Kettleborough, Jamie
> >> > >> > Cc: go-essp-tech at ucar.edu; martin.juckes at stfc.ac.uk;
> >> > >> > stephen.pascoe at stfc.ac.uk
> >> > >> > Subject: Re: [Go-essp-tech] Incorrect dataset names
> >> > >> >
> >> > >> >
> >> > >> > oh, yes, I agree, but it should be on the list ... and
> >> it needs
> >> > >> > revision before folks start writing papers and casting this
> >> > >> in stone
> >> > >> > ... as happened last time, to the detriment of anyone
> >> > >> having any clue
> >> > >> > which data folks used ....
> >> > >> >
> >> > >> > .. and it may be one more straw for some republishing camels.
> >> > >> >
> >> > >> > Cheers
> >> > >> > Bryan
> >> > >> >
> >> > >> > > Hello Bryan,
> >> > >> > >
> >> > >> > > Yep I appreciate its different.  I just wonder of all the
> >> > >> > issues CMIP5
> >> > >> > > has to deal with how high up the priority list this one
> >> > >> > should be - it
> >> > >> > > feels (to me - based on the knowledge I have, which is
> >> > >> incomplete)
> >> > >> > > like a problem you can try to manage with a very low cost
> >> > >> > solution -
> >> > >> > > at least for a while.
> >> > >> > >
> >> > >> > > Jamie
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > > -----Original Message-----
> >> > >> > > > From: Bryan Lawrence [mailto:bryan.lawrence at ncas.ac.uk]
> >> > >> > > > Sent: 05 October 2011 12:19
> >> > >> > > > To: go-essp-tech at ucar.edu
> >> > >> > > > Cc: Kettleborough, Jamie; martin.juckes at stfc.ac.uk;
> >> > >> > > > stephen.pascoe at stfc.ac.uk
> >> > >> > > > Subject: Re: [Go-essp-tech] Incorrect dataset names
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > Hi Jamie
> >> > >> > > >
> >> > >> > > > CMIP5 is different from IPCC!
> >> > >> > > >
> >> > >> > > > Data in the cmip5 archive will arrive after the ipcc
> >> > >> > cut-off etc ...
> >> > >> > > > and we will tag specific version of data that were
> >> > >> > present in time
> >> > >> > > > as IPCC later ...
> >> > >> > > >
> >> > >> > > > Cheers
> >> > >> > > > Bryan
> >> > >> > > >
> >> > >> > > > > Hello,
> >> > >> > > > >
> >> > >> > > > > I may be missing something here (either political or
> >> > >> > > > technical that I
> >> > >> > > > > don't know about) but this feels like a problem you can
> >> > >> > > > live with.  So
> >> > >> > > > > what about:
> >> > >> > > > >
> >> > >> > > > > Method C:
> >> > >> > > > >
> >> > >> > > > > Don't fix anything for these old data sets.
> >> > >> > > > > Start a **single** errata page that all the
> >> gateways point
> >> > >> > > > to.  Item 1
> >> > >> > > > > on on the errata page says something like:
> >> > >> > > > >
> >> > >> > > > > 'Some early datasets have been assigned an incorrect
> >> > >> Project in
> >> > >> > > > > the xxx database which affect the yyy view on the data.
> >> > >> > > > > For
> >> > >> > > > datasets with
> >> > >> > > > > Project = CMIP5 / IPCC Fifth Assessment Report,
> >> please read
> >> > >> > > > Project =
> >> > >> > > > > CMIP5'
> >> > >> > > > >
> >> > >> > > > > Jamie
> >> > >> > > > >
> >> > >> > > > > > -----Original Message-----
> >> > >> > > > > > From: go-essp-tech-bounces at ucar.edu
> >> > >> > > > > > [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
> >> > >> > > > > > martin.juckes at stfc.ac.uk
> >> > >> > > > > > Sent: 05 October 2011 10:17
> >> > >> > > > > > To: stephen.pascoe at stfc.ac.uk; go-essp-tech at ucar.edu
> >> > >> > > > > > Subject: Re: [Go-essp-tech] Incorrect dataset names
> >> > >> > > > > >
> >> > >> > > > > > Hello Stephen,
> >> > >> > > > > >
> >> > >> > > > > > yes, it should be "project=CMIP5, model=....".
> >> > >> > > > > >
> >> > >> > > > > > On A vs. B I don't have a strong preference. I
> >> suppose A
> >> > >> > > > would make
> >> > >> > > > > > sense if there were other issues which could be
> >> > >> > addressed in the
> >> > >> > > > > > same republishing exercise -- so it may be
> >> worth waiting
> >> > >> > > > for a few
> >> > >> > > > > > more opinions. The problem I can see with B is
> >> > >> > ensuring that all
> >> > >> > > > > > gateway databases are updated -- but maybe I'm
> >> > >> imagining more
> >> > >> > > > > > difficulty here than there is,
> >> > >> > > > > >
> >> > >> > > > > > cheers,
> >> > >> > > > > > Martin
> >> > >> > > > > > ________________________________________
> >> > >> > > > > > From: Pascoe, Stephen (STFC,RAL,RALSP)
> >> > >> > > > > > Sent: 05 October 2011 09:50
> >> > >> > > > > > To: Juckes, Martin (STFC,RAL,RALSP);
> >> > >> > > > > > go-essp-tech at ucar.edu
> >> > >> > > > > > Subject: RE: Incorrect dataset names
> >> > >> > > > > >
> >> > >> > > > > > Hi Martin,
> >> > >> > > > > >
> >> > >> > > > > > Sorry, that's an old configuration setting.  To
> >> > >> > confirm should
> >> > >> > > > > > it just say "project=CMIP5, model=..."?  There are
> >> > >> > > > > > several
> >> > >> > > > ways to fix
> >> > >> > > > > > this problem, some quicker than others.
> >> > >> > > > > > I would suggest:
> >> > >> > > > > >
> >> > >> > > > > > Method A (slower, more correct):
> >> > >> > > > > >
> >> > >> > > > > >  1. Change the ESG Publisher configuration  2.
> >> Regenerate
> >> > >> > > > > > THREDDS catalogs with ESG Publisher  3. Republish
> >> > >> to Gateway
> >> > >> > > > > > with ESG Publisher
> >> > >> > > > > >
> >> > >> > > > > > Method B (quick-and-dirty, once you've scripted it):
> >> > >> > > > > >
> >> > >> > > > > >  1. Change the ESG Publisher configuration  2.
> >> > >> Patch THREDDS
> >> > >> > > > > > catalogs  3. Patch the Gateway database
> >> > >> > > > > >
> >> > >> > > > > > I'm happy to use Method B and contribute the
> >> > >> > necessary scripts.
> >> > >> > > > > >
> >> > >> > > > > > Stephen.
> >> > >> > > > > >
> >> > >> > > > > > ---
> >> > >> > > > > > Stephen Pascoe  +44 (0)1235 445980 Centre of
> >> > >> > Environmental Data
> >> > >> > > > > > Archival STFC Rutherford Appleton Laboratory,
> >> > >> Harwell Oxford,
> >> > >> > > > > > Didcot
> >> > >> > > > > > OX11 0QX, UK
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > -----Original Message-----
> >> > >> > > > > > From: go-essp-tech-bounces at ucar.edu
> >> > >> > > > > > [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
> >> > >> > > > > > martin.juckes at stfc.ac.uk
> >> > >> > > > > > Sent: 05 October 2011 09:23
> >> > >> > > > > > To: go-essp-tech at ucar.edu
> >> > >> > > > > > Subject: [Go-essp-tech] Incorrect dataset names
> >> > >> > > > > >
> >> > >> > > > > > Hello,
> >> > >> > > > > >
> >> > >> > > > > > A number of datasets, including some at BADC,
> >> have been
> >> > >> > > > > > published with "project=CMIP5 / IPCC Fifth
> >> > >> Assessment Report,
> >> > >> > > > > > .... " in the dataset name. A search for "IPCC Fifth"
> >> > >> > > > > > suggests we have 4021 datasets with this error, from
> >> > >CCCMA,
> >> > >> > > > > > CNRM-CERFACS, CSIRO-QCCCE, IPSL, MOHC, and NOAA-GFDL.
> >> > >> > We should
> >> > >> > > > > > develop a plan to correct this -- I guess this means
> >> > >> > > > > > republishing the affected datasets,
> >> > >> > > > > >
> >> > >> > > > > > cheers,
> >> > >> > > > > > Martin--
> >> > >> > > > > > Scanned by iCritical.
> >> > >> > > > > > _______________________________________________
> >> > >> > > > > > GO-ESSP-TECH mailing list GO-ESSP-TECH at ucar.edu
> >> > >> > > > > > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >> > >> > > > > > --
> >> > >> > > > > > Scanned by iCritical.
> >> > >> > > > > > _______________________________________________
> >> > >> > > > > > GO-ESSP-TECH mailing list GO-ESSP-TECH at ucar.edu
> >> > >> > > > > > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >> > >> > > > > >
> >> > >> > > > > _______________________________________________
> >> > >> > > > > GO-ESSP-TECH mailing list
> >> > >> > > > > GO-ESSP-TECH at ucar.edu
> >> > >> > > > > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > > --
> >> > >> > > > Bryan Lawrence
> >> > >> > > > University of Reading:  Professor of Weather and Climate
> >> > >> > Computing.
> >> > >> > > > National Centre for Atmospheric Science: Director of
> >> > >> > Models and Data.
> >> > >> > > > STFC: Director of the Centre for Environmental
> >> Data Archival.
> >> > >> > > > Ph: +44 118 3786507 or 1235 445012;
> >> > >> > Web:home.badc.rl.ac.uk/lawrence
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >> > --
> >> > >> > Bryan Lawrence
> >> > >> > University of Reading:  Professor of Weather and Climate
> >> > >Computing.
> >> > >> > National Centre for Atmospheric Science: Director of Models
> >> > >> and Data.
> >> > >> > STFC: Director of the Centre for Environmental Data Archival.
> >> > >> > Ph: +44 118 3786507 or 1235 445012;
> >> > >Web:home.badc.rl.ac.uk/lawrence
> >> > >> >
> >> > >> --
> >> > >> Scanned by iCritical.
> >> > >>
> >> --
> >> Scanned by iCritical.
> >>
-- 
Scanned by iCritical.