[Go-essp-tech] http 403 errors
Kettleborough, Jamie
jamie.kettleborough at metoffice.gov.uk
Thu Oct 6 03:26:27 MDT 2011
Hello Martin,
I'm not sure I understand your email well enough to get my reply right -
but I *think* it's a terminology thing. As I understood it there are 2
(main?) reasons we can get 403's back from the nodes
1. because we are trying to get a dataset that we've seen in the thredds
catalogue, but which is not in the gateways (either because 1a. it
hasn't been published or 1b. because it got into a funny state during
publishing).
2. because the authorization is timing out (which I thought seemed to
happen at times of higher loads - so I've been calling this a load
problem, but maybe that's not the right term).
I can live with 1a. - I'm just taking this as a risk in the way we are
accessing the data. We can sort of live with 2 up to a point - we just
keep retrying - though it is a pain and I'd really rather not have to
(or at least get a different response so I know this is more like a
service unavailable than a 403). 1b. May be the hardest to detect and
deal with. We've had several cases of this.
Jamie
> -----Original Message-----
> From: martin.juckes at stfc.ac.uk [mailto:martin.juckes at stfc.ac.uk]
> Sent: 06 October 2011 09:44
> To: Kettleborough, Jamie; bryan.lawrence at ncas.ac.uk
> Cc: go-essp-tech at ucar.edu; stephen.pascoe at stfc.ac.uk
> Subject: RE: [Go-essp-tech] http 403 errors
>
> Hi Jamie,
>
> Prioritisation is essential, but that doesn't have to mean
> not doing other stuff.
>
> On the 403 issue: are you sure this is a data-node load
> problem, rather than an authorisation service problem? I've
> increased the delay between submission of wget scripts from
> 15 seconds to 30 in the last couple of weeks, because it
> appears that the time taken for a request to get security
> clearance, and hence the likelihood of interference with
> subsequent requests, has increased. There have been some
> changes at STFC which may be contributing to this increased
> security clearance time, on tip of any
> data-node/authorisation service congestion,
>
> Cheers,
> Martin
>
> > >-----Original Message-----
> > >From: Kettleborough, Jamie
> > >[mailto:jamie.kettleborough at metoffice.gov.uk]
> > >Sent: 05 October 2011 16:47
> > >To: Juckes, Martin (STFC,RAL,RALSP); bryan.lawrence at ncas.ac.uk
> > >Cc: go-essp-tech at ucar.edu; Pascoe, Stephen (STFC,RAL,RALSP);
> > >Kettleborough, Jamie
> > >Subject: RE: [Go-essp-tech] Incorrect dataset names
> > >
> > >Hello Martin,
> > >
> > >Sorry if I've been unclear. I don't 'not want to fix' the
> problem, I
> > >was trying to highlight the fact you have to prioritise, and
> > >somethings you'd like to do better with you have to
> compromise on.
> > >From my point of view PKI, checksums, understanding and
> elimintating
> > >those 403's we keep getting when the data nodes are under load are
> > >much higher priority issues. I'm not even sure if for other users
> > >fixing netcdf base time and forcing problems aren't higher
> priority -
> > >I *suspect* these are MUCH higher used meta-data elements
> than this
> > >project. (Yes I know we may have different views on the
> risks posed
> > >by these).
> > >
> > >But as I've said before I do not contribute/manage any
> real effort to
> > >this so you are very entitled to (down) weight my opinions
> based on
> > >this.
> > >
> > >OK I'll shut up on this issue - I think I've more than had my say.
> > >
> > >(Hope this didn't come across more forceful or grumpy than
> I meant -
> > >oh email can be such a pain)
> > >
> > >Jamie
> > >
> > >> -----Original Message-----
> > >> From: martin.juckes at stfc.ac.uk [mailto:martin.juckes at stfc.ac.uk]
> > >> Sent: 05 October 2011 15:54
> > >> To: Kettleborough, Jamie; bryan.lawrence at ncas.ac.uk
> > >> Cc: go-essp-tech at ucar.edu; stephen.pascoe at stfc.ac.uk
> > >> Subject: RE: [Go-essp-tech] Incorrect dataset names
> > >>
> > >> The difficulty is that putting information which we known to be
> > >> wrong in a central catalog is sure to lead to confusion.
> > >> That doesn't mean we know where or when the confusion
> will arise,
> > >> but it is certain to cause problems. The longer we leave it, the
> > >> harder it will be to fix. Rather than trying to image all the
> > >> places it might cause problems and trying to path up the
> symptoms,
> > >> we need to correct the catalog.
> > >>
> > >> I'm not sure why you want this not to be fixed -- are
> you thinking
> > >> that it is a big diversion of effort?
> > >>
> > >> cheers,
> > >> Martin
> > >> ________________________________________
> > >> From: Kettleborough, Jamie [jamie.kettleborough at metoffice.gov.uk]
> > >> Sent: 05 October 2011 15:48
> > >> To: Bryan Lawrence
> > >> Cc: go-essp-tech at ucar.edu; Juckes, Martin
> (STFC,RAL,RALSP); Pascoe,
> > >> Stephen (STFC,RAL,RALSP); Kettleborough, Jamie
> > >> Subject: RE: [Go-essp-tech] Incorrect dataset names
> > >>
> > >> Hello Bryan,
> > >>
> > >> If the problem with this project name is *only* within
> the context
> > >> of citation then I think you may be able to manage the
> situation by
> > >> modifying the guidelines on CMIP5 Data Citation
> > >> (http://cmip-pcmdi.llnl.gov/cmip5/citation.html?submenuheader=
> > >> 3). I don't see that the guidelines suggest using the
> project name
> > >> as a term to use in citation, but even if they did you could add
> > >> some text to the web page to help minimise the risks you
> mention.
> > >> In this case modifying this web page is probably better than the
> > >> errata page I suggested.
> > >>
> > >> As you say you can still add it to a list somewhere to fix when
> > >> it's higher up the priority list.
> > >>
> > >> Thanks,
> > >>
> > >> Jamie
> > >>
> > >> > -----Original Message-----
> > >> > From: Bryan Lawrence [mailto:bryan.lawrence at ncas.ac.uk]
> > >> > Sent: 05 October 2011 13:36
> > >> > To: Kettleborough, Jamie
> > >> > Cc: go-essp-tech at ucar.edu; martin.juckes at stfc.ac.uk;
> > >> > stephen.pascoe at stfc.ac.uk
> > >> > Subject: Re: [Go-essp-tech] Incorrect dataset names
> > >> >
> > >> >
> > >> > oh, yes, I agree, but it should be on the list ... and
> it needs
> > >> > revision before folks start writing papers and casting this
> > >> in stone
> > >> > ... as happened last time, to the detriment of anyone
> > >> having any clue
> > >> > which data folks used ....
> > >> >
> > >> > .. and it may be one more straw for some republishing camels.
> > >> >
> > >> > Cheers
> > >> > Bryan
> > >> >
> > >> > > Hello Bryan,
> > >> > >
> > >> > > Yep I appreciate its different. I just wonder of all the
> > >> > issues CMIP5
> > >> > > has to deal with how high up the priority list this one
> > >> > should be - it
> > >> > > feels (to me - based on the knowledge I have, which is
> > >> incomplete)
> > >> > > like a problem you can try to manage with a very low cost
> > >> > solution -
> > >> > > at least for a while.
> > >> > >
> > >> > > Jamie
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > > -----Original Message-----
> > >> > > > From: Bryan Lawrence [mailto:bryan.lawrence at ncas.ac.uk]
> > >> > > > Sent: 05 October 2011 12:19
> > >> > > > To: go-essp-tech at ucar.edu
> > >> > > > Cc: Kettleborough, Jamie; martin.juckes at stfc.ac.uk;
> > >> > > > stephen.pascoe at stfc.ac.uk
> > >> > > > Subject: Re: [Go-essp-tech] Incorrect dataset names
> > >> > > >
> > >> > > >
> > >> > > > Hi Jamie
> > >> > > >
> > >> > > > CMIP5 is different from IPCC!
> > >> > > >
> > >> > > > Data in the cmip5 archive will arrive after the ipcc
> > >> > cut-off etc ...
> > >> > > > and we will tag specific version of data that were
> > >> > present in time
> > >> > > > as IPCC later ...
> > >> > > >
> > >> > > > Cheers
> > >> > > > Bryan
> > >> > > >
> > >> > > > > Hello,
> > >> > > > >
> > >> > > > > I may be missing something here (either political or
> > >> > > > technical that I
> > >> > > > > don't know about) but this feels like a problem you can
> > >> > > > live with. So
> > >> > > > > what about:
> > >> > > > >
> > >> > > > > Method C:
> > >> > > > >
> > >> > > > > Don't fix anything for these old data sets.
> > >> > > > > Start a **single** errata page that all the
> gateways point
> > >> > > > to. Item 1
> > >> > > > > on on the errata page says something like:
> > >> > > > >
> > >> > > > > 'Some early datasets have been assigned an incorrect
> > >> Project in
> > >> > > > > the xxx database which affect the yyy view on the data.
> > >> > > > > For
> > >> > > > datasets with
> > >> > > > > Project = CMIP5 / IPCC Fifth Assessment Report,
> please read
> > >> > > > Project =
> > >> > > > > CMIP5'
> > >> > > > >
> > >> > > > > Jamie
> > >> > > > >
> > >> > > > > > -----Original Message-----
> > >> > > > > > From: go-essp-tech-bounces at ucar.edu
> > >> > > > > > [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
> > >> > > > > > martin.juckes at stfc.ac.uk
> > >> > > > > > Sent: 05 October 2011 10:17
> > >> > > > > > To: stephen.pascoe at stfc.ac.uk; go-essp-tech at ucar.edu
> > >> > > > > > Subject: Re: [Go-essp-tech] Incorrect dataset names
> > >> > > > > >
> > >> > > > > > Hello Stephen,
> > >> > > > > >
> > >> > > > > > yes, it should be "project=CMIP5, model=....".
> > >> > > > > >
> > >> > > > > > On A vs. B I don't have a strong preference. I
> suppose A
> > >> > > > would make
> > >> > > > > > sense if there were other issues which could be
> > >> > addressed in the
> > >> > > > > > same republishing exercise -- so it may be
> worth waiting
> > >> > > > for a few
> > >> > > > > > more opinions. The problem I can see with B is
> > >> > ensuring that all
> > >> > > > > > gateway databases are updated -- but maybe I'm
> > >> imagining more
> > >> > > > > > difficulty here than there is,
> > >> > > > > >
> > >> > > > > > cheers,
> > >> > > > > > Martin
> > >> > > > > > ________________________________________
> > >> > > > > > From: Pascoe, Stephen (STFC,RAL,RALSP)
> > >> > > > > > Sent: 05 October 2011 09:50
> > >> > > > > > To: Juckes, Martin (STFC,RAL,RALSP);
> > >> > > > > > go-essp-tech at ucar.edu
> > >> > > > > > Subject: RE: Incorrect dataset names
> > >> > > > > >
> > >> > > > > > Hi Martin,
> > >> > > > > >
> > >> > > > > > Sorry, that's an old configuration setting. To
> > >> > confirm should
> > >> > > > > > it just say "project=CMIP5, model=..."? There are
> > >> > > > > > several
> > >> > > > ways to fix
> > >> > > > > > this problem, some quicker than others.
> > >> > > > > > I would suggest:
> > >> > > > > >
> > >> > > > > > Method A (slower, more correct):
> > >> > > > > >
> > >> > > > > > 1. Change the ESG Publisher configuration 2.
> Regenerate
> > >> > > > > > THREDDS catalogs with ESG Publisher 3. Republish
> > >> to Gateway
> > >> > > > > > with ESG Publisher
> > >> > > > > >
> > >> > > > > > Method B (quick-and-dirty, once you've scripted it):
> > >> > > > > >
> > >> > > > > > 1. Change the ESG Publisher configuration 2.
> > >> Patch THREDDS
> > >> > > > > > catalogs 3. Patch the Gateway database
> > >> > > > > >
> > >> > > > > > I'm happy to use Method B and contribute the
> > >> > necessary scripts.
> > >> > > > > >
> > >> > > > > > Stephen.
> > >> > > > > >
> > >> > > > > > ---
> > >> > > > > > Stephen Pascoe +44 (0)1235 445980 Centre of
> > >> > Environmental Data
> > >> > > > > > Archival STFC Rutherford Appleton Laboratory,
> > >> Harwell Oxford,
> > >> > > > > > Didcot
> > >> > > > > > OX11 0QX, UK
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > -----Original Message-----
> > >> > > > > > From: go-essp-tech-bounces at ucar.edu
> > >> > > > > > [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
> > >> > > > > > martin.juckes at stfc.ac.uk
> > >> > > > > > Sent: 05 October 2011 09:23
> > >> > > > > > To: go-essp-tech at ucar.edu
> > >> > > > > > Subject: [Go-essp-tech] Incorrect dataset names
> > >> > > > > >
> > >> > > > > > Hello,
> > >> > > > > >
> > >> > > > > > A number of datasets, including some at BADC,
> have been
> > >> > > > > > published with "project=CMIP5 / IPCC Fifth
> > >> Assessment Report,
> > >> > > > > > .... " in the dataset name. A search for "IPCC Fifth"
> > >> > > > > > suggests we have 4021 datasets with this error, from
> > >CCCMA,
> > >> > > > > > CNRM-CERFACS, CSIRO-QCCCE, IPSL, MOHC, and NOAA-GFDL.
> > >> > We should
> > >> > > > > > develop a plan to correct this -- I guess this means
> > >> > > > > > republishing the affected datasets,
> > >> > > > > >
> > >> > > > > > cheers,
> > >> > > > > > Martin--
> > >> > > > > > Scanned by iCritical.
> > >> > > > > > _______________________________________________
> > >> > > > > > GO-ESSP-TECH mailing list GO-ESSP-TECH at ucar.edu
> > >> > > > > > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> > >> > > > > > --
> > >> > > > > > Scanned by iCritical.
> > >> > > > > > _______________________________________________
> > >> > > > > > GO-ESSP-TECH mailing list GO-ESSP-TECH at ucar.edu
> > >> > > > > > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> > >> > > > > >
> > >> > > > > _______________________________________________
> > >> > > > > GO-ESSP-TECH mailing list
> > >> > > > > GO-ESSP-TECH at ucar.edu
> > >> > > > > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> > >> > > > >
> > >> > > >
> > >> > > > --
> > >> > > > Bryan Lawrence
> > >> > > > University of Reading: Professor of Weather and Climate
> > >> > Computing.
> > >> > > > National Centre for Atmospheric Science: Director of
> > >> > Models and Data.
> > >> > > > STFC: Director of the Centre for Environmental
> Data Archival.
> > >> > > > Ph: +44 118 3786507 or 1235 445012;
> > >> > Web:home.badc.rl.ac.uk/lawrence
> > >> > > >
> > >> > >
> > >> >
> > >> > --
> > >> > Bryan Lawrence
> > >> > University of Reading: Professor of Weather and Climate
> > >Computing.
> > >> > National Centre for Atmospheric Science: Director of Models
> > >> and Data.
> > >> > STFC: Director of the Centre for Environmental Data Archival.
> > >> > Ph: +44 118 3786507 or 1235 445012;
> > >Web:home.badc.rl.ac.uk/lawrence
> > >> >
> > >> --
> > >> Scanned by iCritical.
> > >>
> --
> Scanned by iCritical.
>
More information about the GO-ESSP-TECH
mailing list