[Go-essp-tech] output1 vs. output2

[Kettleborough, Jamie]

Hello Karl,
 
I think its an error if someone tries to publish an unknown diagnostic
with product=output.  *But* do we know the scale of the problem of data
already in the system?  How much data has the wrong product assigned
(either in the DRS id, or in the files netCDF attribute)?  Is there an
expectation that this data will be reassigned to the correct product?
Or will we just live with it?  If its sufficiently large volume/number
of files *and* we are going to 'live with' the legacy problem then I
don't see a strong reason to fix new data - users already have to deal
with the wrong products.  (but happy to be wrong on this)
 
Jamie
 
________________________________

From: go-essp-tech-bounces at ucar.edu
[mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Karl Taylor
Sent: 28 February 2012 16:51
To: stephen.pascoe at stfc.ac.uk
Cc: Drach, Bob; go-essp-tech at ucar.edu
Subject: Re: [Go-essp-tech] output1 vs. output2



	Dear all,
	
	 thanks for identifying the likely bug. 
	
	While fixing this problem, we should check whether the publisher
correctly treats product="unsolicited" data sets correctly.  Is there an
"if test" for this case?  What should we do with data where product as
been set by the data writer to "output", but "the product cannot be
determined" by the publisher?  Should we set product to "output" as the
publisher apparently now does, or should be throw an error, or should be
set it to "unsolicited"?
	
	Please vote. (and give reasons for your view).
	
	thanks,
	Karl
	
	On 2/28/12 3:14 AM, stephen.pascoe at stfc.ac.uk wrote: 

		Karl, Bob,

		

		I think this is a bug in ESG publisher.  I've just taken
a look at the code and it converts the variable to lower-case before
looking it up in an internal table, however the internal table has
"sfcWind" so it never finds the variable.  Other variables that may be
affected are tasAdjust, tsAdjust and parasolRefl

		

		### In esgcet.config.cmip5_product

		def getProduct(cmor_table, variable, experiment, year1,
year2):

		    """Get the DRS product value associated with the
file.

		    Returns

		      'output1' for datasets to be replicated,

		      'output2' for datasets outside the replicated
datasets,

		      'output' if the product cannot be determined.

		    """

		    cmor_table = cmor_table.lower()

		    variable = variable.lower()

		

		### In esgcet.config.cmip5_tables

		# cmor_variables : cmor_table => { variable =>
(priority, dimension_names)}

		cmor_variables = {   '3hr': {   'clt': (1, ['longitude',
'latitude', 'time']),

		    # ...

		    'amon': {   'ccb': (1, ['longitude', 'latitude',
'time']),

		                # ...

		                'sfcWind': (1, ['longitude', 'latitude',
'time', 'height10m']),

		                'ta': (1, ['longitude', 'latitude',
'plevs', 'time']),

		                'tas': (1, ['longitude', 'latitude',
'time', 'height2m']),

		                'tasAdjust': (1, ['longitude',
'latitude', 'time', 'height2m']),

		

		

		Cheers,

		Stephen

		

		---

		Stephen Pascoe  +44 (0)1235 445980

		Centre of Environmental Data Archival

		STFC Rutherford Appleton Laboratory, Harwell Oxford,
Didcot OX11 0QX, UK

		

		From: go-essp-tech-bounces at ucar.edu
[mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Karl Taylor
		Sent: 28 February 2012 00:52
		To: go-essp-tech at ucar.edu
		Subject: [Go-essp-tech] output1 vs. output2

		

		Hi all,
		
		I have noticed (using the new p2p search engine, which
works much faster, and I'm pretty sure is accurate) that
sfcWind_Amon_historical output is designated as "output 1" for some
models (from IPSL, MOHC, MPI-M, and NOAA-GFDL), but "output2" for the
rest.  I think "output1" is correct.  
		
		Can anyone explain why the publisher correctly
characterized the output from some groups, but not for the rest?
		
		thanks,
		Karl


		-- 
		Scanned by iCritical. 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20120229/c6e00a74/attachment-0001.html