<div dir="ltr">Dear Dave,<div>Many thanks for your help and guide, my problem has been fixed now. Very useful suggestions!</div><div>Best wishes,</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 12, 2021 at 11:57 PM Dave Allured - NOAA Affiliate <<a href="mailto:dave.allured@noaa.gov">dave.allured@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>If you can't update the model, then I suggest method 3 to write a new file. This starts with Dennis's code to convert to packed in memory. Untested:</div><br><div class="gmail_quote"> f = addfile ("<a href="http://air.2019.nc" target="_blank">air.2019.nc</a>", "r")<br> xFloat = f->air<br> xShort = pack_values (xFloat, "short", False)</div><div class="gmail_quote"> fout = addfile ("<a href="http://new_air.2019.nc" target="_blank">new_air.2019.nc</a>", "c")</div><div class="gmail_quote"> fout->air = xShort</div><div class="gmail_quote"> exit</div><div class="gmail_quote"><br></div><div class="gmail_quote">This method 3 is not in the documentation, but it should be. This only works when all necessary metadata -- especially coordinate variables -- is preserved or attached in memory before writing output. Fortunately in this case, the <b>pack_values</b> function should preserve metadata. Use <b>printVarSummary</b> liberally to check intermediate results and verify assumptions.</div><div class="gmail_quote"><br><div dir="ltr" class="gmail_attr"><br></div><div dir="ltr" class="gmail_attr">On Tue, Jan 12, 2021 at 10:18 AM Dennis Shea via ncl-talk <<a href="mailto:ncl-talk@mailman.ucar.edu" target="_blank">ncl-talk@mailman.ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>As noted before, the BEST solution is to have the model code altered too accommodate the original values. IE: to not require packed values.</div><div><br></div><div>====</div><div>The approach of created netCDF files containing variables with the scale_factor and add_offset attributes is cumbersome. You will have two files (original: netCDF4) and a (much larger) "packed" file. That is for EVERY variable you need. It is best to fix the model source code.</div><div><br></div><div>Also, deficiencies<b> </b>in the model code will have to be continuously addressed by NCL thumb-in-the-dike solutions.<br></div><div>=====</div><div><br></div><div>Your <span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000">netcdf new-hgt.2019</font></span> file has</div><div><br></div><div><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000">dimensions:</font></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000"><span style="white-space:pre-wrap"> </span>time = UNLIMITED ; // (0 currently)</font></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><br></p></div><div>I have no idea why there is no 'time' dimension associated with 'hgt',</div><div>Is the source variable just (lat,lon)</div><div><br></div><div>If so, delete</div><div><pre>;===================================================================
; make time an UNLIMITED dimension; recommended for most applications
;===================================================================
<a href="http://www.ncl.ucar.edu/Document/Functions/Built-in/filedimdef.shtml" target="_blank"><strong>filedimdef</strong></a>(ncdf,"time",-1,True) </pre></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 12, 2021 at 6:16 AM Setareh Rahimi <<a href="mailto:setareh.rahimi@gmail.com" target="_blank">setareh.rahimi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><font face="arial, sans-serif" color="#000000">Dear Dennis,</font><div><font face="arial, sans-serif" color="#000000">Many thanks for your help. Once I follow the <span style="white-space:pre-wrap"><a href="http://www.ncl.ucar.edu/Applications/method_1.shtml" style="white-space:pre-wrap" target="_blank">http://www.ncl.ucar.edu/Applications/method_1.shtml</a> method to save the new nc file, I face some errors, in fact I am too new in writhing nc files. That method just give me :</span></font></div><div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000">netcdf new-hgt.2019 {</font></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000">dimensions:</font></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000"><span style="white-space:pre-wrap"> </span>time = UNLIMITED ; // (0 currently)</font></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;min-height:16px"><font style="background-color:rgb(147,196,125)" face="arial, sans-serif" color="#000000"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000">// global attributes:</font></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000"><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>:creation_date = "Tue Jan 12 16:33:58 +0330 2021" ;</font></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000"><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>:Conventions = "CF-1.0 " ;</font></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000"><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>:source_file = "NCEP/DOE AMIP-II Reanalysis (Reanalysis-2) Model" ;</font></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(147,196,125)"><font face="arial, sans-serif" color="#000000"><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>:title = "4x daily NCEP/DOE Reanalysis 2" ;</font></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif" color="#000000"><br></font></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif" color="#000000">How can I have the correct nc file?</font></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif" color="#000000"><br></font></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif" color="#000000">Best wishes</font></span></p></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 11, 2021 at 11:35 PM Dennis Shea <<a href="mailto:shea@ucar.edu" target="_blank">shea@ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I was just typing a response that contained the same information as Dave's.</div><div><br></div><div><div>Likely, you have no control "the model I am running only accept packed nc file with scale factor."</div><div>As noted by DaveA, the file you are using is netCDF-4</div><div>[ Perhaps too much info but netCDF-4's compression is described <a href="https://www.unidata.ucar.edu/blogs/developer/entry/netcdf_compression" target="_blank"><b>here ]</b></a><br><div>===============</div></div><div> <br></div><div>%> <b>ncdump -k</b> <a href="http://air.2019.nc" target="_blank">air.2019.nc</a></div><div>netCDF-4 classic model</div><div><br></div><div>In fact, I believe *all* the netCDF files created by "NCEP-DOE AMIP-II Reanalysis" (and others) will be netCDF-4.</div><div><br></div><div>Maybe you could have the model developers alter the code.</div><div>Really, they should have checked if the 'scale_factor' and/or 'add_offset' attributes are present.</div><div>If not, use the variable directly.</div><div>====================</div><div><br></div><div> f = addfile("<a href="http://air.2019.nc" target="_blank">air.2019.nc</a>","r")</div><div> x = f->air</div><div> printVarSummary(x) ; full uncompressed values</div><div> ; *** LOOK AT THE SIZE****</div><div> ; It is MUCH LARGER than the ENTIRE netCDF file<br></div><div><pre> xShort = <strong>pack_values</strong>(xFloat, "short", False)
<a href="http://www.ncl.ucar.edu/Document/Functions/Built-in/printVarSummary.shtml" target="_blank"><strong>printVarSummary</strong></a>(xShort)<br><br> xShort = pack_values(x, "short", False) <br> printVarSummary(xShort) ; *** LOOK AT THE SIZE****<br> ; HALF the size of 'x' <br> ; but much larger than the compressed value!<br><br></pre><pre> The use (say) the following to write 'xShort' a new (much larger) file that the original<br><br><a href="http://www.ncl.ucar.edu/Applications/method_1.shtml" target="_blank"><b>http://www.ncl.ucar.edu/Applications/method_1.shtml</b></a><br><br></pre><pre>Then use that new file for the model.<br><br></pre><pre>Likely,<b> EVERY </b>variable will have to be done like this for the "model you are using<b><br>===<br><br></b></pre><pre><b>It is much easier for the model code to be altered.<br><br></b></pre><pre><b>Good Luck<br></b></pre></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 11, 2021 at 12:46 PM Dave Allured - NOAA Affiliate via ncl-talk <<a href="mailto:ncl-talk@mailman.ucar.edu" target="_blank">ncl-talk@mailman.ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I see. I suggest the best solution would be a simple upgrade to the model code, so that it could read both packed and unpacked air temp files. This will enable the model to use future data updates without extra processing.</div><div><br></div><div>In fortran, this would be an if statement to check whether there is a scale_factor attribute. If not, then skip reading scale_factor and add_offset, also skip the unpacking statement or loop that converts packed data values to unpacked. Just read in the unpacked values directly. You would need two read statements, the original one for the packed data type, and a new one for the unpacked data type (fortran real).</div><div><br></div><div>If something prevents you from changing the model code, then just convert the air temp file to packed using the NCL <b>pack_values</b> function. See the general documentation for how to read and write a netcdf file.</div><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 11, 2021 at 12:00 PM Setareh Rahimi <<a href="mailto:setareh.rahimi@gmail.com" target="_blank">setareh.rahimi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Dear Dave,</div><div dir="auto">Thank you for your reply. Yes I am sure, because the model I am running only accept packed nc file with scale factor. </div><div dir="auto">Please have a look at attached file showing the error I face while running the model with mentioned nc file.</div><div dir="auto">Best regard</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 11, 2021 at 22:20 Dave Allured - NOAA Affiliate <<a href="mailto:dave.allured@noaa.gov" target="_blank">dave.allured@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>That file is already compressed to 31% of its natural array size, using netcdf-4 internal compression. Are you sure you want to do that? If yes, then use the NCL function <b>pack_values</b>.</div></div><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 11, 2021 at 11:07 AM Setareh Rahimi via ncl-talk <<a href="mailto:ncl-talk@mailman.ucar.edu" target="_blank">ncl-talk@mailman.ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear all,<div>I need to convert an unpacked netCDF file to a packed one (<a href="ftp://ftp2.psl.noaa.gov/Datasets/ncep.reanalysis2/pressure/air.2019.nc" target="_blank">ftp://ftp2.psl.noaa.gov/Datasets/ncep.reanalysis2/pressure/air.2019.nc</a>), I wonder if this possible using NCL?</div><div>Many thanks in advance,</div><div>Best wishes,<br>-- <br><div dir="ltr">S.Rahimi</div></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">S.Rahimi<br><br></div>