[Met_help] [rt.rap.ucar.edu #90516] History for Error in mode_quilt_plot.R in v8.1
David Fillmore via RT
met_help at ucar.edu
Tue Jul 9 15:26:54 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
running the R script mode_quilt_plot.R with new v8.1 mode output gives an error, which is related to the new columns added in the output files. I will try to get a fix working. I also noted that the changes I suggested for the R script do not seem to be included, though I haven't checked in detail. I can confirm that my modified v8.0 script (attached) has the same failure, so I am guessing the main issue is the change in the number of columns.
Error in rbind(deparse.level, ...) :
numbers of columns of arguments do not match
Calls: rbind -> rbind -> rbind
Execution halted
Regards
Marion
--
Dr Marion Mittermaier Manager: Model diagnostics and novel verification
Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392 885681
E-mail: marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk/>
http://www.metoffice.gov.uk/research/people/marion-mittermaier
I am co-chair of the WMO WWRP Joint Working Group on Forecast Verification Research (JWGFVR) http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Error in mode_quilt_plot.R in v8.1
From: David Fillmore
Time: Fri Jun 07 11:29:15 2019
Hi Marion -
John and I are looking at the issue mode_quilt_plot.R.
Can you provide some sample mode output data in a tarball so we can
replicate the error?
thanks!
David
On Fri, Jun 7, 2019 at 8:46 AM marion.mittermaier at metoffice.gov.uk via
RT <
met_help at ucar.edu> wrote:
>
> Fri Jun 07 08:46:32 2019: Request 90516 was acted upon.
> Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> Queue: met_help
> Subject: Error in mode_quilt_plot.R in v8.1
> Owner: Nobody
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90516 >
>
>
> Hi,
>
> running the R script mode_quilt_plot.R with new v8.1 mode output
gives an
> error, which is related to the new columns added in the output
files. I
> will try to get a fix working. I also noted that the changes I
suggested
> for the R script do not seem to be included, though I haven't
checked in
> detail. I can confirm that my modified v8.0 script (attached) has
the same
> failure, so I am guessing the main issue is the change in the number
of
> columns.
>
> Error in rbind(deparse.level, ...) :
> numbers of columns of arguments do not match
> Calls: rbind -> rbind -> rbind
> Execution halted
>
> Regards
> Marion
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
885681
> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am co-chair of the WMO WWRP Joint Working Group on Forecast
Verification
> Research (JWGFVR)
>
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
>
>
>
------------------------------------------------
Subject: Error in mode_quilt_plot.R in v8.1
From: marion.mittermaier at metoffice.gov.uk
Time: Mon Jun 10 07:39:08 2019
Hi,
just to confirm the v8.1 script works with v8.0 output, but not with
v8.1 output. The script still has the same issues as outlined/fixed in
the attached. I resolved some of the issues with the script as was.
One of the recommendations was to fix the filenames upstream (i.e. in
MODE) to ensure that things get sorted as expected etc
I've bundled up v8.1 output and attached it here. If you look closely
then some of the output files have the new UNITS columns, whilst
others do not. This inconsistency is breaking the R script rbind
command when the number of columns change in the ingest part. The
files are here:
https://www.dropbox.com/s/k9dncbcbf58ovnq/mode_obj_txt.tgz?dl=0
Regards
Marion
-----Original Message-----
From: David Fillmore via RT <met_help at ucar.edu>
Sent: 07 June 2019 18:29
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North, Rachel
<rachel.north at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #90516] Error in mode_quilt_plot.R in
v8.1
Hi Marion -
John and I are looking at the issue mode_quilt_plot.R.
Can you provide some sample mode output data in a tarball so we can
replicate the error?
thanks!
David
On Fri, Jun 7, 2019 at 8:46 AM marion.mittermaier at metoffice.gov.uk via
RT < met_help at ucar.edu> wrote:
>
> Fri Jun 07 08:46:32 2019: Request 90516 was acted upon.
> Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> Queue: met_help
> Subject: Error in mode_quilt_plot.R in v8.1
> Owner: Nobody
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90516
> >
>
>
> Hi,
>
> running the R script mode_quilt_plot.R with new v8.1 mode output
gives
> an error, which is related to the new columns added in the output
> files. I will try to get a fix working. I also noted that the
changes
> I suggested for the R script do not seem to be included, though I
> haven't checked in detail. I can confirm that my modified v8.0
script
> (attached) has the same failure, so I am guessing the main issue is
> the change in the number of columns.
>
> Error in rbind(deparse.level, ...) :
> numbers of columns of arguments do not match
> Calls: rbind -> rbind -> rbind
> Execution halted
>
> Regards
> Marion
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
> 885681
> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am co-chair of the WMO WWRP Joint Working Group on Forecast
> Verification Research (JWGFVR)
>
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
>
>
>
------------------------------------------------
Subject: Error in mode_quilt_plot.R in v8.1
From: marion.mittermaier at metoffice.gov.uk
Time: Mon Jun 10 07:39:08 2019
Hi all,
I've found some time to figure out what's wrong with the quilt plot
script. Attached is the script (running in interactive mode to debug,
so it's not beautiful and tidy).
There were a number of issues.
1. At the most fundamental level, when inputting a file list the file
names get sorted as strings, not numbers, thus the "ls" command is
putting the file R10_T10 and R10_T11 ahead of R10_T1, and R1 _T10 and
R1_T11 above R1_T1 etc. See the attached listed. When read in in this
order, the ordering is numerically incorrect. This obviously only
becomes an issue of there are more than 9 thresholds/radii. Maybe this
could be fixed by adding leading zeros to the filenames, i.e. calling
the files R01_T01?
2. Given the file naming convention and implications thereof, two
options remained, either sort the files differently in a script before
reading them in or sorting the data after reading it in. I tried to
order the rows after reading them in, but because R functions also
sort strings so that when you have thresholds such as ">=1.62" and
">=16.0", even if you specify decreasing=FALSE, the ">=16.0" appears
before ">=1.62". I have taken away the >= and returned them to numbers
in the big mode_data data frame to ensure that when I order the array
it is actually in the correct numerical order. In the end I concluded
that this was the only way because the aggregate function also does
this and was literally undoing my earlier reordering because it also
uses the character/string ordering. This is what foxed me for a while.
I couldn't understand why, after having put the data in the right
order at the start of the script it still got screwed up. It was only
when I looked at what the aggregate function did in detail, that I
discovered the issue. Changing the thresholds to numbers removes the
use gsub just about everywhere.
3. Reuse of "ind" should ok but I was getting concerned that ind was
reused inappropriately, so I basically reinitialised it every time.
4. Lastly I have sorted out the plotting itself. I have changed the y-
axis array. Maybe with all the other fixes I have made the existing
code would be ok, but when I looked at it I couldn't work out how it
had ever worked correctly? I haven't tested putting back the original
y-axis array as what I've got definitely works. I have also made sure
that all the rows are plotted and given sensible labels. We were
losing rows because the axis was plotted in whole numbers whereas we
have thresholds such as 1.62, 2,2.5, 3.2, 4, 5 which is 6 thresholds
but if you can only plot 1,2,3,4,5 then a row is lost somewhere.
I am now pretty sure that the MODE input, extracted info and quilt
plot match up!
Regards
Marion
-----Original Message-----
From: Julie Prestopnik via RT <met_help at ucar.edu>
Sent: 21 December 2018 16:17
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Cc: North, Rachel <rachel.north at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #87963] Resolved: MODE quilt plot
Thank you for understanding and for your patience. We appreciate it!
Julie
On Fri, Dec 21, 2018 at 5:59 AM marion.mittermaier at metoffice.gov.uk
via RT < met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87963 >
>
> Hi Minna and Julie,
>
> Yes that's fine. No one here either... 😊
>
> Have a good Christmas!
>
> Marion
>
> -----Original Message-----
> From: Minna Win via RT <met_help at ucar.edu>
> Sent: 20 December 2018 17:02
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Cc: North, Rachel <rachel.north at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87963] Resolved: MODE quilt plot
>
> Hi Marion,
>
> John is out of the office for the next two weeks. We can reopen the
> ticket for you, but it won't be addressed until John returns to the
office.
>
> Thanks for your patience,
> Minna
> ---------------
> Minna Win
> NCAR
> Research Applications Lab
> Phone: 303-497-8423
> Fax: 303-497-8401
>
>
>
> On Thu, Dec 20, 2018 at 9:05 AM marion.mittermaier at metoffice.gov.uk
> via RT < met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87963 >
> >
> > Hi John,
> >
> > Rachel and I have been looking at the revised plot and are still
not
> > sure that it's correct.
> >
> > 1. The y-axis labelling of the plot is just a bit strange! The
> > threshold sequence is 1.6, 2.0, 2.5, 3.2 .... etc 2. We would
expect
> > the number of simple objects to decrease with increasing
convolution
> > radius, yet the largest radius (10) has the most objects. The
> > colourbar colours and the pixels in the actual plot do not match
the
> > values in the mode output file. E.g. for radius 10 at 1.62 the
> > number of objects is 76 but the colour suggests it's over 80. Also
> > the # objects for radius 9 is 14 and for radius 1 is 62...! That's
> > odd given the first sentence/expectation.
> > 3. Rows 6 and 8-10 appear empty / NA. There are no -9999 in the
> > input file so there should be no NA.
> >
> > So, in short, I think we still don't understand what is
happening...
> > can you please have a look again? i.e. reopen the ticket!
> >
> > Regards
> > Marion
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 14 December 2018 17:22
> > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> > Subject: [rt.rap.ucar.edu #87963] Resolved: MODE quilt plot
> >
> > According to our records, your request has been resolved. If you
> > have any further questions or concerns, please respond to this
message.
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Error in mode_quilt_plot.R in v8.1
From: David Fillmore
Time: Tue Jul 09 15:26:53 2019
User changes and bug fixes incorporated into mode_quilt_plot.R
master_8.1 and develop MET branches.
------------------------------------------------
More information about the Met_help
mailing list