05.06.08

Wish list: wonder-code-coverage tool

Posted in Eclipse, programmer productivity at 4:06 pm by ducky

In previous posts, I talked about wanting a code-coverage tool to use while debugging, and how EclEmma did most of what I wanted. After my user study, where I dissected in detail how seven professional programmers did four twenty-minute programming tasks, I am even more convinced that a wonder tool would be highly, highly useful to people. I want the wonder tool to colour and present source based on whether code was executed between a user-defined start and stop point.

1. As I mentioned in the first post, I’d like to be able to start and stop collecting code coverage information. It would be nice if I could set START LOGGING and STOP LOGGING breakpoints, so that lines of source would be coloured based on whether they were executed inside the “active” regions. Being able to do this would mean that I could greatly narrow down what code was involved in a particular feature, and not have all the code touched by initialization routines touched.
2. I’d like the number of times that a method gets executed to influence the Mylyn degree of interest. I’d like Mylyn to restrict what classes/methods Eclipse showed me (initially) to only those which were executed between the START LOGGING and STOP LOGGING breakpoints.

For example, I could bring up a GUI application, hit the Suspend button, set a START LOGGING breakpoint, hit resume, and resize a window, and hit the STOP LOGGING button. The PackageExplorer would then only show me the methods involved in event handling and resizing a window. Most of the event-handling code would probably be done by .class files, so it probably would only be the code related to event handling.

I believe that this would have helped my user study subjects enormously. They overwhelmingly used static tracing (i.e. jumping to references or declarations of selected methods) to find where things happened in the code, but they frequently got caught by differences between what static tracing told them and what actually happened. Examples:

• All seven subjects searched for an unusual string related to the given task in a GUI, which seemed perfectly reasonable. However, that string appeared in a superclass of the object that was actually instantiated. Jumping to declarations (via F4 or hyperlinks) took them preferentially to methods in the superclass. Four of the seven subjects never noticed that there were interesting subclasses. If, however, they had been using my wonder tool, the first time they jumped to a method that the subclass overrode, they would have seen from the source code colouring that that method never got hit.
• In one task, important code was not broken, but flat-out missing. Subjects usually found the right method pretty quickly, but then assumed that — because that method didn’t have the needed code — that it didn’t get executed. “Oh, here’s where the start-of-line decoration code is, now I just have to figure out where the end-of-line decoration code is.” Had they seen that that method got executed, they might have realized faster that the end-of-line decoration code was simply missing.
• In another task related to incorrect output, to statically trace through to bug’s location meant going through six methods whose names and comments made it look like they were related only to input. Setting a START LOGGING breakpoint where it wrote “Writing output…” to stdout and a STOP LOGGING breakpoint at the next stdout status message, then the subjects would have been able to see that the “input” methods were involved in writing the output, so they might have been willing to push through them.
• One task that asked the subjects to do something when a pane got resized was so difficult that nobody solved it. In addition to problems with language (Is that pane a Window, a View, a Drawing, a Pane, or a Panel?), it can be tricky to get from the place where a GUI widget gets set up to where the event gets handled. (In this case, they needed to find what methods were executed when a pane was resized.)

With my wonder tool, subjects would be able to get a list of all the classes and methods that were involved in the resize. By browsing that list, they would have a better shot at finding a method that they could use.

I think it wouldn’t be too difficult to implement this — EclEmma has already done the lion’s share of the work — but I doubt it’s compatible with finishing my thesis. 🙁

Permalink