02.16.07
Posted in programmer productivity, review at 5:12 pm by ducky
I got started on looking at productivity variations again and just couldn’t stop. I found Programmer performance and the effects of the workplace by DeMarco and Lister (yes, the authors of Peopleware). The paper is well worth a read.
In their study, they had 83 pairs of professional programmers work all on the same well-specified programming problem using their normal language tools in their normal office environment. Each pair was two people from the same company, but they were not supposed to work together.
- They found a strong correlation between the two halves of a company pair, which may be in part from
- a pretty stunning correlation between the office environment and productivity
- (or perhaps due to different companies having radically different tools/libraries/languages/training/procedures, which they didn’t discuss)
- The average time-to-complete was about twice the fastest time-to-complete.
- Cobol programmers as a group took much longer to finish than the other programmers. (Insert your favorite Cobol joke here.)
Over and over, I keep seeing that the median time to complete a single task is on the order of 2x to 4x times the fastest, not 100x. This study seems to imply that a great deal of that difference is due not to the individual’s inherent capabilities, but rather the environment where they work.
Permalink
Posted in programmer productivity, review at 4:17 pm by ducky
To reiterate, there’s a paper by Sackman et al from 1966 that people have seized upon to show a huge variation in programmer productivity, a paper by Dickey in 1981 that refuted Sackman pretty convincingly, and an article by Curtis in the same issue as Dickey’s. I didn’t talk much about the Dickey paper, but Tony Bowden has a good blog posting on the Dickey paper, where Dickey reports on a more reasonable interpretation of numbers from the Sackman’s data.
(Basically, Sackman compared the time it took to complete a task using a batch system against the time it took using a timeshare system. This was interesting in 1966 when they were trying to quantify the benefit of timeshare systems, but it’s not good to look at those numbers and say, “Ah, see, there is a huge variation in programmers!”)
Because I like making pretty histograms, here are the Sackman numbers via Dickey via Bowden — the way they ought to be interpreted. These show the time to complete for two tasks, the “Algebra” task and the “Maze” task.
The small sample size hurts, but (as in the Curtis data and the Ko data) I don’t see an order of magnitude difference in completion speed.
Permalink
Posted in Random thoughts, University life at 10:16 am by ducky
Today, my alma mater, the University of Illinois, announced that Chief Illiniwek would no longer perform at athletic events.
The Chief’s performances had been hugely divisive, with one side contending that the Chief was offensive to Native Americans and the other insisting that his portrayal demonstrated respect for Native Americans. Both sides were correct, which made it so very difficult.
The Chief was not a mascot. He didn’t stand on the sidelines and lead cheers, he didn’t clown around, he didn’t mug for the cameras, or throw t-shirts into the audience. He came out at halftime, danced, and left. He comported himself with dignity and gravity at all times. The dance was physically very challenging. The students respected and honored the Chief. It was very respectful, compared to what it could have been.
On the other hand, the Chief was not authentic. The dance, while containing some elements of Native dances, was not totally authentic. In particular, the last thing the Chief did before leaving the field was mid-air splits — jumping up, spreading his legs up and out in a V, and touching his toes. The costume was Lakota, not Illiniwek. (The Illinois Natives were pretty efficiently disposed of, so there aren’t a lot of records of what they wore or how they danced.) Furthermore, it is a strange borrowing to put Native culture into this context.
What pushed me over to the anti-Chief side was to think about how I would feel on seeing a bad adaptation of important Western culture out of context. Imagine that you’re at a soccer game in Japan, and at halftime, they announce that it’s time for The Pope. Imagine the crowd going wild as a guy in Greek Orthodox regalia solemnly runs out onto the field, and does something sort of like an Irish jig, ending with mid-air splits. Even as a non-Catholic, that would make me highly uncomfortable and perhaps a bit angry. If I were Catholic, I’d be furious.
Permalink
02.15.07
Posted in programmer productivity, review at 6:18 pm by ducky
(See my previous programmer productivity article for some context.)
Martin Robillard did a study in conjunction with my advisor. In it, he had five programmers work on a relatively complex task for two hours. Two of the programmers finished in a little over an hour, one finished in 114 minutes, and two did not finish in two hours:
Robillard carefully looked at five subtasks that were part of doing the main task; there was a very sharp distinction between the three who finished and the two who did not. The two who didn’t finish only got one of the subtasks “right”. S for “Success” means everything worked. I for “Inelegant” means it worked but was kind of kludgy. B for “Buggy” means that there were cases where it didn’t work; U for “Unworkable” means that it usually didn’t work; NA for “Not Attempted” means they didn’t even try to do that subtask.
| Coder ID |
Time to finish |
Check box |
Disabling |
Deletion |
Recovery |
State reset |
Years exper. |
| #3 |
72 min |
S! |
S! |
S! |
S! |
S! |
5 |
| #2 |
62 min |
S! |
S! |
S! |
S! |
B |
3 |
| #4 |
114 min |
I |
S! |
B |
S! |
B |
5 |
| #1 |
125 min (timed out) |
S! |
U |
U |
U |
NA |
1 |
| #5 |
120 min (timed out) |
S! |
U |
U |
NA |
NA |
1 |
Because coder #1 and coder #5 timed out, I don’t know how much of a conclusion I can draw from this data about what the range of time taken is. From this small sample size, it does look like experience matters.
This study did have some interesting observations:
- Everyone had to spend an hour looking at the code before they started making changes. Some spent this exploration period writing down what they were going to change, then followed that script during the coding phase. The ones who did were more successful than the ones who didn’t.
- The more successful coders (#2 and #3) spread their changes around as appropriate. The others tried to make all of the changes in one place.
- The more successful coders looked at more methods, and they were more directed about which ones they looked at. The second column in the table below is a ratio of the number of methods that they looked at via cross-references and keyword searching over the total number of methods that they looked at. The less-successful coders found their methods more frequently by scrolling, browsing, and returning to an already-open window.
| Coder ID |
Number of methods examined |
intent-driven:total ratio |
Time to finish |
| #3 |
34 |
31.7% |
72 min |
| #2 |
27.5 |
23.3% |
62 min |
| #4 |
27.5 |
30.8% |
114 min |
| #1 |
8.5 |
2.0% |
125 min (timed out) |
| #5 |
17.5 |
10.7% |
120 min (timed out) |
- From limited data, they conclude that skimming the source isn’t very useful — that if you don’t know what you are looking for, you won’t notice it when it passes your eyeballs.
Permalink
Posted in Family, Married life at 11:34 am by ducky
Before nephew Yeshe came to live with us for year, my husband and I talked a bit about getting his buy-in on how the family should run. Being Silicon Valley corporate geeks, we wrote a mission statement for our family:
Family is a happy place where everyone works together to help each other achieve their goals.
Yeshe came and went, but the mission statement lives on. I really like it, and am really glad we sat down and wrote it out.
Permalink
02.12.07
Posted in Canadian life, University life at 10:51 pm by ducky
I gave a talk last week at Web Directions North that I didn’t get to rehearse as much as I’d wanted due to snow, flooding, a rock slide, and laryngitis.
Snow: This might not sound like a biggie, but the Annual Green College Ski Trip happened on Feb 3 and 4. I had been ignoring my husband for days on end as I worked on my presentation, and I couldn’t really spend the domestic credits that it would take to skip the trip.
Flood: While Jim, I, and 26 other residents were on the ski trip, one of my neighbor’s sprinklers failed and gushed water into his room for 40 minutes before the firefighters and residents got it shut off. (Yes, we collectively are looking into the emergency procedures around here.) While our room ultimately had no water damage, nobody knew if they other sprinklers were also going to go off or not, or if the water would leak into our room.
Resident Mika McKinnon, who impresses me more and more as time goes on, basically took charge and organized posses of people to go into neighboring rooms (via the we’ll-let-you-back-into-your-room-if-you-get-locked-out master key holders), unplug all of our electrical appliances, and take laptop computers out and to hold them in a dry place.
Note that they had zero legal right to do this. If I wanted to, I could probably sue them up one side and down the other for breaking and entering, and trespass to chattels if not outright theft. And yet, they had absolutely every moral right to do so. As Mika’s father is a lawyer, I presume that she realized that she could get in legal trouble for doing so, yet did so anyway. I greatly admire her for that, and I am very grateful to her and all the others for rescuing our laptops.
Rock slides: On Sunday, the day we were supposed to come back from the ski trip, a rock slide closed the only road back to home for eight hours. This meant that we got back late. Not only did that mean that I didn’t get Sunday evening to rehearse, but tapping lightly on the door of the guy who had our laptops failed to rouse him. It took another day to get our laptops back.
Laryngitis: On Tuesday evening, my co-presenter emailed me to tell me that he had lost his voice. It looked like maybe I would have to give his presentation and mine… and while I knew about some of his stuff, there were some detailed technical issues that I didn’t actually know anything about. So I tried to cram knowledge about his area.
Fortunately, his voice came back by Thursday, so I didn’t have to give his talk.
My talk went okay. I got some trustworthy feedback that at least some people liked it, but it wasn’t nearly as polished and professional as I am capable of. Next time.
Permalink
02.11.07
Posted in Random thoughts at 10:16 pm by ducky
I drove down to the U.S. today to return Mom’s car. (We’d swapped our car for her minivan in order to carry more people on the Green College Ski trip). I had this unsettling interaction with a U.S. border guard:
Guard: Where are you headed?
Me: To return my mother’s car.
Guard: What’s your mom’s name?
Me: Natalie Sherwood. [Note: Mom’s name changed in this posting for privacy reasons — but I told the guard the truth.]
Guard: Natalie Frieda Sherwood?
Me: Uh, yes.
How did the guard know my mom’s middle name?
The idea I find least objectionable is that they took a picture of Mom’s license plate, did pattern recognition on it, and looked it up in a database of Washington State license plate numbers.
The idea I find most objectionable is that they noticed that Jim and I crossed the border frequently, and had the FBI go check us out.
Unsettling.
Permalink
01.24.07
Posted in University life at 10:53 pm by ducky
I recently needed to select two faculty members to be on my supervisory committee, and one to be the second reader on my thesis. Because I am doing data mining of user studies, I wanted there to be one HCI person and one data mining person on my committee; the second reader needs to be able to understand both.
For the HCI person, I dithered between Kellogg Booth and Joanna McGrenere, ultimately picking Kelly because he’s got a bit broader depth of experience, but Joanna would have been just fine. Karon MacLean also would have been fine. (I know from having been a TA for Joanna and Karon that they are both very easy to work with.) For the data mining person, Raymond Ng was the obvious choice, but my supervisor (Gail Murphy) pointed out that Rachel Pottinger would be a good backup.
I worried a little bit about the second reader, but a bit of sniffing about assured me that Cristina Conati had background in AI and HCI.
After I had done this exercise, I was blown over by the realization that my MS career at UBC could have easily had these people in the seven important slots:
- Advisor (i.e. person who gives advice on classes): Karon MacLean
- First term TA instructor: Karon MacLean
- Second term TA instructor: Joanna McGrenere
- Supervisor (what in the U.S. is called “advisor”): Gail Murphy
- Supervisory committee #1: Joanna McGrenere
- Supervisory committee #2: Rachel Pottinger
- Second reader: Cristina Conati
All women! And the thing that most surprised me was that I had not invested one iota of effort in trying to find women faculty — it just happened. It’s not as if before I got to UBC, I said, “I want to make sure that I find women to mentor me.” (If I had been looking for women, this is the list that would have happened instead of what could have easily happened.)
Not only that, but these aren’t even all the women in the department. There are almost as many women on the faculty who aren’t on my list as are on the list: Tamara Munzner (information visualization), Anne Condon (theory), Irmtraud Meyer (bioinformatics), and Alla Sheffer (graphics).
Twenty-five years ago, when I was getting my undergraduate degree in Metallurgical Engineering, there wasn’t even one woman faculty member in my department. Ten years ago, when I was getting my MS in General Engineering, there was exactly one woman faculty member in my department.
Times have changed!
Permalink
Posted in Technology trends at 10:39 am by ducky
Coté riffed on the cake analogy in my last posting. If I understand him right, he seems to be saying that, just as you scramble to figure out what cake your boss wants by consulting other information sources, you should scramble to figure out what to do when your software process is underspecified.
I’m not sure that’s a perfect analogy, since in the cake analogy, you had other sources of information about the boss’ desires. You had access to your boss’ schedule, phonelist, and permission to act on their behalf. For underspecified software, you would have to use a lot more imagination to figure out just what it should look like, and your management might not give you the time or budget to do so. 🙁
Permalink
01.22.07
Posted in Email, Technology trends at 9:57 pm by ducky
This is a response to Joel Spolsky‘s review of Dreaming in Code, a book about the development of Chandler at the Open Source Applications Foundation.
I was at OSAF for two years. Many of the things Joel says are consistent with my view of how things happened, some are not, and some are fully correct but don’t tell the whole story. Some of the things people on the discussion group are speculating are wrong. I’m going to write responses down in no particular order with no particular story arc. I’ve tried to boldface the thesis topics of each idea.
(Note that I haven’t read Dreaming in Code yet, so am not commenting on the book, just Joel’s review of it and the comments on the discussion group.)
There was an implication that the people at OSAF were lazy and interested in doing as little as possible to keep getting their paycheck. I think that is a grave disservice to the people there. Getting a paycheck isn’t usually that difficult — especially for people of the calibre that OSAF could attract. Top coders want to ship stuff!
However… in a normal company, you have the VP of Marketing representing the customer’s interests and the VP of Engineering representing the developers’ interests (i.e. could it actually ship). The two act as check and balance, so that the customer gets something they want in a reasonable timeframe. Unfortunately, Mitch was acting as the VP of Engineering, and the VP of Marketing, and the customer, and the venture capitalist.
Furthermore, Mitch had been very successful. It seemed reasonable to believe that if he thought we should do X, we should do X. This meant that Mitch didn’t get particularly fierce pushback when perhaps he should have. For example, he was adamant that we should do Chandler peer-to-peer, without a server. Doing a collaborative calendar without a server is really, really, REALLY hard. He eventually clarified that by “server”, he meant “something difficult to administer”, and so OSAF started working on a server.
Mitch has many many many fine points. He is a good and caring human being. He is clever. He has an ambition to make the world a better place. However, he has also been very very wealthy for a very long time, and I think that Mitch’s environment has taught him that that he does not need to be particularly clear in his instructions.
If my husband says to me, “please get me a cake”, I’m going to insist that he tell me what kind of cake, what event it is for, how many people it is for, and when it needs to be ready, and where it needs to end up. However, if my Powerful Boss says, “oh yeah, and order a cake” right before she hops in a cab to go to meet with the President, then I’ve got to figure everything out myself — and I have strong incentive to do so. I’ll go find my boss’ calendar, look through upcoming events on the schedule, guess at which one is likely to need a cake, and phone the boss’ cook to find out what kind of cake the boss likes. If I am really on the ball, I might phone the other participants’ people to find out if there are any food allergies, likes, or dislikes. If I am good, my boss will come back and find out that in fact she did not need to give clear instructions; if I am not good and there is no cake, I most likely would (eventually) find myself no longer employed by Powerful Boss.
This vagueness might be acceptable for cakes, but it’s death for developing code efficiently. For example, Mitch said that by “server”, he meant “something difficult to administer”. Either he changed his mind or he just never realized that he needed to be clear that he meant “something difficult to administer” instead of “always-running software that responds to requests from other computers on the network”.
I think Joel was right that too many designers slowed us down. Something that made it even worse was that the team grew person by person, i.e. was not wholly in place at the onset. Every time we got a new person on the team, there was a significant adjustment period while they lobbied for their own ideas for how things could be done better. For example, Lisa Dusseault is a big WebDAV person. After she started, she lobbied for doing more with WebDAV, saying that Life Would Be Better with WebDAV. She convinced us, and we started doing a lot more with WebDAV. She was probably right, Life Probably Is Better. But changing things around certainly cost us some time.
I think Joel was too harsh on stamping. He is right that being able to send things around by email is not exciting. It’s going the other direction that is more interesting: taking email messages and being able to turn them into things easily. An email message might also be a task: “Could you please frizbat the hongtholt?” An email message might also be a calendar item: “Can we meet next Thursday at 12?” Yes yes, with Outlook you can drag an email message onto your to-do list or calendar, but you still have to (had to?) fill in all the to-do information by hand. Tedious. The text recognition was supposed to do that for you.
I have said before that current to-do list managers suck rocks, and email clients could be much better. Chandler in the Grand Vision would have / could have addressed those. However, we found out very early on that people didn’t care about to-do managers or email clients: in 2003, people’s pain was centered around calendaring: you really had to buy Outlook and run an Exchange server to get halfway decent one, and even Outlook was only halfway decent. There was not a way at the time for a small business or non-profit without an IT staff to coordinate calendars. Period. So we focused on calendaring, which IMHO was the least exciting and revolutionary part of Chandler.
Something that Joel doesn’t mention as being a barrier to getting help from the outside world was that in the time while I was there, we never had a contributor agreement/license in place. So Jane wants to contribute some code: what sort of agreement governs the code that Jane contributes? For most open source projects, that issue is just not thought about because there is no money involved. But with OSAF, there was serious money involved.
First, Mitch Kapor was (is) a wealthy man, and wealth attracts lawsuits. Second, OSAF pays genuine salaries. Third, the plan was always for OSAF to become self-sustaining by providing different (i.e. non-GPL) licenses in addition to (not instead of!) the free GPL licenses.
Hence, we couldn’t really accept much code without a contributor license. That didn’t get done, in large part because our counsel, Mitchell Baker was stretched waaaaay thin also running the Mozilla Foundation. (She’s now only running the Mozilla Foundation, and appears to be doing a fine job of that.)
(Side note — I was absolutely amazed at how much volunteer support we got for NON-coding tasks. From help installing our phone system to CSS stylesheet help on our wiki to advice on video capture tools, the open source community was breathtakingly generous.)
Chandler is an open-source cathedral, not a bazaar. It’s a very different way of doing an open-source project, but that is not inherently wrong or bad. You need cathedrals sometimes.
Finally: Chandler most definitely does have a mascot — one of Mitch’s dogs. 🙂
N.B. I have zero negative connotations associated with the word “boss”. It means exactly the same thing as “manager” to me, except is less formal and fewer syllables. I thus forget that some people attach a negative connotation to it. I meant no negative connotation in the Powerful Boss paragraph.
Permalink
« Previous Page — « Previous entries « Previous Page · Next Page » Next entries » — Next Page »
