Friday, May 22, 2009

Some Folks are Unreachable

We have a perennial debate at work (going on 3 years now) about the "proper" and official Source Code Control tool in our company. Originally it was a crappy tool called CCC Harvest (yeah, sue me, it sucks I can't say it any other way), we then had a brief renaissance and used CVS (not great, but a big productivity boost!), then moved to Clearcase (shoot me now, it has a lot of potential and it theoretically GREAT, but in practice sucks almost as bad as Harvest), then began the migration to SVN (whopee, next step Hg!).

Unfortunately, every step of the way has been met with irreconcilable folks in the change management area who seem to think that developers are simply a whiny bunch of prima donnas who are trying to rob the company blind. I say this because I keep hearing about the "superman" exploit as a reason we should use Clearcase. In case you don't know, the superman exploit is where a clever developer sneaks code in to steal money by taking money that is lost due to rounding errors and deposits it in their personal account.

First off, unless someone was criminally negligent in setting the tool up, this would be nearly impossible in SVN. In fact, it probably MORE difficult in SVN than clearcase. I say this because it is possible (and easy) to administratively erase things in clearcase that are very difficult (by design) in SVN.

What really burns me though is that every time we attempt to have an objective discussion about the relative merits of either tool, the change management team spends their effort spinning things instead of really helping make them better. They are a perfect example of a team that has totally forgotten who their customers are and why they exist. Worse yet, they refuse to acknowledge that the complaints of the development teams are even worth addressing.

ALERT! EMERGENCY FIX REQUIRED!!!

I'm taking a vacation day today. I woke up at 5:00 to take my daughter to school (trip to six flags), then went downstairs to put some finishing touches on some html I was fiddling around with to help with some new screens.

While doing this a developer on my team started IM'ing me in a panic about a production problem that they where going to do an emergency deploy to fix. I attempted to calmly ask her to please explain the exact situation to me, but it came out more like "ARE YOU F'ING JOKING!? TELL ME YOU AREN'T SERIOUS!, WHAT THE HELL?".

Whoops, no more coffee for me, my bad.

So then I opened my email inbox and discovered no less than a dozen emails about about how the day before there was some sort of error that cropped up based on a specific data condition and stopped an extract process dead in it's tracks.

For some reason, this software has run without exception for 2 months and suddenly yesterday at 5:00pm this data condition started to happen. What puzzles me is that this tiny miniscule detail seems to have eluded everyone involved. Nobody seems to have stopped and asked "Hmmmm, WHY did this start happening? What changed? We all just started digging to get ourselves out of the little hole we found ourselves in.

Some fixes I heard where "patch the the code to convert nulls to 0 or -1" or "update the production database and change the nulls to 0's or -1". What I didn't hear (still waiting by the way) is "Hmmmm, I wonder what changed to cause this to happen?" and worse yet, what I didn't hear was "I wonder what we'll break if we suddenly change all this data to a different (invalid) value?"


Now that I'm a bit calmer I'm also asking "what is the business impact to this?" From what I can tell, we had a "red alert all hands on deck emergency" response to a relatively trivial problem. I'm glad everyone want to help and react quickly, but in my experience reacting quickly without thought (Oh, your arm hurts? cut it off!) more often than not has some serious side effects.

Now, someone might come back with an explanation about why this particular problem garnered the response it did, but I'm still waiting on that one. It's now 10:00am and I'm ready to take a break....I think I'll go outside and mow the lawn.

As a followup, we ended up on a conference call about this and it turns out the problem was caused by a bad data condition that really shouldn't have happened. In this case it is probably good that it blew up because we now can figure out how people are putting data into the system that is incorrect. While it's not ideal that we had such a bit TODO, it is good that we didn't put in one of our "fixes". They would have masked the real problem and caused even MORE bad and incorrect data to be put into the system.

My learning from this is that sometimes I need to sit everyone down and keep asking questions. There are a lot of people who are in hammy mode and sometimes someone needs to sit down with them and help get calm. It's funny because I have a reputation as a hot head, but it usually takes quite a bit to work me up if everybody's being reasonable. I took the ARSE test a while back and I'm not TOO bad (yet).

Saturday, May 16, 2009

Meetings gone bad

Below is a mostly fictional schedule that mimics a pattern I see in my office (names and places are fictional, just illustrating a pattern).

8:00am-9:15am Daily Status meeting (room B342) - In attendance: Susan, Bob, Bill, Steven, Mike. Review where we are in the project... Bill is going to fix bug 123 Steven is going to set up server, Mike needs to review the Spec doc and make recommendations, Susan needs to catch up on emails.

9:00am-10:00am Project Planning meeting (room B231) - In attendance: Susan, Bob, Bill, Steven, Mike (all late), and Jim. Plan the work for this week... Bill is going to fix bug 123 Steven is going to set up server, Mike needs to review the Spec doc and make recommendations, Susan needs to catch up on emails. Jim needs Mike to get with him and explain how to use the new Foobernator component. Mike agrees to do it in the afternoon.

10:00am-11:00am Requirements Review Meeting (room B243) - In attendance: Susan, Bob, Bill, Steven, Jim. Review the requirements documents for the upcoming project. Find a few spelling errors (huh?) Move some paragraphs around... uncover some issues that need to be figured out. Bob and Bill are going to get back with clients to figure out what we should do about the issues. Agree to meet tomorrow to re-review the doc.

11:00am-12:15 Issues Meeting (room C333) - In attendance: Susan, Bob, John and Jim. Review "Issues list" to determine what we should be working on for this release. Attendees realized they need Mike and Steven so they pull them into the meeting.


The anti pattern is what I'll call the Night of the living dead. You know this is happening if you see herds of people moving from one meeting room to the other. They meander from meeting to meeting eating at the brains of folks... slowing turning everyone into lethargic zombies incapable of anything useful.

If you have 4 meetings per day with the same 6 people, get a room! No, really, I mean it, book a conference room and move all six people to the same place. I realise we're all overweight and need the exercise, but walking around with the same horde of people all day is a huge time-waster.

In addition, if you have technical staff that you rely on to do things, you cannot invite them to 8 hours of meetings per day and expect them to get anything done. I have literally (not recently) attened continuous meetings where at 9:00am I was given a task, and asked ever hour or so if I had completed it. This was being asked by people who where continuously in meetings with me!

My personal solution to this problem I have implemented over the last year or so has been to start being VERY aggressive about meeting attendance and internally putting a value on my time. Too many people seem to use status meetings as a warm blanket to protect them from the harsh reality of "getting things done". They feel safe in spending 50 hours per week replying to email and attending meetings, and have wondrous excuses why things don't get done. Fact is, many folks could be a heck of a lot more productive if they would just do the work at hand instead of Talking about
doing the work at hand.

In addition, there are people who love to Hijacks meetings to further their own personal agenda. Typically this happens when the person organising the meeting hasn't clearly stated the agenda so everyone makes up their own. Very entertaining to watch, not too entertaining to participate in unless your goal in life is to be a member of the PFD.