Monday, December 17, 2007

Entropy or Energy?

We try so hard to get our IT infrastructure organized, but I wonder how useful that is. In a typical business environment, we see these high costs of software "overhead" and try as hard as we can to reduce the cost to zero.
The problem with this is that we got our original gain from technology, not necessarily from the advantage of the technology itself, but also from the chaos created by changing from one paradigm to another. It is almost like a nuclear reaction and is just about a difficult to control. Too many restrictions and the energy disappears... Not enough restrictions and ... well BOOM!!!
Finding the balance between the highest energy and the highest entropy is a difficult task in IT indeed.

Sunday, December 9, 2007

Why experts suck

I'm reading "supercrunchers" and it has reconfirmed something I've thought for a long time. Experts suck.

Time and time again I find myself at odds with authorities in varying disciplines. Usually the situation develops when we have a problem, consult the experts, and they postulate that the problem is something that can only be solved with a very big and very expensive project.

I will spend some time, form some models, run some analysis and reach some alternate conclusions. I am met by a stubborn and braying pack of obstinate mules who have no argument against my data except that they are "the experts" and I should stop questioning their authority on this subject.

I have, to date, simply thought these people where just jerks and not worth my time. It turns out, however, that the problem is very widespread and many people seem afraid that if we can build a statistical model that can effectively do what their "expertise" provides, then they'll be out of a job. I need to start showing that if we free these folks from this menial task, they can then start doing the creative part of their job and let the computer do the remembering for them.

Sunday, December 2, 2007

New Songs

I wrote some new music and realized and audio engineer might be helpful. I terribly confused at how to compress the dynamic range into something that sounds OK on a radio.

Saturday, December 1, 2007

Enterprise Scalability and Architecture

I've recently spent a lot of time explaining scalability and architecture to people in our organization and am being largely unsuccessful. I blame hardware vendors for this, they shove new technologies at unsuspecting infrastructure professionals without any high level explanation about the real advantages.
As my first example, I will talk about logical partitioning. Listen folks, this is not a scalability feature! Don't listen to the salesman, he doesn't know what the hell he is talking about. I will admit I can imagine some scenarios where this might be able to help you scale a collection of applications, but do NOT try and tell me that adding a new LPAR is somehow scaling my application. Also do not tell me I should buy a machine that is 20x more expensive because it's more "scalable".
The problem with machines that share disk, buses, cache, or other resources is that you need to manage who is allowed to use the resource at any given time. It is very dangerous to allow two different processes to modify a memory location at the same time so you need to manage this. This management is not free and as you add more things (processors) sharing resources the overhead typically increases at an order of magnitude that is greater than linear. Put another way, this means the system will scale at an order of magnitude that is less than linear.
No matter what manner of wizardry your hardware vendor has put into his machines, if you are sharing anything, you are going to incur overhead. This overhead will also increase as you add nodes that need to contend with each other. So any salesman who tells you otherwise is simply wrong.
The number one most sure fire super hot way to scale an application is to partition the data and/or work and do the processing on independent machines. This is the holy grail of scalability, it is a sure fire, can't lose proposition. It is hard to do and does require thought, creativity, and a good deal of elbow grease. It is, however, also a huge reward for the effort and is the ONLY sure fire path to scalability.