Thursday, September 17, 2020

What the heck is source code, environments, and versioning for non technical people

Had an interesting conversation today and thought I'd share some insight to business people dealing with technical folks

The crux of the question centered around a request we were making for "read only access to a staging environment". I pondered why that line item was there because from a technical perspective what we needed was access to the source code for a web application, we really didn't need access to anything in the environment (though it wouldn't hurt). Moreover, with only access to the environment, we wouldn't necessarily have access to the source code so the actual need wouldn't even be met.

In conversation, a light began to come on in my head realizing that "source code" and "environment" are almost meaningless to many non-technical people, and in todays "As A Service" world, sometimes they get mingled together.

Brief Overview

Sourc Code is essentially the instruction that tell a computer "what to do". So, for example: if date > today display "date must be today or in the past" is source code. What happens on many platforms is that those text instructions translate to a string of "0's and 1's" that tell the computer how to do this. so the instructions above might be translated to: 00010101010010101010010101001010000000101010011011110101010101011100000101010101010101001010101010010101001010010100 and the computer, when you feed that string of 0's and 1's to a computer, it will do what the programmer intended the textual instructions "should" do. This string of 0's and 1's is colloquially called the "binary" by tech weeenies.

Now for some wrinkles...this translation is known as "compilation" some languages "interpretation". So languages like C, Fortran, or Go are "compiled" and languagees like Ruby, Javascript, or Python are "interpreted". And (as a side note) languages like Java are actually a hybrid of both. There are literally hundreds if not thousands of programming languages and all of them use some degree of compilation or interpretation (even if they are a visual language). but the important detail is that the "source code" isn't necessarily the same "file" as the "binary". A simple way to think of it is if you're using a computer program and you take a screen shot of a powerpoint presentation, you can then hand that screen shot to someone else (or post online, or whateever) and continue to edit the source code (and even store the sourcee code and binary (the screen shot) in different locations.

So for example, using our powerpoint example, you could be creating a presentation, take a screen shot and post a "work in progress" on the web, while continuing to edit the powerpoint presentation (the source code). Moreover, you may save "versions" of the source code (and/or binary) so that you can explore different fonts/layouts/colors, and display/edit them differently in different "environments"...maybe one environment is an internal web site with a bunch of unbranded pictures, but there's another public site with the final product.

So what?

So putting it all together using our powerpoint example.
  • Source Code = Our PPT file that we edit to create screen shots to display elsewhere
  • Binary = Our screen shot of the presentation at a particular point in time
  • Version = A particular revision of either the PPT or the Screen Shot

OK, got it, but again, so what?

Because anything beyond a trivial Hello World program will have multiple versions, with perhaps diffeerent variations that change over time. So when building things like ecommerce sites (or almost any non-trivial app), folks need the ability to test and validate new features before turning the feature on for the world to see. In our powerpoint example, there might be a QA or workflow to validate the resulting image from the powerpoint uses the right fonts/colors/branding before displaying to the public. Becausee of this need, most modern platforms have the notion of different environments for different purposes, some common examples are:
  • "Local" - the developer local machine, they can only see the code and the images
  • "Development" - the environment where all the developers can see each others work put together. sometimes there can many "development" environments, depending on how complicated the solution is, but the point is, it's a remote environment or version that isn't exclusive to a single developer
  • "Staging/Integration/QA" - these are other environments used for a variety of purposes sometimes they don't exist, sometimes there are dozens of these.
  • "Production" - this is where the world gets the final product
The process of moving the binaries between these environments is generally known as "deployment" and the workflows around deployment are myriad, but the point is that once you've created a version, you move that version between environments.

OOOOOhhhh Kaaaay, I think I got it, so what's your point?

So, the confusion arises because of something I mentioned earlier about "interpretation" and "compilation". In our powerpoint example, an interpreted language creates the screen shot automatically in the environment when someone tries to view the content. In a compiled language, the picture is created ahead of time and only the picture moves between environments (it might be tied to a version of the original PPT, but this is only a loose association).

So for example. Suppose I have a PPT called "Mikes_presentation.ppt" and while building it I create 3 versions "Mikes_presentation_v1.ppt", "Mikes_presentation_v2.ppt", and "Mike_presentation_v3.ppt". In this, I have 3 versions of the source code, and for the purposes of this discussion I store them on my local I have 3 files. Furthermore, let's say I want to take a screen shot of each of these and I want to send them to someone to take a look (maybe they don't have ppt) and I put them out in three different places...two of them are "For internal use only" and the last one is "for the world to see" I might put one at "", "", and ""...let's just pretend those are web sites or "environments". At any given point, if I point someone to those "envirornmeents" they might see a screen shot of any of the versions of the source code because I've deployed different versions to the environments.

However, for "compiled" versions, I may (and routinely would) only send the screenshot to the environment, because the environment itself doesn't need to know anything about the original PPT, it just needs to be a picture. So if I wanted someone to edit that PPT, or enhance it, access to the file in the environment won't be useful because I can't change the original PPT used to generate the picture.

For "interpreted versions" all someone needs is acess to the environment, and if you don't have additional controls, they might edit the PPT that I called "Mike_presentation_v3.ppt" to have completely different content than the one sitting on my hard drive.

OK, is that good or bad?

It's honestly neither, but it does illustrate (I hope...I know it's been a bit of a ramble) that access to the "environment" doesn't necessarily give you access to the "source code"...and an "environment" might not actually reflect what your "source code" (or the copy with the same version) can generate.

So honestly what's the big deal

Well, it gets confusing because some tools (the "interpreted" examples) inherently store the source code in an "environment". This means "environment equals source code", but other tools (the "compiled" examples) don't necessarily equate the two. It's furtheer compilicate by the fact that this is a trivial overview and reality is MUCH more complicated (some "compiled" code is also store in the environment, "environment" also includes things like operating systems, device drivers, the source code doesn't necessarily give you everything you need to reproduce it).

At the end of the day, hopefully I gave a (not so) brief primer to the semantics behing "source code", "environments", and "versioning"... and my full apologies to all the folks who will be coming out of the woodwork to explain the million different ways this is technically not 100% them, I just say "the business people don't care, we just need a better way to explain the concepts.

Programming languages by type

Friday, July 10, 2020

Amazon busted?

Trying to find a battery on amazon and no matter what I try, I get this (logout, login, incognito):

Thursday, December 5, 2019

MQTT/AMQP design implications

If you're working with embedded devices or telematics solutions, you might be hearing noise about a fairly new protocol called MQTT. It's a relative newcomer to the network protocol world having been invented in 2009 and first published in the wild in 2010, it is roughly speaking to tcp based binary wire protocols what SPDY is to HTTP.

At the heart of the protocol and why you might use it instead of say..AMQP is its simplicity. There are only 5 operations that must be implemented, it's wire format is minimal, and because of it's simplicity, it is theoretically able to use less power.

A close examination of the differences between AMQP and MQTT show that the low power or low memory devices (think Arduino class) will certainly be more likely to easily speak MQTT rather than AMQP. As an example of how an ideal architecture leveraging the strengths of each protocol might look, take a look at the following diagram:

When looking at this stack, let's talk about the implications of this approach over a SPDY/HTTP implementation from the device perspective.

For devices living in a low power lossy environment (on the right) using MQTT makes a lot of sense. If you periodically transmit 10 bytes and need to know if a device is connected or well as maintain a small footprint for the libraries doing the connection management, MQTT wins hand down versus AMQP or HTTP. On the other hand, once these messages are delivered to an MQTT broker it becomes more important to handle message queuing, reliability, and a host of other things that an embedded device typically won't have the power or inclination to manage. Additionally, in a low memory/power situation, maintaining application message level transaction state for the life of the operation is often rife with error

In short, it seems for many use cases a combination of these protocols is generally going to be the "best" solution, not one or the other by themselves.

Friday, August 23, 2019

The dark side of git...

Git is Great!

As a distributed source code tool, git is great. I love that when I'm on an airplane I can commit code without a wireless connection and have be able to unwind what I was doing. It was clearly designed with the "offline" model in mind. The idea I can create a quick branch, experiment, make massive sweeping changes, and just drop the whole thing if I realize it AWERSOME! For a fact this puts it ahead of it's open source predecessors (namely...SVN, CVS, and RCS).

But perhaps a victim of it's success

What I observe, however, is that a lot of folks have taken up a development model where "everything is a branch" and we end up with roles like "pull request approval engineer" (not a real title, but if you end up doing that job, you'll know that your doing it). This problem happens when the number of public branches/forks reaches a count and complexity that far exceed any value they could have possibly served.

What is productivity?

I'm going to take a somewhat unpopular stance here, but in general my stance is that branches are antiproductive... before everyone gets their pitchforks out, let me explain my version of "productivity" for a software project. Productivity is producing software that accomplishes the business purpose at a marginal cost that provides positive value. While that might have been wordy or even technically incorrect, the overall equality formula I wan to use is: sum of all activities to product the software must be less than the value the software provides. In other words, if it costs 2 million dollars to build/deploy some software, but the business can only recoup 1 million dollars in value (via cost saving or new sales or whatever) the I would consider that a failure.

The branching use case

As a software engineer, I want to create a branch so that other developers cannot see my changes in their builds.

Well that sucks because:

  1. First of all, the activity of creating the branch, merging in everyone else's branch to your branch (through possibly a different branch) is all stuff that you would get instantaneously for free if you were all working on the same mainline.
  2. Second, you're deliberately delaying visibility of changes from the rest of the team...which means the whole notion of continuous integration is getting thrown out the window
Which brings me to a key question

Are you operating with agility or fragility?

I would contend if you're branching for every feature or bug and merging them back in, your codebase and/or process is more fragile than agile.

Your Thoughts?

Saturday, January 13, 2018

The state of programming languages and frameworks

As a professional software delivery person, I like to keep on top of technology trends and "where the market might be going". Over the last decade and a half, quite a few languages and frameworks have come and gone and very few have had any real staying power. In order to be marketable and knowledgable in things that "people want to know", I generally find the Tiobe index and Google Trends to excellent resources in gauging popularity. In my analysis this year, I've established that relatively speaking, they are in agreement, so I'm going to use google trends (as the charts are easier to embed) to elaborate.

Programming Languages

Before digging into frameworks, there is the notion of "which language" is most popular? In this regard, java has been dominant and looks to remain so for a long time. While there is a downward trend, every major language has had it's mindshare diminished, I can only imagine because of the explosion of alternate languages in recent years. Assessment: learn java, become an expert because while the market is crowded, there will always be work and/or people who want to know something about it. To be clear, I disregarded C, though it does roughly correlate to C++ in is used more in embedded markets and that's not one I'm deep into [yet].

Alternate languages

While I would recommend any newcomers pick one of the "big 5". It really helps to have a "specialized" language you are at least passingly familiar with and can be productive in. In that regard, I also tend to take the "short term" view as these tend to come an go with great regularity. In that regard, I'd say that Python (technically in the big 5 if you go by many sources) is a solid first choice, but ruby is still a viable alternative. Outside those two, almost any other modern language would be a good idea to pick up and have as there are always specialty areas that will have a need [even for legacy languages like ADA or Fortran].

Legacy Languages

One area that is often neglected are so called "legacy languages". These are languages that have fallen out of style and/or been superseded by more modern alternatives. One reason I recommend adding a member of this group to your portfolio is that many experts in these fields are retiring but the systems running on them will continue to live on. Additionally, when doing a migration from a legacy platform, being able to quickly be able to read and understand what the old platform did is a valuable skill. One area to look at is the "area under the curve" as this represents the "amount of code potentially written". In this regard, perl is a clear winner.


Programming languages, however are only one dimension. Beyond this, the frameworks available to deliver higher level functionality are a key factor. From that perspective, I grabbed a few notable frameworks and did a comparison (realizing node.js isn't really a framework). In this regard, ruby on rails, while declining in popularity (and surpassed by spring boot), has a HUGE installed based and would clearly be a good choice. The winner's a little unclear here, but coupled with java's popularity as a language, I think one would not go wrong with spring-boot, perhaps having ruby on rails as a backup (and it IS the dominant framework in ruby).


From my perspective, I have a good familiarity with java and spring-boot, plus a deep understanding of ruby on I'm still fairly well positioned and I think I could easily recommend these as "go to" choices. Beyond those, I think I may spend some time playing around with perl again as it strikes me as a market that is set to be underserved at some point in the next 5-10 years...and will be a prime candidate for "need to know to make legacy migrations go smoothly".

Friday, July 14, 2017

The Technical Estimation Paradox

Face it, we've all been're asked for an "estimate" that you KNOW you're going to be held to, but there are a hundred variables you have no way to control. The client is sitting at the other end of the table tapping their fingers and you think to yourself either: #1 "they don't understand, it's unreasonable, I don't have enough information", or #2 "Hmmmm, how much do I think based on my current information it might take?".

At the end of the day, neither of those matter beyond the psychological value they have for yourself...the real question that matters is "how much is it worth to them for you to deliver this?". Yes, if you're billing time and materials, there are practical problems: If you estimate too low, your client is going to be disappointed that you couldn't deliver in the agreed to cost/time...if you estimate too high, you client might be happy, but often they also think that you cut some corners (especially if you were the "middle of the road" estimate). On the flip side, if it's a "fixed bid", if you estimate too low, your margins are going to dwindle and you could possibly lose money and if you estimate too high you may end up in an ethical dilemma where you are making 99% margin (which is arguably good or bad, depending on your perspective). But at the end of the day, as a consumer of services, you should be happy if you get the contractually agreed to qualities you care about (without assumptions) for the agreed to amount (or less), and as a service provider, you should be happy to deliver at the agreed upon price (or less) with the agreed upon qualities (or more).

Wednesday, July 12, 2017

Software Architectural Decision Making

A common question I get asked is "How do I make architectural decisions?" and my standard answer is "it depends". While it's a tongue in cheek answer, there is a bit of truth to it. While there are frameworks and methodologies to try and reign this problem in, the reality is that the practice of "software architecture" is inherently a mess and certainly a wicked problem. That having been said, I'll give some insight into "how I do it".

First off, let me say, often many "decisions" are predetermined by your primary objective or have such a strong force behind them that there is little value in contemplating alternative solutions. A good example would be "which programming language should I use to develop an Android application?". You really have one decision and it's pretty binary: "Do I use java or javascript?" Yes, from a technical perspective it's possible to use ANY programming language (either through cross compiling or using a runtime interpreter), but if your primary goal is to release an application that allows a user to "do something on their phone" agonizing over every possible option is a HUGE waste of time. On the other hand, if your primary goal is to illustrate how to write applications using ruby to be run on an android device, the decision is preordained (and frankly not really a decision). Moreover, in the latter case, the decision switches from "which language should I use?" to "what approach should I use for running ruby applications on android?".

In the former case above, suppose our primary objective is to write an application that allows users to track their workouts on their phone. In that case, the "language" you use is relevant, but only as a secondary concern. Some questions you have to now concern yourself with are "how many people know java versus javascript?" or "will I want to ultimately release the application in both iOS and Android?". Additionally, you have to concern yourself with "are javascript developers cheaper/faster or java developers?" and "which approach is easier to test and debug?".

However, in the latter case, some questions are: "do I want to highlight using ruby's dynamic nature?" or "illustrate how Ruby can lead to higher quality code?" or something else like "do I want to illustrate how rapidly Ruby applications on android can be developer relative to java?". This also opens up another can of worms you need to consider, such are "is the pool of developers for Ruby such that developing in that language is even VIABLE?".

As we can see, the number of considerations grows at an exponential rate and is extremely sensitive to initial conditions (i.e. what is the primary problem). If you change the "primary problem" entire swaths of decisions are irrelevant (why worry about javascript if your objective is to write things in ruby?). This problem is what makes architectural decision making particularly pernicious and I would contend mean it exhibits fractal characteristics of a nonlinear system. This is also why no one has yet come up with a comprehensive "system" for making these decisions. While many lay claim to methodologies and approaches (take a look at what IBM has to say about it, the fact is it is an extremely difficult problem to reason about.

My best advice is this: Architecture is like software meteorology, you can't predict the future, but you CAN some up with general models that work for defined scopes. What does this mean? Simply that trying to define software architecture is like trying to define what the weather is for the entire globe. The fact is, "it doesn't matter" if your primary objective is to determine if you want to go to the pool or not. All that really matters is the local weather and your preference for the kind of weather you like to go to the pool in. Moreover, you don't need to necessarily explain "why you chose to go to beach instead of a pool" because you realized your original desire was "to go swimming" and limiting your option to a pool might have been a mistake (for other reasons). Put another way, "software architecture is understanding what's important" and "making important decisions", that art is learning how to figure out what is important, the science is too complicated to think about.

What the heck is source code, environments, and versioning for non technical people

Had an interesting conversation today and thought I'd share some insight to business people dealing with technical folks The crux of th...