The premium game model of development has a general cadence: pre-production, production, alpha, beta, and certification. There are variants of course, but that tends to be the gist of it. Alpha, beta, and cert are, of course, where we divert our attention from making features to the grueling task of fixing bugs. And, dear lord are those weeks painful. One house of cards to the next.
But that’s just how it’s done, right?
Yes that’s how it’s done. But it’s also incredibly inefficient. This model of delayed quality assurance means we fix bugs when it is maximally expensive to do so: at the end. After they’ve been buried under other bits of code that rely on those bugs being broken in exactly the way they are broken.
By Reading this Post, You Will Learn:
- What the “time value of money” is
- Its software analog, the “time value of fixes”
- What “technical debt” is
- The value of a disciplined, consistent QA process
A Bird in the Hand: The Time Value of Money
Have you ever wondered why you pay interest on credit card debt? Or why you earn interest on a savings account? I mean, mechanically, why interest rates exist?
The answer is the time value of money, one of the most fundamental business concepts. Simply put, $1 today is more valuable than the same $1 in the future.
Example, I ask to borrow $1 from you with the promise of giving you $1 in exactly one year. This is not a good deal for you.
Why? Two reasons: risk and opportunity cost. Risk, because you don’t know for certain that you will get that dollar back in a year. Opportunity cost because there will be chances to spend that dollar over the next year that you won’t be able to take advantage of because that dollar is in my pocket.
The Nitty Gritty
Let’s look at this mechanically.
First, let’s say you know that there is a 60% chance that you get the $1 back, irrespective of time. That means that, right off the bat, your offer to pay me back is only worth 60% of $1, or 60 cents.
Now, let’s say that you know you could make a 10% profit on the at $1 over the course of a year. In other words, if you held on to that dollar and invested it, you would have $1.10 in a year. This 10% is known in business speak as your discount rate (or, alternatively, your cost of capital). It has that name, simply enough, because it is the rate at which you discount cash flows in the future. So, if your discount rate is 10%, you have to discount my risk-adjusted promise of a $0.60 payment based on the fact that that cash flow won’t occur until a year from now. So, value today = $0.60/(1+10%) = $0.54
In short, if you account for risk and opportunity cost, my offer of a $1 in the future is only worth $0.54 today. In finance speak, this is known as the present value of a future cash flow. And it also means I profit off of the venture at your expense.
Adjusting for Risk and Opportunity Cost
In order for my offer break even, the amount I pay you in a year needs to compensate you for your risk and opportunity cost. Accounting for risk means I need to pay you $1.67 ($1/60%). And adjusting for your discount rate, I need to promise you $1.83 ($1.67*(1+10%)). In other words, the future value of the $1 cash flow is $1.83, so I need to offer at least that much for the exchange to break even.
TLDR: In order for my offer to be fair to you, if I borrow $1 today, I need to pay you at least $1.83 in the future.
It is for those reasons, risk and opportunity cost, that you pay interest on your credit card. Well those and profit.
The Time Value of Fixes
I wrote the above two reasons:
1) I think financial literacy is good for everyone
2) I think there’s an analog for game development
Allow me to introduce you to what I like to call “the time value of fixes.”
You see, I think that a fix today is more valuable than that same fix in the future. And for the same two reasons: risk and opportunity cost.
Deferring a fix is risky because, as time goes on, more and more code gets built around it. And that new code may be written in a such a way as to depend on the initial bug being broken in precisely the way that it’s broken. This is where the famous “house of cards” phenomenon of game development comes from.
In other words, you now no longer just need to fix the bug, you need to fix the issues that relied on bug. So the risk associated with the defect lingering in your codebase increases over time.
And the opportunity cost? Well, the longer the bug remains at large and the greater the scope of the impact as code is built around it, the greater the effort that will be required to implement a fix. And that effort represents dev-hours that could have gone to some more valuable form of content creation.
I’m not alone in my view of analogies between finance and software development.
The term “technical debt” was coined by Ward Cunningham as a direct reference to finance. Ward felt that, if you take the quick and dirty solution, rather than the robust, time consuming one, you are essentially racking up a balance on your project’s credit card. And the longer that debt stays on the ledger, and the more systems get built up around the hack, the more expensive it will be to eventually re-factor it.
In other words, technical debt, much like its financial variant, accrues interest. The longer it lingers, the more it’ll cost you to zero out the balance. Just in terms of dev-hours instead of money. And dev-hours cost money.
What to Do Instead
Simple: fix as you go. Keep the paying down the tech debt and squashing the bugs as you find them. Your ideal should be to keep your bug count so low and inconsequential that you could ship at any second.
Now, some might baulk at the notion of slowing down development to allow for a parallel QA process.
But I’m not actually saying you should slow down.
I’m saying you should consolidate the work. QA should be the last step in the development of each individual feature, not of every feature. I’m saying you should de-frag your production process much as you would de-frag a computer hard drive.
In other words, I’m saying your production process should account for the time value of fixes and resolve issues when it is least expensive: as soon as possible.
A Script for a Robust QA Process
As I write this, I am making my living as a consultant. And one of my clients has an amazingly robust QA process that gets run on EVERY code submission. In simple terms, the process goes like this:
Step 1: Buddy Testing
- What? Developers pull another team member aside and has that person review the changes locally
- When? Before submitting code to the database
- Why? To catch obvious sources of error, especially missed requirements
Step 2: Unit Testing
- What? Developers run automated tests on their submission to catch errors
- When? Before requesting that their changes are merged into a build
- Why? To catch defects before consuming significant testing dev-hours
Step 3: Peer Review
- What? Other developers review the submission in the database
- When? Before a merge is approved
- Why? To catch missed technical requirements or potential technical risks before they can contaminate the build
Step 4: QA Review and Regression
- What? Dedicated team members test the submission in the updated build
- When? Before the change goes to leads for review
- Why? To catch defects before other code gets build over them and to avoid wasting leads’ time with buggy code
Is the above onerous? Perhaps. But that client has one goddamn clean, stable code base. I’ve seen them deploy massive projects within two weeks of the final code submission. No crunch. No midnight oil. Just a superbly disciplined process.
Further Reading if You Enjoyed this Post
Sunk-Costs and Ugly Babies: On The Value of the Scientific Method
The Law of One Price – A Business School Mini-Lesson
The Takeaway: Discipline Equals Freedom
If you’re a fan of Tim Ferriss or Joe Rogan, you may have encountered a character by the name of Jocko Willink. He’s a business consultant. And a retired Navy SEAL. Oh, and he’s a black belt in Brazilian Jiu-Jitsu. Basically, homeboy keeps his shit wired pretty tight. His mantra? Discipline equals freedom.
There is genius in that simplicity.
If you have the discipline to exercise and eat a balanced diet, you will have the freedom availed from good health into your golden years. If you have the discipline to study consistently, you will have the freedom to sleep the night before the final because you won’t need to cram.
And, if you have the discipline to follow a well structured, consistent QA regime, to keep a clean and stable database, you will have the freedom to create more content rather than fight fires.
- Fixing bugs at the end of a project means fixing them when it is the most expensive
- The time value of money states that a dollar today is more valuable than that same dollar in the future
- The time value of money is based on risk and opportunity cose
- In software we have an analog that I call the time value of fixes
- A fix today is more valuable than that same fix in the future, for the same reasons: risk and opportunity cost
- Technical debt is another financial analog: hack code is like credit card debt and is more expensive to pay off the longer it lingers
- A disciplined, efficient QA process needs to account for the time value of fixed
Pingback: The Muda of Defects, Or: Finding Freedom Through Discipline - Game Planning With Science! Part 13
Pingback: Kanban: The Counter-Intuitive Value Of Pull-Based Production - Game Planning With Science! Part 11
Pingback: Conflict of Interest: The Fancy Mess of Scrum, Part 3