A few weeks ago Roger Ehrenberg of IA Ventures wrote a great post on data driven planning and execution, called Plan Well, Execute The Plan. In that post Roger outlines the benefits of “keeping one’s head down” and the process of executing off of a well formulated, data-driven, plan. His post is great, and so I won’t repeat it, but the process is essentially to formulate a hypothesis and associated tests, run the tests, analyze the data, implement changes, formulate a new hypothesis, and repeat. He also outlines the benefits of a hypothesis-driven process, which I believe boil-down to focus and smart decision making. Since reading Roger’s post, I’ve been thinking quite a bit about, and am writing this post to expand upon, why this process really works: Why does hypothesis- and data-driven development really help drive focus and smart decision making?
Much like a sprint-cycle deters distraction for a development team, a hypothesis-driven model helps to manage and deter distractions for the group that utilizes it. Sprint-cycles help developers to focus on the critical features as determined in the beginning of the sprint, while deferring other initiatives to the backlog. Upon completion of the sprint, options for the next sprint are reviewed and the sprint is set based upon the current priority. Often, backlog items that once seemed like high priority, or that otherwise would have served as a distraction, are no longer relevant and are either leap-frogged in the queue or pushed out entirely. Sometimes these features remain high-priority, but if pushed out often it is because the feature is no longer requested, because the team’s focus has shifted, or because an alternative has been discovered with additional time for thought.
In much the same way, a hypothesis-driven culture helps to avoid having (as Roger puts it in his post) “good people and companies knocked off kilter by glamorous, shiny stuff happening in their external environment.” The reality is that these are the types of things that can kill a company by a thousand tiny cuts. I discussed some of this in my post on Parallel-Process Product Development, but focus is important because that “shiny stuff” can continually distract from core product development. The problem is, that “shiny stuff” can also be so difficult to ignore; a competitor releases a press-grabbing feature, a major customer makes demands ahead of a contract, or a strategic brainstorming session yields lots of exiting new ideas. These distractions can often take the company away from its focus, and just as a sprint-cycle maintains focus, a hypothesis-driven culture keeps the focus of the organization on the key hypothesis and the related tests. If after completion of the test, the feature/product/direction is still deemed relevant, then it can be tackled in the next set of hypotheses and tests.
Smart Decision Making
When a company is first founded, its generally done on the back of a hypothesis or a set of hypotheses. Its not always referred to as such, but it usually is just that – an idea that you think will work, in a specific marketplace. However, as the great Steve Blank says, “No plan survives the first contact with customers.” Inevitably, as you go to market, things change and your original set of hypotheses are either proven or dis-proven. The more formal your processes around these hypotheses and the data generated, the smarter your decision-making going forward will likely be. Each successive “test” brings about a set of data that you can use to formulate the next test, ensuring that you are utilizing the “focus” discussed above to your best advantage.
Without hypotheses, testing and data, the only way to determine your direction is by gut and instinct. Sometimes your gut will lead you in the wrong direction, sometimes in a tangential direction, and at the least, if you do head in the right direction, sometimes you’ll zig-zag you way there. Data has the wonderful characteristic of never having opinions and if you extract the right data for your test, it’ll direct you accordingly (it could, of course, have biases, and the individual reviewing the data has opinions but lets assume perfect data and unbiased analysis for now). When you test multiple elements, either simultaneously or over time, often some of your hypotheses prove true, while others are dis-proven. As a result, the data will often dictate a scenario where some of your forward direction remains rooted in what you’ve proved, and where other elements are shifted – the classic pivot. The definition of a pivot point is “a point upon and about which something rotates,” and a proper data-driven, hypothesis-based process helps to determine which elements to pin, and which to shift about that pin. Once you rotate, you can reset, and determine the next set of hypotheses and tests, moving forward with focus from there.
There are likely additional benefits to a hypothesis-driven process, but these are the ones I’ve personally experienced. Additionally, while most easily relatable to product development, hypothesis-driven processes are useful in a number of business areas. Sales and marketing are great examples and its applicable in many additional ways. I’d love to hear about other people’s experience utilizing hypothesis-driven processes and welcome related stories both positive and negative.