Digest The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses

I recently finished The Lean Startup by Eric Ries and in attempt to reinforce what I learned and to share with the world, I figured I’d document the core concepts I took away, as well as sharing all the highlights I made.

The Build-Measure-Learn Loop

One of the first concepts Eric goes over is what he calls the “Build-Measure-Learn” loop and its one of the most prominent pieces that startups can do faster than large companies. His pitch is, startups should focusing their effort on testing their assumptions to learn more about their customers and the market. In order to do that he breaks up a cycle of work into three steps:

1. Build: Build the smallest amount of content needed to measure metrics
2. Measure: Metrics that give you feedback about your assumptions, customers or market
3. Learn: Let the data guide you on whether your assumptions were correct or invalid

What’s interesting about this loop is one actually performs the steps backwards. You first should plan out what you want to learn, than decide what you’ll measure to help you gather information and then build the smallest amount of features required to measure those metrics.

This notion of building the least features is touched upon on the next core component, called “The Minimum Viable Product (MVP)”.

The Minimum Viable Product (MVP)

The minimum viable product is something I imagine most software folk are not new to. That being said, I believe i’d have a very hard time following Eric’s advice, even with having bought in to the idea.

Eric pitches that one should take the list of features they plan to develop and strip them down to almost an embarrassingly low amount of content. What we’re trying to do here is avoid spending months or years building out a product and then “waiting and seeing” what happens by launching.

Instead, we should hyper focus our work on only tasks that are absolutely necessary to validate or invalidate our assumptions about our product, customers and the market.

For a silly example, if we were to build a social network that only allowed one to be invited by finding an in person business card with a special code. Instead of building out the entire social network, we should validate the assumption that:

People would pick up a random business card they found and navigate to the website listed on the page

That way, if we found out that this assumption was wrong, we’d save ourselves all the time spent building out the real feature and the supporting content.

An example from the book where building out a minimum viable product worked out was where an instant messenger addon was being developed. Users were going to be able to create 3d avatars and chat with their friends. By developing the smallest amount of features, the business was able to learn that the customers were more interested in making new friends using their 3d avatar. They abandoned the instant messenger add on and created their own instant messenger platform.

Vanity Metrics vs Actionable Metrics

This piece is something I think I’ll have to go revisit in order to be able to identify vanity vs actionable metrics in the wild. The author suggests that a lot of metrics business measure and tout are strictly only used for PR reasons and do not provide themselves the correct information to garner how we’ll they’re doing.

He states that for a metric to be actionable, it must exhibit three core components:

1. “actionable”: The metric should give an individual an idea on how to replicate the results.
2. “accessible”: The data must be accessible and understandable by everyone in the company.
3. “auditable”: The accuracy of the report must be verifiable.

Additionally, it must demonstrate clear cause and effect. An example of a vanity metric that is mentioned in the book is website visits per month. He states that this is a vanity metric, as we cannot say what we need to do in order to make this metric grow.

Split Testing

The author mentions how important it is to perform split testing on all new features. This idea really drives home how Eric is suggesting one applies the scientific method to their business, allowing one to make data driven decisions. This concept really ties back into the build-measure-learn loop as well.

He pitches that companies should be performing many split tests all at the same time. Allow split to gain access to a small change in the product, ensuring that any difference in performance can be attributed to the change itself, as everything else is static.

There are many tools out there to hep one perform split testing. One thing that is very important is that whether a customer is put into split A or split B must be randomly chosen.

Pivot vs Persevere

Eric later goes on to one of the most important components of the book, which he calls whether to pivot or persevere. Its my belief that this decision making ability will only come with time and experience in a particular market.

Eric pitches that when results of your build-measure-learn loop are seemingly not developing, one needs to decide one of two things

1. Pivot: We modify our hypothesis in search of an alternative approach that may work
2. Persevere: We continue to explore our existing hypothesis

He explains many different types of pivots, for example, one is where a small feature in a product can become a much more prominent part of the product. One that comes to mind is the game “Tribes Ascend”. One day the developers introduced a bug where the players could hold down the jump button to “skate” across the field. The promptly removed it but found out that the community feedback was they loved the new “feature”. After adding it back in, the game has been known as the “first person shooter, with skates”.

Five Whys

An old boss of mine would use this technique to help guide engineering decisions, extracting why we were actually doing a task. The idea is that we should be able to ask ourselves “why” five times, each time getting to a lower and lower level, and closer to the root cause.

This concept can be used in conjunction with a post mortem, where we walk through what caused an outage. Each answer to why can allow us to make a plan to mitigate the issue from ever arising again.

Leveraging Cohorts

Eric walks through the idea of splitting our users up into cohorts, separating them by chunks of time. For example, we could define every week as a separate cohort. Week after week, as we continue to perform our build-measure-learn loop, we should be making progress on improving our actionable metrics. If we’re performing these steps correctly, each cohort should be displaying strong and stronger metrics.

Notes & Highlights

I learn best by highlighting key components that resonated with me and then reflecting on them. To share those with you, I’ve decided to quote each of the highlights I’ve made in the book and record any and all thoughts I had to each of these highlights.

While there are a lot of them, one can think of these highlight as a core summary of the book or a refresher.

The fundamental activity of a startup is to turn ideas into products, measure how customers respond, and then learn whether to pivot or persevere. All successful startup processes should be geared to accelerate that feedback loop.

When people are used to evaluating their productivity locally, they feel that a good day is one in which they did their job well all day. When I worked as a programmer, that meant eight straight hours of programming without interruption. That was a good day. In contrast, if I was interrupted with questions, process, or—heaven forbid—meetings, I felt bad. What did I really accomplish that day? Code and product features were tangible to me; I could see them, understand them, and show them off. Learning, by contrast, is frustratingly intangible.

I found that this quote really resonated with me as someone who does a lot of professional programming. I think its hard to remember that your job isn’t just to program, even if you work on a product team at a software company. One viewpoint is at the end of the day your job is to convert business process and rules into software. The other thing I have to remind myself is that the customer does not care how the product was built, nor what language or framework one used.

The goal of a startup is to figure out the right thing to build—the thing customers want and will pay for—as quickly as possible.

[large companies] are very good at creating incremental improvements to existing products and serving existing customers, which Christensen called sustaining innovation, but struggle to create breakthrough new products—disruptive innovation—that can create new sustainable sources of growth.

[some company] holds themselves accountable for their new innovation efforts by measuring two things: the number of customers using products that didn’t exist three years ago and the percentage of revenue coming from offerings that did not exist three years ago.

I found this metric incredibly interesting, as its something I have not thought of before. The novel-ness of the revenue coming from recently developed offerings can really highlight how innovative the company is or is not being.

changes in TurboTax enabled the Intuit team to develop five hundred experiments per tax season. Before that, marketers with great ideas couldn’t have done those tests even if they’d wanted to, because they didn’t have a system in place through which to change the website rapidly.

This really highlights how spending the blood, sweat and tears on making the build-measure-learn loop as effortless as possible can really pay off.

moving leaders from playing Caesar with their thumbs up and down on every idea to—instead—putting in the culture and the systems so that teams can move and innovate at the speed of the experimentation system.”

I found this quote interesting, as I’ve been in a scenario where most, if not all features, had to be ran up the pyramid and back down. It really caused each team to not be autonomous and did not scale well.

Metcalfe’s law: the value of a network as a whole is proportional to the square of the number of participants. In other words, the more people in the network, the more valuable the network.

As launch day approached, our fears escalated. In our situation, many entrepreneurial teams give in to fear and postpone the launch date. Although I understand this impulse, I am glad we persevered, since delay prevents many startups from getting the feedback they need.

Our previous failures made us more afraid of another, even worse, outcome than shipping a bad product: building something that nobody wants.

Out of further desperation, we introduced a feature called ChatNow that allows you to push a button and be randomly matched with somebody else anywhere in the world.

This feature ended up being a very core component of the product being described. The company originally planned on having customers chat with friends, but the customers were most interested in being connected with random people.

We had a mental model for how people used software that was years out of date, and so eventually, painfully, after dozens of meetings like that, it started to dawn on us that the IM add-on concept was fundamentally flawed.3 Our customers did not want an IM add-on; they wanted a stand-alone IM network. Dlopment methods (known collectively as agile development), which promised to help drive waste out of product development. However, despite that, I had committed the biggest waste of all: building a product that our customers refused to use. That was really depressing. I wondered: in light of the fact that my work turned out to be a waste of time and energy, would the company have been just as well off if I had spent the last six months on a beach sipping umbrella drinks? Had I really been needed? Would it have been better if I had not done any work at all?

Woah. Pretty much all I have to say about this quote.

Lean thinking defines value as providing benefit to the customer; anything else is waste. In a manufacturing business, customers don’t care how the product is assembled, only that it works correctly.

I think this is parallel to the whole “customers do not care what language or framework you used” thought.

I’ve come to believe that learning is the essential unit of progress for startups. The effort that is not absolutely necessary for learning what customers want can be eliminated. I call this validated learning because it is always demonstrated by positive improvements in the startup’s core metrics. It is also the right way to think about productivity in a startup: not in terms of how much stuff we are building but in terms of how much validated learning we’re getting for our efforts.

Once we formed this hypothesis, our experiments became much more likely to produce positive results. Whenever we would change the product to make it easier for people to find and keep new friends, we discovered that customers were more likely to engage. This is true startup productivity: systematically figuring out the right things to build.

it is often easier to raise money or acquire other resources when you have zero revenue, zero customers, and zero traction than when you have a small amount. Zero invites imagination, but small numbers invite questions about whether large numbers will ever materialize.

if the plan is to see what happens, a team is guaranteed to succeed—at seeing what happens—but won’t necessarily gain validated learning. This is one of the most important lessons of the scientific method: if you cannot fail, you cannot learn.

Traditionally, the product manager says, ‘I just want this.’ In response, the engineer says, ‘I’m going to build it.’ Instead, I try to push my team to first answer four questions:

1. Do consumers recognize that they have the problem you are trying to solve?
2. If there was a solution, would they buy it?
3. Would they buy it from us?
4. Can we build a solution for that problem?” The common tendency of product development is to skip straight to the fourth question and build a solution before confirming that customers have the problem.

if we’re building something that nobody wants, it doesn’t much matter if we’re doing it on time and on budget.

Finally, and most important, there’s the pivot. Upon completing the Build-Measure-Learn loop, we confront the most difficult question any entrepreneur faces: whether to pivot the original strategy or persevere.

Although we write the feedback loop as Build-Measure-Learn because the activities happen in that order, our planning really works in the reverse order: we figure out what we need to learn, use innovation accounting to figure out what we need to measure to know if we are gaining validated learning, and then figure out what product we need to build to run that experiment and get that measurement.

STRATEGY IS BASED ON ASSUMPTIONS. Every business plan begins with a set of assumptions. It lays out a strategy that takes those assumptions as a given and proceeds to show how to achieve the company’s vision. Because the assumptions haven’t been proved to be true (they are assumptions, after all) and in fact are often erroneous, the goal of a startup’s early efforts should be to test them as quickly as possible. The first challenge for an entrepreneur is to build an organization that can test these assumptions systematically. The second challenge, as in all entrepreneurial situations, is to perform that rigorous testing without losing sight of the company’s overall vision.

The Japanese term genchi gembutsu, which is one of the most important phrases in the lean manufacturing vocabulary. In English, it is usually translated as a directive to “go and see for yourself” so that business decisions can be based on deep firsthand knowledge.

Handmade PDFs, a pizza coupon, and a simple blog were enough to launch Groupon into record-breaking success;

A minimum viable product (MVP) helps entrepreneurs start the process of learning as quickly as possible.3 It is not necessarily the smallest product imaginable, though; it is simply the fastest way to get through the Build-Measure-Learn feedback loop with the minimum amount of effort.

Before new products can be sold successfully to the mass market, they have to be sold to early adopters. These people are a special breed of customer. They accept—in fact prefer—an 80 percent solution; you don’t need a perfect solution to capture their interest. Early adopters use their imagination to fill in what a product is missing.

Minimum viable products range in complexity from extremely simple smoke tests (little more than an advertisement) to actual early prototypes complete with problems and missing features. Most entrepreneurs and product development people dramatically overestimate how many features are needed in an MVP. When in doubt, simplify.

Somewhere in the business model, probably buried in a single cell in a spreadsheet, it specifies the “percentage of customers who see the free trial offer who then sign up.” Maybe in our projections we say that this number should be 10 percent. If you think about it, this is a leap-of-faith question. It really should be represented in giant letters in a bold red font: WE ASSUME 10 PERCENT OF CUSTOMERS WILL SIGN UP. Most entrepreneurs approach a question like this by building the product and then checking to see how customers react to it. I consider this to be exactly backward because it can lead to a lot of waste.

The lesson of the MVP is that any additional work beyond what was required to start learning is waste, no matter how important it might have seemed at the time.

file synchronization was a problem that most people didn’t know they had. Once you experience the solution, you can’t imagine how you ever lived without it.

The context here was talking about Dropbox.

they used Wizard of Oz testing to fake it. In a Wizard of Oz test, customers believe they are interacting with the actual product, but behind the scenes human beings are doing the work.

If we do not know who the customer is, we do not know what quality is.

Customers don’t care how much time something takes to build. They care only if it serves their needs.

Five dollars bought us a hundred clicks—every day. From a marketing point of view this was not very significant, but for learning it was priceless. Every single day we were able to measure our product’s performance with a brand new set of customers. Also, each time we revised the product, we got a brand new report card on how we were doing the very next day.

cohort analysis. This is one of the most important tools of startup analytics. Although it sounds complex, it is based on a simple premise. Instead of looking at cumulative totals or gross numbers such as total revenue and total number of customers, one looks at the performance of each group of customers that comes into contact with the product independently. Each group is called a cohort.

the sign of a successful pivot: the new experiments you run are overall more productive than the experiments you were running before. There is an antidote to this misuse of data. First, make the reports as simple as possible so that everyone understands them. Remember the saying “Metrics are people, too.” The easiest way to make reports comprehensible is to use tangible, concrete units. What is a website hit? Nobody is really sure, but everyone knows what a person visiting the website is: one can practically picture those people sitting at their computers.[…] Instead of housing the analytics or data in a separate system, our reporting data and its infrastructure were considered part of the product itself and were owned by the product development team. The reports were available on our website, accessible to anyone with an employee account. Second, those building reports must make sure the mechanisms that generate the reports are not too complex. Whenever possible, reports should be drawn directly from the master data, rather than from an intermediate system.

Failure is a prerequisite to learning.

The one envelope at a time approach is called “single-piece flow” in lean manufacturing. It works because of the surprising power of small batches. When we do work that proceeds in stages, the “batch size” refers to how much work moves from one stage to the next at a time. For example, if we were stuffing one hundred envelopes, the intuitive way to do it—folding one hundred letters at a time—would have a batch size of one hundred. Single-piece flow is so named because it has a batch size of one. Imagine that the letters didn’t fit in the envelopes. With the large-batch approach, we wouldn’t find that out until nearly the end. With small batches, we’d know almost immediately. Because of its smaller batch size, Toyota was able to produce a much greater diversity of products. It was no longer necessary that each product be exactly the same to gain the economies of scale that powered mass production. The biggest advantage of working in small batches is that quality problems can be identified much sooner. The ideal goal is to achieve small batches all the way down to single-piece flow along the entire supply chain. Each step in the line pulls the parts it needs from the previous step. This is the famous Toyota just-in-time production method.8

Sustainable growth is characterized by one simple rule: New customers come from the actions of past customers.

I ask teams to adopt these simple rules:

1. Be tolerant of all mistakes the first time.
2. Never allow the same mistake to be made twice.

I strongly recommend that startup teams be completely cross-functional, that is, have full-time representation from every functional department in the company that will be involved in the creation or launch of their early products. They have to be able to build and ship actual functioning products and services, not just prototypes. Handoffs and approvals slow down the Build-Measure-Learn feedback