Category: Lean Startup Experiment

This is a series of posts outlining my experience launching a company using the Lean Startup methodology, from MVP to customer validation to product development.

Lean Startup Experiment – Part 4 – Analyzing the Results of My Lean Startup Hypothesis

I’m excited to announce that the adjustments to my hypothesis (described in Part 2 – How to Make an Effective Lean Startup Hypothesis) and the corresponding changes to my website (described in Part 3 – Adjusting your Experiment to Test your Hypothesis) have provided me with some useful data.

In my original goal for the experiment, noted in Part 1 of this series, I pledged to move forward with my product idea if two people actually pay for my product.

The results

I shouldn’t be surprised by the results because of my confidence in the product idea, but (drumroll, please…) nine people have paid for my product! It was an amazing feeling receiving those first few emails from PayPal saying that someone paid me $25 for my product. Of course, since I don’t actually have a product to provide them yet, I sent them a nice email thanking them for their interest and apologizing that I was unable to fulfill their order. I refunded their money right away.

Why I’m especially excited

As if being paid actual money for my product idea wasn’t exciting enough, there are a few additional reasons why I’m especially excited:

  • Aside from the nine people who paid me for my product, an additional three people contacted me expressing serious interest in my product. For example, one person asked if I would ship to Canada. While the nine paying customers are much more meaningful than the people who expressed interest, it’s exciting to know that it’s even more than just those nine people with interest in the product.
  • All paying customers reached my website from organic search results. So, not only were they paying customers, but I spent $0.00 acquiring them (not counting for the minimal time I spent creating the site and optimizing it for SEO).
  • And possibly the most exciting thing to note: the people who paid me for my product bought it sight unseen! Indeed, my website does not yet have any pictures of the product on it. Creating an image of what I think the product will look like would have required too much of a time investment for me to figure out how to manipulate Photoshop enough to alter an existing product image.


It might still be premature, but I’m ready to consider this product idea validated after this experiment. After all, nine people have paid me cash money for my product, sight unseen. In other words, the pain point that this product solves is great enough that people are willing to pay for a solution without even knowing what the solution looks like. For all they know, the product could be hideously ugly, but that didn’t stop nine people from shelling out real cash.

I will discuss my next steps in the next post in this series.

Lean Lesson Flashback: The Concierge MVP

The Lean Startup includes a great case study of a startup that used what Eric Ries refers to as the ‘concierge MVP’. The company envisioned a grocery delivery web and mobile application that would suggest particular groceries based on recipes, feature sale items, and allow the customer to order through this app.

Rather than invest their time and money into building even a primitive version of the web app, the team went to grocery stores, approached potential customers in person, and started to offer them the services that they envisioned the web app would eventually do. When they found someone to accept their services (for free to start), they continued providing value to the customer. Eventually, they asked the customer to pay for the service, and the customer did. The act of the customer actually paying for the service was a crucial validation of their hypothesis about the usefulness of their web app.

The concierge version of a minimum viable product requires manually taking care of initial customers with the attention that a personal concierge would provide at an upscale hotel. While the vision for your product or service probably includes more automation, this personal approach allows you to experiment and learn about the validity of your hypotheses without investing time and money in development.

Looking back on the founding of my previous startup, Bungolow, my cofounder and I definitely could have used this technique before we invested serious money and three crucial months on development of our website. The bulk of the three months of web development was spent on the calendar / booking / payment processing system on the website, which allowed a customer to select an available hotel room date from a calendar and book the room using their credit card.

Bungolow's prematurely complex calendar booking system

Above: Bungolow’s prematurely complex calendar booking system took three months to develop, when just a sentence telling customers to call us would have sufficed early on.

While the calendar booking system was always part of the long-term vision of the site, we could have launched an MVP without the complex (and costly) system in place yet. We could have spent two days – rather than three months – putting up a simple website with a phone number and a line of text: “If you are interested in our current hotel deal, simply call us, and we will personally walk you through payment and any questions you may have about the sale.”

We would have learned a ton of information about our customers and whether our product solved a problem for them. Three months and serious money later, the complex booking system was live, but it didn’t put us in a position to learn more than a concierge MVP would have.

Lean Startup Experiment – Part 3 – Adjusting your experiment to test your hypotheses

As discussed in Lean Startup Experiment – Part 2 – How to Make an Effective Lean Startup Hypotheses, I learned the hard way that my original hypotheses were too vague, too unfocused, and too many. I cut the original four hypotheses down to one specific hypothesis: that people would buy one unit of my product for $25.

In this post, I will walk through what I did to tweak my experimental website to account for my new hypothesis.

Minimize Options

The pricing page on my original website included the monthly subscription model and three pricing options: an Individual plan with one of the products delivered per month, a Couples’ plan with two of the products delivered, and a Family plan with 3 or more of the products delivered per month.

This was a sloppy product page for many reasons:

  • As discussed in Part 2, by hypothesizing about the subscription model, I was getting way ahead of myself.
  • Not only was the subscription model premature, but it added unnecessary noise into my data. At this stage, if people were not buying, it wouldn’t be clear whether they were turned off by the product itself or by the idea of feeling locked into a monthly plan.
  • I don’t know what I was thinking adding in the Individual, Couples’, and Family plan. I had too many hypotheses to begin with, and here I was trying to test out another hypothesis that wasn’t on that already-too-long list.

To adapt the pricing page to my new, singular, more concise hypothesis, I got rid of the Individual, Couples’, and Family plan nonsense and the necessary monthly subscription. What was left was one option: a single unit of my product for $25.

Simplify the Checkout Process

The checkout process on my original website was long in hindsight. After clicking Buy Now, the customer would be taken to a page where they had to enter their name and email address and click continue; which would take them to a new page where they would enter their shipping information and click continue; which would take them to the Paypal checkout page where they would have to go through Paypal’s payment process.

When originally mocking up this process, I had what I thought was good reason for making it so thorough. I wanted to make sure I had all of the customer’s information at the time of purchase (a valid thought). In reality though, I should not have cared so much (yet) about collecting all that information because I didn’t actually have a product to send the person anyway. Google Analytics revealed that visitors were abandoning the process before making payment.

I shortened the process. Clicking ‘Buy Now’ would now send the visitor directly to the Paypal page rather than through my own information-collection pages first.

A brief side note: I stood by – and still stand by – my decision to actually require myself to receive funds (i.e. have the customer actually pay) for me to consider the experiment a success. Relying on the analytics of how many people click the Buy Now button is not really sufficient, because non-customers like to look around, even past the Buy Now button.

Add a Phone Number

My original site had my email address as the only way to reach me.  That’s not enough. I threw my phone number up there as well, figuring that having customers call me would give me more opportunities to learn.

In part 4, I’ll hopefully have some data to discuss. I’ve learned a lot about the Lean Startup process itself – now I’m hoping to learn a lot about the viability of my product idea.

Lean Startup Experiment – Part 2 – How to Make an Effective Lean Startup Hypothesis

I’ve learned a few things so far in my lean startup experiment, but the biggest lesson so far would be: When conducting a lean startup experiment, keep your hypothesis singular, focused, and testable. Looking back at my first post, Lean Startup Experiment – Part 1, I made four hypotheses. I even tried to play it off as three hypotheses (by using 1a and 1b) – almost as if I knew I was being a bad lean startup experimenter.

I’ll comment quickly on those original hypotheses:

#1: People will pay for a natural version of this common product… yada yada yada.

– Hypothesis #1 is on the right track, but is a little more vague than it needs to be. I will clarify by the end of this exercise why that’s the case.

#1b:  The reason people will buy will be for health purposes… yada yada yada.

– While confirming this hypothesis will be helpful for marketing purposes somewhere down the line, it really doesn’t matter at this early stage. Frankly, this early in the process, we don’t care why someone would buy the product; we just care about if someone will buy the product.

#2: People will subscribe to automatic monthly purchases of the product… yada yada yada.

– This hypothesis will also be really useful… but LATER. Let’s not get ahead of ourselves. For now, let’s just worry about whether someone will buy one, because if they won’t buy one, they won’t buy them each month.

#3: People will pay $25 for the product and will pay $25 each month… yada yada yada.

– This hypothesis was off to a great start until it tried to do too much. Again, we’re not at the stage where we should worry about the monthly payment thing. Also, you can see now that this hypothesis makes Hypothesis #1 vague and redundant.

Taking into account my comments on my original hypotheses, here is my new set of hypotheses: People will pay $25 for the product. (Period. That’s it. Just one hypothesis.)

I’m happy with my new set of hypotheses (i.e. my one hypothesis) because:

a) It’s not too vague to be testable
b) It will be very clear if it’s not proving to be right
c) It won’t bring a lot of variables into the picture, because it focuses on one thing at a time and not on future hypotheses. The focused nature of the hypothesis will help determine if the hypothesis is not proving to be right (see ‘b’).

In the next post, I’ll talk about how to adjust your experiment (in my case, my website is the tool for the experiment) to correspond to your hypothesis.

Lean Startup Experiment – Part 1

I am starting an experiment using the principles of The Lean Startup to see if there is a business possibility around a product idea that I had. I’m not going to go into detail on the product because I don’t want traffic from this blog to affect my data, but the product is a natural version of a common item associated with sleeping.

The product idea is just an idea at the moment – so far, I have invested zero money and very little time researching the physical development of the product (prototype creation, sourcing of raw materials, manufacturing, packaging, etc.). Before I make the investment in the physical development of the product, there are a number of hypotheses that I’d like to prove either right or wrong.

Hypothesis #1

My first hypothesis is that people will pay money for a naturally produced version of the item in question, as the existing market only includes synthetic products.

Hypothesis #1b

A follow-up to the first hypothesis is that the demand mentioned above will come mostly from the inherent health benefits of using the natural version over the synthetic version, with some additional motivation coming from the environmental benefits of the natural version.

Hypothesis #2

The little bit of research I have done on the development of the product has revealed that the natural version of the product would have to be replaced for a new one about once a month. My next hypothesis is that people will not only pay for the natural version of the product (as mentioned in Hypothesis #1) but they will pay for a new one every month in the form of a monthly subscription.

Hypothesis #3

My last hypothesis for this learning cycle is that people will pay $25 for the product and will pay that each month (see Hypothesis #2).

Experiment A to prove or disprove hypotheses 1-3

In the spirit of Lean Startup, my goal is to get through the Build-Measure-Learn cycle for the above hypotheses as quickly as possible. I have spent approximately 3 days putting together a website that should prove or disprove the hypotheses. I want to see if people will actually pay $25 for the natural version of the product and subscribe to receive a new one (and pay) each month, so I am making that possible (in a very basic way) on the website.

The goal:
I am going to use a free Google Adwords credit to send $100 worth of clicks to the site. I will move forward if I can get 2 people to actually pay for my product.

The risk:
It should be noted that I might be making a bad mistake by grouping hypotheses 1 and 3 together in one experiment rather than approaching them one after the other. Grouping them together could lead to bad data. For example, if nobody ends up paying, I wouldn’t know for sure whether it was because they had no interest in the product itself or no interest in subscribing to monthly deliveries of the product. This is a calculated risk that I’m going to take, though, with the reward being speed in getting through this learning cycle if it ends up working. If the bad situation in the example looks like its starting to pan out, I will make a few tweaks to the website to test single purchases rather than subscriptions.

Tools I used for the experiment

My focus in putting together the site was on tools that are free/cheap and quick to implement:

  • Domain name – $8
  • Hosting account – $5/month
  • Twitter Bootstrap theme from – $12 – Given my slow web design skills so far, this purchase made building the website a lot faster for me. I could have used WordPress and one of its free templates, but I wanted the practice of building a site from scratch.
  • Google Adwords $100 Free Credit – Free – These are pretty common. Mine came with my hosting account purchase.
  • Google Analytics – Free
  • Google Docs – Free – I set up a quick customized form for a registration page. I will experiment removing this to see if it had an affect on conversions.
  • Mailchimp – Free – Added an email newsletter sign-up on the site.
  • Paypal – Free before payment fees – I created a Buy Now button using my personal PayPal account and added the code to the site.

I will follow up with a post on the progress of the experiment.