Quickly and Cheaply Validating Your Ideas

So you have a great product idea, but how do you prove that the world
wants it? Well, one way to do this is to spend 3-6 months building
that product (for your sake, we assume that's all it will take). You
hack away, building the features that you want. When it is finished,
you attempt to get some people to come to your site and try it out. If
all goes well, you get a bunch of people to try it, and they give you
feedback on whether they like it. Hopefully they do, and if not, well,
it's back to the drawing table. Maybe they even tell you what they
want to build next time. However, you have just burnt 3-6 months of
runway (which could be coming out of your pocket). It's also pretty
dejecting to build a product that no one wants.

There has to be an easier way to do this. Well, there is. Until
recently, it was pretty expensive and difficult to validate your ideas
(pay someone to run a focus group), but recently this has become easy,
thanks to two inventions: Google Adwords and rapid development
frameworks (Ruby on Rails and Django are two that come to mind).

So there are two things that every web startup needs to do: user
behavior studies and experiments. These can save you a lot of time and
money, and can help put your product on the right track in minimal
time. If you haven't already started. you should be doing them now. So
what are these things, and why are they important?

User Behavior Studies
I'll start with these first, because they are simple. This could more
simply be called "talking to users," but they are actually better than
talking to users, because you get to see how a user truly feels about
your application (and this is pretty obvious when you do a user
behavior study). To your face, a lot of users will tell you that your
product idea sounds great, but that doesn't mean that they truly mean
it (or that they will use it).

If you were a big company like Google, you would have a fancy
usability lab. You would have people who were hired just to do
usability testing. They would run users through a set of actions, and
the product managers and engineers would watch behind one-way glass
(the product managers would furiously scribble notes about every move
that the user made). There would even be fancy computerized systems
tracking where the user is looking at all time. In the end, you would
be able to see real users use the product. It would be a big
production. No one would get any work done for days, and then you
would have a big meeting to discuss the "learnings."

How do startups hack this process to their advantage? Well, you can
put together a quick demo of your product, and load it on your laptop.
Build the simplest possible demo that exposes the functionality you
want to test. Then head out to the nearest busy coffee shop. Talk to
some random people, and ask them if they will take part in a quick
test (most people are pretty nice about this). I know it's scary to
talk to random people, but much less scary than spending 6 months on a
product that flops. Start by asking general questions about how they
already accomplish what your product wants to help them do. If, for
example, your product helps them to shop, show them a blank browser,
and ask them to shop for a product that they want to buy. Observe what
they do. Encourage them to speak their thought process aloud. Observe
what pains they encounter.

Next, whip out your demo. Ask users to attempt to "use" it. Give them
as little instruction as possible. Mostly, just watch. Encourage them
to narrate. If they get stuck, observe that before you tell them what
to do.

Watching a few users can give you invaluable feedback. My current team
had some conflicting hypotheses about how users would use our product,
so we resolved this by showing the product to five random people at a
coffee shop. In two and a half hours, and pretty much for free (I got
a parking ticket for being over time in a two hour zone), we learned
key facts about how people shop for products on the Internet. And we
probably saved ourselves one to two months of wasted effort.

If you haven't done this yet, do it tomorrow.

So while user behavior studies are descriptive, these give you hard
data. To do this, you hack together several possibilities, and you put
it out to the users to vote. But you don't tell them what you are
doing - you can randomize the behavior of your site. First of all,
build the minimum possible experience required to expose a feature.
Then build a few alternatives, and make your application randomly
serve one of the possibilities (be sure to keep track of which
possibility you are testing). Then, use analytics software such as
Mixpanel to record what the user does in each situation.

Finally, drive some Google or Facebook Ads to your page. Don't pay a
lot of money - you often don't need all that many impressions to
figure out what is going on (I think it cost us $6 to run one test
that gave us valuable information). You may need to do some work
figuring out which creatives lead to the highest click-through rates.
Click-through rates don't matter terribly much, although Google will
stop serving your ads if your CTR is too low. Plus, you will probably
need to learn how to do this for when your product launches.

In the end, you can analyze the results, and draw conclusions based on
what happens. It can be surprisingly quick - each time we have run an
experiment, we have had fairly convincing results within a few hours.
Of course, it is important to control the variables. If you change too
many things between conditions, it may be hard to figure out what the
true differences are.

In our case, we looked to see what would lead users to give us their
email address (a big portion of our functionality involves emailing
updates to users). We tried three conditions:

1) We immediately prompt the user for their email address.
2) We wait until the user does something on our site, and then we
prompt them for their email address.
3) We display some site functionality related to the query term that
the user clicked through on, and then we ask for the email.

Interestingly, the third condition had a massively higher conversion
rate (actually pretty respectable), followed by the second condition,
and no users gave us their email in the firstcondition (big shocker).

Ego Check
If you love your product idea more than your first-born child, you
probably don't want to do this. There is nothing more crushing than
having your product idea completely devalidated by an experiment or
user behavior study. But it is even more crushing to spend six months
to a year working on something that you end up killing because no one
wants it. I've done both, and I would drastically prefer the former to
the latter.

So get out there and start already.