How we test fake sites on live traffic

2013-11-04 by Darren Nix
  • Upvote on Hacker News

If you clicked on a 42Floors ad for “new york office space” a while back, there’s an 89% chance that you landed on one of these eight versions of our site.

They’re all fake: just throwaway static HTML mockups.

Design evolution

When 42Floors launched back in March 2012, we had relatively few listings so we could  fit them all on a list/map combo like this.


As our database grew, though, the map  started to bog down with data points.  When we added 1,300 listings for our New York launch, render times were hitting 12 seconds for Midtown.  Plus, it was visually overwhelming to have so many data points on the map.

screenshot-map-neighborhoodsWe took the obvious next step and started clustering.

Clustering solved our speed and density issues but we still weren’t happy with the design because it took too many clicks to drill down to individual listings.

So we came up with Unified View.  Unified View was a three-month undertaking that burned up a lot of dev hours but it seemed worth it.  The finished product was blazing fast and had all the goodies: big photo cards, infinite scrolling, a mini navigation map, and the ability to switch to a full-page list or map view just like AirBnB or Yelp.


Conversion rates

Given the unquestionable awesomeness of the new site, we expected a conservative 50% lift in conversion rates when we deployed. But there was nothing — not even a blip in bounce rate, time on site, or conversion rates to mark the deploy date.

Unsurprisingly, enthusiasm for working on the front end cratered for a while and our focus shifted over to the supply side of the business — adding more listings, making tools for building owners, etc.

We eventually did start talking about iterating on the design again but nobody wanted to sign up to build yet another version of the site and see it flop.

Ignoring the user

At this point, you might be suspicious that we’d been designing in a vacuum. Not so. We’d done plenty of interviews and mockup sessions with users trying to tease out what they wanted, it just never got us anywhere.

Here’s where I issue a disclaimer, lest I get trolled: talking to users is really important, just not about conversion funnels.  Here’s my handy user feedback weighting algorithm:

  • Knowing something broken: 100% accuracy
  • Knowing what they want: 30% accuracy
  • Knowing what they’ll actually click on: 2% accuracy

With this in mind, we set out to disregard the focus groups and drive the design, instead, based entirely on browsing data from live traffic.

The problem was: how to get data?  The types of wholesale redesigns that we needed to split test aren’t possible with Optimizely.  Feedback services like 5SecondTest are unrepresentative of real users. We also couldn’t dedicate months to building multiple working versions of the site that would, in all likelihood, fail.

So we decided to fake it.

Faking it

Over the course of a few days we sketched out eight designs in Moqups and converted them into sloppy Photoshop mockups.  We were pretty optimistic that the design called Modals would win.  The modal-based UX paradigm was intended to act like a one-page app and solve the common back and forth problem of loading search results, clicking on an item, going back, etc.

In addition to ignoring our users’ design preferences, though, we also wanted to try ignoring our own design instincts.  So we designed versions that we hated: Ugly Banking Site, Landing Big Buttons, etc.

Office space mockup 1

We shipped the PSDs off to PSD2HTML and got back the quotes: $295 for the simplest design, $972 for the most complex one.

We added two extras to the typical package:

  1. Copy and paste in listing descriptions and photos from specific real listings from 42Floors instead of lorem ipsum placeholders
  2. Use Bootstrap to make editing easier for us

Even though these were going to be static mockups, they had to feel real to first-time visitors, otherwise the test results would be worthless.  So, each mockup got rollovers, hovers, lists, and modals that showed real but pre-populated data.

Live testing

PSD2HTML finished in nine days at a cost of $4,197.  The next day we uploaded eight folders of static HTML into /public/land/v1-8 in our Rails app.  We cloned eight copies of our AdWords campaign and changed the destination URLs to match.

Then we waited.

The winner

The winning variation was Google Hover Clone — a 1:1 mockup of Google’s site preview pane.

Office space mockup 1

We call that static test Hover1.  Our criteria for picking the winner came down to one number: tour request rate.  In other words, what percentage of users found a listing they liked and contacted it using a particular design.  We use other data points to augment this core metric (bounce rate, time on site, number of listings viewed, search criteria revisions), but ultimately a variation only wins if it’s better than the control at getting users to an office space that they like.

We went on to build a working version called Hover2 and put it in production back in August.  With it, the long-awaited conversion rate lift finally materialized.  As of this writing, we’re on Hover3 and about to deploy a split test for Hover4.

PS – The strangest outcome of the process was that Ugly Banking Site had the lowest bounce rate of all the designs by a wide margin.


About Darren Nix

Post-apocalyptic book junkie. Founder at 42Floors. Previously founded Leaky and Silver Financial.

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 100 other subscribers

  • Upvote on Hacker News
  • josep2

    Sweet post. Thanks!

  • eenoog (one-eye)

    I love this. Brilliant in its simplicity. Gonna apply it right away. Thanks.

  • András Sallai

    The ugly banking site is spot on and really cracked me up :D
    Nice post! Thank you!

    • Tibor Halter

      haha same thoughts here :)

  • livejamie

    Great post. Can you go a little bit more into detail on the results?

  • dr3wid

    The usability problem I see is with the vertical content of the hover pane. Ok I’m hovering over a listing but due to the vertical real estate on my monitor I need to scroll ( mouse wheel ) to see the listing content over on the right. This action looses the focus and instantly hovers me over the next listing, thus loading other content.

    • DarrenNix42Floors

      Totally agree; that’s one of the UX issues that hover4 is going to fix.

  • Corey Dubeau

    One thing I don’t quite get.. Were these functioning HTML websites? Like what happened after someone would click one of the buttons? Would they actually be able to access the full website using the same mock-design, and then you would just populate it with your database content?

    • anon

      Looks like the buttons did absolutely nothing, from the link at the top of the article

    • DarrenNix42Floors

      If you tried to do some functionality that required a database call… like sorting, filtering, or contacting a listing, it would redirect you to the corresponding page on the live site. So it would feel like the site glitched but it would eventually work the second time.

      • Adam B

        If you’re sent to the live site for sorting, filtering, or contacting, doesn’t this defeat the purpose of using the tour request rate as the deciding metric? I’m not familiar with 42floors but I imagine requesting a tour is far enough downstream that users would have to take a lot of activity on the live site before requesting a tour.

  • BraveNewCurrency

    Ugly banking site had a low bounce rate because people were slowed down trying to parse it all.

  • Stefan Wrobel

    RSS feed plz!

  • john

    Could you share the % of each variant, to get a feel how they vary?

  • Ken Smith

    Great post – thanks for sharing! I’d like to know about your traffic. How many unique visitors were required before you declared that a change in behavior was statistically significant?

  • Heather

    Thank you for sharing your process! The photos are fantastic. =)

  • Stuart @

    Love posts like this that go against the very grain of design trends, we all think that beautiful flat design is the way to go – but until you actually test you have no idea what your customers will respond best to :)

  • Abinash Mohanty

    I love the entire process and this how the lean principle should be. Others who don’t care about the user testing or user survey would always ask “How to get good data or matrix?” without leaving your 24/7 desk. Thanks Darren for the lovely insights. These are quite helpful as case studies.