The Math of Split Testing Part 3: The Chance of being Similar

2014-06-24 by Aaron O'Connell

Sometimes we want to verify that a new design will convert at nearly the same rate as an old design. Split tests of this type are not intended to find conversion rate wins, but rather to ensure that the new design is not “too much worse” than the old design. Here, we demonstrate how to analyze a split test of this sort and present an approximate formula to quickly calculate the chance that there is an acceptable difference between the conversion rates of the new and old designs.

Let’s say that we think that the design of our website is starting to look a little outdated and needs to be redone. In this scenario, we’re not really looking to increase conversion rate. We’d be happy to keep our current conversion rate and just make the website look a little more hip. To verify that our new design (branch B) converts at a similar rate Continue…

In Defense of Recruiters

2014-06-17 by Jason Freedman

Recruiter on line three cartoon

Ever since we wrote that post where we trolled annoying recruiters, I’ve had quite a few conversations with entrepreneur friends about this subject.  Overwhelmingly, most of these founders said something along the lines of, “recruiters are just worthless.” Obviously, I could see why they would think so; I used to feel the same way.

I didn’t use a recruiter to hire anyone for either of my first two companies or for the first ten people at 42Floors.  The entirety of my interaction with recruiters was trying to get them to stop spamming us about their “MIT Alumni at Google who is a Ruby Ninja Rockstar.” Or even worse, that same recruiter spamming every engineer in our company promising to help them find a better job.

However, we’ve actually used recruiters very successfully at 42Floors, and all my friends who are claiming that all recruiters are worthless tend to be seed stage or first-time entrepreneurs that have not yet gotten to the best use cases for recruiters.

So I thought it might be helpful to show the different ways we have successfully used recruiters to scale 42Floors.   Continue…

The Math of Split Testing Part 2: Chance of Being Better

2014-06-10 by Aaron O'Connell

Split tests are a powerful way to determine if a new design has outperformed an old design. Here, we present the math necessary to empirically state the chance that a new design (branch B) is better than an old design (branch A). We walk through the problem with numerical simulations and present an approximate formula to easily determine the winner of an A/B test.

Split testing is the preferred way to collect data for fact based decision making. Longitudinal studies are subject to underlying market trends and are not well suited for conversion rate experiments. If, for example, you roll out a new version of your website in April and notice a decrease in conversion rate, you may be tempted to say that your new website design is a failure. But maybe April was just a bad month. If instead you had set up a split test for your new design, you could eliminate this concern. While overall conversions may have fallen in April, it could easily be the case that the results of a split test measurement would have shown the new website design performed 20% better than original design, with 95% chance that the new design is better than the original.

In this post, we’ll introduce the math necessary to declare branch A or branch B the winner of the split test. Continue…

Node FTW

2014-06-09 by Aaron O'Connell

Want to crunch some numbers? Node is 10x faster than Ruby.

I used to do a lot of scientific computing. My work environment had both Matlab and NumPy/SciPy all tricked out. These days I don’t do too much number crunching, spending most of my time with Ruby and JavaScript. Continue…

The Math of Split Testing Part 1: Statistical Sampling Uncertainty

2014-06-05 by Aaron O'Connell

Here we introduce the idea of statistical sampling uncertainty in the content of conversion rate optimization for website design. We begin by illustrating how successive trials of coin flips result in the binomial mass function. Then, we turn the problem on its head and show how to describe the conversion rate of a funnel event (the true bias of a weighted coin) by examining the results of a series of conversions. Finally, we discuss the validity of the Wald and Wilson confidence interval approximations to further quantify the conversion rate confidence interval.

This series of blog posts focuses on the math needed to make fact based decisions as quickly as possible. Many websites have clear actions they wish their visitors would take. A commerce website, for example, would like it if their visitors would purchase a product. While it may be reasonable to focus solely on the number of unique visitors that purchase a product (total conversions), in many cases it is useful to to think about the customer experience as a series of actions that lead down a funnel. Continue…

Striking back at recruiter spam

2014-06-04 by Darren Nix

Screenshot 2014-05-30 14.40.06
We’ve decided on a new approach to handling recruiter spam.

Before I go on, let me say that we like working with high quality recruiters from time to time, including one fantastic freelance recruiter, Oli Ryan, who helped us find several of our key hires.

That being said, we get bombarded daily with spam calls and emails from recruiting companies. Most of these we just ignore, but some bad actors decide to contact every person on the engineering team including calling our mobile numbers. That’s unacceptable.

So, it’s time to strike back: if they waste our time, we’ll waste theirs. Continue…

Conditional Custom Templates with Action Pack Variants

2014-05-30 by Aaron O'Connell

Variants were introduced in Rails 4.1 making it possible to conditionally render a view template. Before Rails 4.1 we employed a custom technique dubbed MultipleTemplates to provide similar functionality. Here we present our MultipleTemplates approach and why we’re switching to Action Pack variants.

We’ve used conditional templates in our Rails API backend for quite some time now. Our main use for conditional templates is to reduce rendering times by including only the minimal amount of data. A careful profile of our response times initially led us to this approach. Continue…

Startup Offer Letters

2014-05-04 by Jason Freedman


I still remember getting my first ever offer from a real life company.  It was a pretty big moment. I was 21, about to embark on my career as a consultant (oh, how silly 21-year-old Jason was) and I had just emerged victoriously through four rounds of interviews.  I correctly estimated the gallons of ketchup used per day by the average McDonalds.  I beat out 137 other candidates.  It was a big deal to me.

I got the call from my interviewer, congratulating me.  She told me the offer letter was in the mail.


The absurdity of LinkedIn

2014-04-30 by Darren Nix

This morning I had a call with someone I hadn’t met so I Googled their name and, inevitably, clicked through to their LinkedIn profile.

Screen Shot 2014-04-24 at 1.28.19 PM

I was bemused to note that, although the search result in Google showed their full name in the title, their profile page obscured their name until I “Upgrade for full name”.

Screen Shot 2014-04-24 at 1.37.21 PM

Obscuring the name seemed especially silly given that the URL of the page includes the full name. Continue…

How to break into startups

2014-04-29 by Jason Freedman


With startups finally popular as potential career paths, I’m starting to get a lot of people that are frustrated with their current job and are looking to make a jump. I just had one such call with this guy Eric – super bright guy, very nice, graduated from an Ivy League school, and now works for a top tier investment bank. And after only 6 months on the job, he realizes he made the wrong decision.  He called me because he wanted to get into startups.

I’ve been posting a lot about our account management jobs recently because we’re rapidly growing that team.  He’s pretty interested in commercial real estate and he wanted to know how he could go after one of those jobs.

In the back of my head I had two thoughts rolling around.  The first is this guy could be really awesome for us.  He’s ambitious.   He’s smart. And the second thought was I bet he’s never going to leave his job.