An Ode to Experimental Design

If you search the Internet to learn about A/B testing, you’ll find scads of articles bursting with tips for cranking your business performance into the stratosphere.

You’ll get blazing hot secrets like…

BOOST YOUR CONVERSION RATE BY 300% WITH THIS TINY TWEAK

and…

DESTROY YOUR COMPETITION USING A STATISTICALLY SIGNIFICANT SHADE OF BLUE

…and it just keeps going like that, into an overenthusiastic pit of armchair psychology and semi-authoritative pseudoscience.


As the gurus tell it, A/B testing is like Vegas slots: plunk some crap into a machine, score a handful of 🍒🍒🍒s, and voilà, Easy Bake Revenue!

With a pitch like that, who could resist? It sounds so simple. If you don’t do it, you’re obviously a fool who’s leaving money on the table.

Welp, I have a couple of hard truths for ya:

  1. It’s not as easy as it sounds, and those big gains aren’t such a sure thing.
  2. Setting out to dramatically boost some arbitrary metric (signups, conversions, revenue, whatever) is exactly the wrong way to approach a design problem.

How do I know this?

I spent most of last year testing dozens of conversion-related design ideas in Basecamp, and I found that the JACKED UP PERFORMANCE aspect is the least interesting part of the process, by far.

The most interesting part is how it can change your thinking. Testing is a ticket to ride. It energizes your adventurous spirit, introduces you to uncharted territory, and lands you in cool places you never expected to go.

Here’s what I learned, and why I’ve come to love doing experimental design.


Testing turns your designs into trash.

After running tests for a while, you’ll find yourself throwing away mountains of design work for just a handful of meaningful improvements.

Do that enough, and you’ll notice something: Maybe design isn’t such a special endeavor after all!

The truth is…a lot of design is ephemeral, malleable, disposable…garbage.

Anything you make today merely represents one moment in time. Maybe it’s your best idea now, but it’s not necessarily the best idea you’ll have in another two days or two months.

Testing makes this painfully obvious on a shorter time scale. You’ll soon become less emotionally invested in your precious creations, and more focused on the problems you care about.

Testing strengthens your gusto.

I don’t know about you, but it took me years to build up the confidence to make hard design decisions. I still struggle with it sometimes.

You have to succeed and fail a lot. You have to take criticism a lot. And you have to trust your gut and keep at it, day after day.

Experimentation is a great way to build that muscle. It’s an opportunity to try things you aren’t completely sure about, and gives you a sweet little safety net for failures.

Testing makes you thoughtful.

It can be tempting to do experiments like the lottery: throw a bunch of random shit at the wall and then declare victory when one thing performed best by random chance.

You might get lucky a few times that way, but it’s a terrible long-term approach. Without some overarching vision, you’ll be left with a gnarly mess of test results born from guesses, and no clear plan for what to do next.

The better way, of course, is to start with good ideas! Do some research, come up with educated hypotheses and concepts you believe in, then build and test them to verify your thinking instead of defining it.

Testing destroys perfectionism.

It’s so freeing to ship a bare-bones version of an idea because it’s “just a test” that you’ll either improve or throw away when it’s done.

If you thought that same design had to stick around permanently, you’d probably never launch it with a lot of known flaws or incomplete parts. You’d want to fix up every last detail and make everything perfect first.

Amazingly, those rough, imperfect tests often outperform the supposedly perfected version you already had in place. When you see that, you’ll realize your outsized attention to detail might not matter as much as you thought.

A license for imperfection is an extremely useful defense against Fussy Designer disease. We should all be vaccinated early and often.

Testing builds empathy.

This sounds counterintuitive: running an experiment is mostly a pragmatic and statistical kind of thing. How is that related to empathy?

It’s related because you’re forced to learn what happens when real human people interact with your work. Your choices all have directly measurable effects, so you can’t hide behind bullshitty designerspeak or vague justifications when the data shows you’re just flat wrong.

That means you have to get outside your insular designer bubble, stop thinking of people as numbers, and get in their shoes a bit.

When you do that, the business boosts you want will happen as a natural side effect of continually tuning your product to serve your customers’ real-life situations. Making things clearer or more efficient for your customers always pays off.

Testing helps you make space.

One tough challenge in UI design is making physical space for new things you want to do. There’s only so much room on a screen!

You might have ideas that require injecting steps into an existing UI flow, adding more screens, revising a visual hierarchy, or rearranging certain navigational elements.

Doing stuff like that is a gamble. You might be confident that your new version is better in some way, but are you sure your improvements are worth the extra steps or added complexity?

Testing lets you dip your toes in the water. You can run a short experiment and see if you’re busting your business before committing to a direction.

Testing tells the truth.

The truth is weird. Sometimes common sense wins out. Sometimes a wild idea succeeds. Sometimes a version you hated performs the best. Sometimes your favorite design turns out to be a total stinker.

Where else in the design world can you get opinion-free feedback like this? There are no Art Directors or Product Managers or App Store reviewers telling you what they think is right. It’s real human nature telling you what’s right!

It’s a fascinating, powerful, bizarre reality.


Do you use testing as part of your design process, and NOT just for improving some Haha Business metrics? I’d love to hear about that. Drop me a comment below or holler on Twitter.

Does Know Your Company actually work?


The most common question I hear from a business owner about our software is something to the effect of: “What are the results I’ll see in my company because of Know Your Company?”

In other words, “Does Know Your Company actually work?”

Admittedly, when a business owner would ask me this, I’d sometimes struggle to answer the question. I’d share anecdotes and success stories from our 200+ CEOs and more than 12,000 employees who use our product in 15 different countries (stories of which we had plenty)… but those stories alone didn’t feel like enough. I wanted a bit more data to share.

To better understand exactly what impact Know Your Company has, we conducted a survey with all of our customers — from which 96 CEOs (28% of our customer base) and 143 employees (7.9% of our customer base) across 30 different companies ended up participating. Here’s a breakdown of the size and industry of the companies who participated in the survey:



Based off this, here are our most interesting findings (you can view the full survey results here)…

#1: What Know Your Company does best is create a more connected company.

The most consistent finding that was revealed in the survey responses is that both employees and CEOs feel connected to one another as a result of using Know Your Company.



The consistency of those responses — it is such a high percentage for both employees and CEOs — stood out to me.

In particular, a lot of the feedback we got was about how helpful Know Your Company is for connecting remote teams or companies with several offices. Employees reported feeling especially connected to co-workers they don’t see regularly, and to new hires just joining the company.


Creating a greater sense of connection in a company matters. Studies show that 33% of employees don’t trust their CEOs — but when they feel connected to their CEO, that trust is increased. As the study describes, the more employees trust their CEO, the more-likely they are to be loyal, work harder and speak highly about the company.

Connection between co-workers increases employee engagement, as well. As stated in this HBR article: “Companies and leaders who want productive, happy employees should make it their job to foster more intimacy at the office.”

#2: The more your employees knows each other, the more engaged they are.

Another finding from our survey is an overwhelming majority of CEOs see higher employee engagement and a positive impact on company culture as a result of using Know Your Company.



These results are important, given how critical employee engagement and a healthy company culture are to a team’s success. Without a healthy company culture, turnover increases… and that’s expensive. It costs the average company 150% of an employee’s salary to find a new person to fill that position.

Employee disengagement is expensive, as well. A 2009 Gallup poll of more than 1,000 U.S.-based employees found that for every disengaged employee, a company loses between $3,400 and $10,000 in salary due to decreased productivity.

#3: Know Your Company gives employees a voice.

The survey also revealed to us how much Know Your Company enables employees to have a voice. Almost 8 out of every 10 employees feel the Know Your Company gives them more of a voice. This has always been an intention of our product — to help employees speak up at work — so it was rewarding to see employees’ outcomes matching that intention.


Numerous employees also shared how Know Your Company helps create a safe, friendly environment where they feel more comfortable voicing their opinions. In fact, some employees admitted noticing how employees are more honest, because of Know Your Company.


When employees are given a voice, they can help you overcome your blindspots as a CEO. By contrast, when employees lack a voice, they are more likely to be disengaged. Gallup found that 4 out of 10 workers become actively disengaged when their managers don’t communicate or ask for feedback.

#4: Know Your Company helps you make more informed business decisions.

One finding that I found particularly interesting is the number of CEOs who have used Know Your Company to make more informed business decisions.


Through dozens and dozens of phone calls and meetings over the past three years, I’d heard how CEOs use Know Your Company to improve their companies’ benefits, employee onboarding, marketing, sales, design, product development, customer support, etc. But the survey data gives us the ability to objectively say that this is an outcome that a large majority of our customers are experiencing.


#5: Know Your Company helps employees know what’s going on in the rest of the company.

Another statistic that surprised me was the number of employees who better understand what is going on in the company because of Know Your Company.


This isn’t the primary function of Know Your Company as a tool (the primary one being to help a company feel more connected). But it’s crucial — the bigger a company gets, the easier it is for employees to feel silo-ed from other areas of the company. When employees feel like they’re “on the same page,” it’s easier for them to see how their individual contribution is helping to the company forward. This can lead to increased motivation and a sense of fulfillment in their own work.


#6: The importance of ease – employees love Know Your Company because it’s easy.

The survey also showed that ease of use is one of the core reasons Know Your Company works so well. We didn’t ask any specific questions about Know Your Company’s ease of use or user interface. But, when asking employees, “What’s the one thing you love about Know Your Company?”, there were many comments in this vein…


Overall, the survey we conducted ended up being formative for us. It confirmed what’s really working with Know Your Company (e.g., we help CEOs feel more connected, have more engaged employees, and make better decisions). This way, we can better market our tool (we recently built a page on our marketing site to address this), and make smarter product development decisions that benefit our current customers (we recently built the Agreements feature, in part based off feedback we got back from survey data).

And, it now gives me a much better answer when someone asks me, “Does Know Your Company actually work?”


Big news! We’re now Know Your Team. Check out our new product that helps managers become better leaders, and get the full story behind our change.


P.S.: If you did indeed enjoy this piece, please feel free to share + give it ❤️ so others can find it too. Thanks 😊 (And you can always say hi at @clairejlew.)