We write code, not documents

Recently a student asked me:

Could you describe one instance where you had to use a diagramming tool (eg. Google Slides Drawings, Lucidcharts, Miro, Whimsical, Gliffy etc) to accomplish a task?

They also provided an example answer I could follow, which consisted of creating a chart to map a user flow, presenting it, getting feedback, adding it to another larger document, and creating Jira tickets.

I was a little surprised (though simultaneously not surprised) — is this how software development is still being taught?

Without being critical of academia, this seemed like a good opportunity to try to shatter by-the-book software development ideas for some future engineers by sharing a different way — our way.

Here was my answer (slightly edited for clarity and typos):

I hope this answer is helpful, but I actually don’t use a lot of diagramming tools, and I think it’s safe to say they’re not commonly used at Basecamp. We don’t write specs and stuff like that as they’re not “real” enough. We will do high level sketches and rough drawings (usually pen and paper or an iPad and a sketching app), but that’s typically it. So more often I’ll grab a pen and paper and sketch out a rough flow of what I need to do, or write out pseudo code of the steps that I need to take. 

Keep reading “We write code, not documents”

Demand Side Sales 101, a new book on sales by Bob Moesta.

Bob Moesta is a dear friend, mentor, and all around original thinker. He’s helped me see around corners, shine lights on things I didn’t know were there, and approach product development from unusual angles. Every time we talk, I come away inspired and full of optimism.

So when he asked me to help him with something, I jumped at the chance. In this case, it was writing a foreword for his new book Demand-Side Sales 101: Stop Selling and Help Your Customers Make Progress. Bob and I have talked sales for years, and I’m so pleased his ideas are finally collected in one place, in a form anyone can absorb. I highly recommend buying the book, reading the book, absorbing the book, and putting some new ideas in your head.

To get you started, here’s my foreword in its full form:

I learned sales at fifteen.

I was working at a small shoe store in Deerfield, Illinois, where I grew up. I loved sneakers. I was a sneakerhead before that phrase was coined.

I literally studied shoes. The designs, the designers, the brands, the technologies, the subtle improvements in this year’s model over last year’s.

I knew it all, but there was one thing I didn’t know: nothing I knew mattered. Sure it mattered to me, but my job was to sell shoes. I wasn’t selling shoes to sneakerfreaks like me, I was selling shoes to everyday customers. Shoes weren’t the center of their universe.

And I wasn’t alone. The companies that made the shoes didn’t have a clue how to sell shoes either.

Companies would send in reps to teach the salespeople all about the new models. They’d rattle off technical advancements. They’d talk about new breakthroughs in ethylene-vinyl acetate (EVA) which made the shoes more comfortable.

They’d talk about flex grooves and heel counters and Texon boards. Insoles, outsoles, midsoles.

And I’d be pumped. Now I knew everything I needed to know to sell the hell out of these things.

But when customers came in, and I demonstrated my mastery of the subject, they’d leave without buying anything. I could show off, but I couldn’t sell.

It wasn’t until my manager encouraged me to shut up, watch, and listen. Give people space, observe what they’re interested in, keep an eye on their behavior, and be genuinely curious about what they wanted for themselves, not what I wanted for them. Essentially, stop selling and start listening.

I noticed that when people browsed shoes on a wall, they’d pick a few up and bounce them around in their hand to get a sense of the heft and feel. Shoes go on your feet, but people picked the shoe with their hands. If it didn’t feel good in the hand, it never made it to their foot.

I noticed that if someone liked a shoe, they put it on the ground next to their foot. They didn’t want to try it on yet, they simply wanted to see what it looked like from above. Companies spend all this time making the side of the shoe look great, but from the wearer’s perspective, it’s the top of the shoe against their pants (or socks or legs) that seem to have an outsized influence on the buying decision.

I noticed that when people finally got around to trying on a shoe, they’d lightly jump up and down on it, or move side-to- side, simulating some sort of pseudo physical activity. They were trying to see if the shoe “felt right.” They didn’t care what the cushioning technology was, only that it was comfortable. It wasn’t about if it “fit right,” it was about if it “rubbed wrong” or “hurt” or felt “too hard.”

And hardly anyone picked a shoe for what it was intended for. Runners picked running shoes, sure, but lots of people picked running shoes to wear all day. They have the most cushion, they’re generally the most comfortable. And lots of people picked shoes purely based on color. “I like green” was enough to turn someone away from a blue shoe that fit them better.

Turns out, people had different reasons for picking shoes. Different reasons than my reasons, and far different reasons than the brand’s reasons. Hardly anyone cared about this foam vs. that foam, or this kind of rubber vs. that kind. They didn’t care about the precise weight, or that this brand shaved 0.5oz off the model this year compared to last. They didn’t care what the color was called, only that they liked it (or didn’t). The tech- nical qualities weren’t important – in fact, they were irrelevant.

I was selling all wrong.

And that’s really what this book is about. The revelation that sales isn’t about selling what you want to sell, or even what you, as a salesperson, would want to buy. Selling isn’t about you. Great sales requires a complete devotion to being curious about other people. Their reasons, not your reasons. And it’s surely not about your commission, it’s about their progress.

Fast forward twenty-five years.

Today I don’t sell shoes, I sell software. Or do I?

It’s true that I run a software company that makes project management software called Basecamp. And so, you’d think we sell software. I sure did! But once you meet Bob Moesta and Greg Engle, you realize you probably don’t sell what you think you sell. And your customers probably don’t think of you the way you think of yourself. And you almost certainly don’t know who your competition really is.

Over the years, Bob’s become a mentor to me. He’s taught us to see with new eyes and hear with new ears. To go deeper. To not just take surface answers as truth. But to dig for the how and the why—the causation. To understand what really moves someone to want to make a move. To understand the events that drive the purchase process, and to listen intently to the language customers use when they describe their struggles. To detect their energy and feel its influence on their decisions.

Everyone’s struggling with something, and that’s where the opportunity lies to help people make progress. Sure, people have projects, and software can help people manage those projects, but people don’t have a “project management problem.” That’s too broad. Bob taught us to dig until we hit a seam of true understanding. Project management is a label, it’s not a struggle.

People struggle to know where a project stands. People struggle to maintain accountability across teams. People struggle to know who’s working on what, and when those things will be done. People struggle with presenting a professional appearance with clients. People struggle to keep everything organized in one place so people know where things are. People struggle to communicate clearly so they don’t have to repeat themselves. People struggle to cover their ass and document decisions, so they aren’t held liable if a client says something wasn’t delivered as promised. That’s the deep down stuff, the real struggles.

Bob taught us how to think differently about how we talk, market, and listen. And Basecamp is significantly better off for it. We’ve not only changed how we present Basecamp, but we’ve changed how we build Basecamp. We approach design and development differently now that we know how to dig. It’s amazing how things can change once you see the world through a new lens.

Sales is everything. It’s survival. From selling a product, to selling a potential hire on the opportunity to join your company, to selling an idea internally, to selling your partner on this restaurant vs. that one, sales touches everything. If you want to be good at everything else, you better get good at this. Bob and Greg will show you how.

Options, Not Roadmaps

Since Shape Up came out, many people asked some version of this question:

I understand you make bets six weeks at a time. But how do you plan in the longer term? Don’t you have some kind of a roadmap?

The short answer is: no. We don’t have roadmaps. We think about what to do at the timescale larger than single bets, but we do it in a different way.

Why not a roadmap?

No matter how you try to hedge it, a roadmap communicates a plan—a series of commitments—to other people.

We of course have lots of ideas about what we’re going to do next. But we don’t want to make any commitments, internally or externally. This is very important.

What’s wrong with committing to future work?

First, there’s the uncertainty. We don’t have a crystal ball. Say we have features A, B, and C we want to build. We don’t know if A is going to work out as planned, and what that would mean for B. We don’t know if we’ll have a eureka in the bathtub one day, and new idea X will suddenly feel much more important than B or C. Or we might start building out B, only to realize that it’s not what we want or it’s harder than we thought and want to bail on it.

In short, we don’t know enough to make good on any promises.

Second, there are expectations. Leadership might be ok with changing course, but what about everyone who saw the roadmap and was eagerly awaiting feature C? What about the conversations with customers where someone in the company assured them to just hold tight because C is coming? Despite our best intentions, if we say we’re going to do something, it’s going to be really hard to back out of that, both internally and externally.

Third, there’s the guilt. Yeah, guilt. Have you ever looked at a long list of things you said were you going to do but haven’t gotten around to yet? How does that list make you feel? The realities of life and uncertainty show us that 100% of the things on the roadmap are not going to happen on time the way we imagine. And meanwhile, the world is not going to stop and wait for us to finish the old list. New ideas are constantly coming up. New requests and new problems constantly arise. If we hold ourselves completely to the roadmap, we’ll have to say no to new important things we actually want to do. And if we interrupt the roadmap to do those new important things, we’ll have to push back other things we promised. And that won’t feel good.

Our solution was to stop making commitments and start collecting options.

A portfolio of options

An option is something you can do but don’t have to do. All our product ideas are exactly that: options we may exercise in some future cycle—or never.

Without a roadmap, without a stated plan, we can completely change course without paying a penalty. We don’t set any expectations internally or externally that these things are actually going to happen.

That means no explicit promises and no implicit promises either. A list on the wall or in some official strategy document is an implicit promise: “this is what we’re doing next.“ There is no official list of what we’re doing next anywhere in the company.

When Jason (CEO) and David (CTO) decided the company would spend X cycles building out HEY, they didn’t have a roadmap. They had what they thought was a good set of options. There were enough good ideas for how to flesh out the app that they felt confident saying “we’ll dedicate X cycles to this.” They decided on which actual ideas to build one cycle at a time.

The overwhelming majority of our good ideas have never been built and never will be. There are things we have badly wanted to build for years that still don’t exist. Why? Something always came up. And that’s ok!

Because we aren’t committing to a roadmap, we aren’t setting expectations. And because we don’t set expectations, we don’t feel guilty when that great idea never gets any build time because we decided something else was more important.

Inside a CODE RED: Network Edition

I wanted to follow up to Jeremy’s post about our recent outages with a deeper, more personal look behind the scenes. We call our major incident response efforts “CODE REDs” to signify that it is an all-hands-on-deck event and this definitely qualified. I want to go beyond the summary and help you see how an event like this unfolds over time. This post is meant for both people who want a deeper, technical understanding of the outage, as well as some insight into the human side of incident management at Basecamp.

Keep reading “Inside a CODE RED: Network Edition”

Three Basecamp outages. One week. What happened?

Basecamp has suffered through three serious outages in the last week, on Friday, August 28th, on Tuesday, September 1, and again today. It’s embarrassing, and we’re deeply sorry.

This is more than a blip or two. Basecamp has been down during the middle of your day. We know these outages have really caused issues for you and your work. We’ve put you in the position of explaining Basecamp’s reliability to your customers and clients, too.

We’ve been leaning on your goodwill and we’re all out of it.

Here’s what has happened, what we’re doing to recover from these outages, and our plan to get Basecamp reliability back on track.

What happened

Friday, August 28

  • What you saw: Basecamp 3 Campfire chat rooms and Pings stopped loading. You couldn’t chat with each other or your teams for 40 minutes, from to 12:15pm to 12:55pm Central Time (17:1517:55 UTC). Incident timeline.
  • What we saw: We have two independent, redundant network links that connect our two redundant datacenters. The fiber optic line carrying one of the network links was cut in a construction incident. No problem, right? We have a redundant link! Not today. Due to a surprise interdependency between our network providers, we lost the redundant link as well, resulting in a brief disconnect between our datacenters. This led to a failure in our cross-datacenter Redis replication when we exceeded the maximum replication buffer size, triggering a catastrophic replication resync loop that overloaded the primary Redis server, causing very slow responses. This took Basecamp 3 Campfire chats and Pings out of commission.

Tuesday, September 1

  • What you saw: You couldn’t load Basecamp at all for 17 minutes, from 9:51am to 10:08am Central Time (14:5115:08 UTC). Nothing seemed to work. When Basecamp came back online, everything seemed back to normal. Incident timeline.
  • What we saw: Same deal, with a new twist. Our network links went offline, taking down Basecamp 3 Campfire chats and Pings again. While recovering from this, one of our load balancers (a hardware device that directs Internet traffic to Basecamp servers) crashed. A standby load balancer picked up operations immediately, but that triggered a third issue: our network routers failed to automatically synchronize with the new load balancer. That required manual intervention, extending the outage.

Wednesday, September 2

  • What you saw: You couldn’t load Basecamp for 15 minutes, from 10:50am to 11:05am Central Time (15:5016:05 UTC). When Basecamp came back online, chat messages felt slow and sluggish for hours afterward. Incident timeline.
  • What we saw: Earlier in the morning, the primary load balancer in our Virginia datacenter crashed again. Failover to its secondary load balancer proceeded as expected. Later that morning, the secondary load balancer also crashed and failed back to the former primary. This led to the same desynchronization issue from yesterday, which again required manual intervention to fix.

All told, we’ve tickled three obscure, tricky issues in a 5-day span that led to overlapping, interrelated failure modes. These woes are what we plan for. We detect and avert these sorts of technical issues daily, so this was a stark wake-up call: why not today? We’re working to learn why.

What we’re doing to recover from these outages

We’re working multiple options in parallel to recover and manage any contingencies in case our recovery plans fall through.

  1. We’re getting to the bottom of the load balancer crash with our vendor. We have a preliminary assessment and bugfix.
  2. We’re replacing our hardware load balancers. We’ve been pushing them hard. Traffic overload is a driving factor in one outage.
  3. We’re rerouting our redundant cross-datacenter network paths to ensure proper circuit diversity, eliminating the surprise interdependency between our network providers.
  4. As a contingency, we’re evaluating moving from hardware to software load balancers to decrease provisioning time. When a hardware device has an issue, we’re days out from a replacement. New software can be deployed in minutes.
  5. As a contingency, we’re evaluating decentralizing our load balancer architecture to limit the impact of any one failure.

What we’re doing to get our reliability back on track

We engineer our systems with multiple levels of redundancy & resilience precisely to avoid disasters like this one, including practicing our response to catastrophic failures within our live systems.

We didn’t catch these specific incidents. We don’t expect to catch them all! But what catches us by surprise are cascading failures that expose unexpected fragility and difficult paths to recovery. These, we can prepare for.

We’ll be assessing our systems for resilience, fragility, and risk, and we’ll review our assessment process itself. We’ll share what we learn and the steps we take with you.

We’re sorry. We’re making it right.

We’re really sorry for the repeated disruption this week. One thing after another. There’s nothing like trying to get your own work done and your computer glitching out you or just not cooperating. This one’s on us. We’ll make it right.

We really appreciate all your understanding and patience you’ve shown us. We’ll do our best to earn back the credibility and goodwill you’ve extended to us as we get Basecamp back to rock-solid reliability. Expect Basecamp to be up 24/7.

As always, you can follow along with live updates about Basecamp status here and follow the play-by-play on Twitter, and get in touch with our support team anytime.

How Basecamp Became a 100% Remote Company

Moving is never fun. It’s bad enough when it’s your stuff, but ten years of stuff at an office you only spent two years in can be daunting! I’m Navid, and part of my job at Basecamp the last two years has been taking care of our office in Chicago. As folks outside of Basecamp learned of our impending office closure, I began to get some questions. The most common being “what did you do with the stuff? What about mail and important documents?” Of course we had to work out some logistical puzzles to keep things running smoothly. Here’s how we used Basecamp and a new service to bid adieu to our office, to make my job remote, and to become a 100% remote company.

We didn’t close down our office because of COVID-19, though it certainly factored in the decision. Basecamp has always been remote. Remote is Basecamp. We wrote the book on it, literally. Our lease was due to expire, and it just didn’t seem worth it to keep it going at the new price. We’d outgrown it as a space for meetups, and it was always too big for the number of Basecampers that reside in Chicago. On a busy day there’d be six people working from the office.

On the other hand, having an office afforded us the standard ways of handling a lot of day-to-day business items. Mail, packages, meetings, storage. It was simple, easy, and the path that most of the world has taken. Losing the office and going 100% remote would take us further down a path less travelled.

Once we came to terms with leaving the office, I got to work on figuring out what to do to meet this goal. I won’t bore you with the minute details that are common to every move, you want to know what we are doing now. How we got to 100% remote. The biggest hurdles to jump were: 1. Primary business address (as most government agencies require a physical address), 2. How to handle the mail/packages, and 3. How to manage key document storage. 

I looked at a few options for our business address and for mail/packages. When the pandemic started, we re-routed our mail to a PO Box near my home. This eliminated the need for me to take public transit or a ride-share to check the mail. The PO Box would’ve been a great long-term solution if it weren’t for two things. 1) We need a new business address, and 2) it still ties me to Chicago.

I also looked at a UPS Store Mailbox. UPS is a great service! You can use it as your business address and they receive your mail and packages, then forward it at your request to anywhere you want. The drag on this is that all the mail will be bundled and shipped, creating further delays in getting the items. So if there is any urgency, you’ll need to get to the mailbox yourself.

In the midst of all of this, someone from Earth Class Mail (ECM) reached out to David via Twitter. ECM, like UPS, offers a business address and they receive your mail and packages. The main, and biggest, difference is that they scan all of your mail for you to review online. If you need any originals, they ship it to you. They also deposit checks for you via overnight shipping to your bank.

Of course, I opted for ECM in the end. They tick all the boxes to make Basecamp 100% remote, and they meet needs we hadn’t considered, like the check deposits. In the first few weeks, I have only tested the mail scanning service, which is working great. I’m looking forward to seeing how mail/package forwarding and check deposits go.

Another question I’ve answered recently is how we handle document retention. I’m definitely not holding onto these items in my home. We use Basecamp! Not long after I started here I began saving digital copies of everything important to Basecamp. I save each document in Basecamp with a name, the amount, and any relevant notes. Keeping only digital copies of invoices, checks, and tax paperwork saves on office space, a luxury we no longer have, and more importantly the documents are secure, searchable, and accessible to anyone who needs them.

When I’m not sure, I check in with our accountants about anything we should keep hard copies of. If there is any chance we would need an original paper copy, we keep it. At the moment we don’t have a permanent solution for these instances (honestly, it isn’t much), so they are locked up in storage. The goal will be to eventually not need a storage space.

That covers how we are remote now! Did I miss anything? Feel free to leave a comment.

We’re hiring Rails programmers

We have two rare openings on our Core Product team for Rails programmers. We’ll be accepting applications for the next two weeks, aiming for a flexible start date in October.

We strongly encourage candidates of all different backgrounds and identities to apply. This is an opportunity for us to bring in a different perspective and we’re eager to further diversify our company. Basecamp is committed to building an inclusive, supportive place for you to do the best work of your career. We aren’t looking for ideological clones, but for people who share our beliefs about writing software well.

Keep reading “We’re hiring Rails programmers”

Remote work is a platform

Back in the mid-90s, just as Netscape Navigator was giving us our first look at what the visual internet could be, web design came in two flavors.

There was the ultra basic stuff. Text on a page, maybe a masthead graphic of some sort. Nothing sophisticated. It often looked like traditional letterhead, or a printed newsletter, but now on the screen. Interactions were few, if any, but perhaps a couple links tied a nascent site together.

And there was the other extreme. Highly stylized, lots of textures, 3D-style buttons, page curls, aggressive shadows, monolithic graphics cut up with image maps to allow you to click on different parts of a single graphic, etc. This style was aped from interactive CD/DVD interfaces that came before it.

Both of these styles — the masthead with text, and the heavily graphical — were ports. Not adaptations, but ports. Designs ported from one medium to another. No one knew what to make of the web at that time, so we pulled over things we were familiar with and sunk them in place. At that time, Web design wasn’t web design – it was print design, multimedia/interactive design, and graphic design. It took years for native web design to come into its own.

The web became great when designers started designing for the web, not bringing other designs to the web.

Porting things between platforms is common, especially when the new thing is truly brand new (or trying to gain traction). As the Mac gained steam in the late 80s and early 90s, and Windows 3 came out in 1990, a large numbers of Windows/PC developers began to port their software to the Mac. They didn’t write Mac software, they ported Windows software. And you could tell – it was pretty shit. It was nice to have at a time when the Mac wasn’t widely developed, but, it was clearly ported.

When something’s ported, it’s obvious.

Stuff that’s ported lacks the native sensibilities of the receiving platform. It doesn’t celebrate the advantages, it only meets the lowest possible bar. Everyone knows it. Sometimes we’re simply glad to have it because it’s either that or nothing, but there’s rarely a ringing endorsement of something that’s so obviously moved from A to B without consideration for what makes B, B.

What we’re seeing today is history repeat itself. This time we’re not talking about porting software or technology, we’re talking about porting a way to work.

In-person office work is a platform. It has its own advantages and disadvantages. Some things are easier in person (meetings, if you’re into those), and some things are harder (getting a few hours to yourself so you can focus, if you’re into that).

Remote work is another platform. It has its own unique flavor, advantages, and disadvantages. Its own efficiencies, its own quirks, its own interface. Upsides, downsides, insides, and outsides. It’s as different from in-office work as the Mac is from Windows. Yes, they’re both operating systems, and methods of computing, but they’re miles apart where it matters. The same is true for the difference between in-office work and remote work. Yup, it’s all still the same work, but it’s a different way to work.

In-office and remote work are different platforms of work. And right now, what we’re seeing a lot of companies attempt to port local work methods to working remotely. Normally have four meetings a day in person? Then let’s have those same four meetings, with those same participants, over Zoom instead. It’s a way, but it’s the wrong way.

Simulating in-person office work remotely does both approaches a disservice.

This is often what happens when change is abrupt. We bring what we know from one to the other. We apply what we’re familiar with to the unfamiliar. But, in time, we recognize that doesn’t work.

The enlightened companies coming out of this pandemic will be the ones that figured out the right way to work remotely. They’ll have stopped trying to make remote look like local. They’ll have discovered that remote work means more autonomy, more trust, more uninterrupted stretches of time, smaller teams, more independent, concurrent work (and less dependent, sequenced work).

They won’t be the ones that just have their waste-of-time meetings online, they’ll be the ones that lay waste to the meetings. They won’t be the ones that depend on checking in on people constantly throughout the day, they’ll be the ones that give their employees time and space to do their best work. They won’t be the ones that can’t wait to pull everyone back to the office, they’ll be the ones that spot the advantages of optionality, and recognize a wonderful resilience in being able to work from anywhere.

And they’ll be the ones that finally realize that there’s nothing magical about the office. It’s just a space where work can happen, but not where it must happen. Anytime a myth is busted is a good time.

Work remotely, don’t port the office.

Spy pixels are evolving like malware, so HEY’s adapting

We knew that spy-pixel pushers might go down the rabbit hole of escalation once we gave HEY users the power to defend themselves. Just like virus and malware makers are constantly trying to defeat anti-virus and other security protections. But I guess we didn’t realize just how quickly it would happen!

Enter GMass, a plugin for Gmail that adds spy-pixel tracking, amongst a grab bag of other stuff. They hadn’t been on our original list of 50+ services we name’n’shame, but thanks to a new blog post where they brag about defeating protections that recipients might take to defend themselves, they came onto our radar.

This lead to an in-depth investigation into how their latest techniques work, and we spent the whole day coming up with a new process of detecting GMass’ spy pixels. It just shipped! And now HEY will name’n’shame GMass, just like we do the other fifty-odd pushers of this kind of surveillance.

Of course, like those virus and malware makers, GMass may try to defeat our protections again. And we’ll then have to adapt once more, and so we will. Internet security is a constantly moving target. But we can hope that Google will soon stop being a conduit for this kind of privacy abuse on the Gmail platform. Just like they don’t tolerate being used for spamming or phishing.

In the mean time, we’ll continue to do the work both on a general level to protect against all forms of privacy attacks against HEY users, but also specifically to identity bad actors, and to call out the users who employ their software for spying.

Have a surveillance-free Friday!