How to waste half a day by not reading RFC 1034

HEY uses a branch deploy system that I’ve written about here on SvN and talked about frequently on Twitter. Plenty of other companies have implemented their own version of branch deploys (typically under a different name), but this was my own implementation, so I’m proud of it. First, a primer on how it works:

  • Developer makes a code change in a git branch and pushes it to GitHub.
  • An automated build pipeline run is kicked off by a GitHub webhook. It builds some Docker images and kicks off another build that handles the deploy itself.
  • That deploy build, well, it deploys — to AWS EKS, Amazon’s managed Kubernetes offering, via a Helm chart that contains all of the YAML specifications for deployments, services, ingresses, etc.
  • alb-ingress-controller (now aws-load-balancer-controller) creates an ALB for the branch.
  • external-dns creates a DNS record pointing to the new ALB.
  • Dev can access their branch from their browser using a special branch-specific URL

    This process takes 5-10 minutes for a brand new branch from push to being accessible (typically).

Keep reading “How to waste half a day by not reading RFC 1034”

Introducing the Basecamp security bug bounty

We’ve run a private security bug bounty program since 2014. Invited testers reported numerous security vulnerabilities to us, many of them critical. We investigated and fixed the vulnerabilities they reported and thanked them with cash rewards. Before 2014, and concurrently with the private bounty program, we ran a public “Hall of Fame” program where we accepted vulnerability reports via email and thanked reporters with credit on our website.

Since the day we launched it, we’ve aimed to take the security bug bounty program public—to allow anyone, not just a few invited hackers, to report vulnerabilities to us for a cash reward. We want to find and fix as many vulnerabilities in our products as possible, to protect our customers and the data they entrust to us. We also want to learn from and support the broader security community.

We’re happy to announce that we’re doing that today. The Basecamp security bug bounty program is now open to the public on HackerOne. Our security team is ready to take vulnerability reports for Basecamp 3 and HEY. Bounties range from $100 to $10,000. We pay more for more severe vulnerabilities, more creative exploits, and more insightful reports.

Here are some of the high-criticality reports we’ve fielded via the security bug bounty:

  • Jouko Pynnönen reported a stored cross-site scripting (XSS) vulnerability in HEY that lead to account takeover via email. We awarded $5,000 for this report.
  • Hazim Aslam reported HTTP desynchronization vulnerabilities in our on-premises applications that allowed an attacker to intercept customer requests. We awarded  $11,437 in total for these reports.
  • hudmi reported that the AppCache web API (since deprecated and removed from web browsers) could be used to capture direct upload requests in Basecamp 3. We awarded $1,000.
  • gammarex reported an ImageMagick misconfiguration that allowed remote code execution on Basecamp 3’s servers. We awarded $5,000.

Check out the full program policy on HackerOne. For information on what to expect when you report a vulnerability, see our security response policy. If you have any questions, don’t hesitate to reach out to security@basecamp.com.

Don’t take their word for it

A few weeks ago, we needed some hardware fast. After some back and forth with the vendor, they promised “expedited delivery”. That sounded like a good thing, but it meant nothing.

To us, expedited delivery meant overnight delivery. That’s what we had in our head. Our experiences elsewhere equated expedited as overnight, but expedited isn’t overnight – it just means faster, prioritized, enhanced, sooner. But it doesn’t mean overnight. Expedited is relative, not absolute. If standard shipping takes 7 days, expedited could mean 5.

Of course, as you’ve probably guessed, the stuff we thought would come overnight didn’t come overnight. A harsh call the next day to the vendor ultimately got us the hardware overnight the next day, but we lost a day in the exchange.

What we had in front of us was an illusion of agreement. We thought a word meant one thing, the other side thought it meant something else, and neither of us assumed mismatched alignment on the definition. Of course we agreed on what expedited meant, because it was so obvious to each of us. Obviously wrong.

This happens all the time in product development. Someone explains something, you think it means one thing, the other person thinks it means something else, but the disagreement isn’t caught – or even suspected – so all goes as planned. Until it goes wrong and both sides look at each other unable to understand how the other side didn’t get it. “But I thought you…” “Oh? I thought you…” “No I meant this…” “Oh, I thought you meant that…”. That’s an illusion of agreement. We covered the topic in the “There’s Nothing Functional about a Functional Spec” essay in Getting Real.

We knew better, but we didn’t do better.

Next time you’re discussing something with someone — inside or outside your organization — and you find the outcome contingent upon a relative term or phrase, be sure to clarify it.

If they say expedited, you say “we need it tomorrow morning, October 3. Will we have it tomorrow, October 3?”. That forces them into a clear answer too. “Yes, you’ll have it tomorrow, October 3” or “No, we can’t do that” or whatever, but at least you’re funneling towards clarity. If they say “Yes, we’ll expedite it” you repeat “Will we have it tomorrow, October 3?” Set them up to give you a definitive, unambiguous answer.

And remember, while we now know that “expedited” is relative, “overnight” can be too depending on where someone’s shipping something from, what time zone they’re in, their own internal cutoffs for overnight shipping, etc. Get concrete, get it in writing, and get complete clarity. Slam the door shut on interpretation. Get definitive.

Basecamp is hiring a Product Designer

Basecamp’s Core Product team is hiring a Senior Designer! We’ll be accepting applications for the next two weeks, with a flexible start date in December.

It’s been over 4 years since we hired for this role, so this is an exciting and rare opportunity to join our team. You’ll start by working on HEY, our brand new email service. In 2021 we’ll also launch a major new update for Basecamp, so there’s lots of impactful design work ahead where you can make your mark.

About the Job

At most companies, product design is split into many different roles: UX, UI, front-end development, and so on. At Basecamp, it’s all one role. We believe the best designs come from someone who can see it all through, from ideas to visuals to the finished product.

That means all of our designers are generalists—excellent visual stylists, front-end developers, project managers, writers, and more. Our projects are design-led, so you’ll have a lot of influence and impact right away.

As a Core Product designer, you’ll be working on the web interfaces for Basecamp and HEY. You might be improving an existing feature, designing something totally brand new, or rethinking how we do something.

Along with having great visual taste and sensibilities, you must be able to write your own HTML, CSS (BEM), and some JavaScript. You’ll work directly inside our Ruby on Rails apps to make your designs come to life. (But note: we don’t expect you to do all the implementation on your own. You’ll be paired with a programmer on most projects, and you’ll consult with the rest of the Core team and our iOS/Android teams.)

As a manager of one, you’ll drive shaped projects, big and small, over six-week cycles. You’ll set direction, take ownership, make calls, and see things through without a lot of oversight. You’ll be able to communicate clearly with your colleagues, work across teams, and lend a helping hand when needed.

You love to write, too. You understand that copywriting is design. The words matter as much as the pixels. Great visuals with weak words are poor designs. You should care about how things are phrased as much as you care about how they look.

Here are some things we’ve worked on recently to give you a sense of what you’ll be doing day-to-day.

  • Built an interface for our email exporting tech, so people can easily export their HEY email into a downloadable MBOX file.
  • Improved color contrast on buttons in dark mode, after hearing bug reports from customers and our QA team.
  • Created novel multi-step onboarding and training flows to help new customers learn how HEY works.
  • Researched the technical requirements for hosting a custom domain name, and then implemented a step-based workflow to help people migrate their company’s email system over to HEY.
  • Extracted reusable CSS components after noticing repeating patterns in our codebase.
  • Prototyped many different variations on HEY’s Screener UI, to make it as clear and simple as possible.

Basecamp is a fully remote company, and this is a remote job. We’re hiring from anywhere with at least 4 hours of overlap with the US-Central Time zone during a normal work day. This could be a 11:00-19:00 schedule from Europe, but we’re not hiring from locations that require a graveyard shift to make the overlap happen.

About Us

Basecamp has been proudly building opinionated, respectful software for nearly 20 years. As an intentionally small, privately-held company, we don’t answer to anyone but ourselves and our customers. We design our products to solve real-life problems, and we strongly defend our customers’ attention and privacy along the way.

We use the concepts we defined in Getting Real and Shape Up to stay focused, keep projects on track, and ship good work on time.

Benefits & Compensation

Basecamp pays in the top 10% of the industry based on San Francisco rates. Same position, same pay, no matter where you live. The salary for this position is $149,442 (Senior Designer).

Benefits at Basecamp are all about helping you lead a healthy life outside of work. We believe quality time to focus on work starts with quality time to think, exercise, prepare a meal, be with family & friends, and of course, time to yourself.

We offer fully paid parental leave. We work 4-day weeks in the summer (Northern Hemisphere), and offer a month-long sabbatical every 3 years. We subsidize your home office, wellness and fitness interests, and continuing education. We also offer an annual charitable contribution match. All on top of top-tier health insurance and a retirement plan with company match. See our full list.

Applicants from outside of the US will be offered a contractor role on comparable terms and equal pay with our domestic employees.

How to Apply

Please submit an application that speaks directly to this position. Tell us about yourself, about what you can bring to Basecamp, and about Basecamp’s role in your future. Tell us about what you’ve done and what excites you. You might be inclined to design something especially for us—that’s fine. Just make sure the content of your application is as impressive as its presentation. We’ll also happily accept a traditional, well-constructed cover letter full of personal touches and that shows us how much you want this job.

We’re accepting applications until Friday, October 16, 2020, at 5PM US-Central time. There’s no benefit to filing early, so take your time.

We strongly encourage candidates of all different backgrounds and identities to apply. We believe that our design work is stronger with a variety of perspectives, and we’re eager to further diversify our company. If you have a background that you feel would make an impact on Basecamp, please consider applying. We’re committed to building an inclusive, supportive place for you to do the best work of your career.

What happens next?

We expect to take a few weeks to review all applications. You’ll hear from us by November 6th, about advancement to the interview stage. Expect 2-3 interviews, all one hour, all remote, with your future colleagues, on your schedule. We’ll talk through your background, your approach to design, and dive into your professional knowledge. No gotchas or surprises.

After the interviews, the final candidates will be given an at-home design challenge. The exercise is representative of the kind of work we do, and it helps us understand how you’d approach new problems from scratch. We invite no more than 5 candidates to this stage, and those candidates should expect to spend about 3 work days completing the project. You’ll be compensated for your time.

We aim to make an offer in November with a flexible start date in December.

Please note that we’re unable to offer individual feedback during the screening process. We usually see 1,000+ applications for open positions, and our hiring team simply doesn’t have the bandwidth to offer personalized feedback before the first interview round.

This is a demanding application process and significant, long-term career move to consider. We appreciate you giving us that consideration, and we promise to give you our full attention in return. We look forward to hearing from you!

APPLY HERE

We write code, not documents

Recently a student asked me:

Could you describe one instance where you had to use a diagramming tool (eg. Google Slides Drawings, Lucidcharts, Miro, Whimsical, Gliffy etc) to accomplish a task?

They also provided an example answer I could follow, which consisted of creating a chart to map a user flow, presenting it, getting feedback, adding it to another larger document, and creating Jira tickets.

I was a little surprised (though simultaneously not surprised) — is this how software development is still being taught?

Without being critical of academia, this seemed like a good opportunity to try to shatter by-the-book software development ideas for some future engineers by sharing a different way — our way.

Here was my answer (slightly edited for clarity and typos):

I hope this answer is helpful, but I actually don’t use a lot of diagramming tools, and I think it’s safe to say they’re not commonly used at Basecamp. We don’t write specs and stuff like that as they’re not “real” enough. We will do high level sketches and rough drawings (usually pen and paper or an iPad and a sketching app), but that’s typically it. So more often I’ll grab a pen and paper and sketch out a rough flow of what I need to do, or write out pseudo code of the steps that I need to take. 

Keep reading “We write code, not documents”

Demand Side Sales 101, a new book on sales by Bob Moesta.

Bob Moesta is a dear friend, mentor, and all around original thinker. He’s helped me see around corners, shine lights on things I didn’t know were there, and approach product development from unusual angles. Every time we talk, I come away inspired and full of optimism.

So when he asked me to help him with something, I jumped at the chance. In this case, it was writing a foreword for his new book Demand-Side Sales 101: Stop Selling and Help Your Customers Make Progress. Bob and I have talked sales for years, and I’m so pleased his ideas are finally collected in one place, in a form anyone can absorb. I highly recommend buying the book, reading the book, absorbing the book, and putting some new ideas in your head.

To get you started, here’s my foreword in its full form:

I learned sales at fifteen.

I was working at a small shoe store in Deerfield, Illinois, where I grew up. I loved sneakers. I was a sneakerhead before that phrase was coined.

I literally studied shoes. The designs, the designers, the brands, the technologies, the subtle improvements in this year’s model over last year’s.

I knew it all, but there was one thing I didn’t know: nothing I knew mattered. Sure it mattered to me, but my job was to sell shoes. I wasn’t selling shoes to sneakerfreaks like me, I was selling shoes to everyday customers. Shoes weren’t the center of their universe.

And I wasn’t alone. The companies that made the shoes didn’t have a clue how to sell shoes either.

Companies would send in reps to teach the salespeople all about the new models. They’d rattle off technical advancements. They’d talk about new breakthroughs in ethylene-vinyl acetate (EVA) which made the shoes more comfortable.

They’d talk about flex grooves and heel counters and Texon boards. Insoles, outsoles, midsoles.

And I’d be pumped. Now I knew everything I needed to know to sell the hell out of these things.

But when customers came in, and I demonstrated my mastery of the subject, they’d leave without buying anything. I could show off, but I couldn’t sell.

It wasn’t until my manager encouraged me to shut up, watch, and listen. Give people space, observe what they’re interested in, keep an eye on their behavior, and be genuinely curious about what they wanted for themselves, not what I wanted for them. Essentially, stop selling and start listening.

I noticed that when people browsed shoes on a wall, they’d pick a few up and bounce them around in their hand to get a sense of the heft and feel. Shoes go on your feet, but people picked the shoe with their hands. If it didn’t feel good in the hand, it never made it to their foot.

I noticed that if someone liked a shoe, they put it on the ground next to their foot. They didn’t want to try it on yet, they simply wanted to see what it looked like from above. Companies spend all this time making the side of the shoe look great, but from the wearer’s perspective, it’s the top of the shoe against their pants (or socks or legs) that seem to have an outsized influence on the buying decision.

I noticed that when people finally got around to trying on a shoe, they’d lightly jump up and down on it, or move side-to- side, simulating some sort of pseudo physical activity. They were trying to see if the shoe “felt right.” They didn’t care what the cushioning technology was, only that it was comfortable. It wasn’t about if it “fit right,” it was about if it “rubbed wrong” or “hurt” or felt “too hard.”

And hardly anyone picked a shoe for what it was intended for. Runners picked running shoes, sure, but lots of people picked running shoes to wear all day. They have the most cushion, they’re generally the most comfortable. And lots of people picked shoes purely based on color. “I like green” was enough to turn someone away from a blue shoe that fit them better.

Turns out, people had different reasons for picking shoes. Different reasons than my reasons, and far different reasons than the brand’s reasons. Hardly anyone cared about this foam vs. that foam, or this kind of rubber vs. that kind. They didn’t care about the precise weight, or that this brand shaved 0.5oz off the model this year compared to last. They didn’t care what the color was called, only that they liked it (or didn’t). The tech- nical qualities weren’t important – in fact, they were irrelevant.

I was selling all wrong.

And that’s really what this book is about. The revelation that sales isn’t about selling what you want to sell, or even what you, as a salesperson, would want to buy. Selling isn’t about you. Great sales requires a complete devotion to being curious about other people. Their reasons, not your reasons. And it’s surely not about your commission, it’s about their progress.

Fast forward twenty-five years.

Today I don’t sell shoes, I sell software. Or do I?

It’s true that I run a software company that makes project management software called Basecamp. And so, you’d think we sell software. I sure did! But once you meet Bob Moesta and Greg Engle, you realize you probably don’t sell what you think you sell. And your customers probably don’t think of you the way you think of yourself. And you almost certainly don’t know who your competition really is.

Over the years, Bob’s become a mentor to me. He’s taught us to see with new eyes and hear with new ears. To go deeper. To not just take surface answers as truth. But to dig for the how and the why—the causation. To understand what really moves someone to want to make a move. To understand the events that drive the purchase process, and to listen intently to the language customers use when they describe their struggles. To detect their energy and feel its influence on their decisions.

Everyone’s struggling with something, and that’s where the opportunity lies to help people make progress. Sure, people have projects, and software can help people manage those projects, but people don’t have a “project management problem.” That’s too broad. Bob taught us to dig until we hit a seam of true understanding. Project management is a label, it’s not a struggle.

People struggle to know where a project stands. People struggle to maintain accountability across teams. People struggle to know who’s working on what, and when those things will be done. People struggle with presenting a professional appearance with clients. People struggle to keep everything organized in one place so people know where things are. People struggle to communicate clearly so they don’t have to repeat themselves. People struggle to cover their ass and document decisions, so they aren’t held liable if a client says something wasn’t delivered as promised. That’s the deep down stuff, the real struggles.

Bob taught us how to think differently about how we talk, market, and listen. And Basecamp is significantly better off for it. We’ve not only changed how we present Basecamp, but we’ve changed how we build Basecamp. We approach design and development differently now that we know how to dig. It’s amazing how things can change once you see the world through a new lens.

Sales is everything. It’s survival. From selling a product, to selling a potential hire on the opportunity to join your company, to selling an idea internally, to selling your partner on this restaurant vs. that one, sales touches everything. If you want to be good at everything else, you better get good at this. Bob and Greg will show you how.

Options, Not Roadmaps

Since Shape Up came out, many people asked some version of this question:

I understand you make bets six weeks at a time. But how do you plan in the longer term? Don’t you have some kind of a roadmap?

The short answer is: no. We don’t have roadmaps. We think about what to do at the timescale larger than single bets, but we do it in a different way.

Why not a roadmap?

No matter how you try to hedge it, a roadmap communicates a plan—a series of commitments—to other people.

We of course have lots of ideas about what we’re going to do next. But we don’t want to make any commitments, internally or externally. This is very important.

What’s wrong with committing to future work?

First, there’s the uncertainty. We don’t have a crystal ball. Say we have features A, B, and C we want to build. We don’t know if A is going to work out as planned, and what that would mean for B. We don’t know if we’ll have a eureka in the bathtub one day, and new idea X will suddenly feel much more important than B or C. Or we might start building out B, only to realize that it’s not what we want or it’s harder than we thought and want to bail on it.

In short, we don’t know enough to make good on any promises.

Second, there are expectations. Leadership might be ok with changing course, but what about everyone who saw the roadmap and was eagerly awaiting feature C? What about the conversations with customers where someone in the company assured them to just hold tight because C is coming? Despite our best intentions, if we say we’re going to do something, it’s going to be really hard to back out of that, both internally and externally.

Third, there’s the guilt. Yeah, guilt. Have you ever looked at a long list of things you said were you going to do but haven’t gotten around to yet? How does that list make you feel? The realities of life and uncertainty show us that 100% of the things on the roadmap are not going to happen on time the way we imagine. And meanwhile, the world is not going to stop and wait for us to finish the old list. New ideas are constantly coming up. New requests and new problems constantly arise. If we hold ourselves completely to the roadmap, we’ll have to say no to new important things we actually want to do. And if we interrupt the roadmap to do those new important things, we’ll have to push back other things we promised. And that won’t feel good.

Our solution was to stop making commitments and start collecting options.

A portfolio of options

An option is something you can do but don’t have to do. All our product ideas are exactly that: options we may exercise in some future cycle—or never.

Without a roadmap, without a stated plan, we can completely change course without paying a penalty. We don’t set any expectations internally or externally that these things are actually going to happen.

That means no explicit promises and no implicit promises either. A list on the wall or in some official strategy document is an implicit promise: “this is what we’re doing next.“ There is no official list of what we’re doing next anywhere in the company.

When Jason (CEO) and David (CTO) decided the company would spend X cycles building out HEY, they didn’t have a roadmap. They had what they thought was a good set of options. There were enough good ideas for how to flesh out the app that they felt confident saying “we’ll dedicate X cycles to this.” They decided on which actual ideas to build one cycle at a time.

The overwhelming majority of our good ideas have never been built and never will be. There are things we have badly wanted to build for years that still don’t exist. Why? Something always came up. And that’s ok!

Because we aren’t committing to a roadmap, we aren’t setting expectations. And because we don’t set expectations, we don’t feel guilty when that great idea never gets any build time because we decided something else was more important.

Inside a CODE RED: Network Edition

I wanted to follow up to Jeremy’s post about our recent outages with a deeper, more personal look behind the scenes. We call our major incident response efforts “CODE REDs” to signify that it is an all-hands-on-deck event and this definitely qualified. I want to go beyond the summary and help you see how an event like this unfolds over time. This post is meant for both people who want a deeper, technical understanding of the outage, as well as some insight into the human side of incident management at Basecamp.

Keep reading “Inside a CODE RED: Network Edition”

Three Basecamp outages. One week. What happened?

Basecamp has suffered through three serious outages in the last week, on Friday, August 28th, on Tuesday, September 1, and again today. It’s embarrassing, and we’re deeply sorry.

This is more than a blip or two. Basecamp has been down during the middle of your day. We know these outages have really caused issues for you and your work. We’ve put you in the position of explaining Basecamp’s reliability to your customers and clients, too.

We’ve been leaning on your goodwill and we’re all out of it.

Here’s what has happened, what we’re doing to recover from these outages, and our plan to get Basecamp reliability back on track.

What happened

Friday, August 28

  • What you saw: Basecamp 3 Campfire chat rooms and Pings stopped loading. You couldn’t chat with each other or your teams for 40 minutes, from to 12:15pm to 12:55pm Central Time (17:1517:55 UTC). Incident timeline.
  • What we saw: We have two independent, redundant network links that connect our two redundant datacenters. The fiber optic line carrying one of the network links was cut in a construction incident. No problem, right? We have a redundant link! Not today. Due to a surprise interdependency between our network providers, we lost the redundant link as well, resulting in a brief disconnect between our datacenters. This led to a failure in our cross-datacenter Redis replication when we exceeded the maximum replication buffer size, triggering a catastrophic replication resync loop that overloaded the primary Redis server, causing very slow responses. This took Basecamp 3 Campfire chats and Pings out of commission.

Tuesday, September 1

  • What you saw: You couldn’t load Basecamp at all for 17 minutes, from 9:51am to 10:08am Central Time (14:5115:08 UTC). Nothing seemed to work. When Basecamp came back online, everything seemed back to normal. Incident timeline.
  • What we saw: Same deal, with a new twist. Our network links went offline, taking down Basecamp 3 Campfire chats and Pings again. While recovering from this, one of our load balancers (a hardware device that directs Internet traffic to Basecamp servers) crashed. A standby load balancer picked up operations immediately, but that triggered a third issue: our network routers failed to automatically synchronize with the new load balancer. That required manual intervention, extending the outage.

Wednesday, September 2

  • What you saw: You couldn’t load Basecamp for 15 minutes, from 10:50am to 11:05am Central Time (15:5016:05 UTC). When Basecamp came back online, chat messages felt slow and sluggish for hours afterward. Incident timeline.
  • What we saw: Earlier in the morning, the primary load balancer in our Virginia datacenter crashed again. Failover to its secondary load balancer proceeded as expected. Later that morning, the secondary load balancer also crashed and failed back to the former primary. This led to the same desynchronization issue from yesterday, which again required manual intervention to fix.

All told, we’ve tickled three obscure, tricky issues in a 5-day span that led to overlapping, interrelated failure modes. These woes are what we plan for. We detect and avert these sorts of technical issues daily, so this was a stark wake-up call: why not today? We’re working to learn why.

What we’re doing to recover from these outages

We’re working multiple options in parallel to recover and manage any contingencies in case our recovery plans fall through.

  1. We’re getting to the bottom of the load balancer crash with our vendor. We have a preliminary assessment and bugfix.
  2. We’re replacing our hardware load balancers. We’ve been pushing them hard. Traffic overload is a driving factor in one outage.
  3. We’re rerouting our redundant cross-datacenter network paths to ensure proper circuit diversity, eliminating the surprise interdependency between our network providers.
  4. As a contingency, we’re evaluating moving from hardware to software load balancers to decrease provisioning time. When a hardware device has an issue, we’re days out from a replacement. New software can be deployed in minutes.
  5. As a contingency, we’re evaluating decentralizing our load balancer architecture to limit the impact of any one failure.

What we’re doing to get our reliability back on track

We engineer our systems with multiple levels of redundancy & resilience precisely to avoid disasters like this one, including practicing our response to catastrophic failures within our live systems.

We didn’t catch these specific incidents. We don’t expect to catch them all! But what catches us by surprise are cascading failures that expose unexpected fragility and difficult paths to recovery. These, we can prepare for.

We’ll be assessing our systems for resilience, fragility, and risk, and we’ll review our assessment process itself. We’ll share what we learn and the steps we take with you.

We’re sorry. We’re making it right.

We’re really sorry for the repeated disruption this week. One thing after another. There’s nothing like trying to get your own work done and your computer glitching out you or just not cooperating. This one’s on us. We’ll make it right.

We really appreciate all your understanding and patience you’ve shown us. We’ll do our best to earn back the credibility and goodwill you’ve extended to us as we get Basecamp back to rock-solid reliability. Expect Basecamp to be up 24/7.

As always, you can follow along with live updates about Basecamp status here and follow the play-by-play on Twitter, and get in touch with our support team anytime.