Signal v. Noise

Since we started working on HEY, one of the things that I’ve been a big proponent of was keeping as much of the app-side compute infrastructure on spot instances as possible (front-end and async job processing; excluding the database, Redis, and Elasticsearch). Coming out of our first two weeks running the app with a real production traffic load, we’re sitting at ~90% of our compute running on spot instances.

Especially around the launch of a new product, you typically don’t know what traffic and load levels are going to look like, making purchases of reserved instances or savings plans a risky proposition. Spot instances give us the ability to get the compute we need at an even deeper discount than a 1-year RI or savings plan rate would, without the same commitment. Combine the price with seamless integration with auto-scaling groups and it makes them a no-brainer for most of our workloads.

The big catch with spot instances? AWS can take them back from you with a two-minute notice.

Keep reading “Running spot instances effectively with Amazon EKS”

Today, on Juneteenth, it is important to us to take a pause and publicly recognize organizations, activists, campaigns, and journalists that have been doing the work, fighting everyday for a more just tomorrow.

The following are contributions from individual Basecampers.

Keep reading “Black Lives Matter”

We recently deleted over three million accounts across all our apps. This was the answer to a question we asked ourselves last year: what should we do about  accounts that weren’t cancelled, but weren’t used either? Should we keep hold of their data forever? 

That felt wrong – we promise to delete data when you cancel your account. Keeping so much data around felt like we weren’t living up to that promise, and felt like a liability, so we decided to do something about it.

Keep reading “Celebrating 3 million accounts (fewer)”

It’s easy to say what a year, what a week.

But that’s a shortsighted, privileged point of view. I’m guilty of holding that occasional perspective. It’s moments like these that jolt me into recognizing the deeper reality.

What we’re seeing is the culmination of years – decades, generations, and centuries – of unjust treatment against black people, minorities, and other marginalized communities.

This country’s racist history is shameful, and so is its present.

Deep systemic racism + the militarization of police (both physically in terms of gear, and mentally in terms of mindset) is a powder keg. We’ve seen sparks before, now we’re seeing the explosion.

If you’re surprised, you’re not paying attention.

I don’t like the violence, but I get it. This is what happens when people are squeezed, compressed, and backed into a corner with no way out. For years, for generations. We’re all humans – if your lot in life was different you just might do the same.

I support peaceful protests, I support the fight against racism, against oppression, and against injustice – wherever it hides.

There’s exceptionally hard work ahead. I recognize this work has been happening for years, often ignored or unappreciated by many people, including me. How frustrating it must be to work so hard, and see such little progress, on something so elemental.

Change will require a massive, sustained effort by millions over many years. A change in perspective, mindset, and approach. And that work will certainly be met with future setbacks, which is why change requires optimism, too (which is in short supply in moments like these). I hope we can find it, and support those who need it.

I’ll be working to educate myself, and break my own patterns of ignorance. This sense of urgency is, embarrassingly, new to me, so I have a lot to learn – which organizations to support, what books to read, what history to absorb, and who to listen to. I’m starting on that today. If you’re like me, I hope you’ll do the same.

-Jason Fried, CEO, Basecamp

We’ve been teaching people how to do remote work well for the better part of two decades. We wrote a whole book about the topic in 2013, called REMOTE: Office Not Required. Basecamp has been a remote company since day one, and our software is sold as an all-in-one toolkit for remote work. Yeah, we’re big on remote work!

So now that COVID-19 has forced a lot of companies to move to remote work, it’s doubly important that we do our part to help those new to the practice settle in. We’ve been hosting a variety of online seminars, done podcasts, and been advocating for healthy ways to do remote right.

Unfortunately, the move to remote work has also turbo-charged interest in employee surveillance software. Drew Harwell’s harrowing report for The Washington Post should make anyone’s skin crawl, but it seems some managers are reading about these disgusting tools and thinking “oh, what a great idea, where can I buy?”.

And as fate would have it, some of those managers would then visit these employee surveillance vendors and see a Basecamp logo! 😱 These vendors promoting their wares by featuring integrations with Basecamp, usually under the banner of “time tracking”. Yikes!

We’ve decided it’s our obligation to resist the normalization of employee surveillance software. It is not right, it is not human, and unless we speak up now, we might well contribute to this cancer of mistrust and control spreading even after the COVID-19 crisis is behind us. That is not something we in good conscience could let happen.

Keep reading “Employee-surveillance software is not welcome to integrate with Basecamp”

There’s no perfect process for hiring great programmers, but there are plenty of terrible ways to screw it up. We’ve rejected the industry stables of grilling candidates in front of a whiteboard or needling them with brain teasers since the start at Basecamp. But you’re not getting around showing real code when applying for a job here.

In the early days of the company, we hired programmers almost exclusively from the open source community. I simply tapped people I’d been working with on the Ruby on Rails project, and I knew that their code would be good, because I’d seen so much of it! This is how we hired Jamis, Jeremy, Sam, Josh, Pratik, Matthew, and Eileen.

But if you only consider candidates from the open source community, you’re going to miss out on plenty of great programmers who just don’t have the time or the inclination to contribute code to open source.

And unfortunately, it’s rarely an option for candidates to submit code from a previous job with an application to a new job. Unlike design, which is at least somewhat out in the open, commercial code is often closely guarded. And even if it wasn’t, it’s often hard to tease out what someone was actually personally responsible for (which can also be a challenge with design!).

So what we’ve started to do instead at Basecamp is level the playing field by asking late-stage candidates to complete a small programming assignment as part of the final evaluation process. I’m going to show you two examples of these projects, and the submissions from the candidates that ended up being hired.

Keep reading “Hiring programmers with a take-home test”

Basecamp’s newest product HEY has lived on Kubernetes since development first began. While our applications are majestic monoliths, a product like HEY has numerous supporting services that run along-side the main app like our mail pipeline (Postfix and friends), Resque (and Resque Scheduler), and nginx, making Kubernetes a great orchestration option for us.

As you work on code changes or new feature additions for an application, you naturally want to test them somewhere — either in a unique environment or in production via feature flags. For our other applications like Basecamp 3, we make this happen via a series of numbered environments called betas (beta1 through betaX). A beta environment is essentially a mini production environment — it uses the production database but everything else (app services, Resque, Redis) is separate. In Basecamp 3’s case, we have a claim system via an internal chatbot that shows the status of each beta environment (here, none of them are claimed):

Our existing beta setup is fine, but what if we can do something better with the new capabilities that we are afforded by relying on Kubernetes? Indeed we can! After reading about GitHub’s branch-lab setup, I was inspired to come up with a better solution for beta environments than our existing claims system. The result is what’s in-use today for HEY: a system that (almost) immediately deploys any branch to a branch-specific endpoint that you can access right away to test your changes without having to use the claims system or talk to anyone else (along with an independent job processing fleet and Redis instance to support the environment).

Keep reading “Seamless branch deploys with Kubernetes”

Spring is emerging in the US and as part of our company spring cleaning, we took a peek at our product policies, noticed some cobwebs, and got out the duster.

You can read our current product policies here. Besides rewriting sections to be more readable, we made four substantive changes:

1. We’ve consolidated our policies across all products owned and maintained by Basecamp, LLC.
That includes all versions of Basecamp, Highrise, Campfire, Backpack, and the upcoming HEY. This change mostly affects our legacy application customers, bringing their (stale) terms and privacy policies up-to-date.

2. We’ve added more details to our privacy policy.
Our customers deserve to be able to easily and clearly understand what data we collect and why. We’ve restructured and fleshed out our privacy policy to do just that, while also adding more details on your rights with regard to your information. Just as important are the things that haven’t changed: that we take the privacy of your data seriously; that we do not, have not, and never will sell your data; and that we take care to not collect sensitive data that aren’t necessary.

3. We’ve introduced a Use Restrictions policy. We are proud to help our customers do their best work. We also recognize that technology is an amplifier: it can enable the helpful and the harmful. There are some purposes we staunchly stand against. Our Use Restrictions policy fleshes out what used to be a fairly vague clause in our Terms of Service, clearly describing what we consider abusive usage of our products. In addition, we outline how we investigate and resolve abusive usage, including the principles of human oversight, balanced responsibilities, and focus on evidence that guide us in investigations.

4. We’ve adjusted how you can find out about policy changes.
In 2018, we open-sourced our policies by publishing them as a public repository on Github. One of the nice things about this repository is it tracks all the revisions we make in our policies so you can see what changed, when, and why. For instance, you can see every change we made to our policies in this refresh. You can also decide whether you want to get an email notification when changes are made by watching the repository. We’ll also be announcing any substantive changes here on SvN; if you prefer email updates, you can subscribe here.

As always, customers can always reach us at support@basecamp.com with questions or suggestions about our policies. You can also open an issue in our policies repository if you’d like to contribute! 

The vast majority of web applications should start life as a Majestic Monolith: A single codebase that does everything the application needs to do. This is in contrast to a constellation of services, whether micro or macro, that tries to carve up the application into little islands each doing a piece of the overall work.

And the vast majority of web applications will continue to be served well by The Majestic Monolith for their entire lifespan. The limits upon which this pattern is constrained are high. Much higher than most people like to imagine when they fantasize about being capital-a Architects.

But. Even so, there may well come a day when The Majestic Monolith needs a little help. Maybe you’re dealing with very large teams that constantly have people tripping over each other (although, bear in mind that many very large organizations use the monorepo pattern!). Or you end up having performance or availability issues under extreme load that can’t be resolved easily within the confines of The Majestic Monolith’s technology choices. Your first instinct should be to improve the Majestic Monolith until it can cope, but, having done that and failed, you may look to the next step.

Keep reading “The Majestic Monolith can become The Citadel”

We had originally planned to release HEY, our new email service, in April. There was the final cycle to finish the features, there was a company meetup planned for the end of the month to celebrate together, we’d been capacity testing extensively, and the first step of a marketing campaign was already under way.

But then the world caught a virus. And suddenly it got pretty hard to stay excited about a brand new product. Not because that product wasn’t exciting, but because its significance was dwarfed by world events.

A lack of excitement, though, you could push through. The prospect of a stressful launch alongside the reality of a stressful life? No.

That’s not because we weren’t ready to work remotely. That we had to scramble to find new habits or tools to be productive. We’ve worked remotely for the past twenty years. We wrote a book on working remotely. Basecamp is a through and through remote company (and an all-in-one toolkit for remote work!).

But what’s going on right now is about more than just whether work can happen, but to which degree it should. We’re fortunate to work in software where the show doesn’t have to stop, like is the case in many other industries, but the show shouldn’t just carry on like nothing happened either.

About half the people who work at Basecamp have kids. They’re all at home now. Finding a new rhythm with remote learning, more cramped quarters, more tension from cooped-up siblings. You can’t put in 100% at work when life asks for 150%. Some things gotta give, and that something, for us, had to be HEY.

And it’s not like life is daisies even if you don’t have kids. This is a really stressful time, and it’s our obligation at Basecamp to help everyone get through that the best we can. Launching a new product in the midst of that just wasn’t the responsible thing to do, so we won’t.

Remember, almost all deadlines are made up. You can change your mind when the world changes around you.

HEY is going to launch when the world’s got a handle on this virus. When we either find a new normal, living within long-running restrictions, or we find a way to beat this thing. We’re not going to put a date on that, because nobody knows when that might be. And we’re not going to pretend that we do either.

In the meantime, we’ll keep making HEY better. We’re also going to put in time to level up Basecamp in a number of significant ways that have long been requested. The work doesn’t stop, it just bends.

If you wrote us an email to iwant@hey.com, you’re on the list, and we’ll let that list know as soon as we open up. If you think you might be interested in a better email experience when that’s something we all have the mental space to think about again, please do send us a story about how you feel about email to iwant@hey.com.

Stay home, stay safe!