Employee-surveillance software is not welcome to integrate with Basecamp

We’ve been teaching people how to do remote work well for the better part of two decades. We wrote a whole book about the topic in 2013, called REMOTE: Office Not Required. Basecamp has been a remote company since day one, and our software is sold as an all-in-one toolkit for remote work. Yeah, we’re big on remote work!

So now that COVID-19 has forced a lot of companies to move to remote work, it’s doubly important that we do our part to help those new to the practice settle in. We’ve been hosting a variety of online seminars, done podcasts, and been advocating for healthy ways to do remote right.

Unfortunately, the move to remote work has also turbo-charged interest in employee surveillance software. Drew Harwell’s harrowing report for The Washington Post should make anyone’s skin crawl, but it seems some managers are reading about these disgusting tools and thinking “oh, what a great idea, where can I buy?”.

And as fate would have it, some of those managers would then visit these employee surveillance vendors and see a Basecamp logo! 😱 These vendors promoting their wares by featuring integrations with Basecamp, usually under the banner of “time tracking”. Yikes!

We’ve decided it’s our obligation to resist the normalization of employee surveillance software. It is not right, it is not human, and unless we speak up now, we might well contribute to this cancer of mistrust and control spreading even after the COVID-19 crisis is behind us. That is not something we in good conscience could let happen.

Keep reading “Employee-surveillance software is not welcome to integrate with Basecamp”

Hiring programmers with a take-home test

There’s no perfect process for hiring great programmers, but there are plenty of terrible ways to screw it up. We’ve rejected the industry stables of grilling candidates in front of a whiteboard or needling them with brain teasers since the start at Basecamp. But you’re not getting around showing real code when applying for a job here.

In the early days of the company, we hired programmers almost exclusively from the open source community. I simply tapped people I’d been working with on the Ruby on Rails project, and I knew that their code would be good, because I’d seen so much of it! This is how we hired Jamis, Jeremy, Sam, Josh, Pratik, Matthew, and Eileen.

But if you only consider candidates from the open source community, you’re going to miss out on plenty of great programmers who just don’t have the time or the inclination to contribute code to open source.

And unfortunately, it’s rarely an option for candidates to submit code from a previous job with an application to a new job. Unlike design, which is at least somewhat out in the open, commercial code is often closely guarded. And even if it wasn’t, it’s often hard to tease out what someone was actually personally responsible for (which can also be a challenge with design!).

So what we’ve started to do instead at Basecamp is level the playing field by asking late-stage candidates to complete a small programming assignment as part of the final evaluation process. I’m going to show you two examples of these projects, and the submissions from the candidates that ended up being hired.

Keep reading “Hiring programmers with a take-home test”

The Majestic Monolith can become The Citadel

The vast majority of web applications should start life as a Majestic Monolith: A single codebase that does everything the application needs to do. This is in contrast to a constellation of services, whether micro or macro, that tries to carve up the application into little islands each doing a piece of the overall work.

And the vast majority of web applications will continue to be served well by The Majestic Monolith for their entire lifespan. The limits upon which this pattern is constrained are high. Much higher than most people like to imagine when they fantasize about being capital-a Architects.

But. Even so, there may well come a day when The Majestic Monolith needs a little help. Maybe you’re dealing with very large teams that constantly have people tripping over each other (although, bear in mind that many very large organizations use the monorepo pattern!). Or you end up having performance or availability issues under extreme load that can’t be resolved easily within the confines of The Majestic Monolith’s technology choices. Your first instinct should be to improve the Majestic Monolith until it can cope, but, having done that and failed, you may look to the next step.

Keep reading “The Majestic Monolith can become The Citadel”

Why HEY had to wait

We had originally planned to release HEY, our new email service, in April. There was the final cycle to finish the features, there was a company meetup planned for the end of the month to celebrate together, we’d been capacity testing extensively, and the first step of a marketing campaign was already under way.

But then the world caught a virus. And suddenly it got pretty hard to stay excited about a brand new product. Not because that product wasn’t exciting, but because its significance was dwarfed by world events.

A lack of excitement, though, you could push through. The prospect of a stressful launch alongside the reality of a stressful life? No.

That’s not because we weren’t ready to work remotely. That we had to scramble to find new habits or tools to be productive. We’ve worked remotely for the past twenty years. We wrote a book on working remotely. Basecamp is a through and through remote company (and an all-in-one toolkit for remote work!).

But what’s going on right now is about more than just whether work can happen, but to which degree it should. We’re fortunate to work in software where the show doesn’t have to stop, like is the case in many other industries, but the show shouldn’t just carry on like nothing happened either.

About half the people who work at Basecamp have kids. They’re all at home now. Finding a new rhythm with remote learning, more cramped quarters, more tension from cooped-up siblings. You can’t put in 100% at work when life asks for 150%. Some things gotta give, and that something, for us, had to be HEY.

And it’s not like life is daisies even if you don’t have kids. This is a really stressful time, and it’s our obligation at Basecamp to help everyone get through that the best we can. Launching a new product in the midst of that just wasn’t the responsible thing to do, so we won’t.

Remember, almost all deadlines are made up. You can change your mind when the world changes around you.

HEY is going to launch when the world’s got a handle on this virus. When we either find a new normal, living within long-running restrictions, or we find a way to beat this thing. We’re not going to put a date on that, because nobody knows when that might be. And we’re not going to pretend that we do either.

In the meantime, we’ll keep making HEY better. We’re also going to put in time to level up Basecamp in a number of significant ways that have long been requested. The work doesn’t stop, it just bends.

If you wrote us an email to iwant@hey.com, you’re on the list, and we’ll let that list know as soon as we open up. If you think you might be interested in a better email experience when that’s something we all have the mental space to think about again, please do send us a story about how you feel about email to iwant@hey.com.

Stay home, stay safe!

The books I read in 2019

Here are all my extracted answers from our monthly Basecamp check-in question of What are you reading? for 2019. (See also my answers from 2016, 2017, and 2018).

The Sane Society
Another Fromm tome! This one starts from the premise of evaluating the different social characters of various societies. But not from the abstract, pretend objectiveness of “everything is equal; everything is just different”, but from a bold perspective of “some societies truly do better than others at promoting human health and flourish. That’s potentially a dynamite perspective, but Fromm handles it with utter grace and respect.

I particularly enjoyed the concept of mental illness or malaise as an act of rebellion against societal pressures and norms. Particularly the refutation that “the sane person” is whoever is performing their productive function in society. Yeah, fuck that.

I also really liked the depiction of a country’s social character to be an expression of what that society thought it needed. A German reputation for stinginess/savings as virtuous? A reflection of what the country needed in order to rebuild after 20th century devastation. It’s a fascinating recast of “stereotype” as something intellectually productive, and not just lazy othering.

Keep reading “The books I read in 2019”

Mailing list software should stop spying on subscribers

The internet is finally coming out of its long haze on privacy, but it’s with one hell of a hangover. So many practices that were once taken for granted are now getting a second, more critical look. One of those is the practice of spying on whether recipients of marketing emails open them or not.

Back in August, we vowed to stop using such spying pixels in our Basecamp emails. And do you know what? It’s been fine! Not being able to track open rates, and fret over whether that meant our subject lines weren’t providing just the right HOOK, has actually been a relief.

But whether these open rates are “useful” or not is irrelevant. They’re invasive, they’re extracted without consent, and they break the basic assumptions most people have about email. There’s a general understanding that if you take actions on the internet, like clicking a link or visiting a site, there’s some tracking associated with that. We might not like it, but at least we have a vague understanding of it. Not so with email spy pixels.

Just about every normal person (i.e. someone not working in internet marketing) has been surprised, pissed, or at least dismayed when I tell them about spy pixels in emails. The idea that simply opening an email subjects you to tracking is a completely foreign one to most people.

When I’ve raised this concern in conversations with people in the marketing industry, a lot of them have taken offense to the term “spy pixels”. Affixing the spying label made a lot of them uncomfortable, because they were just trying to help! I get that nobody wants to think of themselves as the bad guy (Eilish not withstanding), but using the word “spy” isn’t exactly a reach.

Here’s the dictionary definition of a spy: “a person who secretly collects and reports information on the activities, movements, and plans”. That fits pretty well to a spy pixel that tracks whether you open an email or not, without your knowledge or consent!

So. Let’s stop doing that. Collectively. And the best place to instigate reform is with the mailing list software we use. A modest proposal for a basic ethics reform:

1) Mailing list software should not have spy pixels turned on by default. This is the most important step, because users will follow the lead of their software. It must be OK to spy on whether people open my marketing emails if the software I’m using it provides that by default.

2) Mailing list software can ask for explicit consent when the sender really does want to track open rates. Let the sender include a disclaimer at the bottom of their email: “[The sender] would like to know when you open this email to help improve their newsletter. If that’s OK with you, [please opt-in to providing read receipts]. Thanks!”.

That’s it. Don’t do it by default, ask for informed consent if you must. Being respectful of someone’s privacy isn’t rocket science.

And remember, you can still tag your links in those emails with ?source=newsletter or whatever to see whether your call-to-action is working. As we discussed, people have a basic understanding that clicking links and visiting websites – explicit actions they take! – has some tracking involved.

This isn’t going to magically make everything better. It’s not going to fix all the issues we have with privacy online or even all the deceptive practices around mailing lists. But it’s going to make things a little better. And if we keep making things a little better, we’ll eventually wake up to a world that’s a lot better.

Integrated systems for integrated programmers

One of the great tragedies of modern web development over the last five years or so has been the irrational exuberance for microservices. The idea that making a single great web application had simply become too hard, but if we broke that app up into many smaller apps, it’d all be much easier. Turned out, surprise-surprise, that it mostly wasn’t.

As Kelsey Hightower searingly put the fallacy: “We’re gonna break it up and somehow find the engineering discipline we never had in the first place”.

But it’s one of those hard lessons that nobody actually wants to hear. You don’t want to hear that the reason your monolith is a spaghetti monster is because you let it become that way, one commit at the time, due to weak habits, pressurized deadlines, or simply sheer lack of competence. No, what you want to hear is that none of that mess is your fault. That it was simply because of the oppressive monolithic architecture. And that, really, you’re just awesome, and if you take your dirty code and stick it into this new microservices tumbler, it’s going to come out sparking clean, smelling like fucking daffodils.

The great thing about such delusions is that they can keep you warm for quite a while. A yeah, sure, maybe the complexities of your new microservices monstrosity are plain as day right from the get go, but you can always excuse them with “it’s really going to pay off once we…” bullshit. And it’ll work! For a while. Because, who knows? Maybe this is better? But it’s not. And the day you have to really admit its not, you’re probably not even still there. On to the next thing.

Microservices as an architectural gold rush appealed to developers for the same reason TDD appeals to developers: it’s the pseudoscientific promise of a diet. The absolution of a new paradigm to wash away and forgive our sins. Who doesn’t want that?

Well, maybe you? Now after you’ve walked through the intellectual desert of a microservice approach to a problem that didn’t remotely warrant it (ie, almost all of them). Maybe now you’re ready to hear a different story. There’s a slot in your brain for a counterargument that just wasn’t there before.

So here’s the counterargument: Integrated systems are good. Integrated developers are good. Being able to wrap your mind around the whole application, and have developers who are able to make whole features, is good! The road to madness and despair lays in specialization and compartmentalization.

The galaxy brain takes it all in.

But of course, you cry, what if the system is too large to fit in my brain? Won’t it just swap and swap until I kernel failure? Yes, if you try to stick in a bloated beast of an application, sure.

So the work is to shrink the conceptual surface area of your application until it fits in a normal, but capable and competent, programmer’s brain. Using conceptual compression, sheer good code writing, a productive and succinct environment, using shortcuts and patterns. That’s the work.

But the payoff is glorious. Magnificent. SUBLIME. The magic of working on an integrated system together with integrated programmers is a line without limits, arbitrary boundaries, or sully gatekeepers.

Forget frontend or backend. The answer is all of it. At the same time. In the same mind.

This sounds impossible if you’ve cooked your noodle too long in the stew of modern astronautic abstractions. If you turn down the temperature, you’ll see that the web is actually much the same as it always was. Sure, a few expectations increased here, and a couple of breakthrough techniques appeared there, but fundamentally, it’s the same. What changed was us. And mostly not in ways for the better.

If your lived experience still haven’t hit the inevitable wall of defeat on the question of microservices, then be my guest, sit there with your folded arms and your smug pout. It’s ok. I get it. There’s not an open slot for this argument in your brain just yet. It’s ok. I’m patient! I’ll still be here in a couple of years when there’s room. And then I’ll send you a link to this article on twitter.

Peace. Love. Integration.

Testimony before the House Antitrust Subcommittee

My name is David Heinemeier Hansson, and I’m the CTO and co-founder of Basecamp, a small internet company from Chicago that sells project-management and team-collaboration software.

When we launched our main service back in 2004, the internet provided a largely free, fair, and open marketplace. We could reach customers and provide them with our software without having to ask any technology company for permission or pay them for the privilege.

Today, this is practically no longer true. The internet has been colonized by a handful of big tech companies that wield their monopoly power without restraint. This power allow them to bully, extort, or, should they please, even destroy our business – unless we accept their often onerous, exploitive, and ever-changing terms and conditions.

Keep reading “Testimony before the House Antitrust Subcommittee”

The last tracker was just removed from Basecamp.com

Can you believe we used to willingly tell Google about every single visitor to basecamp.com by way of Google Analytics? Letting them collect every last byte of information possible through the spying eye of their tracking pixel. Ugh.

But 2020 isn’t 2010. Our naiveté around data, who captures it, and what they do with it has collectively been brought to shame. Most people now sit with basic understanding that using the internet leaves behind a data trail, and quite a few people have begun to question just how deep that trail should be, and who should have the right to follow it.

In this new world, it feels like an obligation to make sure we’re not aiding and abetting those who seek to exploit our data. Those who hoard every little clue in order to piece of together a puzzle that’ll ultimately reveal all our weakest points and moments, then sell that picture to the highest bidder.

The internet needs to know less about us, not more. Just because it’s possible to track someone doesn’t mean we should.

That’s the ethos we’re trying to live at Basecamp. It’s not a straight path. Two decades of just doing as you did takes a while to unwind. But we’re here for that work.

Every request is now served from our own domains

Keep reading “The last tracker was just removed from Basecamp.com”

Only 15% of the Basecamp operations budget is spent on Ruby

We spend about $3 million every year to run all the versions of Basecamp and our legacy applications. That spend is spread across several on-premise data centers and cloud operations. It does not include the budget for our 7-person strong operations team, this is just the cost of connectivity, machines, power, and such.

There’s a lot of spend in that bucket. The biggest line item is the million dollars per year we spend storing 4.5 petabyte worth of files. We used to store these files ourselves, across three physical data centers for redundancy and availability, but the final math and operational hassle didn’t pan out. So now we’re just on S3 with a multi-region redundancy setup.

After that, it’s really a big mixed bag. We spend a lot of money on databases, which all run on MySQL. There’s ElasticSearch clusters that power our search. A swarm of Redis servers providing caching. There’s a Kafka pipeline and a Big Query backend for analytics. We have our own direct network connections between the data centers and the cloud.

Everything I’ve talked about so far is infrastructure we’d run and pay for regardless of our programming language or web framework. Whether we run on Python, PHP, Rust, Go, C++, or whatever, we’d still need databases, we’d still need search, we’d still need to store files.

So let’s talk about what we spend on our programming language and web framework. It’s about 15%. That’s the price for all our app and job servers. The machines that actually run Ruby on Rails. So against a $3 million budget, it’s about $450,000. That’s it.

Let’s imagine that there was some amazing technology that would let us do everything we’re doing with Ruby on Rails, but it was TWICE AS FAST! That would save us about ~$225,000 per year. We spend more money than that on the Xmas gift we give employees at Basecamp every year. And that’s if you could truly go twice as fast, and thus require half the machines, which is not an easy thing to do, despite what microbenchmarks might delude you into thinking.

Now imagine we found a true silver bullet. One where the compute spend could be reduced by an order of magnitude. So we’d save about $400,000/year, reducing everything we spend running our app and job servers to an unrealistically low $45,000/year. That reduction wouldn’t even pay for two developers at our average all-in cost at Basecamp!

Now let’s consider the cost of those savings. We spend more money on the 15-strong developer team at Basecamp than our entire operations budget! If we make that team just 15% less productive, it’ll cost us more than everything we spend to run Ruby and Rails at Basecamp!

Working with Ruby and Rails is a luxury, yes. Not every company pay their developers as well as we do at Basecamp, so maybe the rates would look a little different there. Maybe some companies are far more compute intensive to run their apps. But for most SaaS companies, they’re in exactly the same ballpark as we are. The slice of the total operations budget spent running the programming language and web framework that powers the app is a small minority of the overall cost.

For a company like Basecamp, you’d be mad to make your choice of programming language and web framework on anything but a determination of what’ll make your programmers the most motivated, happy, and productive. Whatever the cost, it’s worth it. It’s worth it on a pure cost/benefit, but, more importantly, it’s worth it in terms of human happiness and potential.

This is why we run Ruby. This is why we run Rails. It’s a complete bargain.