The Majestic Monolith can become The Citadel

The vast majority of web applications should start life as a Majestic Monolith: A single codebase that does everything the application needs to do. This is in contrast to a constellation of services, whether micro or macro, that tries to carve up the application into little islands each doing a piece of the overall work.

And the vast majority of web applications will continue to be served well by The Majestic Monolith for their entire lifespan. The limits upon which this pattern is constrained are high. Much higher than most people like to imagine when they fantasize about being capital-a Architects.

But. Even so, there may well come a day when The Majestic Monolith needs a little help. Maybe you’re dealing with very large teams that constantly have people tripping over each other (although, bear in mind that many very large organizations use the monorepo pattern!). Or you end up having performance or availability issues under extreme load that can’t be resolved easily within the confines of The Majestic Monolith’s technology choices. Your first instinct should be to improve the Majestic Monolith until it can cope, but, having done that and failed, you may look to the next step.

Keep reading “The Majestic Monolith can become The Citadel”

Why HEY had to wait

We had originally planned to release HEY, our new email service, in April. There was the final cycle to finish the features, there was a company meetup planned for the end of the month to celebrate together, we’d been capacity testing extensively, and the first step of a marketing campaign was already under way.

But then the world caught a virus. And suddenly it got pretty hard to stay excited about a brand new product. Not because that product wasn’t exciting, but because its significance was dwarfed by world events.

A lack of excitement, though, you could push through. The prospect of a stressful launch alongside the reality of a stressful life? No.

That’s not because we weren’t ready to work remotely. That we had to scramble to find new habits or tools to be productive. We’ve worked remotely for the past twenty years. We wrote a book on working remotely. Basecamp is a through and through remote company (and an all-in-one toolkit for remote work!).

But what’s going on right now is about more than just whether work can happen, but to which degree it should. We’re fortunate to work in software where the show doesn’t have to stop, like is the case in many other industries, but the show shouldn’t just carry on like nothing happened either.

About half the people who work at Basecamp have kids. They’re all at home now. Finding a new rhythm with remote learning, more cramped quarters, more tension from cooped-up siblings. You can’t put in 100% at work when life asks for 150%. Some things gotta give, and that something, for us, had to be HEY.

And it’s not like life is daisies even if you don’t have kids. This is a really stressful time, and it’s our obligation at Basecamp to help everyone get through that the best we can. Launching a new product in the midst of that just wasn’t the responsible thing to do, so we won’t.

Remember, almost all deadlines are made up. You can change your mind when the world changes around you.

HEY is going to launch when the world’s got a handle on this virus. When we either find a new normal, living within long-running restrictions, or we find a way to beat this thing. We’re not going to put a date on that, because nobody knows when that might be. And we’re not going to pretend that we do either.

In the meantime, we’ll keep making HEY better. We’re also going to put in time to level up Basecamp in a number of significant ways that have long been requested. The work doesn’t stop, it just bends.

If you wrote us an email to iwant@hey.com, you’re on the list, and we’ll let that list know as soon as we open up. If you think you might be interested in a better email experience when that’s something we all have the mental space to think about again, please do send us a story about how you feel about email to iwant@hey.com.

Stay home, stay safe!

The books I read in 2019

Here are all my extracted answers from our monthly Basecamp check-in question of What are you reading? for 2019. (See also my answers from 2016, 2017, and 2018).

The Sane Society
Another Fromm tome! This one starts from the premise of evaluating the different social characters of various societies. But not from the abstract, pretend objectiveness of “everything is equal; everything is just different”, but from a bold perspective of “some societies truly do better than others at promoting human health and flourish. That’s potentially a dynamite perspective, but Fromm handles it with utter grace and respect.

I particularly enjoyed the concept of mental illness or malaise as an act of rebellion against societal pressures and norms. Particularly the refutation that “the sane person” is whoever is performing their productive function in society. Yeah, fuck that.

I also really liked the depiction of a country’s social character to be an expression of what that society thought it needed. A German reputation for stinginess/savings as virtuous? A reflection of what the country needed in order to rebuild after 20th century devastation. It’s a fascinating recast of “stereotype” as something intellectually productive, and not just lazy othering.

Keep reading “The books I read in 2019”

Mailing list software should stop spying on subscribers

The internet is finally coming out of its long haze on privacy, but it’s with one hell of a hangover. So many practices that were once taken for granted are now getting a second, more critical look. One of those is the practice of spying on whether recipients of marketing emails open them or not.

Back in August, we vowed to stop using such spying pixels in our Basecamp emails. And do you know what? It’s been fine! Not being able to track open rates, and fret over whether that meant our subject lines weren’t providing just the right HOOK, has actually been a relief.

But whether these open rates are “useful” or not is irrelevant. They’re invasive, they’re extracted without consent, and they break the basic assumptions most people have about email. There’s a general understanding that if you take actions on the internet, like clicking a link or visiting a site, there’s some tracking associated with that. We might not like it, but at least we have a vague understanding of it. Not so with email spy pixels.

Just about every normal person (i.e. someone not working in internet marketing) has been surprised, pissed, or at least dismayed when I tell them about spy pixels in emails. The idea that simply opening an email subjects you to tracking is a completely foreign one to most people.

When I’ve raised this concern in conversations with people in the marketing industry, a lot of them have taken offense to the term “spy pixels”. Affixing the spying label made a lot of them uncomfortable, because they were just trying to help! I get that nobody wants to think of themselves as the bad guy (Eilish not withstanding), but using the word “spy” isn’t exactly a reach.

Here’s the dictionary definition of a spy: “a person who secretly collects and reports information on the activities, movements, and plans”. That fits pretty well to a spy pixel that tracks whether you open an email or not, without your knowledge or consent!

So. Let’s stop doing that. Collectively. And the best place to instigate reform is with the mailing list software we use. A modest proposal for a basic ethics reform:

1) Mailing list software should not have spy pixels turned on by default. This is the most important step, because users will follow the lead of their software. It must be OK to spy on whether people open my marketing emails if the software I’m using it provides that by default.

2) Mailing list software can ask for explicit consent when the sender really does want to track open rates. Let the sender include a disclaimer at the bottom of their email: “[The sender] would like to know when you open this email to help improve their newsletter. If that’s OK with you, [please opt-in to providing read receipts]. Thanks!”.

That’s it. Don’t do it by default, ask for informed consent if you must. Being respectful of someone’s privacy isn’t rocket science.

And remember, you can still tag your links in those emails with ?source=newsletter or whatever to see whether your call-to-action is working. As we discussed, people have a basic understanding that clicking links and visiting websites – explicit actions they take! – has some tracking involved.

This isn’t going to magically make everything better. It’s not going to fix all the issues we have with privacy online or even all the deceptive practices around mailing lists. But it’s going to make things a little better. And if we keep making things a little better, we’ll eventually wake up to a world that’s a lot better.

Integrated systems for integrated programmers

One of the great tragedies of modern web development over the last five years or so has been the irrational exuberance for microservices. The idea that making a single great web application had simply become too hard, but if we broke that app up into many smaller apps, it’d all be much easier. Turned out, surprise-surprise, that it mostly wasn’t.

As Kelsey Hightower searingly put the fallacy: “We’re gonna break it up and somehow find the engineering discipline we never had in the first place”.

But it’s one of those hard lessons that nobody actually wants to hear. You don’t want to hear that the reason your monolith is a spaghetti monster is because you let it become that way, one commit at the time, due to weak habits, pressurized deadlines, or simply sheer lack of competence. No, what you want to hear is that none of that mess is your fault. That it was simply because of the oppressive monolithic architecture. And that, really, you’re just awesome, and if you take your dirty code and stick it into this new microservices tumbler, it’s going to come out sparking clean, smelling like fucking daffodils.

The great thing about such delusions is that they can keep you warm for quite a while. A yeah, sure, maybe the complexities of your new microservices monstrosity are plain as day right from the get go, but you can always excuse them with “it’s really going to pay off once we…” bullshit. And it’ll work! For a while. Because, who knows? Maybe this is better? But it’s not. And the day you have to really admit its not, you’re probably not even still there. On to the next thing.

Microservices as an architectural gold rush appealed to developers for the same reason TDD appeals to developers: it’s the pseudoscientific promise of a diet. The absolution of a new paradigm to wash away and forgive our sins. Who doesn’t want that?

Well, maybe you? Now after you’ve walked through the intellectual desert of a microservice approach to a problem that didn’t remotely warrant it (ie, almost all of them). Maybe now you’re ready to hear a different story. There’s a slot in your brain for a counterargument that just wasn’t there before.

So here’s the counterargument: Integrated systems are good. Integrated developers are good. Being able to wrap your mind around the whole application, and have developers who are able to make whole features, is good! The road to madness and despair lays in specialization and compartmentalization.

The galaxy brain takes it all in.

But of course, you cry, what if the system is too large to fit in my brain? Won’t it just swap and swap until I kernel failure? Yes, if you try to stick in a bloated beast of an application, sure.

So the work is to shrink the conceptual surface area of your application until it fits in a normal, but capable and competent, programmer’s brain. Using conceptual compression, sheer good code writing, a productive and succinct environment, using shortcuts and patterns. That’s the work.

But the payoff is glorious. Magnificent. SUBLIME. The magic of working on an integrated system together with integrated programmers is a line without limits, arbitrary boundaries, or sully gatekeepers.

Forget frontend or backend. The answer is all of it. At the same time. In the same mind.

This sounds impossible if you’ve cooked your noodle too long in the stew of modern astronautic abstractions. If you turn down the temperature, you’ll see that the web is actually much the same as it always was. Sure, a few expectations increased here, and a couple of breakthrough techniques appeared there, but fundamentally, it’s the same. What changed was us. And mostly not in ways for the better.

If your lived experience still haven’t hit the inevitable wall of defeat on the question of microservices, then be my guest, sit there with your folded arms and your smug pout. It’s ok. I get it. There’s not an open slot for this argument in your brain just yet. It’s ok. I’m patient! I’ll still be here in a couple of years when there’s room. And then I’ll send you a link to this article on twitter.

Peace. Love. Integration.

Testimony before the House Antitrust Subcommittee

My name is David Heinemeier Hansson, and I’m the CTO and co-founder of Basecamp, a small internet company from Chicago that sells project-management and team-collaboration software.

When we launched our main service back in 2004, the internet provided a largely free, fair, and open marketplace. We could reach customers and provide them with our software without having to ask any technology company for permission or pay them for the privilege.

Today, this is practically no longer true. The internet has been colonized by a handful of big tech companies that wield their monopoly power without restraint. This power allow them to bully, extort, or, should they please, even destroy our business – unless we accept their often onerous, exploitive, and ever-changing terms and conditions.

Keep reading “Testimony before the House Antitrust Subcommittee”

The last tracker was just removed from Basecamp.com

Can you believe we used to willingly tell Google about every single visitor to basecamp.com by way of Google Analytics? Letting them collect every last byte of information possible through the spying eye of their tracking pixel. Ugh.

But 2020 isn’t 2010. Our naiveté around data, who captures it, and what they do with it has collectively been brought to shame. Most people now sit with basic understanding that using the internet leaves behind a data trail, and quite a few people have begun to question just how deep that trail should be, and who should have the right to follow it.

In this new world, it feels like an obligation to make sure we’re not aiding and abetting those who seek to exploit our data. Those who hoard every little clue in order to piece of together a puzzle that’ll ultimately reveal all our weakest points and moments, then sell that picture to the highest bidder.

The internet needs to know less about us, not more. Just because it’s possible to track someone doesn’t mean we should.

That’s the ethos we’re trying to live at Basecamp. It’s not a straight path. Two decades of just doing as you did takes a while to unwind. But we’re here for that work.

Every request is now served from our own domains

Keep reading “The last tracker was just removed from Basecamp.com”

Only 15% of the Basecamp operations budget is spent on Ruby

We spend about $3 million every year to run all the versions of Basecamp and our legacy applications. That spend is spread across several on-premise data centers and cloud operations. It does not include the budget for our 7-person strong operations team, this is just the cost of connectivity, machines, power, and such.

There’s a lot of spend in that bucket. The biggest line item is the million dollars per year we spend storing 4.5 petabyte worth of files. We used to store these files ourselves, across three physical data centers for redundancy and availability, but the final math and operational hassle didn’t pan out. So now we’re just on S3 with a multi-region redundancy setup.

After that, it’s really a big mixed bag. We spend a lot of money on databases, which all run on MySQL. There’s ElasticSearch clusters that power our search. A swarm of Redis servers providing caching. There’s a Kafka pipeline and a Big Query backend for analytics. We have our own direct network connections between the data centers and the cloud.

Everything I’ve talked about so far is infrastructure we’d run and pay for regardless of our programming language or web framework. Whether we run on Python, PHP, Rust, Go, C++, or whatever, we’d still need databases, we’d still need search, we’d still need to store files.

So let’s talk about what we spend on our programming language and web framework. It’s about 15%. That’s the price for all our app and job servers. The machines that actually run Ruby on Rails. So against a $3 million budget, it’s about $450,000. That’s it.

Let’s imagine that there was some amazing technology that would let us do everything we’re doing with Ruby on Rails, but it was TWICE AS FAST! That would save us about ~$225,000 per year. We spend more money than that on the Xmas gift we give employees at Basecamp every year. And that’s if you could truly go twice as fast, and thus require half the machines, which is not an easy thing to do, despite what microbenchmarks might delude you into thinking.

Now imagine we found a true silver bullet. One where the compute spend could be reduced by an order of magnitude. So we’d save about $400,000/year, reducing everything we spend running our app and job servers to an unrealistically low $45,000/year. That reduction wouldn’t even pay for two developers at our average all-in cost at Basecamp!

Now let’s consider the cost of those savings. We spend more money on the 15-strong developer team at Basecamp than our entire operations budget! If we make that team just 15% less productive, it’ll cost us more than everything we spend to run Ruby and Rails at Basecamp!

Working with Ruby and Rails is a luxury, yes. Not every company pay their developers as well as we do at Basecamp, so maybe the rates would look a little different there. Maybe some companies are far more compute intensive to run their apps. But for most SaaS companies, they’re in exactly the same ballpark as we are. The slice of the total operations budget spent running the programming language and web framework that powers the app is a small minority of the overall cost.

For a company like Basecamp, you’d be mad to make your choice of programming language and web framework on anything but a determination of what’ll make your programmers the most motivated, happy, and productive. Whatever the cost, it’s worth it. It’s worth it on a pure cost/benefit, but, more importantly, it’s worth it in terms of human happiness and potential.

This is why we run Ruby. This is why we run Rails. It’s a complete bargain.

Back to windows after twenty years

Apple’s stubborn four-year refusal to fix the terminally broken butterfly keyboard design led me to a crazy experiment last week: Giving Windows a try for the first time in twenty years.

Not really because I suddenly had some great curiosity about Windows, but because Apple’s infuriating failure to sell a reliable laptop reluctantly put me back in the market. So when I saw the praise heaped upon the Surface Laptop 3, and particularly its keyboard, I thought, fuck it, let’s give it a try!

Looks good, doesn’t it?

The buying experience was great. There was nobody in the store, so with four sales people just standing around, I got immediate attention, and typed away a few quick sentences on the keyboard. It felt good. Nice travel, slim chassis, sleek design. SOLD!

The initial setup experience was another pleasant surprise. The Cortana-narrated process felt like someone from the Xbox team had done the design. Fresh, modern, fun, and reassuring. Apple could take some notes on that.

But ultimately we got to the meat of this experience, and unfortunately the first bite didn’t quite match the sizzle. The font rendering in Windows remains excruciatingly poor to my eyes. It just looks bad. It reminded me of my number one grief with Android back in the 5.0 or whenever days, before someone at Google decided to do font rendering right (these days it’s great!). Ugh.

I accept that this is a personal failure of sorts. The Windows font rendering does not prevent you from using the device. It’s not like you can’t read the text. It’s just that I don’t enjoy it, and I don’t want to. So that was strike one.

But hey, I didn’t pluck down close to $1800 (with taxes) for a Windows laptop just to be scared off by poor font rendering, right? No. So I persevered and started setting up my development environment.

See, the whole reason I thought Windows might be a suitable alternative for me was all the enthusiasm around Windows Linux Subsystem (WSL). Basically putting all the *nix tooling at your fingertips, like it is on OSX, in a way that doesn’t require crazy hoops.

But it’s just not there. The first version of WSL is marred with terrible file-system performance, and I got to feel that right away, when I spent eons checking out a git repository via GitHub for Windows. A 10-second operation on OSX took 5-6 minutes on Windows.

I initially thought that I had installed WSL2, which promises to be better in some ways (though worse in others), but to do so required me to essentially run an alpha version of Windows 10. Okay, that’s a little adventurous, but hey, whatever, this was an experiment after all. (Unfortunately WSL2 doesn’t do anything to speed up work happening across the Windows/Linux boundary, in fact, it just makes it worse! So you kinda have to stick with Linux tooling inside of Linux, Windows outside. Defeating much of the point for me!).

So anyway, here I am, hours into trying to setup this laptop to run *nix tooling with Windows applications, running on the bleeding edge of Windows, digging through all sorts of write-ups and tutorials, and I finally, sorta, kinda get it going. But it’s neither fast nor pleasant nor intuitive in any way. And it feels like my toes are so stubbed and bloody by the end of the walk that I almost forgot why I started on this journey in the first place.

I mean, one thing is the alpha-level of the software required to even pursue this. Something else is the bizarre gates that Microsoft erects along the way. Want to run Docker for Windows on your brand new Surface Laptop 3? Sorry, can’t do that without buying an upgrade to Windows Pro (the $1800 Surface Laptop 3 apparently wasn’t expensive enough to warrant that designation, so it ships with the Home edition. Okay, sheesh).

The default Edge browser that ships with Windows 10 is also just kinda terrible. I clocked a 38 on the Speedometer 2.0 test, compared to the 125 that my MacBook Pro 13 ran with Safari. (But hey, there’s another beta version of Edge, the one that now uses the Chrominum rendering engine, and that got it to a more respectable 68.)

Anyway, I started this experiment on a Monday. I kept going all the way through Friday. Using the laptop as I would any other computer for the internet, and my new hobby of dealing with the stubbed toes of setting up a *nix development environment, but when I got to Saturday I just… gave up. It’s clearly not that this couldn’t be done. You can absolutely setup a new Windows laptop today to do *nix style development. You can get your VS Code going, install a bunch of alpha software, and eventually you’ll get there.

But for me, this just wasn’t worth it. I kept looking for things I liked about Windows, and I kept realizing that I just fell back on rationalizations like “I guess this isn’t SO bad?”. The only thing I really liked was the hardware, and really, the key (ha!) thing there was that the keyboard just worked. It’s a good keyboard, but I don’t know if I’d go as far as “great”. (I still prefer travel, control, and feel of the freestanding Apple Magic Keyboard 2).

What this experiment taught me, though, was just how much I actually like OSX. How much satisfaction I derive from its font rendering. How lovely my code looks in TextMate 2. How easy it is to live that *nix developer life, while still using a computer where everything (well, except that fucking keyboard!) mostly just works.

So the Surface Laptop 3 is going back to Microsoft. Kudos to them for the 30-day no questions return policy, and double kudos for making it so easy to wipe the machine for return (again, another area where Apple could learn!).

Windows still clearly isn’t for me. And I wouldn’t recommend it to any of our developers at Basecamp. But I kinda do wish that more people actually do make the switch. Apple needs the competition. We need to feel like there are real alternatives that not only are technically possible, but a joy to use. We need Microsoft to keep improving, and having more frustrated Apple users cross over, point out the flaws, and iron out the kinks, well, that’s only going to help.

I would absolutely give Windows another try in a few years, but for now, I’m just feeling #blessed that 90% of my work happens on an iMac with that lovely scissor-keyed Magic Keyboard 2. It’s not a real solution for lots of people who work on the go, but if you do most of your development at a desk, I’d check it out. Or be brave, go with Windows, make it better, you pioneer, you. You’ll have my utter admiration!

Also, Apple, please just fix those fucking keyboards. Provide proper restitution for the people who bought your broken shit. Stop gaslighting us all with your nonsense that this is only affecting extremely few people. It’s not. The situation is an unmitigated disaster.

Basecamp no longer requires Google for two-factor authentication

When it became clear to us last year that using SMS for two-factor authentication (2FA) was insecure, we kinda panicked. We’d spent a lot of time originally building that SMS-based 2FA login system for Basecamp, and the prospect of having to build an entirely new system compatible with proper authentication apps seemed daunting. Especially with major security liability hanging over our head.

So we went the easy route, and handed the 2FA authentication flow over to Google, using their Google Sign-In APIs. Now, that certainly gave us an immediate and secure solution. Nobody is disputing that Google knows security.

But requiring people to have a Google account to get a 2FA-protected Basecamp was an uncomfortable compromise. There are about a million good reasons for why you wouldn’t want Google to know everything about when you log into apps all over the internet. Google’s business is literally based on collecting as much data as possible, so it can use it all against you for ad targeting. That’s just not a regime we feel comfortable encouraging, let alone requiring.

So I’m thrilled to announce that we got our shit together and built our own, wonderful, and secure 2FA login protection for Basecamp. Google Sign-In still works, but it’s deprecated, and we’ll no longer be recommending it going forward.

Our new secure 2FA solution is built on the TOTP standard with backup codes as a fallback. So you can use any TOTP compatible authentication app, like Authy, 1Password, or Duo, and it works for all versions of Basecamp (here’s how to set it up in Basecamp 3 and Basecamp 2), as well as our legacy apps Highrise, Backpack, and Campfire.

Big kudos to Rosa Gutiérrez from our Security, Infrastructure & Performance team for putting our fears about doing our own TOTP-based 2FA system to shame. She led the project, did the work, and the final result is just great.

Finally, it feels good to have one additional area of the business free from Big Tech entanglement. We also dumped Google Analytics a few months back from Basecamp.com (relying on Clicky.com instead), and we’ll continue the work to untangle ourselves from Google and the rest of the industry behemoths. It’s a long slog, it’s unlikely ever to be fully complete, but every little bit helps.

Oh, and please, if you haven’t already, turn on 2FA to protect your Basecamp account. And if you aren’t already, use a password manager, like 1password. If you’re reusing a password on Basecamp, and you’re not protected by 2FA, you’re at a grave risk of having your account compromised. We work hard to protect everyone at Basecamp, but nothing will protect you online like using 2FA and a password manager everywhere you go.