Keynote on the topic of open source, markets, debts, purpose, and no less than the meaning of life. Delivered at RailsConf 2019. Also available as a long read below.
We pay at the top 10% of San Francisco market rates for salary + bonus (but not stock options) at Basecamp. Those rates apply to everyone here, regardless of the role or where they actually live. We don’t even employ anyone who live in San Francisco!
We picked San Francisco as our benchmark because it’s the highest in the world for technology, and because we could afford it, after carefully growing a profitable software business for 15 years.
For programmers, designers, operations, administration, and the vast majority of other roles at Basecamp, simply following the top of the market rates is a pretty sweet deal: High salaries, benchmarked against the top market, yet most people live in places with far lower cost of living, which means we’re even more competitive in local markets.
(While we might be top 10% for San Francisco, I’m pretty sure we’re in the top 1% for, say, Fenwick, Ontario or Spokane, Washington or Madrid, Spain.)
But where this doesn’t work as well is when the San Francisco market rates reflect how technology hasn’t really raised all boats. While the rates there for programmers and designers are far higher than many other places, this is a lot less true for, say, customer support or other roles that hasn’t seen the same market competition.
Basecamp is hiring a director of operations to run the team responsible for all our technical infrastructure. Our suite of applications is served from a mix of our own servers in leased data-center space and cloud setups in Google Cloud and AWS. The job is to ensure that everything runs smoothly, the lights stay on, and we’ve prepared for bad luck with good planning.
This is a role for someone with experience running a team at least as big as ours and a multi-million dollar budget. You’ll be managing a team of seven and report to the CTO.
Basecamp is a remote-work company, but you’ll need at least 4 hours of overlap with Chicago time in your normal work-day routine.
We strongly encourage candidates of all different backgrounds and identities to apply. Each new hire is an opportunity for us to bring in a different perspective, and we are always eager to further diversify our company. Basecamp is committed to building an inclusive, supportive place for you to do the best and most rewarding work of your career.
It’s desperate times for those still clinging to their workaholic, exploitive ways. From Japan to China to even the US, there’s a growing understanding that working 70-80-90 or 130 hours per week is not glorious. Not virtuous. Not healthy.
So what’s a whoever-works-the-most-wins advocate to do? Sidestep the question of efficiency, of health, of sustainability, of course. Just press the pedal on fear and competition. Here’s your host of terror, Jason Calacanis:
Yeah, that’s it. Those who reject the wisdom of overwork is really helping the ENEMY. This is democracy vs communism!! What is this, 1950? Whatever year it is, it’s stupid.
Rather than support a grassroots rejection of the exploitive abuse of the Chinese workers under the 996 regime, Calanis is doubling down on the premise that to “beat” the Chinese, you must submit to their worst work practices. What?
This is at best a lateral move from “work harder or the kitten gets it”. A trope that’s meant to be a punchline, not a policy recommendation.
Besides being imperial paranoia, urging American companies to adopt Chinese abuses, lest they be left behind in the chase of growth uber alles, is the furthest away you could get from winning. Accepting the terms of engagement by your so-called opponent is a basic, rookie mistake in any form of strategic out-maneuvering.
You’re not going to “beat” the Chinese by one-upping 996 with 997. You’re not going to top Jack Ma’s calls for sacrifice by injecting nationalist fervor and clash-of-civilizations rhetoric into these base pleas for a deeper grind. This is madness.
If you define winning solely as “who has the greater growth”, you’ve already lost. If you dismiss the standard of living enjoyed in Europe – one without medical bankruptcies, crushing college debts, or falling life expectancies – as a “retirement society”, you’re the one who deserves to be dismissed.
The ideological underpinnings of capitalism are already in an advanced state of ethical decay. You don’t save the good parts of said capitalism by doubling down on the worst, most exploitive parts. Racing to the bottom just gets you there faster.
Wouldn’t it be great if there really was just one secret you had to know, and all your professional or entrepreneurial dreams would come true? Then you wouldn’t have to bother trying what works for your or your domain. You’d just have to apply The Secret, and voila!
Needless to say, if there is such a unifying secret, I haven’t found or heard of it. And yet, I keep seeing this false hope powering one of the most common questions I get in interviews: What’s THE ONE THING that you’d tell your younger yourself/other entrepreneurs/new programmers?! This is followed by the breathless anticipation of whatever profound wisdom the questioner most wish they could glean from my life’s experiences.
The boring truth is that the big leaps are all the result of an interwoven tapestry of practices, each contributing a strand of progress or insight. If you pull just one, all you have is that thin, disconnected strand. It’s only together the colors come alive.
That’s why all the books Jason and I have written together really are just a collection of essays. Yes, there’s a theme, but no single, unlocking narrative. REWORK is 88 separate essays, It Doesn’t Have To Be Crazy At Work has 66.
As long as you’re stuck on a quest for that one super-power practice or North Star principle, you’re not going to make space in your brain for the fact that no individual secret is going to make the difference. Only compound wisdom will.
Apple keep insisting that only a “small number of customers have problems” with the MacBook keyboards. That’s bollocks. This is a huge issue, it’s getting worse not better, and Apple is missing the forest for the trees.
The fact is that many people simply do not contact Apple when their MacBook keyboards fail. They just live with an S key that stutters or a spacebar that intermittently gives double. Or they just start using an external keyboard. Apple never sees these cases, so it never counts in their statistics.
So here’s some anecdata for Apple. I sampled the people at Basecamp. Out of the 47 people using MacBooks at the company, a staggering 30% are dealing with keyboard issues right now!! And that’s just the people dealing with current keyboard issues. If you include all the people who used to have issues, but went through a repair or replacement process, the number would be even higher.
Worth noting here is that the 3rd generation membrane keyboard did nothing to fix the issues. Six out of thirteen – nearly half!! – of the 2018+ MacBooks we have at the company have a failed keyboard.
I backed up those figures with a Twitter poll that has over 7,000 respondents already. That’s a 63% failure rate!! But Apple is only seeing 11% of those, as the vast majority of customers are simply just living with their broken computer:
This is a disaster. A complete unmitigated disaster.
But as always, in a time of crisis, the event itself is less indicative of the health of a company than the response. Is Apple going to accept that they’re currently alienating and undermining decades of goodwill by shipping broken computers in mass quantities?
From 2:13am GMT March 13 / 9:13pm Central March 12 until around 4:10am GMT / 11:10pm Central, Basecamp 3 was mostly offline and Basecamp 2 unable to process file uploads and downloads, as our cloud storage provider had a severe, sustained outage. We continued to have minor disruptions in service from 4:10am GMT / 11:10pm Central until everything was cleared at 6:53am GMT / 1:53am Central.
This is the second time in a week that I’m forced to write “I’m so sorry”. That’s incredibly painful. Both because it’s because we’re failing our customers for the second time in a week, but also because it’s showing us just how unprepared we’ve been as an organization to deal with these cloud challenges, despite our belief otherwise.
I’m not going to bother you with platitudes about “lessons to be learnt”, because I’ve already done that just a few days ago. This goes much deeper than just a few lessons. It has called into question our entire risk management and operational structure at Basecamp.
It’s also been a mighty fall. From reaching for 99.999% in uptime – the hallowed five nines! – we’re now scrambling for two of them. From riches of reliance to rags of shambles. To say this is humbling is an epic understatement.
We’re stopping all major product development at Basecamp for the moment, and dedicating all our attention to fixing these single points of failure that the recent cloud outages have revealed. We’re also going to pull back from our big migration to the cloud for a while, until we’re able to comfortably commit to a multi-region, multi-provider setup that’s more resilient against these outages.
I’m sorry. I’m really sorry (and ashamed).
From 4:30am GMT March 7 / 10:30pm Central March 6 until 1:02pm GMT / 7:02am Central, Basecamp 2 and the search feature in Basecamp 3 were mostly offline due to a catastrophic network failure with our cloud provider. Both our primary network link, our backup network link, and several additional ad-hoc network links between critical services needed to run Basecamp 2 were forced offline, as the cloud provider sought to deal with underlying network problems they were having.
Both Basecamp 2 and the search feature in Basecamp 3 are now fully back online.
But this was one of the worst outages we’ve had in the history of Basecamp. We’re incredibly sorry about just how long and broad of an interruption this caused, especially for our European customers of Basecamp 2. We’re so very sorry about this. We know this caused real and deep interruption to many people’s workflow from the early morning to the early afternoon on the main European timezones. And of course to any other customers around the world, including the US, who were also affected.
We’ve learned some hard lessons about network availability, the limitations of redundant, and double redundant backup connections. We’ll be working diligently to change how we work with cloud providers in the future, and how we can insulate ourselves and our customers from any future incidents like this. While this incident may have been triggered by network issues outside of our immediate control, it’s always within our control how we architect our systems, how we prepare for disasters, and how we ensure something like this never has the power to inflict such a traumatic outage.
So I want to make absolutely clear that this is our failure. Even in this new world of cloud services, it’s still always our fault when Basecamp isn’t available. Whatever the underlying problem for an outage, there’s always something you could have done to prevent it. And our list in this case includes a number of both obvious and not-so-obvious steps we could have taken. We will now take them.
Once again, I’m deeply sorry for this terrible outage. We will work as diligently we can to ensure that this doesn’t happen to any version of Basecamp again, neither past or present. Thank you for understanding, thank you for your patience, and thank you for being a customer, even if you with all justification ran out of both understanding and patience during this utterly unacceptable outage.
There are a lot of hoops to jump and obstacles to climb before starting a new business, but a lack of an all-burning passion for the pursuit shouldn’t be one of them. Yes, it’s easier to keep going when you like what you do, but it’s by no means a requirement to profess your love for the endeavor.
I know it often comes from a good place, this advice. That you should just wait until that magic idea comes along. Not be the fool rushing in. But this romantic idea that there’s the perfect opportunity just waiting out there for you to discover it is a mirage.
Most of business, most of the time, is pretty mundane! I’m not still working at Basecamp because, after nearly twenty years, I just spring out of bed every morning yearning to improve todos, events, messages, or project management in general. I like all those things, but the domain itself isn’t a burning bush of passion.
What working on Basecamp allows me to do is keep at the motions I most enjoy: Writing Ruby, sharing lessons and experiences, building a calm workplace, and being fair in dealings with customers. Those aren’t the only things I enjoy in life, but they’re definitely on the high list.
Thing is. I could have pursued all those things in a different domain than, say, helping businesses cope with growth and putting projects in order at Basecamp. In fact, I have! We’ve made quite a lot of applications at Basecamp over the years. Many related to a similar mission as Basecamp, but not all.
There are lots of reasons for why you’d want to start and run a business. Passion isn’t a permit you need to acquire before setting off.
Yesterday at 12:45pm central time, our ops team detected a dramatic spike in login requests to Basecamp. More than 30,000 login attempts were made in the hour that followed from a wide array of IP addresses. Our first line of defense was to block the offending addresses, but ultimately we needed to enable captcha to stop the attack.
After the attack was over, we diagnosed that 124 accounts had unauthorized access from the attack. We immediately reset the password for these accounts, logging out any intruders, and emailed the affected account holders with all the relevant information.
All of the unauthorized access was gained using the correct username and password for the account. It’s highly likely that these credentials were obtained from one of the big breaches, like those collected in combos like Collection #1, Anti Public, or Exploit.in. All the affected accounts showed as “owned” on haveibeenpwned.com.
Our preliminary investigation shows that none of the unauthorized access actually performed any actions within the accounts. It seemed like the attack focused on first validating which accounts were vulnerable, perhaps with a plan to later exploit these vulnerable accounts. Thankfully we were able to detect and stop the attack very quickly, and also ensure that any intruders were prevented further access.
Never the less, this is a serious reminder that you should never share the same password between multiple services. Particularly services such as Basecamp that may contain sensitive information. Here’s what we recommend you do to stay safe:
1.) Use a password manager to ensure you’re using different, secure passwords on every service you use. Then if one service is breached, you don’t have to worry about the rest. We use 1Password at Basecamp and recommend it.
2.) Subscribe to a breach notification service, like the one offered by haveibeenpwned.com. Then you’ll be alerted if your credentials are part of hack known to the public.
3.) Turn on two-factor authentication (2FA) wherever you can! We offer 2FA protection for Basecamp using Google Sign-In. Most services that deal with sensitive information offer 2FA these days. It’s especially important that you enable this for critical services, like your email address.
Our ops team will continue to monitor and fight any future attacks. They did an excellent job detecting and addressing this particular attack. But if someone has your username and password, and you don’t have 2FA protection, there are limits to how effective this protection can be.
Protecting yourself against attacks like this is important. Take the time to learn the basics, and take the steps outlined above to limit the risk.
Update: On January 31st, the mass-attack resumed in much greater strength than before. More than 5,000 IP addresses were used to test stolen credentials. 89 proven correct, but no content was accessed on these accounts, and we followed the same procedure of resetting all logins and writing the people affected. We’ve since beefed up our CAPTCHA protection across all applications and all clients, which has been effective at stopping the attack. CAPTCHA isn’t perfect, and some times it’s annoying, but it has provided effective protection against this wave of attack. We continue to work on shoring up defenses, but do follow the steps outlined above to protect yourself!