There’s no perfect process for hiring great programmers, but there are plenty of terrible ways to screw it up. We’ve rejected the industry stables of grilling candidates in front of a whiteboard or needling them with brain teasers since the start at Basecamp. But you’re not getting around showing real code when applying for a job here.
In the early days of the company, we hired programmers almost exclusively from the open source community. I simply tapped people I’d been working with on the Ruby on Rails project, and I knew that their code would be good, because I’d seen so much of it! This is how we hired Jamis, Jeremy, Sam, Josh, Pratik, Matthew, and Eileen.
But if you only consider candidates from the open source community, you’re going to miss out on plenty of great programmers who just don’t have the time or the inclination to contribute code to open source.
And unfortunately, it’s rarely an option for candidates to submit code from a previous job with an application to a new job. Unlike design, which is at least somewhat out in the open, commercial code is often closely guarded. And even if it wasn’t, it’s often hard to tease out what someone was actually personally responsible for (which can also be a challenge with design!).
So what we’ve started to do instead at Basecamp is level the playing field by asking late-stage candidates to complete a small programming assignment as part of the final evaluation process. I’m going to show you two examples of these projects, and the submissions from the candidates that ended up being hired.
But first it’s important to recognize that we don’t ask applicants to do these assessments as part of their initial application. We simply get way too many applications for most openings to make that practical or fair. The last opening on the Research & Fidelity team got over 1,300 applications. Nobody can review that many code submissions!
So we whittle the group of candidates down aggressively first. This means judging their cover letter and, to a far lesser extent, their resume. For the opening we had on the Research & Fidelity team, we gave 40 people the take-home test, and even that proved to be too many. For the opening we had on the Security, Infrastructure & Performance team, we only gave 13 people the take-home test. That felt better. In the future, we’ll target fewer than 20 for sure.
Then there’s the assessment itself. I’ve heard many fair complaints that companies are asking candidates to complete massive projects that may take 20-30-40 hours of work, which is all unpaid, and which might be difficult for candidates to fit in with their existing job and life. Yeah, don’t do that. Asking someone for forty hours of work product, without pay, which might well go nowhere, is not what we do or advocate at Basecamp.
On the design side, we have asked candidates to complete more substantial projects, perhaps asking 10-20 hours of work, but then we pay them for the work. It’s like getting hired for a small freelance gig, even if you don’t get the job, and even if we’re never going to use the work.
But for programmers, we don’t need a project that large to get a good indication of someone’s programming skills or thought process. With both of the last two openings, we used assignment that were estimated to take 3-5 hours to complete.
That’s still a very substantial commitment! I wouldn’t ask anyone to submit such unpaid work without believing they were clearly in the running for the position. But when you look at our numbers, we usually don’t end up asking more than 1-3% of our applicants for such a commitment.
And it’s worth contrasting that against the work we’re not asking people to do. We don’t run candidates through some recruiter mill, where they have to do phone screening after phone screening. We don’t use any sort of automated tooling to scan resumes or whatever. We don’t even have people travel for interviews.
On our side, we often spend weeks perfecting the job opening itself. We put in an absolutely tremendous amount of work to conduct a fair, human, respectful, and thorough job search.
And the rules of the game are specified up front. You shouldn’t apply to a programming job at Basecamp unless you’re prepared to put in the work writing a considered, tailored cover letter, and then making the 3-5+ hours available to complete the programming assessment, if you make it into that top group of 1-3% of applicants.
The SIP assignment
Anyway, let’s take a look at these take-home assignments. The first is from our Security, Infrastructure & Performance team, and it was designed by Rosa. This was for a senior programmer opening. 13 applicants were asked to complete the assignment, and they were given a full week to do it (such that the estimated 3-5 hours could be spread out over several week nights or maybe the weekend).
It focused on a sliver of a real problem we’d been dealing with at Basecamp: Support for ban in rack-ratelimit. So it wasn’t some Tower of Hanoi abstract, computer-sciency test that you can look up a million solutions to, and which favor recent grads of CS algorithm courses. No, it was for implementing a feature in the same way you might well be asked to do on the job. There was a clear example of the API we wanted, and then candidates could solve it as they saw fit.
Jorge Manrubia, who we ended up hiring, submitted his solution as a full pull request –– complete with system tests and performance tests and documentation! It was a very comprehensive solution, but almost to a fault. I remember one of my concerns with Jorge’s submission was that it was borderline gold plated. We had several other submissions that were also wonderful, but far smaller, which could just as well had made for the final choice.
Jorge admitted to spending closer to 7-8 hours, in total, on his submission. We never policed this, because it really wasn’t a material part of the assessment. The solution would have been as compelling without all the extracurriculars, and several other candidates shone just as brightly with more modest submissions.
Which goes to a second point: Completing a programming assignment is required but insufficient to land a job at Basecamp. You can’t get hired without demonstrating a core competency in writing great code, but you also won’t get hired just because you can write great code alone.
The programming assessment simply unlocks the next step of the evaluation process. Out of the 13 candidates that received the programming assignment, 10 progressed to a first phone interview with Andrea (our head of people ops), then 6 progressed to interviews with the whole team, and finally 2 candidates got an extra interview to determine who we were going to hire.
The R&F assignment
For the programmer opening on our Research & Fidelity team, we followed a very similar process, but made the mistake of giving the programming assignment to too many people –– 40 in total. As discussed earlier, we should, and will in the future, cap that to 20 max.
This assignment was designed by Javan, and focused on another real-life sliver of the work we were doing on HEY: Enhance datalist autocomplete. The challenge came complete with the basic HTML form, and then directions on how to use JavaScript to enhance it to get us the universal autocomplete across the three major browsers.
Nabeelah Ali, who we ended up hiring here, submitted her solution with fewer flourishes than Jorge, but code no less impressive. Everything contained in a single page to make the solution work.
Like Jorge, Nabeelah also ended up spending more than the estimated 3-5 hours to complete the assignment. Somewhere around double there too. Much of that spent refactoring and polishing the solution. As a new mother with a 10-month old, just being back to work, and dealing with nightly wake-ups, this definitely wasn’t easy. But, as she said, “I would do that any day over a doing a live coding challenge”.
Which goes to the point that the alternative to a take-home programming assignment is rarely “nothing”. It’s often spending as much time, or more, traveling for a day of packed interviews. Some times traveling more than once! Dealing with the whiteboard assessments. Or going through abstract programming games or assessments that aren’t reflective of the work the team actually does.
There are of course other alternatives too. Some do half a day of pair programming together with candidates. Others ask for a code review, where the candidate comments on existing code. To me, the former often relies on an in-person meeting to work well (and that wasn’t going to happen, with Jorge hired from Spain, and Nabeelah hired from Norway), and the latter isn’t someone’s own code.
Hiring is hard. Applying is hard. Doing either with programmers without looking at actual code they wrote often risks leading down a path of bias (or, as we call it today, “fit”), credentialism, whiteboard puzzles, and brainteasers.
We’ll stick with the hard work.
So you say “asking candidates to complete massive projects that may take 20-30-40 hours of work” is bad.
So you make a test that is 3-5 hours, because you don’t want to be bad.
Then you hire the person who took 7-8 hours.
SO YOU ARE LOOKING FOR THE PERSON WHO IS WILLING TO PUT IN A LOT OF TIME AFTER ALL!!!
You are just the company you called bad in the first part, wow.
And even post this and think “wow we are so cool and different”…
If you can’t tell the difference between spending 7-8 hours on something vs upwards of 40, I don’t know what to tell. Except maybe programming isn’t a great field for you, given this amount of basic math proving troublesome? 😄
And we don’t designed these assignments to be “not bad”, but because they were the minimally viable assignments that included original code around a realistic domain which would give us the insight needed to make a hiring determination.
It is, however, totally fair to say: I don’t want to audition for a job. I don’t want to spend a day’s work on an assignment, as part of a final selection of candidates. So I’m just not going to apply at all. Completely fair! You certainly control how much effort you want to put into a hiring process. This is the effort we require.
I (kind of) agree with you. The task may be stated as needing 3-5 hours, but when the people who get hired spend 8-10 hours then it’s *actually* an 8-10 hour assignment. It sounds like they looked more objectively at the actual product, as evidenced by Nabeelah getting the job without the “gold-plating,” but both hires spent double the listed time on the assignment.
As someone trying to break into development, these stories discourage me. I can have a really great application and follow directions exactly, but those with the luxury of time will always have the advantage.
Jessica, you’re right in the sense that our targeted range for the implementation was off. WELCOME TO SOFTWARE DEVELOPMENT 😂. This is basically how it works! Estimates suck.
That being said, you could still finish the assignment – even spending twice the estimated time! – in a solid day. If you can’t take a solid day, after you’ve made it into a finalist round of an extremely competitive opening amongst just a dozen other candidates (for the SIP position), then I don’t know what to tell you. That’s the challenge, that’s the game, it was spelled out up front.
But we will definitely spell it out even more clearly for any future openings. And we will revise our estimates to ask for upwards of a solid day’s worth of work. Similar scope of assignment, more realistic estimates, all clarified upfront for people to opt-out on applying under, if they find the conditions too onerous.
What’s funny to me, though, is that there’s such a focus on the work required IF YOU MAKE IT TO THIS STAGE, rather than the work required to get to this stage. When we reject 97-99% of candidates on the basis of their cover letter and CV, you’d think that’d be where the initial focus would be.
Anyway, I detailed with incredible transparency how the process actually goes if you make it into the 1-3% of candidates that might see a programming assessment. So fair enough.
All the best with your career!
Jessica
> I can have a really great application and follow directions exactly, but those with the luxury of time will always have the advantage.
My daughter was 3 years old when I did the exercise, and I also had a full-time job. The exercise caught me flying back to Spain from Canada, in a 2-days flight due to accumulated delays. Considering a jet-lagged weekend, I had more like 3 effective days to complete it, while working for my former employer and being a dad and, working very sparsely, was plenty.
The sentence “the luxury of time” strongly resonated, as you can imagine the free time I had. As I elaborated in a comment below, I think the point of giving you 1 week to do a small exercise is that having more free time doesn’t give you an edge. That people with family can do it. That people working can do it. That was totally my experience at least.
Thanks for sharing what happened after the applications were submitted. I’m sure many people, including me, poured their heart into the application. It’s satisfying to get a peak into what happened afterwards, even though I didn’t make it.
Congrats to Nabeelah!
I second that
It’s a very deep insight. I was also thinking about platforms like Upwork can reduce so many things for a company or individual. I do provide services in Upwork and found it very useful for me in terms of hiring or providing quality services.
I don’t like this pattern how it’s becomes implied you still should spend 10 hours on the assignment. You say you don’t give extra grades for that, but it’s obvious that if everyone puts in 3 hours, and you put 6 hours, you’ll get a better code output. That way, everyone races to get more and more work done.
I don’t blame you specifically for squeezing candidates, but looks like it’s a flaw in this system!
That’s not at all obvious. Code does not magically improve by gold plating. Much code is made worse by needless laboring. We looked at work that’s been deemed “overcooked”. Jorge almost fell under that description!
Different coders work at different speeds as well. Don’t forget that. Plus if anyone thinks they won’t have to work over 40 hours as a programmer without extra pay you are wrong. Most programmers are salary or contract & esp salaried people work until each assignment is done.
When I was interviewed for my first coding job it happened during my last semester of school. I was able to show them some code from my final project which helped me get the job.
Good lord, this is an incredibly unfair approach to programmers looking for a job! Look at this from the perspective of the people you’re hiring. You say that you ask ~20 people to complete a programming exercise that you “estimate” will take 3-5 hours. On it’s own, 5 hours of focused coding is a full day’s work for a programmer. But let’s not forget that these programmers are competing with 20 other highly qualified candidates, and their work needs to be the best to have a chance of getting the job. Not to mention that expected time for software projects is notorious for being underestimated. No programmer who really wants the job is going to only spend 3 hours on these projects. I’m willing to bet that most talented, smart, motivated programmers (the ones you want to hire) are going to dedicate 2-3 8-hour days to working on these projects, for a 1 in 20 chance of getting a job. Multiply that by 19, for the candidates that don’t get the job, and you’ve asked for 456 (3*8*19) hours of people’s time for unpaid work that gets thrown away. That’s $30-50k of people’s time that you’ve wasted ($60-110/hr * 456). Now distribute this practice across every tech company trying to hire programmers and every talented, smart, motivated programmer looking for work. How much time do you think the average programmer can invest in their job search? The dozens of problems created in the industry by this hiring practice should be obvious by now.
Here’s a better approach: judge candidates based on their cover letter, resume, existing code samples, and references. That should be enough to determine at least 80% of the time if someone is a great programmer or not. If you get a lot of applications, narrow down your choices to the top 2 (3 absolute max) candidates, and offer them a PAID 1-3 week long trial contract working on a real project with you. At the end of that trial, offer the best candidate a full time position. Don’t steal time from the talented people who want to contribute to your company and then just throw it away 19/20 times simply because you think some arbitrary competition with loosely enforced time estimates and rules, evaluated in just as arbitrary a manner, is somehow going to result in you having the “best” programmers.
If spending a day auditioning, once you’ve made the cut from over a thousand candidates down to the 13 that got the assignment on the R&F team is too much effort for you, there’s a clear option: don’t apply!
We hire programmers for the long term. The average tenure at Basecamp is 6 years, several have been here for more than a decade. Our salary and benefits package is out there.
At 6 years, we will have spent more than a million dollars in salary and benefits on most programmers. If you don’t want to spend a day’s work auditioning for a job like that out of 13 finalist candidates on R&F, again, totally your call. But I have absolutely zero qualms about the odds, the payoff, and the effort.
Which is partly why we put information like this out there! Complete transparency on what it’ll take to land a job at this company. If that strikes you as overly onerous or unfair, and you chose not to apply because of it, thank you 😄. We’ve made the first cut easier! (The first cut is not amongst the people who apply but in who CHOOSES to apply).
Most people applying likely have a current job. How would they have 1-3 weeks to audition. A single day makes a lot more sense, as described in the article.
Not sure that I would call this approach outright unfair, but it’s clear that the solutions submitted by the hired candidates took far longer than 3-5 hours, and it’s disingenuous to continue to refer to this as a “5 hour” assignment. The pull request description and README updates from the first solution alone would take several hours to write.
Each candidate admits to spending roughly double the time, but that’s likely significantly downplayed to save face or not admit to what some might think of as “cheating”. I agree with your assessment that these solutions probably took 24 hours or more.
Here’s an experiment: next time you offer a take home solution that is supposed to take 3-5 hours, set it up so that the candidate has to submit their work 8 hours after first seeing the problem. This gives them several hours of buffer space and time to take breaks. I can’t be certain, but my guess is you would not see solutions anything like the ones linked above.
I can only talk about the first solution in the post, and I’m pretty sure it didn’t take “24 hours or more”. Also, other solutions we got were simpler, done in the suggested time, and really, really good. We moved 10 candidates to the next phase. As David said, being too elaborate and “gold-plated” almost played against Jorge instead of in favour, but thankfully we didn’t hire him based on the code he submitted alone, that was just another factor among others.
I find this unfair for many people who simply can’t block 3-5 hours on a given day to do a code exercise: people with full-time jobs, families to care for and a bunch of other life obligations. Having a full week allows spending 1 hour one day, maybe 90 minutes another day… which for many people is way more feasible than an uninterrupted bigger block of time.
> “I find this unfair for many people who simply can’t block 3-5 hours on a given day to do a code exercise: people with full-time jobs, families to care for and a bunch of other life obligations. Having a full week allows spending 1 hour one day, maybe 90 minutes another day… which for many people is way more feasible than an uninterrupted bigger block of time.”
Giving everyone a week is unfair to the very same people for the very same reasons. Someone who doesn’t have a full-time job, a family to care for, or other life obligations will be able to spend 40+ hours on a take home project, and as a result submit something closer to the gilded solution.
The fact is that any take home exam that is as open-ended as the tasks outlined above is going to be significantly more difficult for anyone with a lot of obligations. There’s just no way around it. At the very least, capping the task to an absolute 8 hour limit will preclude anyone from spending many multiples of the stated time. And if you think 8 hours is too long to ask for, come up with a 2-3 hour task and cap it at 4 hours. But, as David said, “If you don’t want to spend a day’s work auditioning for a job like that out of 13 finalist candidates on R&F, again, totally your call.”
@DHH
I live these kind of behind the scenes posts.
1. How do you perform hiring for system admins at Basecamp?
2. why do you use naked domains (no “www”) for Basecamp.com & Hey.com?
Doesn’t that create scaling issues?
I used to give a rather simple homework to sys.admin candidates right after the first contact. I estimated it takes about 1 hour to finish it if you do not know the topic at all and need to do research. It worked well as a fence to greatly reduce the number of candidates to talk to. We hired some great people this way who are staying with my team for many years now.
There was a sour apple though. We had a candidate with an excellent resume, went through homework in 15 minutes, passed all interviews easily. We hired this candidate and then he did not produce any results in three months no matter how hard I tried to reach out. It was an interesting experience; I am still mystified by it.
Our last sys.admin came out of a small pool of juniors we have for light-duty tasks. For many years I was against hiring juniors in my team and I recognize now it was a mistake. Some lack of experience is compensated with eagerness to learn and enthusiasm. Some of the “boring” tasks my seniors did not look forward to are getting solved in record numbers now. This young enthusiasm is spreading to the rest of the team. Turned out hiring juniors was not as scary as I imagined.
Hi Oleg
Do you work for Basecamp? I don’t see you listed at https://basecamp.com/about/team
No, I don’t.
Is the enthusiasm you note a product of being in the field for less time, or because the juniors are younger?
In other words, would you consider older juniors who have switched careers?
Congrats to Nabeelah! Totally agree on the hiring process, only would probably not omit the requirement to keep the code also accessible to the impaired users (while still avoiding frameworks and whatnot). Anything on the front-end should have that in mind. Basecamp is the pinnacle of inclusivity when it comes to its employees, and I think it should extend also to its users. Accessibility is not difficult, nor it breaks any standards. It just should be kept in mind in all stages from design to coding.
>” Basecamp is the pinnacle of inclusivity when it comes to its employees”
Is it?
It’s doesn’t appear Basecamp us any black employees. Not trying to troll, just pointing out that’s a major demographic missing for your “inclusive” statement.
https://basecamp.com/about/team
I like the take home small assignment approach it emulates real-life. Just want to comment that I am a junior Go/js programmer i am the age of 43 (Retired from the military). I just hope that being junior and older isn’t a big issue as I embark on this career.
My situation is similar to yours. After some research, the impression I get is that an unfortunate percentage of companies just won’t consider you — thinking that you’ll either lack the endurance or inclination to work long hours; or you’ll be distracted by family obligations, etc; or you’ll be insufficiently subservient; or you’ll be dissatisfied with what they pay juniors; or that you’ll simply be too cognitively calcified in your advanced age to be brought up to speed rapidly.
But it also seems to be the case there are plenty of places that are not that way. Often the person in charge of hiring has had a previous good experience with older entry level hires, or they are otherwise conscientious about not allowing such biases to fester in their organization.
I suppose the trick is figuring out how to not waste your time with the prejudiced companies.
The condescending tone in DHH’s responses to justify the approach, is classic, and really shows the level of arrogance that’s in many tech companies. Just because a company puts their handbook out for everyone to see, doesn’t mean they have a good working environment.
This tells an underling story though…
“Jessica, you’re right in the sense that our targeted range for the implementation was off. WELCOME TO SOFTWARE DEVELOPMENT 😂. This is basically how it works! Estimates suck.”
So Basecamp has been doing software development for over a decade, yet was unable to properly estimate this feature. A feature which was for their upcoming email service, that I’m guessing they will be charging for. Was that piece of information conveyed to the applicants?
But for argument’s sake, let’s say the feature was already implemented in the service and they had no intention of evaluating the free work to improve on the feature. How is it then that a proper estimate couldn’t be provided?
Try to spin it however you want, but it’s hard to justify that the process was fair. And to then attempt to push the blame onto the applicants by telling them they shouldn’t apply or that they shouldn’t be in programming, is simply bad business.
If you tell candidates that the assignment is 3-5 hours of work, and they adhere to that time limit, their solution should be weighted the same as those who went over. It could even be argued that those who stayed within the timeframe should be weighted higher, since they followed the time requirements that were outlined. But when reality is that double that amount of time is actually required, that underestimation is not on the candidates. It’s not the candidate’s responsibility to decipher the hidden rules.
The process was flawed. Own it, learn from it, and do it right the next time, instead of making excuses and insults.
Quite agree. The responses don’t speak well.
Whole I generally like the approach…
Cover letters don’t tell much useful info that couldn’t be garnered in the first 5 minutes of a phone screen.
A 3-5 hour task is too large. It should be 1-3 hours at most. A single evening at most should be the expectation and all that’s require. If you can’t get a good sense of their abilities from that then you have a bigger problem.
While making it a more real world scenario, it shouldn’t be something that contributes to your business in a form. That’s just wrong and nearly as bad as a 20-40 hour project.
Even if someone doesn’t have the time to contribute to open source, they can still pull together a couple small things to out up on GitHub, BitBucket, GitLab, etc to build their own little portfolio.
I would venture to guess that many of the commenters here critiquing and nit-picking the approach have never actually managed a hiring process or sorted out a pile of hundreds of resumes of possibly qualified individuals. Or have been burned with a bad hire who looked good on paper but had very little actual ability to complete work on time and with high quality. There’s simply no silver bullet process to this and DHH’s / Basecamp’s is as reasonable as I’ve seen or designed myself. By the way, love the use of public Gists for the take homes. That has served me well to help the internal team design and review the test itself.
For all the people criticising the hiring process of Besecamp – please read this https://www.jarednelsen.dev/posts/The-horrifically-dystopian-world-of-software-engineering-interviews first and then come back and comment again. Whatever flaws the Basecamp approach has, I’ll take it anyday of the week, compared to what Giant Search and Advertising Company and Giant Social Media Platform Company have do.
I am actually a bit surprised that you use this methodology. Mainly because I think it goes against so much that (I think) you stand for.
In blog posts and books you argue over and over how careful we should be about other people’s time and you sum up meeting time just Jo Sprague did. Still, you don’t seem to see the problem with asking 20 people to put down lots of time that result in nothing.
In my opinion, it’s nothing but naive to think that people will spend 3-5 hours on a home assignment if they are able to spend more. My experience also tells me that a “3-5 hour home assignment” usually takes at least twice the time. And then people (at least I) double that if they want to stand out of the crowd. I think you have reason to believe that when people say they spent 7 hours on a task they might actually have spent 20. They have all the reasons to lie. Do you still think this is a good method?
You strike me as a company that do thing because you think they are “right”, not just because you can. However, in this case, I get the impression that you do this just because you can. You emphasise this by saing “if you don’t like it, don’t apply”. And why should you do things differently? You still get 1300 applications. Well because, to me, this is greed. Taking a lot, probably a lot more than you need, just because you can.
A recruitment process is not a well-balanced relationship. You’re not equal. A lot of companies say “it’s just as much us winning you over as it is you winning us over” but we all know that’s not true. Not with 1300 applicants. Acknologing this and trying your best to level this relationship is rather something I would expact from Basecamp. To say “We understand that we have all the power but we’re not going to abuse it. Because that’s not who we are. We understand that this (asking a lot of people to put in lots of hours resulting in nothing for all but one) is pretty much how the industry looks like. But we are not like everyone else. A recruitment is a commitment we hope to last for a long time and it’s important that it’s an equal give and take from all parties. That’s not starting from day one but from day zero.”
I’m not sure you want suggestions to what I think the process should look like. If you’d like, I’d happily provide it. But I hope you at least “Give if five minutes”.
Well-said, excellent comment. And thanks for the link
By the time we get to the programming assignment, the team involved with the hiring process will have spent far more time than any individual candidate have on this search. It takes a tremendous amount of time to manually filter hundreds or thousands of applications down to a dozen or so finalists. We are not asking for anything in return that we do not put in ourselves.
This idea that it’s “for nothing” is bizarre. It’s the commitment required to get a shot! And what a shot! By the time you reach this stage, you’ve made it into a tiny selection. Now, as with the SIP position, you have a really strong chance to land this position. You’ve gone from many :1000 to 1:13.
Except, of course, this isn’t a lottery! It’s a chance to demonstrate your skill and your character through writing and programming. It’s an opportunity to sidestep the traditional hoops of credentialism or traveling for interviews or dealing with recruiters or any of the other gates presented by other companies that we don’t employ.
We ask nothing that we aren’t putting in ourselves. Not outsourced, but by the actual team of coworkers you’ll be working with afterwards. They’re the ones reviewing applications and judging submissions. They can sniff a gold plating as well as any other programmer.
Take your chance or don’t take your chance, but if a day’s work is too much for you when you’re within striking distance of the job, I’d rather you don’t.
I think a technical exercise like the one I did for Basecamp has 3 benefits:
– It validates candidate skills in a fair and objective way. Asking for references is not objective. Asking for open source is not fair, not everyone does it. CVs are essentially meaningless. This was about solving a real problem in a week.
– You could do it calmly in your own environment, working at your own pace, without the stress of someone watching you while you do it.
– The problem was small enough to make it perfectly doable by someone with family and a full time job in a week (like it was my case).
But all these things are obviated and many commenters focus on the fact that you could spend more than the estimated 3-4 hours doing the exercise. So what? How is that unfair? So, do you expect candidates to start a timer and stop working when it rings? My only concern would be the inverse: that the project was so big, that people with more free time had an edge. That wasn’t the case at all.
We were given 6 days to do the SIP exercise. The 3-4 hours was an estimation they gave us, never a constraint. The didn’t tell us: make sure you don’t spend more than 3-4h. What’s the point of doing that? Stress people out? Measuring who is better in pretending you worked N hours exactly? I really think, of all the things you can extract from Basecamp hiring process, the fact that people are focusing on this really speaks very highly of the whole approach.
As regarding getting paid for your time, the project was small enough to not offer any payment, in my opinion. A 1-week project would be a different story.
If you are going to apply for a position at Basecamp, you really need to be willing to play a game where odds are low and the effort will be huge. This means you are likely to invest a ton of time and get nothing. I know this because I had a history of working hard on applications (like multi-days hard) just to never pass the first filter. I considered the possibility of doing the technical exercise and not passing a likely outcome.
I think a take like “they are making me waste my time if they don’t select me” is pretty negative and self-destructive. I’d rather go with “I have a chance to get my dream-job, I am going to give all I have and, if it does not work, at least I did everything that was under my control”. To me, how to deal with this emotional drain is a far more interesting discussion. And I don’t think getting paid a fee for a few hours of work will help with that at all.
Plus, you can say,”I worked on an interesting technical problem”. Which, after all, is in our DNA as software developers.
To me the core issue of saying this should be 3-5 hours and it taking 2x more, and those people getting the job is that this goes against what DHH and Basecamp seem to preach, which is that you work 8 hours days, not overwork.
This is saying if you put in 2x the effort and time then you are more likely to get the job. How is that different than someone spending 12 hour days to get promotions? Does that not put pressure for others to do more?
This is the same argument VCs and other use. “It’s worth the 12 hour days because you are getting paid 300k”.
It’s fine if you hire someone doing more, but you have to realize this is similar to what VCs and other businesses with demanding but high paying jobs say. Put in the work, if you think it is worth it
The way the OP repeatedly and openly dismisses all of the legitimate criticism that others have raised is a huge red flag. Why would I want to work for someone who is so incredibly narrow minded in their approach?
It is a seller’s market right now, the demand for quality programming talent significantly exceeds demand which means that tests, such as the one proposed by the OP, generally do more harm than good. Sure, you might weed out some unqualified candidates that were able to slip through the initial screening process but you simultaneously pushed away anyone who isn’t interested in a time sink.
Make no mistake, we don’t need -you- to hire us. The vast majority of truly qualified candidates not only already have repatively cushy jobs, they also generally have a line of people waiting for them to be available.
*the demand for quality programming talent significantly exceeds supply
Should’ve proof read before pushing the post button.
*relatively cushy jobs
And another one.
If one absolutely insists on making applicants write code, then have them attempt to fix a bug or implement a feature in your code base and submit a PR; then pay them fairly for it. Hell, the submission doesn’t even have to be to your own code base, it could be for an open source library that you depend on. The idea here is that we’re no longer burning valuable programming hours on code that doesn’t actually interact with the world.
> there’s a clear option: don’t apply!
First, I’m a big fan of DHH/Jason and team since the early 2000s. Lot’s of great stuff they’ve contributed to business thinking over the years–big and small. Thanks for that.
With regard to this thought exercise in hiring programmers with take-home assignments, I find a lot of the responses from DHH to commenters points flippant. I’m not a programmer, PM rather, but have recently been treated to the “only 3-5 hour” set of assignments as part to a job interview. First time in my 20 year job history I had to do it and thought, “well, maybe that’s just how it’s done now.” The amount of time I had to actually spend on my assignment was easily over double. As a candidate, you want to be thoughtful so you naturally spend more time, even if you don’t have to. If you don’t get the job, you’re left with a pretty shitty feeling–not just of losing the job but a bunch of hard assignment work down the drain. I don’t think I will ever again accept such a take-home assignment. My guess, some percentage of your future engineering candidates will think the same and simply not bother applying. If you want to hire the best, I doubt it’s a good idea to lose some percentage of your candidates from even applying.
Interviewing is hard and of course any employer naturally wants to reduce false positives. But asking candidates to jump through extra hoops without compensation for work outside a “normal” interview loop is demeaning. I think your policy for paying interviewing designers is the way to go. If having engineers complete a programming assignment is critical for your interview workflow, then compensate them the same way you do your designer candidates. I believe your firm could easily afford it.
I had many companies asking for projects before. i did one. for a company called iolo back in 2004. spent 20 hours for free, got a low-ball offer.
That was the last time i did anything free for anyone.
Projects are fine as a part of a job interview, but, only if they are paid.
The problem is, recruitment is *really* hard on both sides. Even before you account for the power imbalance due to demand > supply (or occasionally supply > demand), how can you as an applicant demonstrate everything you are and how can you as a company identify who will be a good fit in a short amount of time? While each side will spend a lot of time on their respective processes (hiring or applying), the time spent between one applicant and one company is surprisingly short given what it leads to (working the majority of your waking hours for n years, and the ripple effect that work has on everyone’s personal lives). That’s not even including the fact that you are only looking at the subset of the population that are looking for a job at the specific point in time that you are looking for a candidate!
Kudos to Basecamp for opening up about their hiring process (a lot of companies don’t), acknowledging the current pitfalls (eg brain teasers, theory questions) and for trying something different. However from my experience on both sides of the process, I would suggest three improvements.
The first is to level the playing field of your coding test. As a company that holds “fixed time, variable scope” as a priciple, it’s strange that you are not at the very least weighting each application against the time they (at least they say they) spent on it. A 3-5 hour task is not too far off the time spent for a normal interview (get ready, travel there, wait, interview, travel back), but I would make it real-time for the candidate – just ask them what time they would like the task sent, schedule an email and ask them to send back their work 4 or 5 hours later. That way at least you can compare the quality of each application against the same timeframe (aside from making it harder to compare quality, letting them spend 2-3 times as long just means they will have to spend 80-120 hours a week delivering the quality you are expecting if they get the job).
The second is to be clearer in your expectations. Do you expect unit tests? Do you expect documentation? The worst thing you can do during recruitment is award credit to candidates who guessed correctly what you would like.
The third is a bigger focus on character. In my experience, the key to a good hire is a 60:40 split of attitude to skills, only because it’s harder to improve attitude than it is to improve skills. The best way i’ve found to measure this is a 30-60 minute chat (video/face to face is better than telephone so you can read body language) against a rough set of questions (so you can compare against other candidates) as it’s rarely about what they say and more how they say it.
On a side note, what amazes me the most is that a lot of companies don’t quantify the time spent on each hire, especially those who are keen to keep costs down. The cost of staff writing up an advert, getting budget approval, sifting through applications, doing (sometimes multiple) telephone interviews, doing (multiple) face to face interviews, discussing applicant’s suitability and finally the onboarding process which sucks a lot of time out of everyone in the team. If you added all those hours up and worked out a salary cost to the business of hiring one person, maybe giving that previous person a raise would have been more cost effective!
Just my two cents, YMMV.
Thanks for your feedback, Mike. This post was mostly about the coding exercise, we didn’t detail the other parts of the hiring process. The coding exercise was just one more piece, after it, the hired person went through 3 more interviews, a 30-minutes one and two 60-minute ones, video face-to-face. These were more important than the code exercise. The exercise just unblocked the path to them. We did also take into account the amount of time and the amount of “gold-plating” added to each solution submitted. I was involved in the SIP opening and when I ranked solutions, my top one was one of the simplest submissions we got, where the candidate had spent under 5 hours on it. Throwing hours at a programming task doesn’t necessarily makes the solution better. In many cases, it makes it worse. I do agree that we should have been clearer on our expectations about tests, documentation, etc. We’ll definitely do this better next time and will also improve our time estimation.
We considered doing the real-time approach to this, but we discarded it because it makes hard for candidates who can’t easily block 4-5 hours on a given day to work on a task: people with full-time jobs, children or other family members to care for. This was precisely the case for Jorge, he wouldn’t have been able to do the test in this way, but rather splitting the work in 3 days, working a bit each day. I think to give a full week benefits more people.
Thanks for your response, and I get your point about candidate availability, I was assuming that most candidates would be able to set aside 4 hours within a 3 week period and think the benefit of having a fixed time to accurately compare candidates outweighs the inconvenience (it also removes bias towards candidates whose desire for the job is greater than their ability).
I also agree that throwing hours at coding is not always beneficial but I think most developers would love an extra hour to tidy up/refactor before submitting!
If you want, you could do the first 30 minute interview before the coding task to identify candidates who might not have the strongest coding skills but a better attitude/character than others? Or do you find this is less relevant in a company that works remotely?
Regardless, your process sounds a lot better than most companies so thanks for sharing.
@DHH
Thanks for these insights.
Did you use any tools to support the take-home assignement process? If yes would be interesting to hear your experience on that.
Currently we are working on such a tool trying to make the take-home assignment process as smooth and fair as possible for both sides. CodeSubmit, check it out if you want.
Greets
After reading this I’m most interested what you consider a good cover letter. From the job seeker side I’ve had the impression my cover letter wasn’t looked at. From the hiring side I have used some HR software where its a bit of a pain to even see the cover letter so I’ve been a bit skeptical how effective they are as I am now searching.
Take home code all the way. Thats what we do at work, write code and review code.
I’ve been a big fan and proponent of the materials that Basecamp has written about both project management and having a good corporate culture. Long time listener, first time caller.
This post felt like a gut punch.
I know firsthand that hiring is a time consuming and difficult endeavor; I know all too well the cost of making the wrong decision. But this process is a race to the bottom and antithetical to so much else written about the company’s culture.
Everything about take-home assignments incentivizes spending as much time as possible on it. 3-5 hours is a recommendation, but it’s more than a wink-and-a nod to put in as much time as possible. The time limit is not monitored and it’s a competition. I agree that gold-plating only goes so far, but someone with more time can do more research, work out alternate solutions, read and write more thorough documentation, etc.
The thirteen people should all be around the same technical competency by virtue of having gone through several rounds of elimination prior. With skills being relatively close, time spent becomes even more disproportionately important. The fact that the person you hired spent more than the allotted time on the test despite you suspecting it beforehand only further emphasizes that. It’s the same sort of mentality that leads people to stay long hours at the office because of an unspoken rule that you need to be seen putting out “more effort”.
You propose this process is a beneficial way to sidestep in-person interviews, but I look at it as detrimental. I can’t stay extra after an in-person interview, polishing my answers until I get them more right. I’m a design manager and your comment about 10-15 hour design challenges made me wince. I don’t know what I could possibly learn from that which shouldn’t have been learned through portfolio reviews, discussions, and high-level whiteboard exercises prior.
I don’t know much about your background. I don’t know what you’ve had to go through in the past, and I don’t know when the last time was that you had to find a new job to afford rent or buy food. But a comment like “if a day’s work is too much for you when you’re within striking distance of the job, I’d rather you don’t” makes me sad to read.
You’re not within striking distance. You’re up against a dozen other people. Maybe you’re applying to tens of jobs at once and trying to prepare for as many interviews as possible. Maybe every job you’re applying to is giving you a take-home test where you know you need to put in extra hours to get a leg up over the competition.
Maybe they all say “don’t apply if you can’t do this”.
So you don’t get any job.
I suppose it could be worse. You could have been one of the 13 people who didn’t get the job and who poured in ten, fifteen hours on the assignment that could have been spent instead applying elsewhere. That the eventual hires are here defending it is like a lottery winner advising everyone to play the lotto because clearly it works!
You can make the point that this sort of assignment favors people with disposable incomes or those without families. You could argue – and many have – that these assignments provide little gain that a good, structured interview process wouldn’t have provided earlier. I’m not convinced it’s good tactical decision — you end up hiring the person with a combination of skill and disposable time, not necessarily the best skilled person.
There are a number of people here writing that this feels wrong for a lot of reasons and particularly untrue to the spirit of your other writings. And hey, it’s not our company: it’s yours. But for the good of those who spend time going through your interview process, I hope that these replies give you some pause and an opportunity to reflect on the negative ramifications that real people deal with going through this sort of interview process.
I’m stubborn and I’ve been in my fair share of rooms with people politely trying to suggest that I rethink my stance instead of digging in my heels. But I don’t. It’s usually months later that I realize I should have taken a step back to, at the very least, get a new perspective.
Cheers and best wishes.
> That the eventual hires are here defending it is like a lottery winner advising everyone to play the lotto because clearly it works!
Since it was me the one talking about low odds, let me paste what I wrote again:
> If you are going to apply for a position at Basecamp, you really need to be willing to play a game where odds are low and the effort will be huge. This means you are likely to invest a ton of time and get nothing
If anything, I am suggesting the opposite ☝️: be realistic, don’t apply if you can’t bear with investing time and not being selected. Prepare to be rejected if you do.
By the way, odds are low when you have many applicants for a single position. It’s completely independent of the hiring process. The problem with your “lottery winner” is that it obviates other factors but, of course, luck plays a role when you have to select one human being out of 1000.
Here’s the problem with “don’t apply if you don’t want to invest the time”: what happens when more and more job applications move that direction? This mentality — and blog posts promoting the framework — have ramifications outside of this one company.
I’ll give two examples; one close to home for the topic, and one more contrived but in the same spirit.
A few prominent tech companies propagate the “leetcode” style technical tests. Some, appropriately, question the need for this. Is it providing value, or is it really just a burden on applicants? It doesn’t matter, because if you don’t like it, you’re told to apply elsewhere.
But then more companies start adopting the approach. Not necessarily because it’s right or proven to be helpful, but because it’s been popularized.
A few years later, the majority of programming job openings require this, despite evidence to the contrary that it results in better employees or retention rates. And so new graduates and applicants spend countless hours “grinding leetcode” for something that has little practical effect on problem solving or becoming better employees. Now it’s just part of the game because folks said “if you don’t like it, don’t apply”. And, well, the whole point of a take-home test is supposed to be a rebuttal to that interview style in the first place.
The second example I promised is more contrived, yes, but is an extension of the same logic. A big factory opens with lots of new jobs. The pay is above average, but the hours are grueling and employees are held to standards that put them at risk. They must work in potentially unsafe conditions, at a rate of productivity that can, at times, endanger them. Third-party advocates push back, and the common response is “If they don’t like it, they should just apply elsewhere”
But there’s a scarcity of jobs and there’s good pay. So people take the risks. And, over time, more companies do this because they see it working so well. If they can get away with it by saying “if you don’t like it, apply elsewhere”, why shouldn’t they? So the standards of all places fall over time, because the incentive has pushed the trend downward.
That’s what I meant by my comment about it being a race to the bottom. It’s a similar, though not perfect, comparison to The Prisoner’s Dilemma or the Nash Equilibrium. My point is, coming here to effectively say “Well, I got mine because I put in the [unnecessary extra] work” is exactly the problem.
I’m happy you got the job — really! But here, individuals become incentivized to make an unbalanced risk-reward commitment. Over time, other places will adopt this until it becomes the norm. And ten years from now, we’re stuck with the same sort of problems we have now with tech job interview processes. We’ve just traded one problem for another.
Of course the odds are low with any position like this. The odds are low for *most* open positions and that’s part of the unbalanced power dynamic here. This framework is on the low end of “how badly do you really want it?” for a job opening — but when more folks adopt it and competition gets stiffer, the needle is going to move.
Perhaps the lotto analogy was poor. It’s more like it becomes a blind auction where you’re bidding your *time*. When you can’t see what other folks are bidding, you’re going to be incentivized to overbid. That seems worse.
While still imperfect, I’d be a lot more comfortable seeing a hard, policed cap on time spent. Then it’s at least more of a level playing field.
Great work! I’ve been taking a very similar approach in our 100% company and it’s nice to see it validated from a company I really respect. Thanks for sharing!
100% remote company*
Use take-home coding tests in Java, C#, JS, PHP, Python and more to identify exceptional software engineers easily.
Thanks for the article.
You can visit here for more- https://semidotinfotech.com/
Great to read the effort you take to have real fit developers joining your teams! I was impressed by the number of initial applications you receive first hand. I wonder if the reading of cover letters could be replaced by a reliable online assessment that applicants would do instead. Such an online assessment would contain successful moments at work that candidates can choose from, and thus show their preferred work culture. Based on the requirements around the job, you could shortlist then only those with a ‘cultural fit’ and then continue with the coding assessments etc. What do you think? If you’re interested we can talk, we could make that work together 🙂