What’s new in Basecamp 3 for iOS

Hello from Team iOS here at Basecamp! We’ve been hard at work making Basecamp better for you, here’s just some of what’s new…

Better sharing
Basecamp 3’s share extension has been redesigned and powered-up. Post a funny GIF in a Campfire, Ping Zach that URL, or drop a PDF into Docs & Files all without leaving the app your’re using or launching Basecamp at all! Basecamp even remembers the places you share to most. If you haven’t yet made this part of your Basecamp workflow—definitely give it a try.

Share from Photos or other apps on your iPhone, then choose where to save in Basecamp.

Pin me!
Now organizing the Basecamps and Campfires you use most is much easier. Pinned Basecamps and Campfires stick to the top of the list for quick access—just tap the pin. Now, for instance, you can easily jump into a Campfire even if you’re not following it.

Pinned Basecamps and Campfires stick to the top of the list for quick access.

More control over what’s new
Our Basecamp is a busy place, every day there are tons of new notifications about new Pings, comments, @mentions or Campfires. Swipe on any unread or recent item to see more options for dealing with it quickly. Swipe and Mark as Read to clear it or tap Unfollow to prevent any further notifications from the thread. Make a mistake? You can always swipe to Mark as unread and pop it back into the unread list.

Swipe from the right for quick actions: Unfollow, Mark as Read, or Mark as Unread.

More fun!
Basecamp is a serious tool for getting work done but that doesn’t mean you can’t have a little fun with your team. Basecamp 3 for iOS now supports third-party image keyboards like Bitmoji, KIMOJI and all of your various moji-based keyboards (you’re welcome, millienials!). In fact, you can simply paste images (Hello animated GIFs!) right into Pings and Campfires, too.

Who says work can’t be fun?

And that’s not all…
Swipe-to-go-back is even easier (just swipe toward the right on any screen to go back, not just from the left edge)… see who’s been talking in new Campfire activity… a brand new flow for starting new Pings (even with multiple people)… plus tons of bug fixes and behind-the-scenes improvements to speed and stability.

See who’s talking in Campfires (left), Swipe toward the right to go back (center), easily Ping even several people.

As always Basecamp 3 for iOS is free, get the latest version with all these updates in the App Store. Thank you so much for using Basecamp, for your kind words, and great suggestions! Stay tuned, we’ve got a lot more in the works—let us know in the comments what we can do to make Basecamp even better for you!

♥️ Team iOS,
Dylan Ginsburg, Zach Waugh, and Jason Zimdars


Basecamp 3 works where you do on iOS, Android, Mac, Windows, and anywhere you’ve got a web browser and an internet connection. Your first Basecamp is completely free so try it today, it takes just a minute to sign-up.

Fighting the ticking clock in your head


I have this weird character flaw — I have a clock in my head that’s always ticking. It’s a mechanism I’ve built up over the years because I hate wasting time.

In some ways this internal ticking clock is a really handy tool to have. It gives me a healthy respect for time, which means I’m rarely late, I’m generally pretty productive, and ship work in a timely manner.

And when it comes to shipping work, that mental clock can be really handy too. I always have a sense of how far along I am in my time budget — it sharpens focus and is an excellent tool for hammering scope. It’s a perfect way to find out if I can get 80% of the feature for 20% of the work.

On the flip side, the problem with the internal ticking clock is when things don’t go as planned. Sometimes I’ll spend a couple days of a two-week cycle exploring solutions for a problem, only to find that none of them work. All of a sudden I’m 20% into my time budget and no closer to a working solution or shipping the feature.

At that point my internal clock starts yelling, “you’re wasting time!” It starts getting louder and louder in the back of my head. It can become a major distraction unless I quiet it down.


Knock out the clock by shifting your perspective

If you’re like me, you’ll never be able to turn your internal clock off completely. You can’t just pull the battery on your brain. The best thing you can do is just knock it out for a while. 👊⏰

When my internal clock starts getting loud, there’s one thing that has consistently helped me: I temporarily shift my perspective on how I’m measuring progress. Instead of focusing on feature completion, I focus on the other value my work is creating.

This is helpful because it actively moves you away from the idea that value can only be measured by how much of a feature is working. It’s certainly important to ship features in a timely manner, but there are so many other subtle dimensions of work that are valuable uses of your time. 0% functionality after a day of work sounds rough, but a 100% better understanding of a handful of APIs sounds like a good deal, right?

To help me focus on the other valuable aspects of my work, I’ll usually take a minute to ask myself these kinds of questions:

  • Am I learning anything new? Almost certainly. Every new piece of work through my entire career has been opportunity to learn. If you leave any piece of work saying “I knew it all, I got it totally right”, then you’ve probably missed some key learning moments.
  • Has this been valuable, productive time with my team? “Productive” and “valuable” can mean a lot of things. Maybe by talking with your team you came up a clever new solution nobody has ever done. Or maybe you never solved the problem, but you had a ton of fun working with your team. All of those count as valuable and productive in my book.
  • Have I had at least one sharable, teachable moment? If you’re learning regularly, you’ve probably found at least one thing you can share. It doesn’t matter what scale you share at — at your own company, on Twitter, or at a conference talk. Getting out there and sharing is another way for you to learn and reinforce the importance of your daily challenges.
  • Am I identifying any weaknesses I can work on? Even if I’m not producing a finished feature, I’m almost always finding things to improve. It might be a UI tweak, code clarity, or new tooling. Or maybe I’ve identified a weakness in my own knowledge base that I need to improve. It doesn’t matter what it is, if you’re improving anything, it’s time well spent.
  • Am I prioritizing quality? Quality is a broad term, but we take it seriously at Basecamp. If I ever feel rushed by my internal clock, I always make it a point to stop and consider this. Some level of time pressure is healthy for shipping, but not at the expense of quality.

At the end of the day, shipping a feature a few days later is rarely going to have a significant impact in the long run. So if your internal clock is getting on your nerves and causing you to rush, take a breath and put it aside. Consider all the other value you’re creating with your time — I bet it’s a lot more than you might think. 🤘


We’re hard at work making the Basecamp 3 and its companion Android app the best it can be. Check ’em out!

The Majestic Monolith

Monolith by Rene Aigner

Some patterns are just about the code. If your code looks like this, and you need it to do that, here’s what to do. You’d do well to study such patterns, as they give you a deep repertoire of solutions ready to apply and make your code better every time you hit their context.

Then there are other patterns that are less about the code and more about how the code is being written, by whom, and within which organization. The Majestic Monolith is one of those patterns. But before we dive into all its glory, let’s first examine its opposite pattern: Micro/services oriented architecture.

M/SOA is a prescription to break down an application into many smaller parts, run each of these parts as their own application, and then let the constellation solve the grand problem you really care about.

This is a great pattern. No, really. Not being sarcastic here. If you’re Amazon or Google or any other software organization with thousands of developers, it’s a wonderful way to parallelize opportunities for improvement. Each service can be its own team with its own timeline, staff, and objectives. It can evolve independently, at least somewhat, of whatever else the rest of the constellation is doing.

When you reach a certain scale, there simply is no other reasonable way to make coordination of effort happen. Otherwise everyone will step on each other’s feet, and you’ll have to deal with endless merge conflicts. (Well, at least in theory, I hear Facebook is having a great time with a monolith, whether it’s majestic or not is a different discussion).

In other words, M/SOA fits the organizational shape of very large corporations. So far so good!

Where things go astray is when people look at, say, Amazon or Google or whoever else might be commanding a fleet of services, and think, hey it works for The Most Successful, I’m sure it’ll work for me too. Bzzzzzzzzt!! Wrong!

The patterns that make sense for organizations orders of magnitude larger than yours, are often the exact opposite ones that’ll make sense for you. It’s the essence of cargo culting. If I dance like these behemoths, surely I too will grow into one. I’m sorry, but that’s just not how the tango goes.

This is true of not just technical patterns, but general organizational approaches too. But that you shouldn’t run HR like a 50,000-person company when you have 50 seems obvious to most though (with some exceptions).

The problem with prematurely turning your application into a range of services is chiefly that it violates the #1 rule of distribute computing: Don’t distribute your computing! At least if you can in any way avoid it.

Every time you extract a collaboration between objects to a collaboration between systems, you’re accepting a world of hurt with a myriad of liabilities and failure states. What to do when services are down, how to migrate in concert, and all the pain of running many services in the first place.

As I said, all that pain is worth it when you have no choice. But most people do have a choice, and they do have an alternative. So allow me to present just one such choice: The Majestic Monolith!

Having your system described as “monolithic” is usually a point of derision. Them be fighting words amongst many programmers! I say don’t just turn the other cheek, but embrace the monolith with pride and a salute! Don’t just accidentally waltz your system into a monolithic design, do so with intent and with your head held high. Any monolith worth erecting is worth making majestic!

So what is a majestic monolith exactly? It’s an integrated system that collapses as many unnecessary conceptual models as possible. Eliminates as much needless abstraction as you can swing a hammer at. It’s a big fat no to distributing your system lest it truly prevents you from doing what really needs to be done.

Enter the case study: Basecamp

I’ve been writing Basecamp as a majestic monolith since 2003. The latest iteration, version 3, takes this pattern to new heights and renders under its domain not just the web, but all of our native platforms as well.

Basecamp 3 is available via the web, as native mobile apps on iOS and Android, as native desktop apps on Windows and Mac, and through email as well. That’s a lot of platforms to juggle concurrently! And I believe the only possible way to do so with a small team of ~12 programmers is do explicitly choose and commit to The Majestic Monolith, in all its controversial glory.

In summary, the pressures that drove us to embrace this pattern are as follows:

Basecamp is a large application. There are literally hundreds of screens of various kinds under its domain. We have 200 controllers with a total of 900 methods! This combined with a model of 190 classes with some 1473 methods. And that’s just what’s directly inside app/*, not to talk about our front-end, like the Trix text editor.

Basecamp is a small team. As I mentioned, we have just 12 programmers, and many of those are busy keeping the systems we’ve been creating over the last decade operational. In addition, we have just 7 designers (counting Jason, my partner and our CEO).

Basecamp is available on 6 platforms: Web + iOS + Android + Mac + Windows + Email.

Basecamp has millions of users and a back catalogue of many apps we’ve committed to maintaining until The End of the Internet.

So a very broad scope, deep commitments, and a very small budget, all comparably speaking. The necessity of this situation simply isn’t compatible with a highly labour intensive pattern like M/SOA. It’s not compatible with writing 100% native apps, but it fits the hybrid application model like a glove.

In addition to these formal constraints is the mission to write beautiful, understandable, and succinct code. Code that not only makes us smile while we write it, but also when we later have to extend or patch it. Such a mission is simply not compatible with an accidental monolith. It just gotta be majestic.

One of the benefits to the majestic monolith is that it basically presumes that the people who work on it also understand it. It’s much easier to silo knowledge and responsibilities with a proliferation of smaller systems. We’ve had that happen for the few external, shared services we do have, like Basecamp ID (shared authentication for all generations of the Basecamp app). “Oh, you gotta talk to Jeff about that”.

This in turn puts immense pressure on making the application understandable and changeable by individuals, not teams. Which in turn again forces you to take the time to turn a long letter into a short one. If your standards are slipping, you’re not just pissing in your own corner, you’re pissing all over that majestic monolith we all have to polish every day. Don’t do that!

In many ways, this goes right to the essence of Ruby and Rails’ perception of programmers. That given the right incentives and nudges, most will rise to the occasion and write beautiful code. That our systems should be a red carpet invitation to become a better, more productive programmer. Who cares if those who aren’t invested in walking that path use the tools and patterns to screw themselves over? Maybe everyone needs to do that once or twice before appreciating the ropes that guides you into the gala premiere.

The Majestic Monolith doesn’t pretend to provide a failsafe architectural road to glory. That’s a fool’s errand. Many programmers suffering under many oppressive influences will turn any architecture made with any tool into a big pile of mud. So worrying too much about how to save those who’s likely out of reach anyway isn’t a productive use of energy. And it takes all the ones who do care down the wrong path.

TL;DR: Run a small team, not a tech behemoth? Embrace the monolith and make it majestic. You Deserve It!

Two new email reports in Basecamp

Until today there’s only been one way to stay caught up with all the activity on your Basecamp account, and one way to see what’s on your plate—both required you to log in to Basecamp and check yourself. Now you can have each automatically delivered to your email inbox.

Stay caught up with the daily activity email

Get a daily summary of all the activity across your account—even if you weren’t directly involved in it. Now you’ll get a daily email summarizing everything that happened since yesterday morning. If you’re working with clients, activity on the Clientside will be front and center with a special callout. After that you’ll get a rundown of all the new stuff that was added, and finally any discussions that took place.


Stay ahead with the weekly assignments email

If you’re like me and use assignments to keep you on track, you’ll enjoy starting each week with a full email report of what’s on your plate. Any assignments due this week or overdue — hey, it happens! — will be called out at the top for you.


Starting and stopping your email reports

Decided you don’t need these anymore? Just tap the link to “turn this email off” at the bottom of your email report or visit the Latest activity screen and the My assignments screen and click the orange button that says “Emailing me…”. Want it back on? Just click the “Email me…” button again.

Send me a daily email of what happened

Send me a weekly email of my assignments

We hope these new email reports make it easier to stay caught up, and stay ahead—all without even logging in. Happy Basecamping!

Basecamp 3 for Mac and Windows is here!

One of the best things about Basecamp is you can use it almost anywhere. All you need is a web browser and an internet connection. Laptop, phone, tablet, hotel lounge, school computer lab, Mom and Dad’s den — you know, the one with the CRT monitor… Anyway… Basecamp also is available in fully featured apps for iOS and Android that offer the additional power and convenience of native features on your phone or tablet.

Today, in that same spirit, we’re proud to announce Basecamp 3 for Windows and Mac.

Basecamp 3 for Mac (left) and Windows.

If you’re using Basecamp 3 daily on your computer we think you’ll find the app a big level-up from the browser experience. Here’s why:

It’s right at home

Once installed, Basecamp 3 has its own icon on your desktop, in the dock on your Mac, Windows taskbar, and when switching applications. Instead of being hidden in tab within your web browser, Basecamp is now front-and-center.

Basecamp 3 in the OS X dock (top) and Windows taskbar.

Notifications

Turn on notifications and you’ll see them right there on your computer as they happen. Even when you’re working in another application, a quick glance at the system tray icon or OS X menu bar will let you know when there is something new in Basecamp. Focus on work without missing a thing!

Basecamp Notifications on OS X (top) and Windows.

Available now

Basecamp 3 for Windows and Mac are free and available now. You can download them using the links below. For installation instructions view our help docs.

Download Basecamp 3 for Windows or Basecamp 3 for Mac.


Basecamp 3 works where you do on iOS, Android, Mac, Windows, and anywhere you’ve got a web browser and an internet connection. Your first Basecamp is completely free so try it today, it takes just a minute to sign-up.

First principles

In January of 2006, a young comedian with a taste of doing stand-up in college, moved to New York to make it big.

I got $200 and dreams, let’s do this thing.

But following a stretch of barely getting by and sleeping on the subway, he moved back to Chicago by May of the same year. So what happened to him? Did he give up?



Richard Feynman, one of the greatest physicists and thinkers of our time, was devoted to breaking things down into first principles. He’d strip a problem or idea down to the basics he first could prove to be true before building more on top.

For example, instead of studying theory and papers from other physicists to prepare to take his oral exam for his graduate degree, Richard opened up a blank notebook, titled it “Notebook Of Things I Don’t Know About”, and essentially wrote his own physics text book. Over the course of weeks, he built up his knowledge of physics from the very beginning of what he could prove to be true.

During the exam he was asked the color at the top of a rainbow. Instead of using the ROYGBIV acronym most of us learn in grade school, he used the refractive index of water, wavelengths, and physics to calculate the actual colors.

First principles is why I’m so interested in stand-up comedians, who are often solo acts, and survive on their own good ideas and delivery to make people laugh. They don’t have teams or venture capitalists doing things for them. They can’t make excuses that one of their co-founders blew up the business.

Our comedian had a recent interview where he shares several first principles that have helped him on his journey and we can learn a lot from them…


Generic problems

A lot of my standup is argument: Why are we doing this? Why do I have to do this? Why? Simple example: My buzzer rings yesterday. I go downstairs. UPS has a package. And he’s like “What’s the apartment?” “The one you were just ringing”

Peter F. Drucker wrote The Effective Executive in 1966 but much of it still rings true today. How do the best of us seem to get so much done? Well there are several things, like picking battles that matter and working from strengths. But one that stands out — working on generic problems.

Drucker: By far the most common mistake is to treat a generic situation as if it were a series of unique events; that is, to be pragmatic when one lacks the generic understanding and principle.The effective decision-maker, therefore, always assumes initially that the problem is generic. He always assumes that the event that clamors for his attention is in reality a symptom. He looks for the true problem. He is not content with doctoring the symptom alone.

This comedian is great at observing how even a UPS delivery driver presents a generic problem many of us face.

One of the first things I did at Highrise was to stop all the “favors” we were doing. Customers would come in and ask for unique things they needed handled in their account. Things that could only be done by a developer using our backend tools. It took up a ton of time, but even worse, it forced us to focus on symptoms.

When we instead focused on the problems they represented as generic issues many customers were going through, we were able to make big changes that made a lot of people happier. For example, “Company Tags” is a feature we just added because I refused to run a script for just a single account.

Of course it’s a balance. We still find ourselves doing one off favors, and in my opinion still too many. But we get a lot done at Highrise with a very small team, and this is a big reason: when a support case comes in, our first thought isn’t to delight this specific user at this specific moment, but to figure out if what we’re looking at could be generic and what we can do to stop it from happening in the future for more people. It’s hard to pull off, because we might actually disappoint a user at first, but the time we get back can be used to make a much larger impact for our customers.


Ask

On meeting Mitch Hedberg.

I got to do a show with him, actually, in 2005, a few weeks before he passed. He had a couple nights at Zanies in Chicago. I didn’t really have any social grace at that time. I was 22. I just asked him if I could get a guest spot. And he put me and a couple other comics on for five minutes in front of a sold-out crowd.

An opportunity he just had the guts to ask for. No reason Mitch would say yes, but what would hurt to ask?

Over and over again I see opportunities come my way just because I wasn’t afraid to ask. Our first deal at my first startup resulted from a cold email asking if someone would like to chat. I didn’t expect a reply. Instead, it turned into 35k in revenue, which made all the difference for us when we were making nothing.

Even what I’m doing today was because awhile ago I decided to take a shot and email Jason Fried of Basecamp to see if he’d like to chat about product development and share ideas about what I was working on. No reason I expected him to say yes. But he did. Over time that relationship turned into me taking over Highrise.

Didn’t know me from shit. And I killed it.

Sometimes you just take a bunch of shots and see what happens.


Perspective

Daniel T. Gilbert, a psychologist at Harvard, published an article in 2013 about the “end of history illusion”. They found significant evidence that humans are really bad at predicting our own change. Sure, when we look back a decade we see how much we’ve grown and learned, but when we try to predict how we’ll change over the course of the next decade, most people think they’ll change very little. Our tastes, ideas, hobbies, etc. will just remain the same. Then, give us another decade, and we’ll look back, and again we’ll see how naive we were.

That’s a ripe place to make mistakes. The feeling of not changing makes us resistant to looking for help and new ideas.

We launched a brand new marketing site for Highrise in November of 2014. Of course, I wanted to test the new site to make sure it converted new customers as well or hopefully better than the old site.

And believe me, I know split testing.

I’ve been split testing sites for decades. I know all the tools available. I’ve even written a split testing utility for Ruby.

So as we rolled out the new site, we first tested. And as soon as we saw a statistically significant increase in signups, we deployed the new site to all our traffic. And on we went.

Now, in a new year, we started reviewing how Highrise is doing compared to Highrise of the past, and even more, how does it compare to Basecamp?

This time, to get some help with looking at historical data I reached out to Noah, the data scientist at Basecamp, who sent over a thorough analysis.

Active users — up. Retention of trial customers — up. Yearly retention of users — up.

Awesome. But…

“Nate, the revenue mix is totally off.”

We had increased free signups, but decreased paid signups. The new marketing site has a greater conversion rate but only because we included free signups in our test metric. Overall, we lost a bunch of potential paid customers because of our changes.

Uggh.

The subsequent fix to our marketing site took under 3 hours. Now our plans page looks a lot like the original site again. And it increased our revenue by 63%.

How did I not see this? Hubris. I should have brought in Noah from the get go. I did a poor job of predicting how much I still needed to learn and change in this new role and business.

Our comedian shares in a recent interview with NPR regarding his time sleeping on the subway:

“It was just really goofy on my part,” he admits, in retrospect. “ ’Cause I didn’t have to do that. I could have — one, I could have made up with my sister and apologized for popping in on her place like that. But I was too cocky. And two, I could have just gone back home to Chicago. So it was just goofiness on my part. I don’t glamorize it, like, ‘Oh, I was living on the train for my dream. It wasn’t; I don’t look at it like that at all.”

Now he realizes how easily he could have fixed his situation early in his career. If only he knew then what he knows now. How cocky he was. How easily he could have asked for forgiveness or more help. If only we could predict how little we know 🙂

But maybe just knowing Gilbert’s research is enough. Maybe if we just assume we really never know enough, we’ll make some better decisions.


When I see a comic make headway in the world, like Louis C.K or Mitch Hedberg, they get my attention. Of course because they’re funny. But how do they do it? How do they stay relevant and find new good ideas? How do they market themselves?

Hannibal Buress is a great example. He was that young cocky kid at 23 who moved to New York only to end up sleeping on the subway and moving back home. Today though we can find Hannibal on comedy specials, TV, touring the world, and we can learn a lot from his recent interviews with Scott Raab in Esquire and NPR’s David Greene.

If we understand what’s worked for Hannibal as he struggled to create a life and business in his craft — like looking for generic problems, asking for help, and realizing how little we know — perhaps we can use those as first principles in our own lives, teams and organizations.


P.S. It would be awesome to meet you on Twitter, or see the awesome new stuff we’re doing at Highrise.

New: Better control over email and phone notifications in Basecamp 3

Before today, it was hard to know if Basecamp would tell you about new messages via email or through the app on your phone. Now Basecamp has clearer settings that give you control over how and when you receive new messages from other people on your Basecamps.

The new email notifications setting

Now you can check a box to tell Basecamp if you want emails or not. Want emails? Check it on. Don’t want emails? Check it off. It’s as simple as that.

In the past, Basecamp didn’t send you emails if you had the app installed on a phone or tablet. Now your phone and tablet settings are entirely separate from the email settings. You control if you want emails and phone notifications, just one of them, or neither.

The new notification settings

New option: Receive everything or just Pings and @mentions

Sometimes you don’t want to know about everything on a Basecamp account. You may just want to be notified when people mention you by name or send you a private message. Now we have a setting that allows you to block all notifications except for Pings and @mentions.

New setting to get everything or just Pings and @mentions.

Work Can Wait

All of these settings filter through Basecamp 3’s much-loved Work Can Wait feature. Use Work Can Wait to tell Basecamp to “hold your calls” outside of working hours. All notifications will be silenced until it’s work time again so you can enjoy your free time without interruptions.

Work Can Wait

To access the improved settings screen, click your avatar in the upper-right corner and click Change notification settings.


More details are available on the Notifications help page.

Thanks for using Basecamp 3! We hope these new settings make it easier for you to get notifications from Basecamp in the right way at the right time.

Practical skills that practical data scientists need

When I wrote about how I mostly just use arithmetic, a lot of people asked me about what skills or tools a data scientist needs if not fancy algorithms. What is this mythical “basic math” that I mentioned? Here’s my take on what skills are actually needed for the sort of work that I do at Basecamp: simple analyses focused on solving actual business problems.

The most important skill: being able to understand the business and the problem

I’ll get to actual practical skills that you can learn in a textbook in a minute, but first I have to belabor one point: the real essential skill of a data scientist is the ability to understand the business and the problem, and the intellectual curiosity to want to do so. What are you actually trying to achieve as a business? Who are your customers? When are you selling your product? What are the underlying economics of the business? Are profit margins high or modest? Do you have many small customers or a few large customers? How wide is your product range? Who are you competing with? What challenge is the business facing that you’re trying to solve or provide input towards a decision on? What’s the believable range of answers? Who is involved in solving this problem? Can analysis actually make a difference? How much time is worth investing in this problem?

Understanding the data

Before you look at any data or do any math, a data scientist needs to understand the underlying data sources, structure, and meaning. Even if someone else goes out and gets the data from wherever it’s stored and gives it to you, you still need to understand the origin and what each part of the data means. Data quality varies dramatically across and within organizations; in some cases you’ll have a well documented data dictionary, and in other cases you’ll have nothing. Regardless, you’ll want to be able to answer the following questions:

  • What data do I need to solve the problem?
  • Where is that data located? In a relational database? In a log file on disk? In a third party service?
  • How comprehensive (time and scope) is the data? Are there gaps in coverage or retention?
  • What does each field in the data mean in terms of actual behavior of humans or computers?
  • How accurate is each field in the data? Does it come from something that’s directly observed, self-reported, third-party sourced, or imputed?
  • How can I use this data in a way that minimizes the risk of violating someone’s privacy?

SQL skills

For better or worse, most of the data that data scientists need live in relational databases that quack SQL, whether that’s MySQL, Postgres, Hive, Impala, Redshift, BigQuery, Teradata, Oracle, or something else. Your mission is to free the data from the confines of that relational database without crashing the database instance, pulling more or less data than you need to, getting inaccurate data, or waiting a year for a query to finish.

Virtually every query a data scientist writes to get data to analyze to solve business problems will be a SELECT statement. The essential SQL concepts and functions that I find necessary are:

  • DESCRIBE and EXPLAIN
  • WHERE clauses, including IN (…)
  • GROUP BY
  • Joins, mostly left and inner
  • Using already indexed fields
  • LIMIT and OFFSET
  • LIKE and REGEXP
  • if()
  • String manipulation, primarily left() and lower()
  • Date manipulation: date_add, datediff, to and from UNIX timestamps, time component extraction
  • regexp_extract (if you’re lucky to use a database that supports it) or substring_index (if you’re less lucky)
  • Subqueries

Basic math skills

Once you have some data, you can do some maths. The list of what I consider to be the essential list of math skills and concepts is not a long one:

  • Arithmetic (addition, subtraction, multiplication, division)
  • Percentages (of total, difference vs. another value)
  • Mean and median (and mean vs. median)
  • Percentiles
  • Histograms and cumulative distribution functions
  • An understanding of probability, randomness, and sampling
  • Growth rates (simple and compound)
  • Power analysis (for proportions and means)
  • Significance testing (for proportions and means)

This isn’t a very complicated set of things. It’s not about the math, it’s about the problem you’re solving.

Slightly more advanced math concepts

On occasion, some more advanced mathematical or SQL concepts or skills are of value to common business problems. A handful of the more common things I use include:

  • Analytic functions if supported by your database (lead(), lag(), rank(), etc.)
  • Present and future value and discount rates
  • Survival analysis
  • Linear and logistic regression
  • Bag of Words textual representations

There are some problems that require more advanced techniques, and I don’t mean to disparage or dismiss those. If your business can truly benefit from things like deep learning, congratulations! That probably means you’ve solved all the easy problems that your business is facing.

Remembering how I got here — with a lot of help from others


About a month back, Basecamp put out a call for internship candidates. We’re looking for great people who want to learn about programming, product design, operations, data, or marketing directly from the people who work on Basecamp every day.

But before we went public with it, any interested team had to internally pitch a meaningful project and commit to investing the proper time and energy into teaching, guiding, and helping our interns regularly.

The Android team is made up of three people. And of those three, I was the loudest “no” vote. I think it was something along the lines of, “we don’t have time, it’s not a priority.”

I look back on that now and realize how insanely silly and selfish that was.

At every turn of my 15 year career, people have been helping me. Helping me to learn new skills, advance my career, or just to be a better person. I’ve tried to do the same for others over the years.

And yet when I was given such a clearcut opportunity to help someone else professionally, my first reaction was to punt it away because it wasn’t “a priority.”

At that moment, I had completely forgotten how I got to where I am today— with a lot of help from others. Sure, hard work, persistence, and maybe even some smarts helped pave the way. But I sure as hell didn’t build my career all by myself. I needed to remind myself of that. I needed to remember how I got here.


I have no idea what I’m doing

At my first job out of college, I was thrown onto Java projects. I had no idea what I was doing. For over two years, a bunch of people helped me as I fumbled my way through the basics of programming, business, and being a professional. These years laid the foundation of my career. Without the help, guidance, and friendship of a few key mentors, my career wouldn’t have turned out nearly as well as it did. I’m sure of that.

Truth.

By 2008 I’d been doing Java for a long while, and it was time for a change. I joined an interactive design agency in Chicago, converted to a project manager, and once again had no idea what I was doing. I was running multiple projects with dozens of people, in an industry I didn’t fully understand. I wasn’t doing any programming, but I sure was making a lot of clients angry.

And once again, so many people helped me through it. Veteran project managers taught me the tactics of project delivery, designers helped me understand the intricacies of their work, and account managers helped me handle, persuade, and sell to clients. By the time I’d left, I was a better balanced, more polished professional in all areas of my work, in large part because of the help of others.

Fast forward to today. I’ve been at Basecamp for almost three years, and have been a professional programmer, consultant, and project manager for over 15 years. And yet part of me still has no idea what I’m doing. I’m still getting help from others on a daily basis, from every part of the company. It doesn’t matter what I’m stuck on, there’s always a friendly teammate to help me with all the things that I don’t understand (which is a lot).


Paying it forward

Hopefully by now it’s crystal clear why I felt like such a fool for saying no to the internship idea. I honestly can’t imagine where I’d be today professionally if it wasn’t for for the selfless, generous help of others. But there’s a happy ending to this story.

About a day after I said no to the internship idea, I realized my madness and jumped on board with the idea. And not only did I vote yes, I wrote the formal pitch for our project and will be the point person for our awesome Android intern.

I know I didn’t say thank you enough to everyone who helped me so much along the way. I can only hope that paying it forward now and in the future serves as a small repayment of that debt.

So the next time you’re given the opportunity to help someone professionally, I’d encourage you to really stop and consider it. I know time is short and life is busy. But try to remember those people who helped pave the way for your successes — think of how proud they’ll be to know they’ve taught you well.


If you’re interested in one of our internship projects, applications close on 2/24/16, so you’ve got less than a week to apply. Get a move on!

We’re hard at work making the Basecamp 3 and its companion Android app the best it can be. Check ’em out!

Data scientists mostly just do arithmetic and that’s a good thing

Hi, I’m Noah. I work at Basecamp. Sometimes I’m called a “data scientist.” Mostly, I just do arithmetic, and I’m ok with that.

Here’s a few of the things I worked on in the last couple of weeks, each of them in response to a real problem facing the business:

  • I analyzed conversion, trial completion, and average invoice amounts for users in different countries.
  • I identified the rate at which people accidentally sign up for Basecamp when they mean to sign in to an existing account and how that’s changed over time.
  • I analyzed and reported on financial performance of a few of our products.
  • I ran and analyzed a survey of account owners.
  • I analyzed an A/B test we ran that affected the behavior of a feature within Basecamp.

In the last two weeks, the most “sophisticated” math I’ve done has been a few power analyses and significance tests. Mostly what I’ve done is write SQL queries to get data, performed basic arithmetic on that data (computing differences, percentiles, etc.), graphed the results, and wrote paragraphs of explanation or recommendation.

I haven’t coded up any algorithms, built any recommendation engines, deployed a deep learning system, or built a neural net.

Why not? Because Basecamp doesn’t need those things right now.

The dirty little secret of the ongoing “data science” boom is that most of what people talk about as being data science isn’t what businesses actually need. Businesses need accurate and actionable information to help them make decisions about how they spend their time and resources. There is a very small subset of business problems that are best solved by machine learning; most of them just need good data and an understanding of what it means that is best gained using simple methods.

Some people will argue that what I’ve described as being valuable isn’t “data science”, but is instead just “business intelligence” or “data analytics”. I can’t argue with that arbitrary definition of data science, but it doesn’t matter what you call it — it’s still the most valuable way for most people who work with data to spend their time.

I get a fair number of emails from people who want to get into “data science” asking for advice. Should they get a masters degree? Should they do a bunch of Kaggle competitions?

My advice is simple: no. What you should probably do is make sure you understand how to do basic math, know how to write a basic SQL query, and understand how a business works and what it needs to succeed. If you want to be a valuable contributor to a business, instead of spending your weekend working on a data mining competition, go work in a small business. Talk to customers. Watch what products sell and which ones don’t. Think about the economics that drive the business and how you can help it succeed more.

Knowing what matters is the real key to being an effective data scientist.