Integrated systems for integrated programmers

One of the great tragedies of modern web development over the last five years or so has been the irrational exuberance for microservices. The idea that making a single great web application had simply become too hard, but if we broke that app up into many smaller apps, it’d all be much easier. Turned out, surprise-surprise, that it mostly wasn’t.

As Kelsey Hightower searingly put the fallacy: “We’re gonna break it up and somehow find the engineering discipline we never had in the first place”.

But it’s one of those hard lessons that nobody actually wants to hear. You don’t want to hear that the reason your monolith is a spaghetti monster is because you let it become that way, one commit at the time, due to weak habits, pressurized deadlines, or simply sheer lack of competence. No, what you want to hear is that none of that mess is your fault. That it was simply because of the oppressive monolithic architecture. And that, really, you’re just awesome, and if you take your dirty code and stick it into this new microservices tumbler, it’s going to come out sparking clean, smelling like fucking daffodils.

The great thing about such delusions is that they can keep you warm for quite a while. A yeah, sure, maybe the complexities of your new microservices monstrosity are plain as day right from the get go, but you can always excuse them with “it’s really going to pay off once we…” bullshit. And it’ll work! For a while. Because, who knows? Maybe this is better? But it’s not. And the day you have to really admit its not, you’re probably not even still there. On to the next thing.

Microservices as an architectural gold rush appealed to developers for the same reason TDD appeals to developers: it’s the pseudoscientific promise of a diet. The absolution of a new paradigm to wash away and forgive our sins. Who doesn’t want that?

Well, maybe you? Now after you’ve walked through the intellectual desert of a microservice approach to a problem that didn’t remotely warrant it (ie, almost all of them). Maybe now you’re ready to hear a different story. There’s a slot in your brain for a counterargument that just wasn’t there before.

So here’s the counterargument: Integrated systems are good. Integrated developers are good. Being able to wrap your mind around the whole application, and have developers who are able to make whole features, is good! The road to madness and despair lays in specialization and compartmentalization.

The galaxy brain takes it all in.

But of course, you cry, what if the system is too large to fit in my brain? Won’t it just swap and swap until I kernel failure? Yes, if you try to stick in a bloated beast of an application, sure.

So the work is to shrink the conceptual surface area of your application until it fits in a normal, but capable and competent, programmer’s brain. Using conceptual compression, sheer good code writing, a productive and succinct environment, using shortcuts and patterns. That’s the work.

But the payoff is glorious. Magnificent. SUBLIME. The magic of working on an integrated system together with integrated programmers is a line without limits, arbitrary boundaries, or sully gatekeepers.

Forget frontend or backend. The answer is all of it. At the same time. In the same mind.

This sounds impossible if you’ve cooked your noodle too long in the stew of modern astronautic abstractions. If you turn down the temperature, you’ll see that the web is actually much the same as it always was. Sure, a few expectations increased here, and a couple of breakthrough techniques appeared there, but fundamentally, it’s the same. What changed was us. And mostly not in ways for the better.

If your lived experience still haven’t hit the inevitable wall of defeat on the question of microservices, then be my guest, sit there with your folded arms and your smug pout. It’s ok. I get it. There’s not an open slot for this argument in your brain just yet. It’s ok. I’m patient! I’ll still be here in a couple of years when there’s room. And then I’ll send you a link to this article on twitter.

Peace. Love. Integration.

11 thoughts on “Integrated systems for integrated programmers

  1. In a micro service environment, how do people handle:

    – latency issues given so much is now going back and forth over the wire

    – security

    – centralized reporting / analytics, since now data is spread across multiple systems environment

    1. > how do people handle…

      badly.

      But to be serious, a mixture of docker, kubernetes, reporting and analytics SaaS, and a shit load of configuration files.

    2. Your concerns are basically why kubernetes is so popular. You end up trying to push many of these concerns, such as low latency scaling and security, from the individual development team, to a dedicated SRE team.

      Unfortunately, the term “SRE” has morphed away from a “software engineering” role to a “fancy IT guy” role. My rule of thumb: if your SREs do not code review and/or have the power to reject the incoming work, they’re not really doing SRE work. And unfortunately, people don’t like “real SREs”; they tend to say things like “we’re not running your pile of shit – please fix XYZ”, instead of just gleefully trying to make your shitpile smell like roses.

      Anyhow, if you’re not in a large group with tons of teams, a firehose of communication, and general confusion of who owns what, stay far away from microservices. Or find a new job.

  2. Nice to hear a prominent voice advocate for taking away the Koolaid before the acid test gets too far out of control. I’m currently rewriting an app that started as a bunch of microservices and became so complex and non-performant that we finally decided to throw it away and start over. It ended up being a bunch of black boxes with poorly understood interactions. By the end, the lead developer was trying to explain to me why we couldn’t really test the system because its behavior was “non-deterministic.”

  3. The late Jim Weirich made the point that all engineering is about tradeoffs. So the question is: what are you trading off by doing microservices? The answer: complexity for flexibility in internal implementation of the component/service.

    I think the microservice approach works really well when you have big teams; the anticipation that every company will be the next google is what pushes the architecture to an extreme level of abstraction that is unwarranted.

    Most companies never get to this level of size; unfortunately most devs don’t understand that the big advantage of a small company is that it is small and flexible; this allows it to change its mind quickly. Large institutions, architectures, and processes can’t change on a dime and they actually yearn for a time in which they were smaller.

    I see the same movement going on right now with React; drastic over complexity for very little gained. Are there times where this makes sense? Sure – for true single page apps. If your app has more than two or three pages – guess what? It’s not a single page app.

    What’s interesting here is that coding is hedonistic. If you are enjoying the process, you probably have the right architecture. Bugs get fixed and changes happen easily.

  4. Thank you, DHH, for the good laugh,

    Jos Jong wrote a book about “Vertically integrated architecture”. In it, he suggested integrating a database with the monolith (backend/frontend) and revisited the concept of query language and compilers, which would become a query engine.

    I don’t know if you have any opinions about this?

    It is still at the stage of idea but interesting work: https://www.youtube.com/watch?v=iTG7PrZllFE

  5. I take issue with the term “monolith”. It invokes a sense of being too large. It was a term coined by the cult of microservices. You can’t beat them if you use their language.

  6. For years I have worked in IT and played around with open source stuff from many different projects. Mostly poking and prodding to either better understand or tweak something for my use. Sometimes a project is like dealing with a blackbox even when looking at the source (this could be partly my inexperience). Other times it is a beautiful epiphany of code, streamlined, well documented and easily understood by the passing observer.

    Code should be cogent and lucid or rewritten.

  7. very informative and helpful, keep posting and updating all users!!
    Not all of the Google Play store’s 2.9 million apps are worth downloading.
    Hooey apps picks of the best Android apps in all categories deserve a place on your smartphones.
    https://hooeyapps.com/

Comments are closed.