Stimulus 1.0: A modest JavaScript framework for the HTML you already have

Modern JavaScript doesn’t have to mean single-page, client-side MVC apps.

We write a lot of JavaScript at Basecamp, but we don’t use it to create “JavaScript applications” in the contemporary sense. All our applications have server-side rendered HTML at their core, then add sprinkles of JavaScript to make them sparkle.

This is the way of the majestic monolith. Basecamp runs across half a dozen platforms, including native mobile apps, with a single set of controllers, views, and models created using Ruby on Rails. Having a single, shared interface that can be updated in a single place is key to being able to perform with a small team, despite the many platforms.

It allows us to party with productivity like days of yore. A throwback to when a single programmer could make rapacious progress without getting stuck in layers of indirection or distributed systems. A time before everyone thought the holy grail was to confine their server-side application to producing JSON for a JavaScript-based client application.

That’s not to say that there isn’t value in such an approach for some people, some of the time. Just that as a general approach to many applications, and certainly the likes of Basecamp, it’s a regression in overall simplicity and productivity.

And it’s also not to say that the proliferation of single-page JavaScript applications hasn’t brought real benefits. Chief amongst which has been faster, more fluid interfaces set free from the full-page refresh.

We wanted Basecamp to feel like that too. As though we had followed the herd and rewritten everything with client-side rendering or gone full-native on mobile.

This desire led us to a two-punch solution: Turbolinks and Stimulus.

Turbolinks up high, Stimulus down low

Before I get to Stimulus, our new modest JavaScript framework, allow me to recap the proposition of Turbolinks.

Turbolinks descends from an approach called pjax, developed at GitHub. The basic concept remains the same. The reason full-page refreshes often feel slow is not so much because the browser has to process a bunch of HTML sent from a server. Browsers are really good and really fast at that. And in most cases, the fact that an HTML payload tends to be larger than a JSON payload doesn’t matter either (especially with gzipping). No, the reason is that CSS and JavaScript has to be reinitialized and reapplied to the page again. Regardless of whether the files themselves are cached. This can be pretty slow if you have a fair amount of CSS and JavaScript.

To get around this reinitialization, Turbolinks maintains a persistent process, just like single-page applications do. But largely an invisible one. It intercepts links and loads new pages via Ajax. The server still returns fully-formed HTML documents.

This strategy alone can make most actions in most applications feel really fast (if they’re able to return server responses in 100–200ms, which is imminently possible with caching). For Basecamp, it sped up the page-to-page transition by ~3x. It gives the application that feel of responsiveness and fluidity that was a massive part of the appeal for single-page applications.

But Turbolinks alone is only half the story. The coarsely grained one. Below the grade of a full page change lies all the fine-grained fidelity within a single page. The behavior that shows and hides elements, copies content to a clipboard, adds a new todo to a list, and all the other interactions we associate with a modern web application.

Prior to Stimulus, Basecamp used a smattering of different styles and patterns to apply these sprinkles. Some code was just a pinch of jQuery, some code was a similarly sized pinch of vanilla JavaScript, and some again was larger object-oriented subsystems. They all usually worked off explicit event handling hanging off a data-behavior attribute.

While it was easy to add new code like this, it wasn’t a comprehensive solution, and we had too many in-house styles and patterns coexisting. That made it hard to reuse code, and it made it hard for new developers to learn a consistent approach.

The three core concepts in Stimulus

Stimulus rolls up the best of those patterns into a modest, small framework revolving around just three main concepts: Controllers, actions, and targets.

It’s designed to read as a progressive enhancement when you look at the HTML it’s addressing. Such that you can look at a single template and know which behavior is acting upon it. Here’s an example:

<div data-controller="clipboard">
  PIN: <input data-target="clipboard.source" type="text" value="1234" readonly>
  <button data-action="clipboard#copy">Copy to Clipboard</button>
</div>

You can read that and have a pretty good idea of what’s going on. Even without knowing anything about Stimulus or looking at the controller code itself. It’s almost like pseudocode. That’s very different from reading a slice of HTML that has an external JavaScript file apply event handlers to it. It also maintains the separation of concerns that has been lost in many contemporary JavaScript frameworks.

As you can see, Stimulus doesn’t bother itself with creating the HTML. Rather, it attaches itself to an existing HTML document. The HTML is, in the majority of cases, rendered on the server on either on the page load (first hit or via Turbolinks) or via an Ajax request that changes the DOM.

Stimulus is concerned with manipulating this existing HTML document. Sometimes that means adding a CSS class that hides an element or animates it or highlights it. Sometimes it means rearranging elements in groupings. Sometimes it means manipulating the content of an element, like when we transform UTC times that can be cached into local times that can be displayed.

There are cases where you’d want Stimulus to create new DOM elements, and you’re definitely free to do that. We might even add some sugar to make it easier in the future. But it’s the minority use case. The focus is on manipulating, not creating elements.

How Stimulus differs from mainstream JavaScript frameworks

This makes Stimulus very different from the majority of contemporary JavaScript frameworks. Almost all are focused on turning JSON into DOM elements via a template language of some sort. Many use these frameworks to birth an empty page, which is then filled exclusively with elements created through this JSON-to-template rendering.

Stimulus also differs on the question of state. Most frameworks have ways of maintaining state within JavaScript objects, and then render HTML based on that state. Stimulus is the exact opposite. State is stored in the HTML, so that controllers can be discarded between page changes, but still reinitialize as they were when the cached HTML appears again.

It really is a remarkably different paradigm. One that I’m sure many veteran JavaScript developers who’ve been used to work with contemporary frameworks will scoff at. And hey, scoff away. If you’re happy with the complexity and effort it takes to maintain an application within the maelstrom of, say, React + Redux, then Turbolinks + Stimulus will not appeal to you.

If, on the other hand, you have nagging sense that what you’re working on does not warrant the intense complexity and application separation such contemporary techniques imply, then you’re likely to find refuge in our approach.

Stimulus and related ideas were extracted from the wild

At Basecamp, we’ve used this architecture across several different versions of Basecamp and other applications for years. GitHub has used a similar approach to great effect. This is not only a valid alternative to the mainstream understanding of what a “modern” web application looks like, it’s an incredibly compelling one.

In fact, it feels like the same kind of secret sauce we had at Basecamp when we developed Ruby on Rails. The sense that contemporary mainstream approaches are needlessly convoluted, and that we can do more, faster, with far less.

Furthermore, you don’t even have to choose. Stimulus and Turbolinks work great in conjunction with other, heavier approaches. If 80% of your application does not warrant the big rig, consider using our two-pack punch for that. Then roll out the heavy machinery for the part of your application that can really benefit from it.

At Basecamp, we have and do use several heavier-duty approaches when the occasion calls for it. Our calendars tend to use client-side rendering. Our text editor is Trix, a fully formed text processor that wouldn’t make sense as a set of Stimulus controllers.

This set of alternative frameworks is about avoiding the heavy lifting as much as possible. To stay within the request-response paradigm for all the many, many interactions that work well with that simple model. Then reaching for the expensive tooling when there’s a call for peak fidelity.

Above all, it’s a toolkit for small teams who want to compete on fidelity and reach with much larger teams using more laborious, mainstream approaches.

Give it a go.


Stimulus was envisioned during a two-week deep dive into the current state of JavaScript that I took about a year ago. I mined our patterns within the Basecamp 3 codebase, extracted an archetype, and used the latest-greatest JavaScript techniques to make it real. This deep dive first gave birth to Webpacker for Rails, but this framework couldn’t have happened without the incredible work by Sam Stephenson and Javan Makhmali. They took my rinky-dinky prototype of Stimulus and rewrote it from scratch in TypeScript. Just like they had done with Turbolinks 5. It’s a beautiful piece of code.

Announcing the Basecamp 3 API

Basecamp 3 finally has an API! Head to https://github.com/basecamp/bc3-api for documentation and a guide to getting started.

We’re already seeing some great 3rd party integrations like Tick’s time tracking service and expect many more to come soon. If you have an integration you’d like us to know about, be sure to submit it here.

Some technical details:

  • The API is all part of the same majestic monolith that powers the web interface. Same controllers, new Jbuilder views.
  • Everything is cached to the max with liberal use of Russian Doll caching and ETags. While layering it on, I found and fixed a few related issues so consider the campsite cleaner than I found it!
  • Pagination is implemented following RFC5988’s convention of `Link` headers to provide URLs for the `next` page.
  • In-app Rate limiting is handled by Jeremy’s excellent Rack::Ratelimit middleware.
  • Traffic is served on a separate domain to help shield from CSRF and session vulnerabilities, and to aid with monitoring, routing, and throttling.

That’s all. Happy APIing!

Lufo, Last Used First Out — An easy way to drastically improve the user experience of long select…

Lufo is a jQuery plugin to track the most recent options chosen on a <select> element and display them at the top of the list.

Source on Github.


“Stop being regionally biased”

That’s the subject of an email we received recently at Highrise.

Well that got our intention. What are we doing wrong?

The message explained:

Your current drop down menu for country locations is not very Asia friendly — all those listed at the top of the list are in North America, Europe + Japan. My most frequent country selections are Hong Kong, Singapore, Indonesia, Vietnam, China, etc…

They were right. Our menu is 200 countries long, and we highlight a few, but it’s clearly biased towards that few.

But that got us thinking… this isn’t just a problem for Countries, we have this problem all over our app. We have a bunch of menus, many with more than a few choices, where it’s hard to repeatedly pick things that aren’t at the top of the list.

Sure, browsers try and give you the ability to type some letters to find things faster in the menu, but they fail at anything but rudimentary searches.

What we really need is a solution to have HTML select menus remember what the user last picked and make those easier to pick next time.

Something like:

Last Used First Out

LUFO!!!


Long <select> menus with many options can be a pain. States/territories, countries, currency lists, etc. have so many options it’s cumbersome to scroll through them to find the ones you use often.

Lufo tracks the most recent selections on a long <select> menu and stores them in a browser’s localStorage (it falls back to a tracking cookie if the browser doesn’t support localStorage). When someone revisits a page with that same long menu, their most-recently selected options will be copied to the top of the list for their convenience. No bias on your part. Super-convenient for the people using your site.

You can add Lufo to just about any <select> form control, choose to group the recent selects, customize (or hide) the labels and divider text, and even enable/disable Lufo based on how many options are in a dynamically-generated <select>.

http://i.imgur.com/PQd8JUq.gifv

Usage

  1. Include jQuery in your project
  2. Include the Lufo plugin code:
http://jquery.lufo.js

3. Call the Lufo plugin on a <select> element:

$(‘select’).lufo();

If you are using the plugin with multiple, different, <select> menus, you will want to specify a unique storage name for each menu:

$('select.some-select').lufo({
listStoreName: 'someSelectValues'
});

See below for other available options.

Options

Unless otherwise noted, all option examples below are Lufo defaults.

Check for Initial Placeholder Values

If your <select> menu has a built-in placeholder (for example: “Select a country…”) that resides at the top of the list and has no value, Lufo will check for it, and if found, move it to the very top of the list after adding in the most-recently selected options.

checkInitialValue: true

Set to false to turn off this check.

Strip “selected” from Cloned Values

If a recent option has the selected attribute set (selected=”selected”) it will remain when that <option> is copied to the top of the list.

stripSelected: false

Set stripSelected to true if you want the selected attribute removed from the copy.

A Title for the Recently Selected List

By default, Lufo will place a disabled <option> at the beginning of the most-recently selected options list with the text: “Recently selected:”.

recentsListTitleEnabled: true,
recentsListTitle: 'Recently selected:'

Turn the title off by setting recentsListTitleEnabled to false or change the title text by setting recentsListTitle to something else.

List Divider

Lufo will place a disabled <option> at the end of the most-recently selected options list to use as a divider.

dividerEnabled: true
dividerText: '––––––––––––––––––––––––'

You may disable the divider by setting dividerEnabled to false or modify the divider by setting dividerText to something else.

Recently Selected List Memory

By default, Lufo will remember the 5 most-recently selected items in a <select> menu.

recentsListLength: 5

You may change this number to any positive integer.

Conditionally Enabling Lufo

A <select> list must have at least 5 items in it to enable the tracking and display of recently selected items. This is especially useful for dynamic lists in an app that may need to grow over time before it becomes useful to track recent selects.

listMinimumLength: 5

You may change this number to any positive integer to enable Lufo sooner, or to wait until a list grows beyond five options.

Using <optgroup>

By default, Lufo adds the list of recent selects to the top-level of a <select> menu.

groupList: false

If you wish to group the options into an <optgroup> set groupList to true.

Ignoring Certain Values

Lufo will track clicks on all available <option> in a <select>.

If you wish to ignore a few options in the list, you may create an array of the values (value=) that you do not want Lufo to track.

Example:

ignoredValues: ['dog', 'cat', 'cow']

Setting the above option would not track clicks on any options in a <select> that contain the values of dog, cat, or cow.

Storage & Tracking Cookie Preferences

If you will be using Lufo on a single <select> menu on your site, you don’t need to change any of the storage preferences. Lufo will use a browser’s localStorage (or if that isn’t available, set a tracking cookie) to remember the most recent selections.

Default:

listStoreName: 'recentOptionValues',
cookieAge: 30 // only relevante for browsers without `localStorage`

However, if you are using Lufo on multiple menus with different values, you need to set a unique storage item name for each of the menus.

Example:

$('select.some-select').lufo({
listStoreName: 'someSelectValues'
});
$('select.other-select').lufo({
listStoreName: 'otherSelectValues'
});

You may also increase or decrease the amount of time the tracking cookie persists on your site (for browsers withoutlocalStorage). The default cookie age is 30 days. You may change it by setting cookieAge to another positive integer to represent a number of days.

Example:

cookieAge: 90

The example above would remember the recently selected options for 90 days instead of the default 30.

Examples

Example #1

Set Lufo to track a <select> with the class name countries and set a unique storage item name based on that class name.

$('select.countries').lufo({
listStoreName: 'countriesSelectValues'
});

Example #2

Set Lufo to track a <select> with the ID author_id, and a unique storage item name. Only enable Lufo if the list grows to10 items. Only show 3 of the most recently selected items. Ignore a value of none. Finally, turn off the divider, shorten the recent list title, and move the recent selects into an <optgroup>.

$('#author_id').lufo({
groupList: true,
recentsListTitle: 'Recent',
dividerEnabled: false,
recentsListLength: 3,
listMinimumLength: 10,
ignoredValues: ['none'],
listStoreName: 'authorIdSelectValues'
});

Contributing

We would love to see your contributions to Lufo! Check CONTRIBUTING.md for more information.

License

MIT License

Credit

A ton of thanks to Grant Blakeman doing the vast majority of the work putting this together.

P.S.

You should follow us on Twitter: here, or see how we can help you with contact management using Highrise — a handy tool to help you remove anxiety around tracking who to follow up with and what to do next.

Snapback Cache — What we use to make our infinite scrolling feeds at Highrise awesome.

Many apps today have some concept of an infinite scrolling feed: Facebook, Twitter, LinkedIn and many more. Almost all of them suffer from the same problem. If you click on something in the feed that brings you to a new page, when you hit the back button or try to return to that original feed, your place is lost. All the scrolling is gone.

At Highrise we had that same problem. So this is the library we use to fix that. We call it our Snapback Cache, and it’s made a big improvement to how people can use infinite scroll in our app and still get a lot of work done without losing their place.


Another great thing about this is it operates on the URL, so you can have multiple infinite scrolling feeds to cache. At Highrise we have a “main activity” and then activities for a Contact, etc. They each get their separate cache. To keep a manageable memory footprint for your browser, we keep 10 caches as a maximum.

The basics of how it works

Using this small javascript library, you hook it up to the click events on things in your infinite scrolling feed. For example:

https://gist.github.com/n8/96475a20b36e87973671

Now when people click the links inside our “recordings” container, the stuff inside the current recordings container is cached locally using the browser’s session storage.

Then the javascript library watches the load event of any pages being browsed. If the library sees that that browser’s URL is a url we’ve already cached, and it’s not “too old” (15 minutes), we replace the contents of our container (#recordings in our example) with the cached version, and scroll the browser to the place where it had been cached.

This sounds easy, but there are certain things we bumped into that the library also helps with. Things like disabling autofocus events that mess up scrolling and making sure things in the cache can actually be more granularly ignored or even refreshed.

Syntax and how to use it

var snapbackCache = SnapbackCache({ options });

Here are some example options:

https://gist.github.com/n8/b7036e96761a98709ca7

bodySelector is mandatory. It tells us what on the page you want to cache.

finish is a function of things that you’d like to happen before the page is cached to get the page to get cleaned up. For example, we already try to get jQuery animations to finish, but if there’s anything else on the page that might be animated or dynamically changing when someone is trying to navigate your site, you probably don’t want those “transitional” things cached. In our case we have a search bar that we want cleared up before things are cached.

removeAutofocus is a function that removes any auto focus behavior from your page. autoFocus events can mess with the browsers ability to scroll to the right place. So we want to nip that in this function. In our case we have multiple autofocus things going on, so we clear all that up.

refreshItems is a function to help refresh anything that might have gone stale from the cache. You can use that in conjunction with a method available on snpachbackCache called markDirty.

So in our case, we cache a note or comment or email in our feed. But if someone at some point edits/deletes one of those notes, comments or emails, we have javascript call

snapbackCache.markDirty(id_of_dirty_thing);

Then when the snapbackCache replaces the cached contents it’s saving for us, it makes sure to call the refreshItems function you specify along with an array of “dirty items” you can do something with. In our case, we take all those dirty ids, and issue an ajax call that does all the work to refresh bits of the cached page.

nextPageOffset is a function that the Snapback cache can use to figure out what “page” your user is on. We take that page and store it along the cached contents of the page. That way when the cached page is restored you have the page number the user was on and get pick up infinite paging at the appropriate place. See the page-cache:loaded event below to do that.

Events

There are a couple of events we send out that are useful.

snapback-cache:cached is an event emitted as soon as the contents of the page have been cached into session storage

snapback-cache:loaded is an event emitted as soon as the contents of the page have been replaced. We use this at Highrise to set the appropriate offset for our infinite scrolling:

https://gist.github.com/n8/625c218e7ac8b79fdacb

nextPageOffset was calculated because we had setup a “nextPageOffset” function on the page cache.

Installation

1) Add the snapback_cache.js to your javascript stack.

2) Add a cache variable with the options set:

var snapbackCache = SnapbackCache({ bodySelector: "#recordings", });

3) Call snapbackCache.cacheCurrentPage() whenever you need to, and magically when people return to that url, the cache will do the rest.

Feedback

Source code available on Github. Feedback and pull requests are greatly appreciated. Let me know how we can improve this.

A ton of thanks to everyone at Highrise for helping get this into our stack. Especially Jon Phenow, Grant Blakeman and Michael Dwan for the edits and help getting it open sourced.

P.S.

You should follow us on Twitter: here, or see how we can help you with contact management using Highrise — a handy tool to help you remove anxiety around tracking who to follow up with and what to do next.