00:00:12.259
Thank you! My name is Andrew Cantino.
00:00:19.800
I work for a company called Mavenlink, where we build project and financial management software for consultancies. We're based in San Francisco.
00:00:26.340
I am here today to introduce you to Brainstem, an API presentation library that we've been working on off and on for about a year. We actually just released it publicly on GitHub just a few minutes ago.
00:00:37.800
I am excited to tell you all about it.
00:00:44.399
Mavenlink has a really complicated data model. For example, we have projects that contain tasks, which have budgets that are reflected in time entries. Those belong to timesheets that are included in reports, etc. It's a very relational product with a complex data model.
00:00:57.360
This complexity is necessary because we are ultimately trying to build something that models both business and human relationships. These relationships are incredibly complicated, and while we aim to simplify the model, there’s only so far we can go. We need our API to express these relationships in a meaningful and consistent way, which is how Brainstem evolved.
00:01:21.840
We began developing Brainstem about a year ago as an extraction from our product. We've been using it internally for some time, and now it powers both our internal and external APIs.
00:01:40.259
I wanted to start by addressing what you want in an API for your product. Firstly, and perhaps most importantly, you want an API for your users. They need to be able to get their data out of the product, input new data, and transform it in meaningful ways. Ultimately, you are building a system for them, and it has to be easy for them to interact with it.
00:01:58.920
That's fairly obvious, but sometimes it needs to be reiterated. You also want an API for users who are treating your system as a platform, trying to build products on top of it. We hear a lot about ecosystems, and while it may seem overemphasized, it’s something to support, as it benefits both your users and your business. If others add value to your product, it makes your product more integral to their workflow.
00:02:21.120
Finally, and maybe most importantly, you want an API for yourselves. You need to build a unified platform that you can develop upon as your company grows. Ideally, new feature development should leverage your API, which should be the same API exposed to external users as well as used internally. This unified approach ensures you have hands-on experience with the API, allowing you to make informed statements about its capabilities.
00:02:40.259
At Mavenlink, most of our new feature development is built on our public API. This allows us to confidently state that anything our mobile app can do can also be done through our API since they are the same.
00:03:02.760
Therefore, it should be evident that you want a well-defined API for your product. Ideally, one that you can also build upon. What else do you want in an API?
00:03:18.879
I would argue that you want a consistent API. This should be fairly obvious, but consistency is important in various contexts. For instance, the date formats should always be the same—whether you choose ISO 8601 or Unix timestamps, just pick one and stick to it. Additionally, think about IDs; they should have a consistent format, whether they are numbers or strings. It's also key that routes are predictable and that the response structures across your API endpoints are standardized and somewhat guessable.
00:03:40.740
In our case with Brainstem, we use JSON for responses. While some may choose XML, which I personally wouldn’t recommend, it’s still essential to have a versioned API. Upgrading APIs is always a hassle; it's hard to manage for both your team and the users consuming your API.
00:04:02.640
Quick show of hands: how many of you have had an API change without a version bump? About a third of you. And how many of you have changed an API on someone else? It's a common struggle.
00:04:17.040
Even with versioning, dealing with API changes remains complex, but versioning certainly helps.
00:04:45.240
Finally, you want your API to be fast. What does a fast API look like? In Rails, a fast API closely interacts with the database, performing as few queries as possible and ideally working with some scopes that you've already set up in your models. I assume you are using ActiveModel, probably ActiveRecord, or at least something built on top of ActiveModel.
00:05:02.160
Additionally, loading associations should be quick and done in the same request. This concept, known as side-loading, means if you're trying to get a project and also want to retrieve all the associated posts or participants, you should be able to do that as a single request.
00:05:21.900
It's the equivalent of an ActiveRecord include statement to avoid the N+1 query problem. An efficient API avoids object repetition. In a scenario of loading posts, if these posts belong to users, you wouldn’t want to return each full user object for every post; ideally, you'd reference the user by ID.
00:05:43.020
This keeps the response size manageable, and using a compression algorithm like GZIP can help optimize performance, though it’s important to remember that the data must still be decompressed and parsed on the client side.
00:06:05.640
Additionally, your API should allow for expressive filtering and sorting to help users retrieve only the data they need.
00:06:13.920
These principles guided our motivation for building Brainstem. When we initiated our project, we felt that no existing solution catered to our specific needs. While the API landscape has evolved over the last year, I will discuss that further later.
00:06:32.760
Let me illustrate a motivating example from our mobile app where we need to express complex relational data. Here’s a simple interaction: a user posts a private message to two others, saying, 'Hey guys, I am working on my slides,' and attaches a copy of the keynote file.
00:06:49.860
The post associates with a task called 'Make Presentation.' In response, Jeff replies, 'Looks like a great start.' For the display, I want this post object, which serves as the focus of the view, along with its associated data.
00:07:09.840
To achieve this, I need various data elements: the project title, recipients’ names, the post text, details about the attachment such as size and icon, the task name, and any replies associated with the post.
00:07:32.040
Serializing this relational data presents challenges, as we’ll see in different methods. One naive approach is to create a JSON structure containing all necessary data but none extraneous.
00:07:52.740
This approach is functional until there's a need for additional information—making changing the underlying API cumbersome.
00:08:14.880
If many posts reference the same project, that results in significant repetition across posts. This also begs the question: where did my relational model go that prompted me to use Rails and ActiveRecord in the first place?
00:08:34.200
In a better alternative, we might keep track of everything using IDs—a more efficient method. This approach means every post and reply is included in a more organized JSON structure.
00:08:53.640
This side-loading means that in one request, all related objects are returned. Although this might generate more top-level objects than the naive response, it fosters greater reusability.
00:09:13.800
Furthermore, it's more consistent, enabling predictable object fields without needing project titles scattered throughout the structure.
00:09:32.520
So long as we are retracing necessary IDs, the response won’t inherently lead to longer loading times, particularly since many IDs likely overlap.
00:09:48.960
More importantly, we recover our relational model, which is what we wanted to achieve.
00:10:06.780
Now, let’s explore how to generate a JSON response like that. Brainstem allows you to do this with a single request. You would simply make a request to the posts API endpoint and tell it to include the project, user, recipients, tasks, attachments, and replies.
00:10:23.400
You could also set defaults based on what you think most users would want. The relational structure you saw is serialized from ActiveRecord objects using Brainstem presenters.
00:10:39.840
What is a presenter? A presenter takes an ActiveRecord object and serializes it into the fields you wish to send over the wire—JSON in our case, but it could be adapted for other formats.
00:10:54.360
Let me show you an example of a Brainstem presenter. This presenter inherits from the Brainstem presenter, and it is versioned—you may notice that the module has a version, and changing it will alter the presenter’s version accordingly.
00:11:14.520
Here, we present a post with a method called 'present,' which takes a post and returns a Ruby structure containing the data you want.
00:11:34.920
To add associations in a way that enhances the output, use an association keyword. List the associations from the model that you want to expose to users. For each association, you must create a corresponding presenter.
00:11:55.440
The Brainstem framework infers that these associated records are objects, ensuring they serialize properly, while being cognizant of how to reference them—this is crucial.
00:12:13.020
Let’s move to another area. I will illustrate how this integrates into a Rails controller. For example, you might have an API V1 post controller that inherits from your application controller.
00:12:31.440
This controller should also inherit from an API controller to include shared behaviors, such as authentication, while mixing in Brainstem controller methods.
00:12:49.560
Those methods, for example, enable you to present data from your posts table. You provide a scope in the block that dictates the source of your data modifications. This design keeps operations front-facing and efficient.
00:13:08.520
For instance, you might start with an unscoped post without any restrictions. Alternatively, you could filter it to return posts visible to a specific user, requiring associated authentication and authorization.
00:13:29.160
By implementing this correctly, you gain inclusions as mentioned earlier, and you can request the JSON for a specific post ID—ensuring you only get relevant data back.
00:13:45.160
This method operates consistently across all CRUD actions, and filters and sorts can enhance user experience, allowing users to exert control over received data.
00:14:02.880
Here’s how you can implement sorting in a presenter. We declare sortable columns like 'updated_at' and 'created_at.' The sort order can utilize a lambda for custom sorting while allowing user-defined conditions.
00:14:22.320
For filtering, we can set up a filter on task IDs in the presenter. Filtering takes a lambda to extract relevant scopes, ensuring users only get posts linked to operations they care about.
00:14:42.720
This approach promotes data security while streamlining the logic within the Rails controller, making the backend more efficient.
00:15:02.880
To summarize this section, I have outlined how to craft a straightforward API in Brainstem, exposing filters, sorts, include queries, and presenting comprehensive data effectively.
00:15:19.260
Now, let’s pivot to client-side implementation, relevant for this audience since we all address similar client-server API challenges.
00:15:37.320
This methodology is geared for developers crafting views and employing Backbone to consume the API we’ve developed.
00:15:52.800
Using your own API during development is critical; it's a means of 'eating your own dog food,' ensuring that you are living on your own platform.
00:16:09.280
We realized that combining Brainstem with Backbone enabled us to develop Brainstem.js—an adapter or shim for Backbone to facilitate relational model loading over Brainstem.
00:16:26.040
Similar to Ember Data, Brainstem.js preserves relational models over the wire, while adapting Backbone without existing libraries managing the data as we needed.
00:16:45.040
Let me illustrate a simple example in CoffeeScript, showing how you can consume Brainstem data within your view. We begin by creating a storage manager.
00:17:02.800
The storage manager orchestrates managing different models and collections that come across the wire, maintaining their interrelations.
00:17:21.440
Initially, it functions as a global singleton instance. We're refactoring it to allow for more versatility, but even now, it ensures seamless interactions across various views.
00:17:39.400
Moreover, it acts as an identity map, keeping track of object mappings by type and ID. This enhances efficiency by circumventing re-requests for already-retrieved data.
00:17:58.320
Using this architecture in our mobile app meant that requests were optimized for speed, ensuring data packets sent over the wire remained compact and quick.
00:18:17.240
Essentially, the storage manager can load multiple collections, facilitating pagination while allowing specified inclusions of related data.
00:18:36.840
You could also load single models, applying filters and sorts as required. While Backbone is not mandatory for Brainstem users, if you’re using Backbone, the integration is valuable.
00:18:54.700
The summary here is that Brainstem.js extends Backbone's capabilities with relational models, assuring all necessary associations are self-sufficient in memory before confirming readiness.
00:19:10.080
When utilizing these relational models, it functions very similarly to Backbone's traditional methods, augmenting interactions within the API.
00:19:28.960
Sharing these design experiences, let me touch upon a few design decisions that arose during the Brainstem development that may interest you.
00:19:47.460
First, we opted to use IDs as keys rather than arrays in our API responses, improving performance during data traversal.
00:20:07.320
This means the response structure functions like a lookup table, allowing faster accesses than iterating through arrays, which merely encourages inefficient practices.
00:20:23.680
One common concern with hashes is sorting; we addressed this through a results array that neatly matches the exact response set in a structured manner.
00:20:46.580
Users should loop through this results array, referencing the base objects as needed. Brainstem.js automates this process to ease workflow.
00:21:05.540
Additionally, another decision we made was to include filters within presenters. This allows versioning and upgrades to proceed seamlessly alongside other updates.
00:21:25.620
We wanted Brainstem to know how to filter data without inferring from the controller, ultimately aiming for more extensible options in the future.
00:21:41.760
This becomes invaluable in scenarios where users may want to express specific filters and retrieve comprehensive data sets in single requests, instead of bombarding the server.
00:22:06.600
Regarding additional libraries that emerged, Active Model Serializers is a noteworthy one, offering a mature DSL for defining fields but lacking filtering and sorting capabilities that Brainstem encompasses.
00:22:30.720
Similarly, Ember Data is another robust library, but given our backbone-centric architecture, Brainstem.js integrates some Ember Data features favorably.
00:22:53.640
In summary, Brainstem serves as both a presenter library for serializing models and an API abstraction layer for building efficient APIs. It offers an optional adapter for Backbone.
00:23:11.880
At its best, using Brainstem allows for user-sorted, filterable APIs with efficient side-loading of associations, simplifying the API's overall functionality.
00:23:33.600
We launched Brainstem just a few moments ago—check it out! You can view these slides online, and feel free to follow me on Twitter. I’m @tectonic.
00:23:50.760
I’m happy to answer any questions now or after the talk.
