Introducing Brainstem, your companion for rich Rails APIs

Talks

Andrew Cantino

Introducing Brainstem, your companion for rich Rails APIs

by Andrew Cantino

In the video "Introducing Brainstem, your companion for rich Rails APIs" presented by Andrew Cantino at RailsConf 2013, the focus is on the Brainstem library, which is a tool designed to enhance the development of APIs in Ruby on Rails, particularly for complex data models found in applications like Mavenlink's project and financial management software. Cantino explains the challenges of working with complex relational data and emphasizes the need for APIs that are user-oriented, consistent, fast, and manageable.

Key points discussed include:

- Complex Data Models: Introduced by reflecting on Mavenlink’s relational data structure, which must express intricate user and business relationships, forming the basis for the development of Brainstem.

- Importance of API Design: Three key users of an API are identified: users of the product, developers building atop the product as a platform, and the developers making internal capabilities better. All components must work consistently across these interactions.
- API Characteristics: Emphasized the need for an API to be well-defined, consistent, and fast.

- Consistency includes using standard response formats and predictable route structures,

- Fast APIs minimize database queries and integrate side-loading to fetch associated data in one request.

- Brainstem's Functionality: Cantino describes how Brainstem allows for streamlined serialization of complex relational data into a single JSON response with the use of presenters. Presenters convert ActiveRecord objects into desired JSON formats, allowing for efficient and clear data handling while preserving relational integrity.
- Integration with Backbone: The presentation introduces Brainstem.js, an adapter that allows Backbone.js to consume Brainstem APIs effectively, maintaining relationships between data models and facilitating smoother interactions.
- Design Decisions: Notable decisions include using IDs for faster data access rather than arrays, incorporating filters into presenters for better versioning and extensibility, and ensuring user-centric query management.
- Launch Announcement: Cantino concludes the presentation with an announcement about Brainstem's public release on GitHub, inviting viewers to explore its functionalities further.

This talk showcases the significant impact Brainstem can have on building rich, efficient, and user-friendly Rails APIs, giving developers robust tools to manage complex architectures without compromising performance or usability.

00:00:12.259 Thank you! My name is Andrew Cantino.

00:00:19.800 I work for a company called Mavenlink, where we build project and financial management software for consultancies. We're based in San Francisco.

00:00:26.340 I am here today to introduce you to Brainstem, an API presentation library that we've been working on off and on for about a year. We actually just released it publicly on GitHub just a few minutes ago.

00:00:37.800 I am excited to tell you all about it.

00:00:44.399 Mavenlink has a really complicated data model. For example, we have projects that contain tasks, which have budgets that are reflected in time entries. Those belong to timesheets that are included in reports, etc. It's a very relational product with a complex data model.

00:00:57.360 This complexity is necessary because we are ultimately trying to build something that models both business and human relationships. These relationships are incredibly complicated, and while we aim to simplify the model, there’s only so far we can go. We need our API to express these relationships in a meaningful and consistent way, which is how Brainstem evolved.

00:01:21.840 We began developing Brainstem about a year ago as an extraction from our product. We've been using it internally for some time, and now it powers both our internal and external APIs.

00:01:40.259 I wanted to start by addressing what you want in an API for your product. Firstly, and perhaps most importantly, you want an API for your users. They need to be able to get their data out of the product, input new data, and transform it in meaningful ways. Ultimately, you are building a system for them, and it has to be easy for them to interact with it.

00:01:58.920 That's fairly obvious, but sometimes it needs to be reiterated. You also want an API for users who are treating your system as a platform, trying to build products on top of it. We hear a lot about ecosystems, and while it may seem overemphasized, it’s something to support, as it benefits both your users and your business. If others add value to your product, it makes your product more integral to their workflow.

00:02:21.120 Finally, and maybe most importantly, you want an API for yourselves. You need to build a unified platform that you can develop upon as your company grows. Ideally, new feature development should leverage your API, which should be the same API exposed to external users as well as used internally. This unified approach ensures you have hands-on experience with the API, allowing you to make informed statements about its capabilities.

00:02:40.259 At Mavenlink, most of our new feature development is built on our public API. This allows us to confidently state that anything our mobile app can do can also be done through our API since they are the same.

00:03:02.760 Therefore, it should be evident that you want a well-defined API for your product. Ideally, one that you can also build upon. What else do you want in an API?

00:03:18.879 I would argue that you want a consistent API. This should be fairly obvious, but consistency is important in various contexts. For instance, the date formats should always be the same—whether you choose ISO 8601 or Unix timestamps, just pick one and stick to it. Additionally, think about IDs; they should have a consistent format, whether they are numbers or strings. It's also key that routes are predictable and that the response structures across your API endpoints are standardized and somewhat guessable.

00:03:40.740 In our case with Brainstem, we use JSON for responses. While some may choose XML, which I personally wouldn’t recommend, it’s still essential to have a versioned API. Upgrading APIs is always a hassle; it's hard to manage for both your team and the users consuming your API.

00:04:02.640 Quick show of hands: how many of you have had an API change without a version bump? About a third of you. And how many of you have changed an API on someone else? It's a common struggle.

00:04:17.040 Even with versioning, dealing with API changes remains complex, but versioning certainly helps.

00:04:45.240 Finally, you want your API to be fast. What does a fast API look like? In Rails, a fast API closely interacts with the database, performing as few queries as possible and ideally working with some scopes that you've already set up in your models. I assume you are using ActiveModel, probably ActiveRecord, or at least something built on top of ActiveModel.

00:05:02.160 Additionally, loading associations should be quick and done in the same request. This concept, known as side-loading, means if you're trying to get a project and also want to retrieve all the associated posts or participants, you should be able to do that as a single request.

00:05:21.900 It's the equivalent of an ActiveRecord include statement to avoid the N+1 query problem. An efficient API avoids object repetition. In a scenario of loading posts, if these posts belong to users, you wouldn’t want to return each full user object for every post; ideally, you'd reference the user by ID.

00:05:43.020 This keeps the response size manageable, and using a compression algorithm like GZIP can help optimize performance, though it’s important to remember that the data must still be decompressed and parsed on the client side.

00:06:05.640 Additionally, your API should allow for expressive filtering and sorting to help users retrieve only the data they need.

00:06:13.920 These principles guided our motivation for building Brainstem. When we initiated our project, we felt that no existing solution catered to our specific needs. While the API landscape has evolved over the last year, I will discuss that further later.

00:06:32.760 Let me illustrate a motivating example from our mobile app where we need to express complex relational data. Here’s a simple interaction: a user posts a private message to two others, saying, 'Hey guys, I am working on my slides,' and attaches a copy of the keynote file.

00:06:49.860 The post associates with a task called 'Make Presentation.' In response, Jeff replies, 'Looks like a great start.' For the display, I want this post object, which serves as the focus of the view, along with its associated data.

00:07:09.840 To achieve this, I need various data elements: the project title, recipients’ names, the post text, details about the attachment such as size and icon, the task name, and any replies associated with the post.

00:07:32.040 Serializing this relational data presents challenges, as we’ll see in different methods. One naive approach is to create a JSON structure containing all necessary data but none extraneous.

00:07:52.740 This approach is functional until there's a need for additional information—making changing the underlying API cumbersome.

00:08:14.880 If many posts reference the same project, that results in significant repetition across posts. This also begs the question: where did my relational model go that prompted me to use Rails and ActiveRecord in the first place?

00:08:34.200 In a better alternative, we might keep track of everything using IDs—a more efficient method. This approach means every post and reply is included in a more organized JSON structure.

00:08:53.640 This side-loading means that in one request, all related objects are returned. Although this might generate more top-level objects than the naive response, it fosters greater reusability.

00:09:13.800 Furthermore, it's more consistent, enabling predictable object fields without needing project titles scattered throughout the structure.

00:09:32.520 So long as we are retracing necessary IDs, the response won’t inherently lead to longer loading times, particularly since many IDs likely overlap.

00:09:48.960 More importantly, we recover our relational model, which is what we wanted to achieve.

00:10:06.780 Now, let’s explore how to generate a JSON response like that. Brainstem allows you to do this with a single request. You would simply make a request to the posts API endpoint and tell it to include the project, user, recipients, tasks, attachments, and replies.

00:10:23.400 You could also set defaults based on what you think most users would want. The relational structure you saw is serialized from ActiveRecord objects using Brainstem presenters.

00:10:39.840 What is a presenter? A presenter takes an ActiveRecord object and serializes it into the fields you wish to send over the wire—JSON in our case, but it could be adapted for other formats.

00:10:54.360 Let me show you an example of a Brainstem presenter. This presenter inherits from the Brainstem presenter, and it is versioned—you may notice that the module has a version, and changing it will alter the presenter’s version accordingly.

00:11:14.520 Here, we present a post with a method called 'present,' which takes a post and returns a Ruby structure containing the data you want.

00:11:34.920 To add associations in a way that enhances the output, use an association keyword. List the associations from the model that you want to expose to users. For each association, you must create a corresponding presenter.

00:11:55.440 The Brainstem framework infers that these associated records are objects, ensuring they serialize properly, while being cognizant of how to reference them—this is crucial.

00:12:13.020 Let’s move to another area. I will illustrate how this integrates into a Rails controller. For example, you might have an API V1 post controller that inherits from your application controller.

00:12:31.440 This controller should also inherit from an API controller to include shared behaviors, such as authentication, while mixing in Brainstem controller methods.

00:12:49.560 Those methods, for example, enable you to present data from your posts table. You provide a scope in the block that dictates the source of your data modifications. This design keeps operations front-facing and efficient.

00:13:08.520 For instance, you might start with an unscoped post without any restrictions. Alternatively, you could filter it to return posts visible to a specific user, requiring associated authentication and authorization.

00:13:29.160 By implementing this correctly, you gain inclusions as mentioned earlier, and you can request the JSON for a specific post ID—ensuring you only get relevant data back.

00:13:45.160 This method operates consistently across all CRUD actions, and filters and sorts can enhance user experience, allowing users to exert control over received data.

00:14:02.880 Here’s how you can implement sorting in a presenter. We declare sortable columns like 'updated_at' and 'created_at.' The sort order can utilize a lambda for custom sorting while allowing user-defined conditions.

00:14:22.320 For filtering, we can set up a filter on task IDs in the presenter. Filtering takes a lambda to extract relevant scopes, ensuring users only get posts linked to operations they care about.

00:14:42.720 This approach promotes data security while streamlining the logic within the Rails controller, making the backend more efficient.

00:15:02.880 To summarize this section, I have outlined how to craft a straightforward API in Brainstem, exposing filters, sorts, include queries, and presenting comprehensive data effectively.

00:15:19.260 Now, let’s pivot to client-side implementation, relevant for this audience since we all address similar client-server API challenges.

00:15:37.320 This methodology is geared for developers crafting views and employing Backbone to consume the API we’ve developed.

00:15:52.800 Using your own API during development is critical; it's a means of 'eating your own dog food,' ensuring that you are living on your own platform.

00:16:09.280 We realized that combining Brainstem with Backbone enabled us to develop Brainstem.js—an adapter or shim for Backbone to facilitate relational model loading over Brainstem.

00:16:26.040 Similar to Ember Data, Brainstem.js preserves relational models over the wire, while adapting Backbone without existing libraries managing the data as we needed.

00:16:45.040 Let me illustrate a simple example in CoffeeScript, showing how you can consume Brainstem data within your view. We begin by creating a storage manager.

00:17:02.800 The storage manager orchestrates managing different models and collections that come across the wire, maintaining their interrelations.

00:17:21.440 Initially, it functions as a global singleton instance. We're refactoring it to allow for more versatility, but even now, it ensures seamless interactions across various views.

00:17:39.400 Moreover, it acts as an identity map, keeping track of object mappings by type and ID. This enhances efficiency by circumventing re-requests for already-retrieved data.

00:17:58.320 Using this architecture in our mobile app meant that requests were optimized for speed, ensuring data packets sent over the wire remained compact and quick.

00:18:17.240 Essentially, the storage manager can load multiple collections, facilitating pagination while allowing specified inclusions of related data.

00:18:36.840 You could also load single models, applying filters and sorts as required. While Backbone is not mandatory for Brainstem users, if you’re using Backbone, the integration is valuable.

00:18:54.700 The summary here is that Brainstem.js extends Backbone's capabilities with relational models, assuring all necessary associations are self-sufficient in memory before confirming readiness.

00:19:10.080 When utilizing these relational models, it functions very similarly to Backbone's traditional methods, augmenting interactions within the API.

00:19:28.960 Sharing these design experiences, let me touch upon a few design decisions that arose during the Brainstem development that may interest you.

00:19:47.460 First, we opted to use IDs as keys rather than arrays in our API responses, improving performance during data traversal.

00:20:07.320 This means the response structure functions like a lookup table, allowing faster accesses than iterating through arrays, which merely encourages inefficient practices.

00:20:23.680 One common concern with hashes is sorting; we addressed this through a results array that neatly matches the exact response set in a structured manner.

00:20:46.580 Users should loop through this results array, referencing the base objects as needed. Brainstem.js automates this process to ease workflow.

00:21:05.540 Additionally, another decision we made was to include filters within presenters. This allows versioning and upgrades to proceed seamlessly alongside other updates.

00:21:25.620 We wanted Brainstem to know how to filter data without inferring from the controller, ultimately aiming for more extensible options in the future.

00:21:41.760 This becomes invaluable in scenarios where users may want to express specific filters and retrieve comprehensive data sets in single requests, instead of bombarding the server.

00:22:06.600 Regarding additional libraries that emerged, Active Model Serializers is a noteworthy one, offering a mature DSL for defining fields but lacking filtering and sorting capabilities that Brainstem encompasses.

00:22:30.720 Similarly, Ember Data is another robust library, but given our backbone-centric architecture, Brainstem.js integrates some Ember Data features favorably.

00:22:53.640 In summary, Brainstem serves as both a presenter library for serializing models and an API abstraction layer for building efficient APIs. It offers an optional adapter for Backbone.

00:23:11.880 At its best, using Brainstem allows for user-sorted, filterable APIs with efficient side-loading of associations, simplifying the API's overall functionality.

00:23:33.600 We launched Brainstem just a few moments ago—check it out! You can view these slides online, and feel free to follow me on Twitter. I’m @tectonic.

00:23:50.760 I’m happy to answer any questions now or after the talk.

RailsConf 2013