Better APIs with Pliny

Ruby on Ales 2015

Will Leinweber

#service-oriented-architecture

Better APIs with Pliny

by Will Leinweber

The video titled 'Better APIs with Pliny' features Will Leinweber discussing the development of APIs through the lens of the Pliny framework, which is built on Sinatra and beneficial for Ruby developers. The presentation is structured to first clarify the significance of splitting applications into services and the considerations that come with it, followed by a deep dive into the features and advantages of using Pliny.

Key points discussed include:

- Importance of Services: Pliny allows teams to focus on specific functionalities, improving maintainability and enabling independent scaling. It addresses challenges that arise as teams grow, reflecting Conway's Law which states systems mirror organizational structures.

- Caution Against Microservices: Although microservices have their benefits, Leinweber highlights the potential costs and complexity they introduce, such as the loss of shared databases, increased overhead in API interactions, and disparate deployment processes.

- API Design Principles: The presentation introduces the HTTP API Design Guide, emphasizing best practices like using e-tags, consistent date formatting, and minimizing path nesting. These principles aim to enhance the efficiency and usability of APIs.

- Introduction to Pliny: Pliny is outlined as a solution to manage API complexity, enabling better logging, configuration management, and effective serialization. Features like support for JSON schemas and elegant versioning make it particularly suitable for dynamic applications.

- Use Cases: Leinweber references practical examples such as 'Telex', an open-source project that centralizes notifications, showcasing how Pliny facilitates simpler code management and application interaction.

- Encapsulation and Mediators: He stresses the importance of keeping controllers clean through mediators, simplifying testing and understanding of the application structure.

- Community and Collaboration: The speaker encourages developers to engage with the open-source community surrounding Pliny, promoting collaborative improvement of the tools.

Overall, attendees are encouraged to leverage Pliny for their API projects, emphasizing its philosophy of reducing complexity and enhancing team collaboration, thus leading to better API development.

00:00:15 Thank you.

00:00:29 Will works a lot with Postgres at Heroku. Heroku recently launched support for Postgres 9.4 in January, including JSONB. I'm currently using it for one of my projects, and it is fantastic. So, thank you, Will!

00:00:40 For those of you who have ever used the watch command in Postgres, you can monitor a query like you would in a Linux terminal. For example, you can do a 'SELECT * FROM users' and watch the results, which will continuously update as the UI changes. That is Will's patch, and it's true—this part is a fact.

00:01:01 Now, I'd like to welcome you all. Today, I want to talk about building APIs. Specifically, I will discuss some considerations when doing that and then introduce a tool called Pliny, which is an appropriate tool for this conference because it's named after a fantastic beer. It is also a Ruby library well-suited for a beer-themed Ruby conference.

00:01:22 Pliny is super easy to use—the first step is to open a bottle of Pliny, and the second step is to write your API. This makes the process significantly better!

00:01:38 I brought a couple of cases of Pliny to share with you all because I know that many of you may not have had the chance to try it. While I don’t think it’s the best beer out there, it’s definitely one of my favorites.

00:01:59 Pliny is brewed by Russian River Brewing in California. I checked where they distribute it and found a few places in Portland. Unfortunately, when I visited Belmont Station, I was informed that Russian River had stopped shipping bottles to Oregon. They suggested I try Washington, but I figured that wasn’t necessary. However, if you’re in San Francisco, Pliny is always on tap at Toronado. I'm willing to join you there!

00:02:20 Now, let’s talk about what this session will cover. The first part will discuss services and the reasons to consider splitting your app into multiple services, as well as why you might not want to.

00:02:37 Next, we’ll cover some needs that API-centered applications have, before finally diving into the details of Pliny itself.

00:02:47 Firstly, let’s consider why services are important. There was a great talk yesterday that highlighted how, at times, you can avoid using services while still having a productive monolithic application. That approach is great, but you may want to split certain components into services when your code base becomes increasingly complex.

00:03:06 Having distinct separations can lead to more discipline within the code and often helps mitigate unexpected failures and breaks when adding new features.

00:03:11 Let’s watch that one more time... that’s sad! Let's leave it at that. So, one of the primary advantages of having a separate service is that it can focus on a single purpose. This single-purpose approach makes it easier to understand and maintain the code, especially when it comes to scaling different parts independently.

00:03:42 If your service requires a different type of implementation that benefits from another language or different set of tools, it allows for those decisions to be made as well.

00:04:01 However, I believe the main reason for using services is that as teams grow, the need for services becomes more pronounced. Larger teams should think about how they structure themselves and the code they are working on.

00:04:23 I recall the first time I heard Conway's Law, which states that systems tend to reflect the social structure of the organizations that create them. Initially, I took this as a challenge, thinking there must be a technical way to avoid this. But over time, I recognized that this situation is, unfortunately, inevitable.

00:04:57 Instead of trying to circumvent this occurrence, I’ve realized it's more effective to embrace it. Teams should focus on how they should be structured, acknowledging that software systems will ultimately embody that structure.

00:05:12 One of the things that's easier to coordinate within a team rather than between multiple teams is having their own codebases. This allows for focused iterations without stepping on each other's toes.

00:05:29 That said, there are many reasons to be cautious about using services. Over the past year or so, there has been a lot of talk regarding microservices, which has always given me pause.

00:05:49 Many articles only address the positives of microservices and gloss over the reality that they can be extremely costly. Implementing services takes significant time and effort, which can often be underestimated.

00:06:05 These concepts go through the typical hype cycles, and we need to be mindful of that. I wanted to share some reasons why sticking with a monolithic approach might not be so bad for a while longer.

00:06:24 As I mentioned earlier, I focus a lot on databases. One downside to splitting services into separate applications is losing a shared database, which has many advantages including the ability to conduct transactions and efficient joins without overloading applications.

00:06:54 When everything is centralized in one database, managing reporting is also simpler. Once you have several different services, you often have to make API calls and handle responses, which can lead to inefficiency.

00:07:02 To provide a concrete example, I work on a tool that provisions and monitors production databases at Heroku. When a new database is added, it triggers a series of provisioning actions and error handling that involves a complex interface.

00:07:29 We've encountered various challenges; for instance, internal applications are treated the same as customer apps, making it difficult to separate and prioritize our internal needs over those of our customers.

00:08:02 Additionally, each service requires its own deployment, versioning, monitoring, and other processes, which can require a considerable amount of overhead depending on your organization’s maturity.

00:08:33 It’s vital to note that sometimes it is necessary to split services, but you must carefully consider that decision. You're all talented individuals, so I’m sure this is something you’ve reflected on.

00:09:01 In the next part, I’d like to discuss some considerations when building an API and the basic design principles involved.

00:09:21 The API team has put together a guide called the HTTP API Design Guide, which outlines many points to consider when creating your API. It addresses many straightforward concepts, like using e-tags for caching, accurately interpreting accept headers, and utilizing appropriate status codes.

00:09:43 Also, it's important to ensure that dates are consistently formatted, ideally following ISO 8601. The guide goes on to introduce some more advanced and novel ideas.

00:10:07 One interesting suggestion is to include request IDs. When a request is made, it includes a unique request ID, which gets passed along in the logs and helps trace requests through a distributed system.

00:10:30 Another suggestion is to minimize path nesting in your URLs; keeping paths shallow not only improves readability but also efficiency.

00:10:45 Lastly, the guide discusses the use of range headers for API pagination. If you only need to return a limited set of items, clients can specify ranges in their requests, making the API more efficient.

00:11:04 There are many more considerations in the guide, and if you have any questions or think there’s something amiss, feel free to open an issue on GitHub. The maintainers are usually eager to discuss.

00:11:26 Now let’s transition to the main topic: Pliny. Pliny is a Sinatra-based tool extracted from Heroku's main API over time. Initially, Heroku started as a single massive Rails application.

00:11:49 As the project matured, they began to migrate away from that structure. I remember discovering a heavily bloated class while fixing a patch on their app, which handled state transitions in a cumbersome manner.

00:12:14 Fortunately, they've since restructured the code to adopt new patterns, maintaining active record while favoring Sinatra. This evolution brought with it many valuable lessons that have been implemented in Pliny.

00:12:36 Pliny helps eliminate minor decisions you would otherwise face when structuring your APIs. These small decisions might seem insignificant at first, but they accumulate and can lead to burden.

00:13:02 By using Pliny, remote requests, logging, and configuration management become streamlined processes, all while maintaining effective serialization for outgoing JSON data.

00:13:19 There are three main categories I want to discuss regarding Pliny. The first covers management and operational aspects of your application.

00:13:36 The second covers interactions between your service and others, and finally, I’ll discuss features to help manage growth as your application evolves.

00:14:00 Most of the examples I will use come from an open-source project I created called Telex. Telex centralizes notifications sent from various projects in Heroku, as they were being implemented in many fragmented ways.

00:14:23 I open-sourced this project because I was able to leverage Pliny effectively. I wanted to provide others with practical examples of how to do so, and that advantage isn't available to many others.

00:14:44 Another open source project some colleagues worked on is Transferrato, a backup and restoration service designed to replace the PG Backups add-on.

00:15:00 What I find most appealing about Pliny is that it is Ruby-based. I appreciate Ruby, but one aspect that isn’t often discussed is the ability to open a production console.

00:15:20 This feature allows you to work interactively within your production environment. Having IRB at my disposal has proven extremely useful during challenging situations, like outages. It enables you to quickly write small scripts to manage and recover applications.

00:15:45 Controversially, some argue that you should expose your service to every possible API, but I believe that approach isn’t particularly efficient. Instead, having a basic API for your production console can be a lifesaver.

00:16:07 Pliny uses Postgres as its backend, which I’m a big fan of. There’s also a wonderful SQL library by Jeremy Evans that I highly endorse.

00:16:22 SQL is incredibly well-maintained and presents a unique way to write queries in a similar format to raw SQL, making it easy to work with PostgreSQL effectively.

00:16:45 When implementing notification retrieval, I found the eager loading performance and clarity with SQL exceptional. It allows me to execute more complex queries efficiently without excessive overhead.

00:17:21 Additionally, the API design guide emphasizes the importance of using JSON schemas, which Pliny supports well. Thus, you can generate machine-readable interfaces that streamline interactions through your API.

00:17:38 Moreover, employing JSON schemas allows for automated validation of requests and responses, helping prevent regressions while providing standardized error messages when validating incoming data.

00:17:59 Lastly, versioning and serialization are handled elegantly within Pliny. Pliny makes it straightforward to manage multiple versions of your API.

00:18:20 It allows you to name and track them effectively, as most applications will generally support at least two versions—one being phased out and another one currently implemented.

00:18:45 Finally, one of Pliny's greatest strengths is its support for mediator objects. Mediators encapsulate interactions to keep application models and controllers clean and focused.

00:19:08 This structure helps streamline testing and enhances understanding of what each piece of your application does, as you manage interactions more effectively.

00:19:45 An example of a mediator in Telex is the component that handles incoming messages and notification creation to ensure notifications are dispatched to the correct users.

00:20:06 Pliny provides several additional features, including CORS support for your JavaScript applications, a request store that leverages thread-local variables, and robust configuration management.

00:20:29 I recommend anyone interested in using Pliny reach out to me or open an issue on the project. Having a consistent set of tools for building various APIs can help standardize practices and improve teamwork.

00:20:57 Ultimately, it has been enjoyable to use Pliny when building applications, so I encourage everyone to give it a try. Thank you for your time!

Ruby on Ales 2015