Derek Carter
Rails APIs: The Next Generation
Summarized using AI
Rails APIs: The Next Generation
by Derek CarterIn this video titled "Rails APIs: The Next Generation," Derek Carter discusses Procore's journey in building a comprehensive API for their extensive web application. Procore, a cloud-based construction management software company, operates with a large team and a complex application architecture that consists of over 40 distinct tools and more than 500 controllers. During the talk, Derek elaborates on the challenges faced by their tool-segmented engineering teams while developing a consistent API across their diverse applications.
Key points discussed include:
The Importance of APIs: Building an API creates a natural separation of concerns, fostering a modular architecture while also increasing developer happiness and customer trust. It allows seamless integration with other systems and expands the application's capabilities by encouraging third-party development.
Characteristics of a Good API: Derek outlines that a good API should be predictable, consistent, static, simple, and flexible. Each of these qualities reduces developer errors and enhances the overall usability of the API.
Development Strategies: Carter highlights the use of a squad model, where Procore's engineering teams are organized into squads which autonomously own specific tools. He discusses the introduction of a ‘guide squad’ that oversees API development to maintain consistency across squads.
Contracts and Communication: The importance of API contracts is emphasized, where agreements on endpoint functionalities are established amongst stakeholders to ensure clarity prior to development. This aids parallel development, allowing front and back-end teams to work simultaneously on agreed requirements.
Serialization Challenges: Derek discusses the switch from JBuilder to Active Model Serializer for better performance and organizational capabilities, explaining how it allows for clearer API definitions while reducing duplication of efforts across various endpoints.
Documentation: A significant portion of the talk is dedicated to API documentation, where Derek stresses that an undocumented API is virtually useless. He outlines how Procore utilizes Swagger (the Open API Specification) for documenting their APIs, facilitating easier maintenance and integration with third-party tools.
In conclusion, Derek Carter urges developers to prioritize API design quality, consistent documentation, and the establishment of supportive frameworks to enhance API efficacy. His key takeaways include fostering an environment that minimizes unnecessary decisions for developers, building comprehensive style guides, encouraging collaboration among teams, and passionately advocating for thorough API documentation.
00:00:12.830
Let me introduce myself. Instead of just being the guy telling jokes up on stage, my name is Derek Carter, and I work for Procore. I've been a web developer for about 20 years and a Rails developer since 2008.
00:00:21.230
This picture was taken by a colleague of mine, and it is from our office. I kid you not, I love our office.
00:00:26.930
I take a walk out here every single afternoon. For those of you who caught John's talk yesterday, you know a little bit about the history of Procore.
00:00:33.739
Procore is cloud-based construction management software. As our tagline says, "We build the software that builds the world." Today, I'm here to tell you about an endeavor that Procore embarked on in 2016 and into this year.
00:00:46.100
Procore decided to really double down and build out a complete API across all of our projects, applications, and tools. To give you a sense of what that endeavor entailed, I want to talk a little bit about the guts of Procore.
00:01:07.690
Procore has been growing like crazy. Last count, I think we have over 700 employees, about 120 engineers, and at least 22 squads. Our Rails app is over 10 years old, and we have over 40 distinct tools, each of which is basically an application unto itself.
00:01:36.720
Among those, we have over 500 controllers in our application. Procore is big; our application is big. It's all a Rails app—it’s huge! Forget majestic monolith; we are a majestic Uber Lyft.
00:01:46.850
I actually thought this would be a great name for a metal band, but beyond that, we operate with squads. One thing that Procore really believes in is autonomy—not just at the squad level, but at the personal level.
00:02:20.730
We believe in giving direction to people and allowing them to use their experience and knowledge to help us get where we need to go. We strongly adhere to the idea of autonomy, which is something we share with every new hire.
00:02:39.840
It's a strong part of our culture, and we believe very strongly in it. Let me elaborate on our autonomy concept.
00:02:53.310
We follow the Spotify squad model. Essentially, we break up our R&D department into squads that each have a product manager, UX designer, QA, and a handful of engineers.
00:03:04.140
These squads have ownership over certain tools. They have full control, making decisions about the product's direction and what to work on next.
00:03:11.720
This idea of autonomy is crucial. If you aren’t familiar with the squad model, squads roll up into tribes.
00:03:18.000
For things that cross squads and relate to common interests, we roll up into guilds. Some examples of guilds we have at Procore include a front-end guild that addresses front-end concerns.
00:03:30.750
We also have a performance guild that monitors our controllers and database performance. Additionally, we have a guild for master failures because, with an application as large as ours, we encounter a lot of random failures.
00:03:47.870
We have at least 22 squads, each owning their own tools within our extensive application. We aim to make a consistent effort to create one large API across the entire application. How do we achieve this without it feeling like we're herding cats? While it wasn't always a smooth ride, we ultimately got there.
00:04:13.200
As a spoiler, the end result is that we've created a full, consistent, and beautiful API—at least I think it's beautiful. But before I dive into how we accomplished this, I want to discuss why we did it.
00:04:32.070
Why did Procore decide to double down on investing in an API? There are many benefits to building out an API like this—both internal, which relates to developer happiness and code health, and external, which relates to customer satisfaction and sales health.
00:04:39.180
One internal benefit of building an API is the forced separation of concerns. You effectively create a distinction between your API layer and your view layer, which leads to a more modular architecture.
00:05:08.880
This leads to cleaner architecture. Another internal benefit is change tolerance. If we get a new Rails framework every three months, a good, consistent, and well-designed API can endure several iterations of different JavaScript frameworks or front-end libraries. You can replace those at any time, and yet your API remains relatively unchanged.
00:05:27.630
I like to use the analogy of televisions: we've had them in our living rooms for almost a century now, and the API for how a television gets its power—the power plug—has remained constant, even as television shapes have drastically changed.
00:05:40.470
When you design an API well, it can stand the test of time. One significant external benefit is that APIs foster customer trust. When customers compare products, a well-maintained API gives them confidence that they can easily input and extract their data from your system.
00:06:07.530
This can really aid in closing deals and foster trust between you and the customer, even before they've engaged with you. Furthermore, an API can significantly expand your capabilities. By empowering others—whether they are integrators, developers, or simply individuals interested in your business space—to build on your platform, you vastly increase what your application can do without investing extra resources.
00:06:34.140
When things function properly, as they do at Procore, you become the ecosystem, fostering an entire suite of applications that operate on your platform. Everything works seamlessly, and you can reap the benefits of that.
00:07:06.870
Now that I've discussed the benefits, let’s explore what makes a good API. When we were initiating this endeavor, what was our end goal?
00:07:14.159
A good API is predictable and consistent. Developers do not want to write new code for every endpoint you present. This issue is compounded when developing SDKs for your application. The more inconsistent your endpoints are, the more complexity you introduce into your code.
00:07:26.909
A good API should be static. If your API changes in a breaking way, congratulations—you’ve just broken everything that integrates with it. One saying I really like is that writing an API is like sex: make one mistake, and you’re stuck supporting it for the rest of your life.
00:07:42.090
A good API is also simple and clear. Writing an API is not the time for cleverness; it should give exactly what someone expects—nothing more and nothing less. Additionally, a good API is flexible. It may seem contradictory to assert that an API should be both flexible and static, but think about steel.
00:08:09.419
The goal is to make it strong without breaking its flexibility. So we’ve discussed what a good API should look like; now let’s address how we actually achieved it. Building this beautiful API wasn’t always easy or pretty. There was a lot of discussion and arguments among many smart and talented engineers at Procore.
00:08:41.578
Different engineers had varying views on how to approach things, leading to passionate debates about style and architecture. A principle that resonates with me, and that I've distilled through these discussions, is something Jeff Bezos states: "Disagree and commit." This means, even when there seems to be an impasse, someone has to give in for the sake of productivity.
00:09:07.000
You must be willing to accept a decision you don’t fully believe in and commit to it 100% for the sake of progress. I want to backtrack a moment to share some endeavors we undertook within the squad model at Procore that contributed to our success.
00:09:36.040
One approach we introduced is the concept of guide squads. A guide squad is essentially a squad that doesn’t own its own tools but oversees the process that spans across other squads. It functions like a guild but with more ownership. For instance, I am a part of the API squad, and while I don’t handle most endpoints directly—those are owned by developers in the specific tools—we act as shepherds in the API development process.
00:10:03.460
We help facilitate the 'disagree and commit' process and make important decisions to maintain productivity. So, what does this entail? As Steve Krug famously says, "Don’t make your developers think about matters that don’t concern them." This is crucial.
00:10:49.089
One of the most common questions from developers at Procore is, "Should I do this or that?" This usually is an abstract decision that doesn’t pertain to them. They want clarity on how to stay consistent with the rest of the application, so we must ensure they aren’t bogged down by irrelevant problems.
00:11:14.960
One solution we implemented is having a style guide. This stems from the Rails principle of convention over configuration. We have built a style guide for our API and for many different sections of our application.
00:11:38.900
It’s important to note that you do not have to have a comprehensive style guide planned out in advance; that would actually be detrimental. You won’t know all the decisions at once. Instead, I encourage you to maintain a wiki or some way to record every question that arises and its answer.
00:12:04.220
It’s imperative to document these decisions because you can't make a decision and then walk away. The following day, it’s common to forget—"Wait, what did we agree upon?" So write it down, and eventually, you’ll have a style guide. Ours looks quite nice.
00:12:25.460
This snippet is just a small section of it, formatted to look appealing. This style guide particularly helps new developers within our application who begin writing APIs, ensuring many of their questions are answered prior to seeking guidance from another developer.
00:12:46.600
Another key aspect is boilerplate and examples. Let’s be honest: when writing code, we often copy existing code. Therefore, let’s ensure that the source is as close to what we envision the final product should be.
00:13:12.190
So, consider a scenario where a developer is writing an API. This individual completes their API and sends it over to a front-end developer—this could be for a mobile team or integrators. Finally, once they receive the API, they realize it lacks significant features they actually need.
00:13:40.440
Thus, the backend developer returns to the API to add those necessities. Once it's finally done, the front-end developer celebrates, only for the product manager to realize that customers have different requirements.
00:14:03.650
You find yourself returning to square one. How do you overcome this repetitive cycle so developers can remain productive? One significant approach is creating API contracts. For those unfamiliar with an API contract, it is essentially an agreement between stakeholders regarding the expected appearance of the API when it’s finished.
00:14:28.070
In our scenario, we simply utilize JSON files. An example of a contract would be for a simple to-do list application—a common concept within the Rails community. We circulate these contracts for agreement and buy-in so front-end developers, mobile developers, and integrators can quickly verify whether all attributes they need are included.
00:14:54.100
This allows you to bypass prolonged discussions. Once everyone agrees to the central contract, developers can start working against it while other backend developers simultaneously write tests for it. This method empowers parallel development.
00:15:19.800
Next, I want to delve deeper into building static APIs flexibly. We found ourselves creating endpoints that did not entirely meet customer needs. Sometimes, it isn’t feasible to have contracts stipulating every requirement for the endpoints before they're fully developed.
00:15:54.220
This is where we introduced support levels for our APIs. We label them as alpha, beta, or production. Many times, when a developer is writing an API for a tool and isn’t certain whether it meets all stakeholder expectations, they'll tag it as beta.
00:16:06.940
Internal or external customers wishing to explore these beta tools are fully aware that these APIs could change, allowing us to gather valuable feedback from a product perspective. This model allows us to be more flexible, akin to pouring concrete, giving us time to modify it before it solidifies.
00:16:32.040
Once we gather sufficient feedback, we then promote it to production. An additional key point I want to emphasize is the necessity of using reusable components.
00:17:00.520
In my opinion, it’s impossible to build a consistent application without developing common components. Taking, for example, our to-do list app, we may have a product requirement to filter this endpoint by the completion status of items. In doing so, that creates a substantial amount of code.
00:17:26.430
This becomes particularly cumbersome when we have over 500 controllers across our application, increasing the likelihood of excessive and confusing code everywhere.
00:17:45.720
To combat this, we created a gem we call 'filterable,' which we include in our controllers. It offers a simple, user-friendly interface: just include the concern, and you can specify filter attributes and provide a type for validation.
00:18:04.000
This not only enhances code readability but also relocates all complex operations to a centralized location, allowing you to provide thorough testing around them. More importantly, it creates uniformity, as all endpoints across your application filter in the same manner.
00:18:24.670
We expanded this gem to support sorting and offer various data types, as well as scopes for more advanced filtering. I'm happy to share that we will soon release this gem as open source for the Rails community.
00:18:50.040
An integral part of developing APIs is serialization. From my experience, serialization often constitutes one of the biggest performance costs for APIs and is a common area where newcomers to API development struggle.
00:19:12.640
We used to work with Jbuilder for some time, but while it is flexible, it did not meet our performance requirements, and it failed to sufficiently organize our development process. Therefore, we transitioned to Active Model Serializers.
00:19:38.520
Active Model Serializers is a gem actively supported within the Rails application. You simply install it in your Gemfile, and create a serializer by listing your attributes. It also supports relationships, and if those relationships have their own serializers, it integrates those seamlessly.
00:19:59.230
The beauty of using Active Model Serializers is its compatibility with Rails. If the serializer has the same name as the model, it will automatically apply, providing immediate benefits.
00:20:27.120
For illustration, a sample output from a serializer could show attributes alongside included associations. Additionally, if we were to add another endpoint to our API—this could be a 'show' endpoint—there may be times when we do not want to display every field.
00:20:51.130
Accordingly, we can modify the Active Model Serializer to utilize flags for fields and includes for attributes and relationships, respectively. However, I find this design paradigm unsatisfactory, as it places too much view logic within the controller.
00:21:15.270
Maintaining consistency across different API endpoints becomes notably challenging. To address this, we extended the Active Model Serializer—similarly to how Application Controller extends simply—and built an Application Serializer.
00:21:49.230
This allows us to create views within the Active Model Serializer itself. These views would enable us to define distinct attribute sets, easily wrapped in those views, calling them from any controller.
00:22:07.340
With a single-line change and one serializer, we can now support multiple views, and significantly improve consistency. In addressing the problems we once faced—creating multiple serializers for various representations—this adjustment streamlined our approach.
00:22:28.120
The output is simple and clean. Developers utilizing it can now focus on their core task rather than getting bogged down in serialization code.
00:22:50.970
One other essential aspect of building APIs is documentation. Let me be completely frank: an undocumented API is practically worthless. Documentation’s importance cannot be understated.
00:23:10.780
Documenting your API is crucial—no matter how you do it, just do it. We initially documented our API using wikis and markdown, which is a great starting point due to its low barrier to entry.
00:23:39.700
Additionally, many developers are already familiar with wikis and markdown, making it easier for everyone to contribute to documentation, which can result in a polished final product.
00:23:54.940
However, a common issue arises: as new endpoints are added, for instance, there is a risk that documentation updates may be forgotten, leading to discrepancies between the documentation and the API.
00:24:19.870
When this unaligned documentation stacks up across all applications, what was once a well-maintained resource can quickly become a disorganized mess.
00:24:38.390
At Procore, we utilize Swagger—also known as the OpenAPI Specification—to combat this issue. Swagger provides a formal specification for describing RESTful APIs.
00:25:10.300
In our case, we write our Swagger documentation in YAML, and it outlines every endpoint, including the necessary parameters.
00:25:27.950
This approach offers numerous advantages. First, it provides structured documentation, which is highly valued among developers. Swagger sensitizes us to separate our style from our content.
00:25:49.640
Additionally, because Swagger outputs in JSON, it is machine-readable, and we're able to automate various tasks, including SDK generation.
00:26:01.160
While there are great benefits to using Swagger, it's worth mentioning that it also comes with challenges. One of the biggest hurdles is that it's another tool developers need to learn.
00:26:16.890
Not all developers are familiar with the JSON spec and will need to be trained on Swagger's specific parameters.
00:26:34.020
Yet, we’ve found that the advantages significantly outweigh the challenges. Regarding Procore's implementation of Swagger—keep in mind, our application is complex and already has numerous moving parts.
00:27:04.800
We aspire to synchronize our writing documentation with how we develop the API, ensuring style guides are in place and striving for reusable components.
00:27:30.210
However, we encountered a problem: our swagger documentation became quite large, reaching 7.7 megabytes, which is considerable for a JSON file. Our extensive coverage led to challenges with various swagger renderers.
00:27:53.230
Due to this, we developed our own architecture diagram for a documentation generator, running on AWS Lambda—a subject which could merit its own talk.
00:28:15.220
Kicking off with our continuous deployment and integration processes, it creates static JSON that can be rendered and is also human-readable. This data is then stored on a CDN.
00:28:41.450
We subsequently employ a React rendering application for our documentation site, ensuring rapid and complete access to our endpoints.
00:29:03.420
In summary, please remember to empower your developers to focus on what they do best.
00:29:21.180
Avoid encumbering them with unnecessary decisions unrelated to their expertise. Establish a style guide for your endeavors so that questions have preformed answers.
00:29:40.860
Prioritize the practice of 'disagree and commit.' This notion is crucial when dealing with passionate and intelligent developers.
00:30:01.960
Make contracts your priority to yield significant advantages; after trying it, you'll likely never revert to a different method. Additionally, favor reusable components as a general programming principle.
00:30:23.830
Finally, ensure that you document your API effectively. An undocumented API is a challenge for everyone, including internal developers.
00:30:42.340
My name is Derek Carter. I work at Procore, where we are hiring like crazy. If you'd like to engage with us about opportunities like this, please reach out. Thank you!
00:30:50.070
That's a great question. The question was whether we have internal routes that we don’t want to expose to customers.
00:30:58.230
Yes, we certainly do. One way we handle that is by using an "X Internal Only" flag in our Swagger documentation, allowing API developers to mark specific routes as internal.
00:31:19.950
On our documentation rendering side, we implement LaunchDarkly to filter out those features—ensuring that only approved internal and beta customers can access these endpoints.
00:31:35.670
Regarding contracts, the question was whether this concept is specific to Procore. Currently, yes, it is. It has been newly introduced in our system and is somewhat experimental.
00:31:57.260
However, the method exhibits great potential, and I believe this could warrant further discussion as we desire to contribute valuable insights to the Rails community.
00:32:23.270
Additionally, the application serializer will soon be contributed as a pull request to Active Model Serializer, due to its significant value.
00:32:41.660
I appreciate the insight regarding the challenges surrounding Jbuilder, as it does hinder the sharing of partials.
00:32:53.550
Active Model Serializer allows for more versatile sharing of serializers, either explicitly or implicitly by name.
00:32:59.920
To address the question on how we document endpoints that are subject to change, we generally employ comments in the JSON files.
00:33:17.310
In summary, we strive to stay ahead of the curve and maintain clarity within our API contracts to enhance developer experience.
00:33:27.250
Finally, the timeline for our Filterable gem is already approved; we just need to finalize all details before the release. Thank you for your understanding!
00:33:44.990
We currently document our Swagger manually due to our application being quite old. Consequently, handwritten documentation is the most reliable approach for now.
00:34:00.000
All changes to Swagger are reviewed and integrated into our repository to maintain high standards. We ensure comprehensive documentation and consistent updates.
00:34:17.940
Thank you all, and enjoy the rest of your day at RailsConf!
Explore all talks recorded at RailsConf 2017
+105