Boutique machine generated gems

by CJ Avilla

In the talk titled "Boutique Machine Generated Gems," CJ Avilla discusses the potential for automating the generation of Ruby gems using familiar UI building blocks. The speaker illustrates the evolution of Stripe Ruby, the official client library for Stripe's API, highlighting improvements made over time to enhance developer experience (DX). Key points covered include:

Historical Context: The speaker reflects on how the Stripe Ruby gem started as a single file with minimal resources in 2011 and evolved into a more organized and comprehensive library.
Developer Experience (DX): Emphasis is placed on external DX, focusing on how third-party developers interact with APIs, utilize SDKs, and the documentation available to them.
Challenges of Manual Code Generation: Initially, changes to API endpoints required labor-intensive manual updates across multiple SDKs. This led to the realization of the need for automation to improve efficiency.
Introduction of OpenAPI Specification: The integration of OpenAPI expanded testing capabilities and streamlined API updates for various programming languages, significantly reducing the workload.
Automating Code Generation: The use of a pipeline to generate SDKs saves hundreds of hours in development time, allowing for quick updates in response to API changes.
Custom Tooling with Prettier Poet: The talk introduces Prettier Poet, a tool inspired by React components, designed to produce clean and readable code across different languages without the complexity of massive template builds.
Live Demos: CJ conducted live demonstrations to showcase how Prettier Poet can generate Ruby code through a component structure, displaying its application across different programming languages while stressing readability and maintainability.
Automated Documentation and Changelog Generation: Automation extends to producing changelogs and validating parameters required for API endpoints, easing the technical writing process.
Stripe CLI UI: A feature enabling interactive API usage directly from the browser, providing easy access to documentation.

The session concludes with a reinforcing message about the significance of automating the generation of SDKs, which fosters better integration experiences for developers. The advancements in tooling streamline processes while enhancing maintainability and clarity in code and documentation.

00:00:00 Ready for takeoff.

00:00:18 All right, these are some pictures that Dolly gave me of machine-generated gems. It's a collection of machine-generated images presented in a graphic novel style. I thought it was pretty fun. However, I want to roll back to 2019 for a second.

00:00:38 I was out on a walk while attending an on-site event in Seattle. I was walking from the event center to dinner, chatting with my friend Alex about this really cool tooling that he was building. He was using React, but he was writing in several different programming languages.

00:00:50 It sort of blew my mind a little bit. I was really fascinated, much like we are when talking about gems. I have been chomping at the bit, waiting to talk about this. So, I'm super excited to share it with you all today.

00:01:08 Before we dive into automatically generating some of our SDKs, I want to discuss the hard version of the process—writing things by hand.

00:01:20 During practice, I told a colleague that we used to write all our gems by hand. They reacted as if I meant we were actually using pencil and paper, which is not true.

00:01:32 I work for a company called Stripe. The main thing you need to know about Stripe today is that we have a web API, and a huge lion's share of those who integrate with Stripe use our official client library. We have various client libraries, with Stripe Ruby being one of them.

00:01:46 It's a very popular gem, and our APIs are utilized by millions of businesses worldwide, ranging from super small startups to massive enterprise companies for accepting payments, making payouts, and managing their businesses online.

00:02:04 Now let's talk about the developer experience, particularly the hard way. When we talk about DX, we often refer to our internal developer experience when we're working on our applications, feeling the pain of slow CI or error messages that we are encountering.

00:02:23 Today, I want to focus on external DX—how third-party developers integrate with our APIs and use our tools. This includes everything from third-party developers using an SDK, a Ruby gem, to interacting with our API, the error messages they receive, to the documentation they read, and even the shape of the actual API response.

00:02:42 All of this contributes to what we call external DX. There's a lot we can build into this automatically so we don't have to do endless chore work.

00:03:01 Let's discuss the hard way first. In 2011, version 1.5 of the Stripe gem was as far back as the public GitHub history for Stripe Ruby goes. At that time, Stripe Ruby was primarily a single, massive `stripe.rb` file with just four resources available to interact with the Stripe API.

00:03:32 As time progressed, we dedicated a lot of effort to hardening the core of Stripe Ruby. From 2012 to 2014, you see PRs focused on cleaning up organization, making sure we adhered to the patterns we wanted for our classes, and the overall implementation of Stripe Ruby, so that using the SDK would result in a nice experience.

00:04:06 For instance, we had PRs that organized our modules into their own files. As we progressed to 2015 and 2016, the PRs mirrored the API itself. Every time a new endpoint or property was released, we would have to create a PR to update not just Stripe Ruby, but also Stripe Python, PHP, Node, Go, Java, and others.

00:04:31 These were labor-intensive tasks, often costing us hours or even days for each change, depending on the size of that change. An example would be adding a new resource, like a source object, that needed to follow strict CRUD patterns. If that resource had additional methods, such as 'verify', we would add those manually.

00:04:54 In 2017, we introduced the OpenAPI Specification into Stripe Ruby. Initially, we incorporated it for testing, using it as the fixture backing our automated tests to ensure Stripe Ruby behaved as expected. This was also the first time we had an artifact that represented the API's structure in the SDK.

00:05:14 If you aren't familiar, the OpenAPI Specification was formerly known as Swagger, which underwent a name change due to ownership issues. At Stripe, we describe all our APIs using a Ruby DSL and use some internal tools to output YAML and JSON files representing different versions of these specs.

00:05:32 For instance, we can generate specifications for beta features, public features, and more. All these specs are available on GitHub, specifically at github.com/stripe/openapi.

00:06:02 Fast-forwarding to 2019, we still observed many high-effort PRs being created for all our SDKs. However, that also marked the beginning of automation in our process, as we started to realize that code generation would come to our rescue.

00:06:35 Now, if you check the repository, you'll notice we just ship one API update PR and bump the version, and that update is now generated. Every time the API changes, it flows automatically out to the open API spec, which generates the SDKs, saving us hundreds of hours in engineering work per year.

00:07:06 You might be wondering what alternatives we considered and how we determined the need for this tooling. It's worth discussing the requirements we had going into this.

00:07:30 We wanted to build a pipeline that would support seven official SDKs and generate code that closely resembled the handwritten version.

00:07:43 We consider our SDKs forms of documentation; they should be clear and readable. In case there's a bug, developers can easily reference the code on GitHub to understand the issue.

00:08:02 Additionally, we wanted to establish a straightforward pipeline that could be maintained by a small group of engineers at Stripe, responsible for SDK generation. These aspects formed the baseline requirements for our approach.

00:08:27 With an OpenAPI spec established, we explored using tools like Swagger Codegen or the OpenAPI Code Generator. These tools offer a range of open-source utilities that allow you to transform your spec into client libraries or SDKs in various programming languages.

00:08:46 For example, you can easily install Swagger Codegen via Homebrew and provide a JSON blob sourced from GitHub as input to generate the desired output in Ruby.

00:08:57 Utilizing an off-the-shelf tool might be a beneficial route for newcomers looking to develop in multiple languages easily.

00:09:20 Another reason for wanting to align our generated code with existing formats is to minimize breaking changes for third-party developers. The output of Swagger Codegen produces a massive Ruby file named `default_api.rb`, which can exceed 30,000 lines.

00:09:50 Aside from this giant file, there are over 2,000 model files representing the parameters we send to our API or the results returned, leading to a challenging debugging process. If bugs arise, it becomes tricky to navigate large files and numerous model files.

00:10:34 One of the resources generated is the customer object, which provides a well-documented representation of different attributes.

00:10:52 If anyone is starting fresh today, using the automatic tools available is a great option. Despite this, we noticed formatting inconsistencies and repetitive code patterns that necessitated revision.

00:11:15 Computers excel at generating and understanding repetitive structures, while humans often struggle with that task.

00:11:38 There are various features unique to the Stripe API that complicate the use of off-the-shelf tools. One example is that we have the concept of expandable fields. This means if you make a request to the API and receive a JSON response with an ID for a related field, you can alter your request parameters.

00:11:58 Instead of receiving an ID in the response, you may receive a fully expanded, nested object within the JSON response. This creates polymorphic scenarios where a property could either be a simple string or a complex object, complicating implementation.

00:12:23 Although using off-the-shelf tools is convenient for developers, it also can present challenges due to specific features of our APIs.

00:12:43 Swagger Codegen utilizes templates for generating code. The source of Swagger Codegen consists of massive, mustache-like templates that populate based on the OpenAPI specs, resulting in method names and class names.

00:13:07 For a long time, we wrote code snippets within our documentation using ERB, integrating raw Ruby code directly into templates. We wanted to steer clear of the string concatenation or template approach because it became messy and cumbersome.

00:13:37 On the flip side, we considered using AST (Abstract Syntax Tree) builders and printers. Note that these tools tend to be specific to the programming language, requiring our team to learn multiple AST builders across various supported languages such as Go, Java, or C#.

00:13:57 Now, let me introduce you to something called Prettier Poet. This is one of the components in our pipeline that I’m particularly excited to share with you.

00:14:17 Prettier Poet is a React-like tool for generating readable code across different languages, powered by the core of Prettier. Many of you in this room may have used React before.

00:14:45 Typically, React is viewed primarily as a front-end framework that allows us to construct components which return HTML. However, we don't utilize the event handling or reactivity aspect; we merely leverage the components.

00:15:09 This design enables us to have isolated, highly testable components that we can then integrate into larger constructs. For instance, we create a welcome component that processes props and renders accordingly. We subsequently build an application component composed of simpler components.

00:15:30 Using Prettier Poet, we achieve conditional rendering and a variety of other beneficial features. Beyond in-browser applications, React Native allows us to share code across mobile platforms, targeting both iOS and Android.

00:15:51 Moreover, tools like Raycast provide interesting use cases for writing extensions using React components.

00:16:11 Next, I planned a live demo utilizing Prettier Poet. Here we import React and rendering functions to convert components into strings, allowing us to implement Ruby.

00:16:34 We're building a higher-order component called 'Say Hello Ruby,' which accepts a message prop. In turn, we introduce a subcomponent called 'Method Call,' where we can define the arguments to be printed.

00:17:02 We use the render-to-string function, which takes care of outputting Ruby code. A notable feature of this function is 'printWidth', which allows for custom line-breaks based on desired width.

00:17:30 If you’re curious about Prettier's implementation, there is a paper we can share that describes its elegant methods for achieving well-formatted code.

00:17:49 Using Prettier is a common practice among many developers to auto-format their code, alongside alternatives like Ruby Format.

00:18:11 Now, let's see our output: 'puts hello rubyconf.' Although valid if 'hello' and 'rubyconf' were defined as constants, we intended to display them as a string.

00:18:26 For this reason, we'll incorporate 'ruby.string' to ensure the message is interpreted appropriately.

00:18:45 Upon re-running the code, we finally achieve the output: 'puts "hello"' and our code becomes valid.

00:19:20 What if we wanted to print the number 42? Running this generates an exception because the props passed are not valid hard-coded strings, allowing us to enforce integrity in our rendering.

00:19:50 Instead of 'ruby.string', we replace it with 'ruby.literal' to specify that we want to print the literal value without enforcing string limitation.

00:20:15 Now, we can print 'puts 42' along with another message, presenting both in one output.

00:20:33 Let's look at another example where we import both Ruby and JavaScript. We create functionality for languages similar to the previous demonstrations, allowing us to illustrate how code is structured across multiple languages.

00:20:50 We define a 'say hello' function that prints outputs conditionally based on language. Thus, we can finally print statements in Ruby and JavaScript, as shown in our demo.

00:21:17 Let’s add Go into the mix. I can easily declare a new function call for Go, utilizing its formatting conventions to achieve 'println' output.

00:21:40 Now we can print greetings in three languages, each formatted appropriately for their respective syntaxes.

00:22:00 Moving on, I would like to discuss another example involving rendering models from a given OpenAPI spec. This demonstrates how our specs can become complex when representing all routes and operations.

00:22:20 We generate resource methods with well-defined properties and associations, applying a modular structure in our example implementations.

00:22:40 Using Prettier Poet, we can create a Ruby class for a customer resource, allowing us to see how output gets formatted based on character line limits.

00:23:10 Here's another example from our implementation for rendering routes in frameworks like Sinatra and Express, using modular components.

00:23:29 We define HTTP methods and their associated functioning logic, catering to individual teaching demonstrations, successfully streaming output to both frameworks.

00:23:48 Next, we explore how to illustrate the usage of SDKs by printing examples of how to call the Stripe API using Ruby, complete with method traits.

00:24:08 Eventually, we return to our documentation, displaying code pieces reflecting API requests using Stripe Ruby, Python, PHP, Java, etc.

00:24:30 Our tooling allows technical writers to generate code in various languages without needing extensive programming knowledge. This seamless integration is a part of the magic we have created.

00:25:00 We can validate that the required parameters are provided for specific API endpoints and appropriately generate and format documentation for various languages.

00:25:30 Moreover, we have automated changelog generation based on diff comparisons, allowing us to eliminate redundant examples from our documentation as we optimize the content.

00:26:00 The Stripe CLI UI is another exciting feature we introduced, enabling users to interact with the API directly from the browser while accessing all documentation.

00:26:30 Thank you so much for your time and attention. I appreciate you joining and hearing about these advancements.

00:26:50 I am happy to take questions if you have any, and I'm eager to share my experiences further.

00:27:12 Thank you again!