Talks
GraphQL Migration: A Proper Use Case for Metaprogramming?
Summarized using AI
GraphQL Migration: A Proper Use Case for Metaprogramming?
by Shawnee GaoIn her presentation at RubyKaigi 2019, Shawnee Gao discusses the migration of Square's largest Ruby application to GraphQL, exploring metaprogramming as a solution to simplify the process. This talk begins with an introduction to GraphQL, comparing it to REST and highlighting its capacity to manage complex data queries efficiently, particularly in environments with numerous interlinked data sources.
Key points of the discussion include:
- Complexity in Application Architecture: Gao illustrates how Square's monolithic application, consisting of over 273 thousand lines of Ruby and numerous controllers, required a more scalable solution to model increasingly intricate business relationships.
- GraphQL Basics: A foundational understanding of GraphQL is established, outlining how it allows clients to tailor queries to their needs, significantly reducing under-fetching and over-fetching issues associated with REST APIs.
- Metaprogramming for Efficiency: Gao emphasizes the use of metaprogramming to automate repetitive tasks in defining GraphQL types and endpoints, thus improving developer experience while maintaining flexibility in the codebase.
- Implementation of a GraphQL Layer: The process includes the transition from a REST layer to GraphQL for handling client-server interactions, highlighting the flexibility in maintaining both systems during the transition.
- Defining Targets and Schema Creation: The concept of targets as a unified data access layer is introduced, alongside the creation of GraphQL type classes through metaprogramming techniques to enhance the coding process.
- Handling Permissions and Security: Structured authentication and permission checks were stressed as essential for maintaining data security, particularly when handling sensitive information like credit card details.
- Testing and Documentation: The complexity of writing tests in a metaprogrammed environment is discussed, emphasizing how efficient testing methodologies can validate the functionality of APIs without overcomplicating the testing process.
In conclusion, Gao highlights the advantages of using metaprogramming in GraphQL implementations, including improved flexibility, reduced boilerplate code, and a positive impact on developer efficiency. She encourages attendees to reach out for further discussions about opportunities at Square, particularly for those interested in development.
Questions from the audience focus on handling documentation within generated schemas and performance optimizations, both of which play critical roles in a successful GraphQL implementation.
00:00:00
Hello everyone, thank you so much for coming to my talk. Before I get started talking about some really exciting new features my team at Square is working on, I'd like to say thank you to RubyKaigi.
00:00:06
RubyKaigi is actually the first conference that I've ever been accepted to present at, so this is quite an honor to be here with you all today. Thank you all for coming.
00:00:13
A little bit about me: this is me with my dog. They say pets take after their owners, and I think this is evidence of that. I was born in Beijing but I grew up in a place called St. Louis, Missouri, in America.
00:00:25
Now I live in San Francisco. Since being in San Francisco, I've traveled around a lot. I've been to Japan and I love the animals here. But something else I noticed is how much Japan loves cats.
00:00:33
I work at a company called Square, which is a payments company. We started off as a small device that could plug into your phone and allow merchants to take card payments easily, which has become the norm in America.
00:00:54
Over time, we've evolved into a larger company and now we offer omni-channel solutions. We provide APIs that facilitate offline payments, payment SDKs, and also APIs for processing payments on e-commerce websites.
00:01:17
Most people are familiar with our hardware, which is probably what we're known for the most. However, what may not be well-known here in Japan is that we offer a whole ecosystem of products that include payroll and everything you need to run a business.
00:01:37
The bigger point I want to make is that as any business grows and matures, your scope widens as you enter more markets. For us as engineers, this means the business objects we need to model and the relationships between these business objects become increasingly complex.
00:02:02
We need to build applications that can keep up with our evolving business logic. We have an internal dashboard to track all our entities to ensure that our merchants are legitimate and their businesses are healthy.
00:02:28
My team works on getting data from all our product teams, which are very specific. We acquire all that data and present it on a dashboard so that our agents can verify the legitimacy of merchants and confirm that payments are successful.
00:02:54
This application has grown to become a monolithic application with 273 thousand lines of Ruby and 22 thousand lines of Camel. It has over 200 controllers communicating with more than 150 internal services.
00:03:16
This complexity led us to look for a solution, and that's where GraphQL comes in. For those who are not familiar with GraphQL, it is similar to how REST is a set of ideas for querying resources on your server.
00:03:43
With GraphQL, your server has to structure your app data like a schema, giving your client the power to query anything it wants as long as it falls within the constraints of the schema. It's akin to a database query except your resource is no longer your database; your resource is your app data.
00:04:02
Consider the limitations of REST: for instance, if you look at a REST endpoint, you can guess what it responds with, but you can't be certain of the data returned. This often leads to under-fetching, portraying only a single resource at a time.
00:04:50
To illustrate how GraphQL works differently, let’s look at a GraphQL query. You can define precisely what you want to retrieve, such as an amount for a payment, simplifying the process significantly.
00:05:19
This dramatically reduces the implementation of multiple REST APIs when building your frontend and backend. Instead, you fundamentally have one smart endpoint instead of many separate controllers.
00:05:54
I am not here to convince you that GraphQL is a better choice over REST, as there are many use cases where REST is perfectly fine. However, for us, we are querying data from multiple different sources, and our data is highly diverse.
00:06:24
The value of our data is highly relational. For example, it’s not just about knowing that a payment is a thousand dollars; it’s also important to understand the context around that payment.
00:06:42
In today’s roadmap, I will discuss our existing architecture, provide a step-by-step process of writing a GraphQL API, and we'll cover how we can use metaprogramming to streamline certain parts.
00:07:14
I will also demonstrate that even with metaprogramming, this API remains flexible and avoids introducing bugs. Lastly, I will touch on testing our application.
00:07:34
Here’s an overview of our existing architecture. We gather data from various services, including our applications' databases. Our Rails backend communicates through another REST layer to our frontend, where we use Ember at Square.
00:08:04
Today’s talk will focus on replacing that one REST layer between our backend and frontend with GraphQL. Before diving deeper into GraphQL, we must discuss something called ‘targets,’ which is essential for understanding metaprogramming.
00:08:32
Data comes in from these servers in various shapes and forms, and we translate this into a target layer. A target is merely a class that wraps raw data so that it interacts seamlessly with each other and our backend via a unified API.
00:08:59
All targets inherit from a base target class, which manages how to fetch data with defined methods. Furthermore, all targets possess IDs to uniquely identify them. This is the data model I'm using throughout my talk.
00:09:27
For instance, a payment is associated with a merchant via a merchant ID. To illustrate what a target looks like, I will provide you with a stub that clarifies the type of data we expect, detailing how we retrieve this data.
00:09:59
Here's one for a merchant client, which interacts with my stub. The stub has a field called ‘store name.’ The same goes for the payment, which shares a similar structure with an additional field for the merchant.
00:10:26
It either memorizes the merchant or retrieves it since the merchant is already a target itself. We can easily pull that information using a lookup API.
00:10:56
Now let’s write a GraphQL API for our data model. The data needs to be structured like a graph, which means that it enters through a GraphQL controller.
00:11:21
I want to emphasize that your GraphQL controller can coexist alongside your REST controllers. If some parts of your system are still RESTful, that’s perfectly acceptable.
00:11:46
The first request is to identify whether it’s a query or mutation. We will be focusing primarily on read operations today. After categorizing the request as either payment or merchant, we resolve each field until we reach the leaf nodes.
00:12:16
For payments, the query gets a bit more complex since it has the merchant as a field. What I showed you in the graphical interface earlier is the query we will handle.
00:12:42
We enter a GraphQL controller and execute the demo schema. This process is straightforward; we pass in the query and any relevant variables.
00:13:08
The query type is checked to see if it is a payment, validating according to our schema. The server verifies that it has a payment ID, which is required and must be a certain type.
00:13:34
The validation checks also extend to associated fields to ensure everything is correct. Once validated, it looks for a resolver, which matches the name of the field and operates as a pure function.
00:14:14
So, what parts of this can we metaprogram away? I’d like you to keep three ideas in mind as I navigate this technical aspect: Is it still flexible? Will we encounter strange bugs? And is this a cleaner pattern to follow?
00:14:58
Moreover, does it reduce boilerplate code and improve the developer experience? In looking at the GraphQL controller, we see that it connects to the demo schema and uses type class validation.
00:15:38
If everything passes validation, it will go to the route type resolvers to fetch the response from the app data.
00:16:20
In my earlier demo, I wrote just a few lines of code to illustrate the essence of metaprogramming. The laborious part is what I term as the ‘GraphQL type classes’—this is where writing becomes repetitive.
00:16:50
Ruby isn't inherently typed, leading us to the question of how to annotate our target classes so that we clarify the types of each field. Wouldn't it be helpful to have a class variable indicate metadata about our fields?
00:17:22
In the example I’m providing, I will use annotations to clarify our GraphQL types. Dynamic methods are methods defined during runtime, while unbound methods aren't associated with classes initially and need to be bound later.
00:17:55
I'm using simple examples to demonstrate to you how these concepts work together for metaprogramming our target class annotations.
00:18:23
So, here is my annotated target class. We previously assigned methods within our payment class, using the definition that indicates the field while returning a symbol for our hash.
00:19:01
When we add our annotations, we ensure each field has a type declaration, expanding our annotations further to include whether the field can be null.
00:19:31
Now let's generate the GraphQL type classes. I want to create a GraphQL type generator that takes in my target classes and returns a mapping of the two, simplifying our queries.
00:20:01
We expect that when initialized, it will create these new classes, linking our target classes to their respective GraphQL types.
00:20:32
As a reminder, all GraphQL types reside in a namespace called ‘Types,’ and this gives clarity on where everything originates.
00:21:02
Let’s look at how we define route types—these are starting points for querying our schema. The significant parts are our field definitions and resolvers.
00:21:30
To define fields, we can assign validity to each target, effectively making every target a point into our schema. When we interpolate the field name, we plug the original code into our interpolated response.
00:21:58
Now let’s see how we generate resolvers. We can dynamically create resolver methods by interpolating the target classes for further simplification.
00:22:24
This approach radically reduces boilerplate code, eliminating redundancy in our development. If you can believe it, I've managed to eliminate all explicit target types in my codebase except for the base query type.
00:22:52
We can query values such as merchant IDs and verify their functionality. With an accurate schema preloaded, we can conduct thorough syntactic checks, validating that our API behaves as expected.
00:23:22
Next, I would like to add a new business object into the mix—a card associated with a payment. This new component fits nicely within our existing structure.
00:23:56
With minor adjustment to our code, I can swiftly incorporate the new card object and re-query the structure. As expected, the new card functionality appears correctly.
00:24:18
Now, let's discuss potential significant structural changes in our data. Some queries might inherently require higher permission levels for data access, restricting what can be visible or queryable.
00:24:43
In this case, we tailor our GraphQL schema so that certain objects can only be accessed selectively, granting permissions via the context of related transactions.
00:25:10
For proper data security and privacy measures, essential operations (like viewing credit card information) must go through a payments authentication system.
00:25:31
This emphasis on relationship management in queries exemplifies the customization capabilities inherent in GraphQL. It's essential to implement comprehensive permission and authentication checks within the backend.
00:25:57
Now, let’s touch on testing. As I mentioned during my presentation, tests can be quite challenging to write under the right circumstances.
00:26:22
However, using metaprogramming, testing becomes more manageable. Instead of testing each business object individually, you can focus on one code pathway, ensuring your API remains functional.
00:26:47
Here’s a simple test case: initially, types may not exist until they’re generated, at which point we can confirm that everything works as intended.
00:27:10
Revisiting our checklist on metaprogramming, I can confidently say it preserves API flexibility while reducing repetitive tasks and enhancing the overall developer experience.
00:27:35
In conclusion, I would like to thank my team at Square; without their support and mentorship, this talk would not have been possible.
00:28:03
If anyone is interested in developing at Square, please reach out! I have to point out that our contact actually speaks Japanese, which is a huge bonus, unlike me.
00:28:27
Thank you once again!
00:28:49
Do you have any questions? I am open to answering questions.
00:28:58
Yes, sir?
00:29:01
Thank you very much for your talk. We are using GraphQL for a mostly Rails application. We are pretty happy with it and encourage everyone to use a GraphQL tool.
00:29:25
You demonstrated some impressive capabilities with metaprogramming. How do you handle documentation for the schema of such generated models?
00:29:41
What GraphQL offers is the ability to document every field and type in your schema. In a realistic environment, we'd list these fields in an array or hash format.
00:30:03
These would include defined types along with their nullability. We can integrate documentation fields into our definitions to enhance clarity.
00:30:26
As always, documenting your API aids the frontend in understanding what data they can expect as input/output.
00:30:55
Yes, we can have a structure to ensure that documentation is available alongside each field in our schema.
00:31:10
Any other questions about the metaprogramming or performance side?
00:31:29
Yes, how do you address performance when using GraphQL?
00:31:40
Performance has always been a key concern with GraphQL. We need to ensure that our backend is capable of handling requests efficiently.
00:32:03
Heavy caching is an essential part of our strategy. If a query takes too long, it could cause connection timeouts for users.
00:32:25
Quality introspection helps us anticipate how long queries might take, allowing us to implement restrictions on query depths.
00:32:47
In some cases, instead of sending large queries from the frontend, we might send an identifier that corresponds to a previously stored query on the server.
00:33:10
This helps manage efficiency while ensuring that we can still access relevant data with less overhead.
00:33:30
Any additional questions or comments?
00:33:45
I’m happy to discuss anything else, including fun facts about my dog!
Explore all talks recorded at RubyKaigi 2019
+69