Spice up your life with EQL

Talks

Norbert Wójtowicz

Spice up your life with EQL

by Norbert Wójtowicz

In this talk titled "Spice up your life with EQL" presented at wroc_love.rb 2019 by Norbert Wójtowicz, the focus is on building APIs effectively using concepts from functional programming and immutable data structures. Wójtowicz discusses how his journey from front-end to back-end development has provided insights that culminate in a comprehensive understanding of API development.

Key points discussed include:
- Journey Through Development: The speaker reflects on his experience transitioning from a Ruby back-end developer to embracing front-end technologies like ClosureScript and React, rekindling his passion for front-end development.
- Immutability and Data Flow: He emphasizes using immutable data structures across both front-end and back-end development, detailing the advantages it brings in crafting consistent and predictable applications.
- Decoupling APIs: The talk introduces the notion of building efficient APIs that respect data independence. Wójtowicz illustrates how traditional RESTful designs can be inadequate due to their sequential data-fetching processes.
- Example of YouTube API: The speaker uses YouTube as a case study to showcase how efficient data retrieval is achieved through the use of 'parts' in their API, which allow developers to request only necessary information, enhancing performance and minimizing strain on back-end systems.
- Introduction to GraphQL: Wójtowicz discusses GraphQL as an advanced alternative to REST APIs, explaining its ability to allow clients to fetch only the required data attributes. He acknowledges its advantages but points out the potential complications of type management that can arise.
- EQL as a Solution: The concept of EQL (Entity Query Language) is presented as a flexible and decoupled approach to API development, allowing for a more significant advantage in querying structures effectively without type constraints.
- Practical Implications: The speaker relates how EQL can lead to seamless integrations across APIs, enhancing the capability to build composite applications that can operate efficiently by leveraging data from various services conducting context-free operations.

In conclusion, Wójtowicz encourages developers to explore EQL and associated practices in their projects, which can significantly enhance API functionality and foster a better development process. The overall message is clear: by embracing immutability and adopting a more flexible querying mechanism, developers can create APIs that are both powerful and easy to manage.

00:00:14.799 Okay, so I'm not a bird. This is not my first talk at Wroc Love RB. Although I've been here before, there are only two or three people in the audience who know who I am. For the rest of you, perhaps it's better that you don't.

00:00:20.070 I was here in 2015, and I gave a talk about how, as a Ruby back-end developer, I reignited my love for front-end development when I discovered ClosureScript and React. It turns out that front-end development isn't crazy; it's just that the tooling was. If everything is just 'view equals function of data', then everything becomes nice and pleasant. I actually enjoyed front-end development.

00:00:33.199 Two years later, I returned and shared my experiences with you all. I told you that I had been to the woods, and I came back to discover that you can do even back-end development with Closure and immutable data structures. You can build entire systems with this approach, and it's not just limited to front-end development. There's great tooling available with Closure Spec and other such resources, providing functionality similar to that of type checkers but without some of the hassles.

00:01:01.710 Then, two more years passed, and I gave a talk at a conference, discussing how not only can you do front-end development with this technology, but you can also replace your database with the concept of immutable data. I explored the idea of how databases might work if we only had immutable data and a solid data flow concept. It turns out that they actually look really nice.

00:01:19.440 My talk at that conference was great because it was recorded, and if you're interested in learning how to replace your relational database, NoSQL database, or Redux state machine with something more scalable that adapts to changes in your domain over time, I highly encourage you to check out that video. But I’m not going to repeat it here; instead, we have a new slot and a new topic for this talk.

00:01:52.020 I want to close the loop. I've discussed front-end development, back-end development, and how I replaced my databases. The only missing pieces are APIs. Now we have a fuller picture, and I can finally go home and write Lisp all day long. So, today, this talk will focus on how to build APIs using the concepts I’ve been sharing for the past five years.

00:02:36.410 There’s a recurring theme here. What is that theme? Well, aside from my presence every two years, there was a photo tweeted from my last visit with a caption that read, 'When you use Closure and haven’t told anyone for five minutes.' I was disappointed, and let me explain why.

00:02:51.100 They completely missed the point of my talks. If they had tweeted something else, I would have been guilty as charged, because my facial expressions change every time I hear someone describe their problem. Often, it turns out they could solve their issues using declarative data structures and simple functions. Anyone who's talked to me after a party knows this to be true.

00:03:29.520 So the question remains: how can we take this approach and build APIs with it? I’ve tried to distill this down into about a four-step process because if it takes more than four steps, nobody's going to do it. Here’s how to build an API using the concepts of immutability, EDN, context-free structures, and flat data.

00:03:54.480 Let’s consider YouTube as an example. What does YouTube look like? For someone who has been living under a rock for the last twenty years, it’s a single-page app where a lot of interactions occur. Now, if you’re a front-end developer tasked with building this app, you’d also need to handle back-end requests to render all this information. We're not going to delve into the intricate details because they are irrelevant for the moment. The key here is to consider the structure of the data being fetched.

00:04:41.160 When we query for video data from the YouTube API, we receive detailed information about the video, including title, licenses, views, share links, and all related metadata. We also need to consider the channel associated with the video, which can be a user or a physical channel. In the YouTube API, users are essentially channels, and there isn't a separate concept for users.

00:05:03.910 This means that when rendering a view, you'll need data on the channel title, avatar, number of subscribers, and much more. Additionally, for popular videos, you have the comments section, which includes information about who wrote the comment, the body of the comment, when it was posted, and the author's title and avatar.

00:05:24.070 On top of that, there's a sidebar with recommendations for the next video, which requires information like the video title, thumbnail, and potential reasons for why it was suggested—maybe the channel is popular or similar users have watched the video. All data necessary to render just a single video view must be fetched, including all these details.

00:06:00.760 If you were to use a REST API, fetching all this data would look something like this: fetch the video, fetch the channel-related info, fetch the comments for each one, get the author's info per comment, and repeat this for additional videos. Clearly, this sequential approach is not efficient; it would take ages—potentially crashing back-end systems due to the volume of data needed.

00:06:48.570 Instead, YouTube has created a more efficient way of doing this. They use something called 'parts' in their API, where different pieces of information can be retrieved through different parts. Users can fetch only the information they need by obtaining the correct part, and some parts come with different quota costs depending on their importance to the server.

00:07:20.130 So, if you wanted to get the channel title, which part would you request? The answer is the 'snippet.' The YouTube API has defined various types and their corresponding characteristics, allowing developers to differentiate what data is returned in a single query efficiently. This marks a significant improvement compared to the limitations of REST.

00:08:01.330 Now, how does GraphQL fit into this discussion? GraphQL and other solutions like it, such as those employed by Facebook and Netflix, aim to solve similar issues. The fundamental concept is that you can define types—for example, a video type with attributes like ID, title, description, etc.—and then define relationships between these types.

00:08:42.710 With GraphQL, you can write a query that specifies exactly the attributes you're interested in for a given video ID, using a map notation for related entities like channels and comments. This means you can effectively and intuitively pull the data you need—it's a far more streamlined approach than REST and reduces boilerplate code significantly.

00:09:56.020 GraphQL also empowers the client, allowing them to request only the specific information needed without breaking older clients, as they won't request new features not related to their current needs. This keeps older clients functional even as new features are added over time. The reduction of boilerplate code is a huge advantage since much front-end code is dedicated to just fetching and transforming arbitrary API responses.

00:10:47.940 However, GraphQL isn't perfect. Its main limitation is that it requires defining specific types, and this can lead to type explosions when attributes need to be repeated across different types—especially evident when building large applications. For instance, if you had a comment component that only cared about the author's title and body, you'd need to create three different types because the attributes are always nested within a specific type scope.

00:11:42.740 In contrast, one way to approach this is through 'decomposing'—or taking apart elements that are unintentionally coupled. By examining the components of data and breaking them down, you can achieve clearer structures without redundancy. This is especially useful for keeping elements that you might want to use in diverse contexts separate, thus avoiding a rigid structure.

00:12:34.750 EQL emerges as a concept that allows for clear separation of concerns because it recognizes that attributes often exist independently from their original types. By treating data as independent entities, we can build APIs that provide a more flexible querying experience, effectively reducing unnecessary type coupling.

00:13:25.710 An example of this would be the implementation of pathom within ClosureScript. Using pathom, you can seamlessly pull data based on identifiers that pass the requirements of being context-free, allowing for a much cleaner separation of responsibilities. For example, we can use 'select keys' to pull only the attributes we care about from a hash map, ensuring flexibility when navigating our data.

00:14:27.412 In any API, it’s crucial that we know how to traverse and derive new information without needing to build complex queries. JQ serves as a good model for this since it can pull out specific attributes using a string-indexed notation, making it easy to access nested data structures.

00:15:14.560 Thus, the challenge that remains is how to use these principles effectively while still returning results in an intuitive format. That is where EQL fits in, enabling us to write clear and understandable queries that can derive new kinds of information based on relationships among entities.

00:15:52.950 Within the Closure ecosystem, developers can leverage libraries built on EQL to integrate various data sources efficiently, including third-party APIs like YouTube. The spirit of immutable data is to have predictable systems where side effects are minimized, helping to create applications that are easier to reason about and maintain.

00:17:44.225 One interesting application of this concept is in the integration of different APIs. For example, if you can pull a YouTube video ID from a SpaceX API response, you can enable cross-referencing across services. This flexibility allows developers to create robust applications that can compile and manipulate data from various sources.

00:18:29.520 Imagine crafting a front-end dashboard for a website dedicated to SpaceX where, alongside launch details, you also fetch information on the associated YouTube video: its title, channel, and view count. This type of seamless integration is possible through the use of context-free identifiers, making it easy to compose rich UIs without overcomplicating our data-fetching logic.

00:19:27.840 You can structure these queries as you wish, allowing you to create meaningful dashboards or widgets without needing stringent rules on joins or structuring in your schema. This flexibility is beneficial for developers who need to tailor the API to their own needs, effectively assisting users in retrieving the specific information they desire.

00:20:20.670 Over the course of this talk, I've illustrated several principles of building effective, context-free, and immutable APIs with EQL and related approaches. While it takes careful consideration and practice to implement these ideas, the benefits will be evident in the applications you create.

00:21:16.600 In conclusion, I'm glad to have shared these ideas with you today, and I hope you've found the concepts of EQL and pathom helpful. I encourage you to explore these in your own projects to see how they can enhance your API's functionality and your overall development process.

wroc_love.rb 2019