rubyday 2023

Summarized using AI

End to end typing for web applications

Frederick Cheung • June 16, 2023 • Verona, Italy

In the video titled "End to end typing for web applications," Frederick Cheung discusses strategies to effectively manage data types and enhance type safety in web applications, minimizing the risk of bugs arising from mismatched assumptions between frontend and backend data. The presentation highlights the significance of having a single source of truth regarding data types and emphasizes using TypeScript alongside JSON Schema for robust type checks.

Key Points Discussed:

- Problem with Type Safety: Cheung opens with an anecdote about a sorting function in JavaScript that fails to handle nullable last names, illustrating the common pitfalls developers encounter when integrating frontend and backend code without clear type definitions.

- Introduction to TypeScript: He explains TypeScript, an open-source language that adds static typing to JavaScript, which helps in identifying type mismatches before runtime. Type annotations allow developers to specify expected data types, helping to catch errors early.

- Static vs. Structural Type Systems: The distinctions between nominal and structural type systems are outlined, showcasing how TypeScript differs from others by focusing on the structure of data rather than its name.
- Practical Applications: Cheung details the various constructs available in TypeScript, such as unions, intersections, generics, and how they help manage complex data structures effectively. He discusses narrowing types through conditional checks and the use of discriminated unions to create more precise type definitions.
- Integrating JSON Schema: Cheung emphasizes the importance of JSON Schema in describing the structure of JSON data, which can maintain sync with TypeScript types. He offers insights into workflow options for generating TypeScript definitions from JSON Schema to ensure front-end safety.
- Progressive Integration: Discussing his team's incremental approach, Cheung suggests adopting TypeScript gradually, integrating schemas without requiring a system-wide overhaul. This strategy fosters confidence during refactoring and enhances type safety across the application.
- Conclusion and Benefits: Cheung concludes by stressing the benefits of this approach, including reduced bugs, increased team productivity, and an overall safer codebase. He invites the audience to explore a sample Rails application to see these concepts in action and highlights the importance of continuous learning and adaptation in development practices.

By the end of the talk, Cheung effectively illustrates how adopting TypeScript alongside JSON Schema fosters a cohesive development environment, ensuring a smoother integration of frontend and backend systems while minimizing the likelihood of errors.

End to end typing for web applications
Frederick Cheung • June 16, 2023 • Verona, Italy

Ever had a bug because the frontend made incorrect assumptions about the shape of response data from the backend?
Or maybe you trod nervously during a refactor? Or perhaps you broke an app by changing the backend data in a way you didn’t think would matter?

This talk will show you how avoid these mistakes, enabling you to keep moving fast, by having a single source of truth for your data types, checked both on the frontend and the backend.

rubyday 2023 happened in Verona on 16th June 2023

Info and details of this edition:
2023.rubyday.it/

rubyday 2023

00:00:00.160 Um, well hello everyone! It's lovely to be here. This is my first in-person Ruby conference since the before times. So yeah, it's great to see so many people, and I'm excited to be here, even though I had to leave my kitten at home.
00:00:06.680 I just got him on Tuesday, and his name is Elrond. Anyway, let's move on from cats.
00:00:12.480 I wanted to start with an unfair question: what's wrong with this code? It's some JavaScript with a function to sort users.
00:00:18.840 I do this by comparing the last name with the users, and I wondered if anyone had a guess about what's wrong with it.
00:00:31.360 Yeah, pretty close. Actually, in this application, the last name is nullable. One way users get created is by inviting them. They fill out a form with their email address, and after they click the link, they enter some details. Until they've completed that last step, users don't have a name or anything other than their email.
00:01:00.120 It's unfair because how are you supposed to know that this flow exists? In fact, looking at this code, you don't even know where 'users' has come from: has it come from an API, or is it just some static data in the front-end code? I think it's unfair, but I also believe it's representative. It's not easy to understand how backend concepts materialize in the frontend. It's very easy to make mistakes.
00:01:38.960 For example, I always have a blind spot in our codebase: is it 'last name' or 'surname'? I type the wrong one, and it happens no worse than 50% of the time—more like 75%. Anyway, today we're going to talk about how we can try to fix this.
00:02:25.120 I am a long-term Rails developer, and I remember Rails 1.0—simpler times. I'm the lead engineer at Skilla Whale, where we provide live team coaching for tech teams to help engineers become better and more productive.
00:02:41.280 The relevance of that to this talk is that since joining Skilla Whale, I've been doing a lot more front-end development, so these problems are becoming my problems too. I want to fix them. You might think that we can add more front-end tests, and by 'more,' I mean some. However, I want something a bit more systematic that doesn't rely on me remembering to add tests to cover all the cases.
00:03:36.080 If you were to ask a front-end person, they'd say, 'Hey, you should be using TypeScript!' That's great, but what is TypeScript? Other than being an open-source project from Microsoft, the Wikipedia page paraphrases it as adding static typing with optional type annotations to JavaScript. It's a bit of a mouthful, and also, it’s worthy of note that unlike SB and RBS in the Ruby world, TypeScript has been very widely embraced in the JavaScript world. In fact, any valid JavaScript is also syntactically correct TypeScript. There might be type errors, but the syntax is still good. This means it interoperates with JavaScript, so if you have an existing codebase, you can convert it to TypeScript one file at a time.
00:04:57.959 Type annotations are things you add to variables, functions, and classes. What happens when you add them is that you'll say, 'Hey, you said this thing needs an array of numbers, but you've given it an array of strings instead.' These are optional, and you can add them gradually; you can have a TypeScript file with absolutely zero type annotations in it. It might not do a lot, but it will still do something because it already has type information for all of the JavaScript standard library, the DOM interface, and so on. But you don't have to add them until you need them, and it's quite good at inferring them. You don't need to add many type annotations for it to start being useful.
00:06:30.360 Lastly, the term 'static' means that TypeScript checks the code without running it. There are no runtime checks—no 'Is this object or whatever?' The TypeScript you write gets checked and is then converted to JavaScript, which has no trace of the TypeScript left inside. TypeScript does understand statically what those runtime checks would do, and we'll talk about that more later.
00:07:16.480 There are two categories of type systems: nominal and structural. If you use SWG, you're using a nominal type system. Although these two classes might seem similar, they are different. For example, if two classes look and behave the same, from the perspective of SB, they're two different classes. You can't take a user and pass it to a function that expects a company. Structural typing is a bit different. For example, two TypeScript types with an ID property of type number and a name property of type string are interchangeable. This is perfectly valid TypeScript: I create a user and assign an object to it, which has an ID and a name, and then I assign the user to a company. This doesn’t make sense when stated out loud, but TypeScript doesn’t care about the names of the types; it only cares about the structures.
00:08:29.760 So what do types look like? These are the building blocks: primitive types like numbers, strings, and booleans; literals, which might seem pointless now but will become useful later; objects, which are simple key-value pairs; and functions, which here take a user and return a string. You can also work with arrays of things. Then we have some special types that act as escape hatches for working with untyped code. These building blocks are combined in interesting ways: you have unions, which can specify a number or a string, intersections that require an object to have properties from both a user and user details, and index types that allow indexing into types. You can also use generics, for instance, defining an HTTP response type with a status code and headers and a data field whose type can be specified via a type parameter. There are lots of complicated things you can do with generics, including conditional generics and utility types.
00:09:21.840 So, what does TypeScript do with this? This is a very quick overview. There are lots of other workflow benefits that TypeScript provides. Really, what you're trying to figure out is what things are. If you have something in your editor, what is it? I've worked with plenty of front-end codebases where this wasn't obvious. Here's something, and it comes from here, and it comes from there—a prop from this thing, and that’s coming from an API call. You have to go through four layers of indirection before you figure out what it is. Then once you know what things are, you have to figure out if you can use them correctly. Can I perform this operation on this type? You can get feedback in the command line, and you can run the TypeScript compiler, which will inform you if you're trying to call a function that might be undefined.
00:10:23.360 In-editor help is also available. This is the same code I just mentioned that checks a parameter called 'delete.' It might be a line item action or undefined. If it was a line item action, that's okay, but if it's undefined, then trying to use it as a function won't work. TypeScript acts as a language server, so any editor supporting the language server protocol will display these checks and suggestions. This includes VS Code, Sublime, Emacs, and basically any mainstream code editor at this point.
00:11:18.119 The live feedback is particularly useful for things like typoing property names; you get notified about any misspellings before you even finish typing. What does this look like in practice? You could have a React component. You can specify the type it expects as arguments. If I pass something that’s not a user, forget to pass a user, or pass something called a user but lacking the right properties, those are all errors. If the user doesn't have a name, that would be an error. While I'm typing 'user.', it will suggest properties that exist. Coming back to the code I started with, if you say lastName is a string and then compare two strings, TypeScript is happy with that. But if you say that lastName is a string or null, it will alert you that this could possibly be null. This would also appear if you ran the TypeScript compiler.
00:12:18.680 I have one last type concept for you: narrowing. This is when TypeScript can infer that your data has a more specific type than what you initially told it. The inference happens statically. Let's consider some code: I have a function that takes either a number or a string. Inside an 'if' block, it recognizes that the value is a number. Although it's not evaluating the type of the value, it understands the JavaScript syntax and knows the type of operator's role. The comparison shows that the value must be a number.
00:13:01.680 You might think this is trivial, but the understanding of types can extend beyond the simple cases. Consider an invoice page with line items—these can represent either money we've received or services we’ve provided. The most important object here is payments. It has a type field, which must have the literal type 'payment'—similar to single-table inheritance or delegated types in Ruby. It includes properties like ID, amount, and date. Coaching sessions are also important items in our invoices, representing the services we provide. In this case, the type must be 'session,' and it will include some properties in common with payments as well as session-specific properties.
00:14:07.059 A line item could be a payment, coaching session, refund, or something else. The technical term for this is a discriminated union, as it's a union of types where there's a property that tells you what type you have. By convention, this property is usually called 'type', but it can have any name. In the following code about line items, we might not know what kind it is, but it doesn't matter because everything in this union has an amount. Therefore, I can print out the amount easily. This code, however, is incorrect because if the line item were a coaching session, logging the session ID would be fine; but if it were a payment or a refund, that wouldn’t make sense.
00:15:21.679 TypeScript can recognize this mistake and state, 'Wait a minute, this property doesn’t exist on this type because it isn't present in this branch of the type.' But if I first check whether the item is of type 'session,' then within that branch, it's been narrowed down to type 'coaching session,' making it okay to access this property. As you can see, even though much depends on static checks, it understands enough about the language and will provide a lot of information to you.
00:16:35.500 Discriminated unions pop up all over the place, and they are very useful. They are good for cases of single-table inheritance or whenever you have similar objects. We often use them when a form can handle either a saved or unsaved object; they are mostly the same, but the saved object has an ID and some extra metadata that the unsaved object does not.
00:17:19.919 Now, let's pivot to an important point. While you can do a lot of work at compile time, there's an elephant in the room: some data will remain unknowable at compile time. If you have a web application that never communicates with an API or any backend, that’s quite limited. At some point, you will get data from the outside world. You might have some code like this in your application where you make a fetch call to get some data. In these examples, you might directly pass JSON.
00:18:09.280 This line can be problematic. This isn't a cast; it's an assertion. You're telling TypeScript, 'Trust me; data.user exists, and it conforms to this type.' It may or may not, and TypeScript doesn't know—it gives you a free pass. There are runtime schema libraries like Zod or TypeBox that provide runtime checks. But part of what I want is to enforce static checks, helping to find mistakes without needing to write a lot of tests that cover all code paths.
00:19:13.600 The question then becomes, how do I ensure my types are correct? If I doubt that user always has a name, why would I write this type? If I’m unaware that a name can be null, I’d write it as though it can’t, meaning I won't get the error flagged to me. So, everything hinges on this aspect. Have we made any progress? I believe so! We've added a lot of safety, created consistency within the front-end world, and probably identified very few places where we're writing down our assumptions.
00:20:29.919 There are likely only a few places where you're fetching data from remote sources, and they’re easy to identify. But can we do more? This is a rhetorical question; if I said no, I'd be severely under time. So I want to talk about JSON schema next. JSON schema is a document that describes the expected structure of JSON data. Sounds a bit circular, doesn’t it? However, there exists meta-schemas, which are JSON schemas that describe the structure of JSON schemas—that's a complex cycle. If you've ever used Open API or Swagger, that's a dialect of JSON schema.
00:21:31.759 This is a simple schema example; it’s an object rather than a string or array. Its properties are mandatory because properties default to optional in JSON schema. This schema emphasizes only human-facing properties. In another schema, the object has a users property that must be an array, and the type of things in the array is defined elsewhere—showing that you can create composite schemas.
00:22:45.959 In our Ruby applications, we can use gems like JSON Schema and JSON Schemas. I typically write all my schemas as plain files on disk, or YAML for readability. Various tools can generate usable API files for you. For example, Great API can generate Swagger files. But, as part of my workflow, I wanted to avoid starting with rewriting my API in the tool; instead, I opted for plain files.
00:23:55.679 So what do we do with them? The first action is inspecting them, fetching data, and expecting it to conform to the schema. We do the runtime assertions probably only in development since they're beneficial for catching issues before specs are written and for corner cases that arise during development. You might think these two processes are similar. But while JSON schema can be useful in the Ruby world with schema files and TypeScript checks, the crux is how to ensure both stay in sync and to avoid repeated definitions.
00:25:30.000 The response to this problem is yes; there are several tools that can help. I’ve been using one called JSON Schema to TypeScript, which works well. I have found that schemas have richer types than TypeScript. For instance, in TypeScript, you have a type called number, while in JSON schema, you can specify integers, floats, and define limits like minimum and maximum values. TypeScript strings are basic, whereas JSON Schema can specify patterns and regular expressions to ensure formatting.
00:27:46.640 It performs tasks you'd expect, and I’m wary of tools that generate code, having dealt with Swagger definitions before. However, I haven't had major issues with this library; it usually does what I want. If it doesn’t, there’s a straightforward escape hatch that allows me to specify my own types. If I start with a schema I’ve defined in YAML for a snooze object and indicate that it may be an ISO 8601 date, when run through the converter, I get the generated type.
00:29:08.880 The generated comment contains helpful descriptions that assist my front-end team in understanding what’s going on. We utilize schemas wherever possible—pages that are HTML or HAML templates, data attributes, API responses, and Action Cable events. Talking about Action Cable, we run a Rails app that delivers coaching sessions predominantly through it. When the coach navigates to the next slide, a message is sent through the channel broadcast to all participants.
00:30:35.090 The schema for the channel is defined in JSON schema format, indicating that whatever data arrives must match specific criteria. After running the input through JSON schema with TypeScript generation, we see generated types for events. A switch statement processes events, narrowing the Type field to identify the specific payload with its properties. If I forget any events, it should trigger a notice of the missing cases during compilation. As a result, adding a new event requires updating both the schema file and the TypeScript type.
00:31:47.000 Initially, I kept all my types in one file to manage. That strategy quickly became unmanageable, so I now maintain a so-called 'type root files' directory, organized by function with each creating a corresponding TypeScript file. Moreover, this directory is clear about what’s generated—encouraging to avoid hand editing unless necessary. These root files can reference types as needed, promoting reused definitions without redundancies. Ultimately, this structure supports effective organization across our application.
00:33:09.960 Summing up, we have Ruby code generating data, schema files defining that data's structure, specifications ensuring everything aligns with those shapes, and TypeScript types enforcing front-end safety. This setup achieves closer to end-to-end type safety and significantly aids in aggressive refactoring without concern for overlooked dependencies.
00:34:24.160 There's still a concern with type assertions, which sometimes feel like accidents waiting to happen. To address this, several colleagues have worked with TypeScript to develop more intuitive mapping from API responses. We've created functions generating TypeScript types, which support our API while maintaining accuracy.
00:36:26.240 Our approach emphasizes progressive integration, not requiring a full rewrite for adoption. Instead of a complete overhaul of systems, we embraced incremental tasks—converting existing components to TypeScript, adding schemas while creating or modifying endpoints. Over time, 95% of our front-end code adopted TypeScript. The benefits came before full transition too. Adding TypeScript before a refactor boosts confidence, avoiding unnecessary errors during implementation.
00:37:32.088 Thank you for your attention! The URL at the top leads to a Rails app showcasing this process, allowing you to see all the code. As for my curious cat, he’s eager to explore but is presently grounded, unlike the freedom you can pursue today. Thank you again, and here’s the feedback link.
00:38:14.760 Hi! Thank you for your talk. I have a question. Did you experiment with using Sorbet or other Ruby signature systems for better integration into the Ruby coding flow?
00:39:03.049 Yes! At our company, we use Sorbet for the backend of an application. While there are points with community support, TypeScript provides more immediate value and user-friendliness.
00:39:27.719 As for Open API, it offers significant potential due to its alignment with JSON schema. We didn’t switch to using it for our new system.
00:39:48.000 We still prefer specifying our schemas manually, but it’s certainly a path we would consider in future developments. Thank you for asking!
00:40:08.160 Thank you once more for your insights. We’ve learned that applying these practices gradually leads to incredible results.
Explore all talks recorded at rubyday 2023