Building a Ruby web app using the Ruby Standard Library

HTTP

Maple Ong

@wildmaples

#ruby-standard-library

Building a Ruby web app using the Ruby Standard Library

by Maple Ong

In this talk titled 'Building a Ruby Web App Using the Ruby Standard Library,' Maple Ong presents a foundational approach to understanding web applications without relying on Rails. Her session explores core concepts of web development by building a simple step tracker app entirely using Ruby and its Standard Library. Key points covered include:

Introduction to the Speaker: Maple Ong, a software developer at Shopify, shares her journey into coding despite not having a formal degree in computer science.
Web Application Basics: The session emphasizes that it will not focus on creating a web app for startups or provide a Ruby coding tutorial but will return to foundational concepts.
Networking Protocols: An overview of how applications communicate using IP, TCP, and HTTP, with a focus on the reliability TCP provides when delivering packets of data.
Server and Client Interaction: Maple explains how to create a TCP server in Ruby using the Socket library and demonstrates a simple echo server that responds to client messages with uppercase text.
HTTP Structure: She discusses the significance of HTTP for web applications, detailing the message structure of requests and responses, including methods like GET and POST, which are crucial for data exchange.
Building the Step Tracker App: Maple outlines the design of a fitness tracker web app that records daily steps and dates, explaining how user input can be managed through HTML forms and processed via POST requests.
Data Persistence: The presentation shifts to how data can be stored persistently using a YAML file, emphasizing the concept of a lightweight database and explaining how to read/write data as transactions.
Conclusion and Recommendations: The session concludes with recommendations for participants to explore concepts such as Rack for enhancing web applications and the potential move towards using more traditional databases like SQLite. Maple encourages learning through building applications from scratch and shares her intention to provide further resources and interaction on Discord.

The audience receives Maple's practical insights positively, reinforcing the educational value of understanding fundamental principles in web development. This session provides a thorough understanding of how a web application functions at its core, leading participants to an appreciation of higher-level frameworks like Rails.

00:00:00 Our next speaker is Maple Ong, and the topic is 'Building a Ruby Web App Using the Ruby Standard Library.' Do you want to understand how a web application works without Rails magic? Let's go back to the basics of web applications and build one from the ground up using the Ruby Standard Library. After this session, you'll appreciate Rails that much more.

00:00:14 About our speaker: Maple is a software developer who made it without a formal education in computer science. She currently works at the Truffle Ruby team at Shopify, where speedy Ruby isn't just a fantasy. Previously, Maple worked on Backwork, a Ruby gem designed to enforce modularity in Rails applications. You can find our lovely speaker on Twitter at @ongmaple, but we drop this to the stream chat. Maple, the floor is yours!

00:01:23 Oh, hello there! Hi everyone, welcome to the talk. Today, we'll be building a web app using Ruby. I was just kidding about that—I'm not going to be looking at my phone in real life.

00:01:40 So just to be clear: this talk is not about how to build a web app for your startup. You can just use Rails; they have awesome guides. This is also not a tutorial on how to write Ruby code because I'm going to assume that you have some experience reading and writing Ruby code already. Lastly, we're not going to go into complex web application concepts; this is going to be a foundational talk.

00:02:06 With that being said, we'll learn about networking protocols, specifically IP and TCP, HTTP messaging, data storage, and persistency. Just a quick introduction to me: my name is Maple.

00:02:20 This is the picture we have on the site; I'm actually not that serious in real life. Right now, I live in Waterloo, Ontario, Canada, which is about an hour away from Toronto. I don't have a computer science degree; I actually studied health sciences and only learned programming in my last year of undergrad.

00:02:36 I currently work on the Code Foundations team at Shopify with a bunch of cool people. More specifically, I work on Truffle Ruby. If you didn't know, Truffle Ruby is a high-performance Ruby implementation, very much like MRI or JRuby. It's built on the Truffle language implementation and framework and GraalVM.

00:02:54 Essentially, what we're trying to do is optimize it for production-level code, which is pretty fun. Outside of work, I really enjoy working on training, which is why I chose our web app today to be a step tracker. This is what we'll be building today!

00:03:13 If you're a fitness enthusiast like me, you know that tracking your daily steps is a good way to do informal cardio, especially if you don't enjoy cardio very much. I aim for at least 10,000 steps a day and you can find me walking on my under-desk treadmill during meetings.

00:03:32 So, obviously, I'm no designer, but the point of this app is to keep it super simple. We're going to focus on the concepts of web applications and keep it simple, silly! What we want our web app to do is to record the date and the number of steps per day and display that on the main page for everyone to see.

00:03:50 Right before we begin, a quick disclaimer: I won't be able to cover every single detail in this talk. However, I highly encourage you to dig into the documentation and source code by yourself if you're curious about any detail I've mentioned.

00:04:07 I will also be sharing the link to this presentation, and the slides will contain links that I found helpful when I was writing this talk. So let's get started, and start from the bottom: how applications communicate.

00:04:25 Just like humans use languages to speak, computers do the same by utilizing various levels of communication protocols. At the lower end of the networking stack, we have the Internet Protocol (IP) within the network layer, we have TCP within the transport layer, and on top of that, we have HTTP within the application layer. We will be talking about all three layers in this talk.

00:04:42 At the IP level, or the Internet Protocol level, little packets are delivered from the source to the destination—your computer. That's basically how you connect to the Internet. The truth is, packet delivery is generally unreliable, though, because it might take different network routes to reach its destination, and even then, packets might arrive out of order.

00:05:02 To overcome this unreliability issue, two protocols were created, and one of them is TCP. It ensures a more stable and reliable connection. The other is UDP, but for this talk, we'll focus on TCP since that's how our web application server communicates with the client.

00:05:21 TCP provides a layer of instructions on top of IP to be more reliable. For example, the sender can add an incrementing number with each IP packet, and the packets are then reassembled in order once they arrive at the destination. TCP also provides a level of error control, where a mandatory field is sent and then checked for corruption at the destination.

00:05:40 The protocol can then discard any corrupted packets, ensuring that our web application won't have to handle any corrupted data. To use TCP, we have to first establish a connection between the host and the client through a port.

00:05:58 There's an actual directory of port numbers for different types of applications. For example, the port number for Simple Mail Transfer Protocol (SMTP) is 25, which is used to connect to an email server.

00:06:12 To establish a TCP connection between the web application, which is your backend, and the client, which is the browser, a socket must be created. That's how a server listens for incoming connections. Once a connection is established between the server and client, the socket turns into a client socket, which allows the server to share information with the client and vice versa.

00:06:28 Let's try writing this in Ruby. Thankfully, a Ruby library called Socket exists. Socket is the library that interfaces with the TCP implementation built into the operating system and provides a good API to create sockets.

00:06:45 Let's take a look at our first code example. The goal is to write a service that returns the same information as the input to the client but in uppercase. We'll first require the Ruby library socket and use the TCP server class to create a TCP server at port 9999.

00:07:06 In a loop, we will accept any connections to the server, and once the connection is established, it will be assigned to the client variable, representing a client socket.

00:07:24 We'll create another while loop that takes any lines of text from the socket using the get method, adds some exclamation points to it, and then sends it back to the client with a put statement. After that, we close the client socket, and that's the end of the loop.

00:07:43 Let's run this and then use Netcat to connect to it. Netcat is a command-line tool for TCP clients that allows us to create TCP connections directly from the terminal. We will use Netcat, or nc, to connect to the server.

00:08:06 You can see here that I'm just running the file with the code we just wrote. Here, we connect using Netcat to the localhost, which is where the server is hosted, followed by the port number. We will enter some text, and you can see it returns the text with exclamation points.

00:08:22 Now that we have a way of communicating with the rest of the world, let's explore a higher level.

00:08:34 Just for the record again, we're sending messages to the client socket, but it's not really a web client or a browser. Let's see how we can actually send messages to a browser this time. Considering what we have right now, we can essentially send and receive streams of information, or bytes, between the server and the client.

00:08:52 However, since that information can literally be anything, we can see that there might be an incompatibility issue. If a browser client reads and writes information to the server, there will be a problem if other servers all return their information in a different structure.

00:09:12 How would a browser communicate with all those other servers? There has to be a standardized way of sending and receiving information. We can't just stream packets to TCP; there must be a way to structure that information, and that's where HTTP was born.

00:09:30 HTTP, or Hypertext Transfer Protocol, is yet another layer on top of our network stack. In fact, we are now in the application layer, which gets its name because HTTP messages come directly to and from the application.

00:09:48 The HTTP protocol is built on top of application-agnostic TCP, meaning that TCP can be used in any type of application, not just the web. TCP is also implemented in the operating system, but we can implement HTTP in Ruby.

00:10:07 Just for the record, there are definitely Ruby libraries out there, such as Net::HTTP, that we can use to implement HTTP, but we are doing this ourselves to learn about it.

00:10:23 An important concept that helped me immensely in understanding HTTP—and helped clear up confusion—is that HTTP does not represent any physical code. Instead, it's just a description of the structure of a message that everyone agreed to use as a standard.

00:10:41 HTTP is a specification, and we use code to parse and structure the message according to those specifications.

00:11:00 HTTP communication can be broken down into two types: requests and responses. A client can make a request to the server, and then the server can return a response. However, every single HTTP message has the same structure. It starts with a start line, then header fields (which can be zero or more), followed by a blank line and an optional message body that usually contains HTML code.

00:11:17 I guess I really want to explain why there's an alien being there, but maybe after I discuss the structure, you can understand why I drew that alien.

00:11:33 The difference between the request and response message is in the start line. The first line of a request is called the request line, while the first line of the response is called the status line.

00:11:50 The request line for HTTP can be broken down into the method token, the request path, and the HTTP version. In this example, the method token is GET, the request path is 'hello there,' and the HTTP version is 1.1.

00:12:07 There are several types of method tokens, also known as request methods, that we will be using in this talk: GET and POST. A GET request is when information is requested from the server, while a POST request is when information is sent to the server, which may modify the database.

00:12:23 There are also other types of methods, like PUT and DELETE. Moving to the HTTP response, the status line can be broken down into the version number of HTTP, the numeric status code (in this case, 200), and the status description, which is 'OK.'

00:12:43 There are other status codes as well. The most common one is the 404 error, or '404 Page Not Found.' You probably remember it because it's very frustrating to see.

00:13:00 The status code itself categorizes the type of statuses. For example, status codes in the range of 200 indicate a success, while codes in the range of 300 indicate redirection to perhaps a different page. Codes in the 400 range indicate a client-side error, and codes in the 500s indicate server-side errors, like a timeout.

00:13:16 Let's take a look at a real example of an HTTP request. We can do this using Netcat, again with the '-l' flag to listen for any incoming connections. In this case, we're looking at localhost at port 9999.

00:13:34 This is a request, and you can see the first line is a POST request to the target path 'add data,' with the HTTP version being 1.1. Below that, you see a bunch of header fields that look very familiar.

00:13:50 Structurally, you can see it's the header name followed by a colon and then its value—it's very much like a Ruby hash. Lastly, we have a blank line and then the message body.

00:14:04 Let's move on to our next code sample, where we will try to receive and send simple HTML. HTML is a way to describe a formatted and interactive document using plain text.

00:14:24 Just like the browser can use the HTTP protocol, it can also parse and display HTML to the users. Let's take a look. Don't feel overwhelmed—we're going to go through it. The goal is to accept and parse a request and then send it back as simple HTML.

00:14:40 To start, instead of just reading the client input, we will take it and split it into three parts for the start line: the method token, the target, and the HTTP version number.

00:14:58 Next, we will decide what to do with it. In the case where the method token is a GET request and the target is 'show data,' which will be our main page, we will craft the response message to say 'Hello, World!' in an HTML <h1> tag.

00:15:15 For any other combination of method token and target, we will return a message saying it didn't hit any endpoints and just shrug.

00:15:32 We need to construct our HTTP response since we know what our message is going to be. You can see here it has the same structure as we've seen before; the first line is the status line, specifically 'HTTP/1.1 200 OK.'

00:15:51 The content type is 'text/html.' There are other content types and header fields you can attach to your HTTP response, but we won't go into those.

00:16:08 Lastly, we provide a blank line followed by the response message. Let's look at the demo of this code after it's run.

00:16:27 We will go to localhost with just the port number without any path, and we'll see 'didn't hit any endpoints' because we weren't on the correct path.

00:16:43 We’ll try a valid path and still nothing, but when we try 'show data,' our main page, we will see 'Hello, World!'

00:17:01 Now that we have a structure for our app, let’s allow users to input some data, specifically steps and dates, and display it on the site. But how can we allow users to input data and send it to the backend?

00:17:18 The simplest way is to use an HTML form. We won't go into detail about how the form element looks and works, but I’ve linked the resources for you to explore.

00:17:40 First, let's take a step back to see how the flow should look. When a user first visits the website, say the main page, it will request information with a GET request to the server. The server then responds with a form for the browser to display. The user inputs their information, and it is sent back to the server with a POST request.

00:17:58 Finally, the server responds with an updated browser page containing the new data. Let's walk through the code now.

00:18:16 We are going to create some default data in a variable called all_data, which will be an array of hashes. Each hash will contain the date and step count as keys.

00:18:32 We will also define a method called daily_steps_form, which will encompass the HTML for the form. The form's action will be set to the target path, which is 'add data,' when the request is made, and the method will be a POST request.

00:18:50 Lastly, the encoding type specifies how the form data will be encoded when it's sent to the server. Here, we will use 'application/x-www-form-urlencoded.'

00:19:06 Now that we’ve defined a variable to store all data, we can display it when a GET request is made. We will use this variable to parse the array and create a response message with the data, organizing it as an HTML unordered list.

00:19:26 We are creating a new endpoint, which will be a POST request at the target 'add data.' Here, we will also set the status code to 303 and redirect the page back to the main page after a user enters new information.

00:19:42 To accomplish this, we will parse the header fields from the request and store them within a hash called headers. The most relevant part of this headers hash is the 'Content-Length' key, representing the size of the request message body in bytes.

00:20:01 Let's refer back to our HTTP request example. Here, you can see that the Content-Length header is 34 bytes long, and the message body below is also 34 bytes long.

00:20:21 The reason we care about this Content-Length header is so that the application can accurately know the size of the body and stop receiving the request once we've received the entire message.

00:20:40 Let’s point out the structure of the message body—it's in the 'application/x-www-form-urlencoded' format we set earlier. The variables here are separated by an ampersand.

00:20:59 Once we’ve got the message body using the Content-Length header, we must decode it. The method from the URI Ruby library provides decoding into an array of key-value pairs.

00:21:16 After we obtain the decoded message, we will store it in our all_data variable, and lastly, we will hard-code the location header into our HTTP response for simplicity. This means we will redirect to the main page after every single HTTP response.

00:21:35 Now, let's run the code. As you can see, I am running the code we just wrote and then going to this main page 'show data.' We can now see the HTML form—let’s enter a date and then the step count.

00:21:53 You got zero steps today! That's cool, and that's it—our app works!

00:22:10 Now let's see what happens with this new GIF. You can see that with a bunch of entries already in place, I will stop the server, refresh to ensure it's soft, run the server again, and refresh the browser.

00:22:26 You can see that the steps are gone. It’s probably not the best idea to run this in production.

00:22:42 Why, though? This is because we are only storing the data in memory, which is not permanent. So where can we store the data instead? A database seems too complicated, so let's store it in everyone's favorite file type: a YAML file!

00:23:00 Only if you're a human, though! We could use a plain text file instead of YAML, but then we'll have to implement a way to represent Ruby data in plain text.

00:23:16 To keep it simple, we will use YAML store, a Ruby library that helps serialize and deserialize Ruby objects to and from YAML files.

00:23:32 Let’s dive into the code; this is a quick one. First, we will create a YAML store and a new file called store.yaml that contains our previous default step count entry.

00:23:48 Under the GET endpoint, we will read from the file, creating a transaction that fetches the step data from the file and stores it within our all_data variable.

00:24:05 A transaction is simply how we read and write data to the YAML store as part of the API.

00:24:21 For the POST endpoint, we will update the YAML file by creating another transaction that incorporates the user's new input into the daily steps hash from the file.

00:24:36 And voilà, we’ve created a basic web application using Ruby and only Ruby libraries! Any user input data will now be persistent because it is stored within the YAML file.

00:24:54 When we want to read any data, we will also be reading it from the YAML file, so everything is good!

00:25:12 Today, we’ve learned about TCP sockets, how to form connections between client and server, how to receive HTTP requests, and send HTTP responses.

00:25:26 Additionally, we've covered how to store data persistently in a file. I feel like these are all fundamental concepts that you’ll encounter at a higher level of abstraction when working day-to-day.

00:25:44 For example, when working with Rails, these concepts can apply to other programming languages and web frameworks as well, not just Ruby and Rails.

00:26:01 As you can tell from the code I shared, writing everything manually is a huge pain in the back end. If you don’t believe me, try writing a legitimate web application, maybe a blog, using only Ruby libraries, and tag me in it because I would love to take a look.

00:26:18 It’s a fantastic educational experience! So what’s next? Spoiler alert: there are more layers to the Ruby ecosystem that will assist you in building your web application.

00:26:35 For example, you can refactor your web application to be a Rack application. Rack is a web server interface on top of HTTP, which is very similar to HTTP because it's just a set of specifications.

00:26:54 Rack is the interface for web servers such as Puma and Unicorn. Once you've refactored the app to be a Rack application, you can delegate the HTTP logic to the web servers, so you won’t have to write it yourself.

00:27:11 You can also use a database such as SQLite or MySQL instead of storing data in a YAML file. If you’re going to do this as a learning experience, I highly recommend using SQLite because it's much simpler, and there’s a Ruby gem for it.

00:27:27 Using a database provides read and write optimizations when dealing with large amounts of data. Lastly, we have Ruby on Rails, which contains the all-in-one framework to build a Ruby web application.

00:27:44 Using Rails means you won’t have to worry about Rack or databases, as everything is built into Rails.

00:28:02 This is the code we’ve written—well, maybe half of it—and I’d like to show you what Rails libraries would replace our current code.

00:28:19 First, the daily steps form can be replaced by Action View, which is the Rails library that collaborates closely with Action Controller to manage view template rendering.

00:28:36 This HTML code would not even be in the same file! Rack application servers like Puma will handle parsing the HTTP request and generating the HTTP responses.

00:28:53 Action Dispatch parses the HTTP request and routes each request to our endpoints. Action Controller handles the business logic within the endpoint.

00:29:11 In this example, it would handle the GET action, retrieving the correct objects to display on our page.

00:29:27 Lastly, we have Active Record, which implements Object-Relational Mapping (ORM) to handle database interactions and map them to Ruby objects for usage or modification.

00:29:44 Thank you so much for listening! You can find me on Twitter at @ongmaple. Thank you, and goodbye!

00:30:01 Thank you! On the backstage, we were discussing your Twitter handle; it should be 'Oh my God, Maple!' because you are such an amazing person. We have a chat full of comments. Let me read this out loud: 'Maple's talk is better than probably 99% of all university courses as an introduction to the web! I really like talks that explain fundamental principles in a practical way. Best presentation award goes to Maple today!'

00:31:07 "Very nice illustrations, explanatory slides, and a great presentation! For your next talk, will you be implementing an email server from scratch? You are truly amazing! So, we hold you accountable for that! There’s so much praise in the comments!"

00:31:26 As a follow-up, there's only one question for you: how many of these techniques do you use in your daily work, like building something from scratch to understand how it works?

00:31:44 That's such a good question! I’d say that this knowledge helps me understand how the system works as a whole. For example, when working on Rails, it's beneficial to understand what's happening underneath.

00:32:05 For example, I've learned about Rack middleware, and in my work with Truffle Ruby, we try to benchmark middleware in production code. Understanding why we need middleware and how it connects to the application was something I found very helpful in my learning.

00:32:24 Would you look into using Rack next? That’s a good idea for a follow-up to this project, though I haven’t tried it yet.

00:32:42 If you try it, remember to share your results with us! There’s a request to share your repository or quizzes with the code?

00:33:04 So, I’ve thought about this. While I want to share the code, I feel it would go against the spirit of this talk. I'd love for you to write it out yourself after hearing what I had to say.

00:33:22 However, I do have a version of this mini web app linked with a similar blog post I've written, so I can share that.

00:33:40 This is an opportunity for the Euruko 2021 challenge! Build a Ruby web app using only Ruby libraries and see how far you can go and how complicated it can get.

00:33:59 I would be very interested in that! As stated in the talk, feel free to tag me; I'd love to take a look!

00:34:15 We can discuss this challenge idea in Discord. Are you ready for that discussion? I hope we can get you on Discord if you are not already there!

00:34:30 Just one question: will the code for this simple app be shared?

00:34:47 We have discussed this! You may share something, but the challenge is to build your own app!

00:35:02 I will share the slides for sure! I have the slides ready, and I'll share that link on Discord.

00:35:20 Yes, we have more comments on the stream chat: 'Maple, you have the best t-shirt so far!' Oh, thank you!

00:35:34 I really like it; I think we all agree that Led Zeppelin is one of the greatest bands EVER! Thank you, Maple, and we will see you on Discord later.

00:35:50 Thank you so much for this. It was a truly amazing talk and presentation.

EuRuKo 2021