Build Realtime Apps with Ruby & Pakyow

RailsConf 2016

by Bryan Powell

In the session "Build Realtime Apps with Ruby & Pakyow" presented by Bryan Powell at RailsConf 2016, the focus was on constructing real-time web applications using the Pakyow framework, emphasizing a traditional backend architecture without relying on JavaScript. Key points discussed include:

Introduction to Pakyow: Pakyow is a server-side, end-to-end web framework for Ruby designed to build applications that keep data synchronized with the server, contrasting with typical client-side frameworks.
Unique Architecture: Unlike Rails, Pakyow allows developers to build server-driven applications where browser views automatically sync with server state changes, enhancing user interaction without the need for JavaScript.
Real-Time Interaction: Powell demonstrated building a blog application where features like live comment updates and viewer counters were implemented simply by subscribing views to server-side states; this was achieved solely with Ruby code.
Separation of View Structure and Rendering Logic: Pakyow's design differentiates the rendering logic from view structure, allowing developers to manage UI transformations efficiently when states change.
Historical Context: Powell shared insights into the inception of Pakyow, discussing their prior experiences with Rails and the challenges they faced in integrating prototypes with backend logic, thereby motivating the creation of Pakyow to focus on view-first development.
Additive Rendering: The framework handles rendering changes to the UI in a manner that minimizes unnecessary DOM manipulation, allowing for seamless updates and maintaining the application's responsiveness.
Enhancement of Real-Time Features: The design philosophy supports real-time functionalities as an enhancement rather than the main structure of applications, ensuring that traditional server-side capabilities remain intact even without JavaScript.
Future Aspirations: Bryan discussed the vision for Pakyow's development, including improvements in documentation and API design to make it more accessible for developers, encouraging community involvement through collaborative issues for those interested in contributing.

In conclusion, the talk illuminated how Pakyow provides a fresh approach to building modern, real-time applications while preserving the fundamental server-driven architecture that has served web development for years.

00:00:10.250 Let's go and get going. So, hello RailsConf! It's been a pretty good conference, and I hope you all have enjoyed it.

00:00:15.509 This morning, we're going to kick off the alternative framework track. We're going to talk a little bit about how to build real-time apps with Ruby and Pakyow. I'm Bryan Powell, one of the creators of Pakyow. I play an active role in maintaining it today.

00:00:29.699 About fifty percent of my time goes towards building new features and fixing problems. I have the privilege of working alongside about a dozen other maintainers. We love this project; we love the little slice of the Ruby community that we've been able to carve off for ourselves over the last year or two.

00:00:40.620 You can follow me on Twitter at @brianp. I'm curious, who here has heard of Pakyow before RailsConf? Okay, awesome— a few people. For everyone else, Pakyow is a web framework for Ruby.

00:00:52.640 Pakyow is an end-to-end framework, so it doesn’t work on top of Rails; you would use it instead of Rails in your stack. In Pakyow, we’ve designed it to provide a different take on building applications for the web, and it’s really different than what you might see in something like Rails or even other ecosystems.

00:01:12.840 There's really nothing quite like it. So, this being RailsConf, I did want to take just a minute to publicly thank DHH. A couple of years ago, he was kind enough to offer his endorsement of some of the concepts behind Pakyow. This is why we do what we do. Thank you, DHH!

00:01:21.000 Pakyow lets you build server-driven applications that automatically stay in sync with the server. What this means is that Pakyow is a server-side framework just like Rails. On the initial request, Pakyow will perform the rendering on the server side, build up a full HTML response, and send that down to the browser. That’s presented in the browser just like any other HTML page would.

00:02:01.040 Now, where things start to differ from other server-side frameworks is that the view in the browser automatically keeps itself in sync with the server. If we’re looking at a list of comments in our application and one of you adds a new comment to the list, we’re going to see it immediately. Anyone else looking at that list will see that change immediately in their browser. The framework handles this for us, so as the developer, we’re not having to move any of our application to the client; it retains this traditional server-driven architecture.

00:02:50.610 Let’s look at an example of that together. We’re building a blog, and there’s a bit of a story behind this. We want to build a blog, but we want it to be more than just a place to throw random thoughts out; we want to think of it more as a place to start a conversation.

00:03:02.250 To make this engaging for the reader, we’re going to do that in a couple of ways. You’ll see the two counters: there’s a blue one and a gray one. The blue counter indicates the number of active readers—how many people are currently reading this page. The gray one shows the total number of readers across the history of this blog post. Underneath that is a comment form, and below that is where the comments will be listed. Right now, there aren’t any, so we see an empty message.

00:03:44.940 Let’s simulate usage of this between two clients and see where things stand. You’ll notice that as I refresh, the comment will show up in one client but not the other. The same thing applies to comments; it’s not until the second client refreshes their screen that they will see that new state.

00:04:07.740 Let’s fix this and make this a little bit more engaging. We’re going to start by opening up our routes. This is going to look very different from something you might see in Rails. We’re not going to try to understand everything that’s going on here right now, but we’re going to focus on our statistics first. The two counters will start to render in real time.

00:04:31.340 Then we’re going to test that out and come back to our comments. There are two steps that we’re going to take here: we’ll find the code responsible for rendering our counters and add a call to subscribe.

00:05:01.140 So what this is doing is telling Pakyow that we want to subscribe this rendered view to any future changes in the state. That’s step one. Step two is we’ll write a client-side JavaScript component that will be responsible for rendering the state on the client—just kidding! We’re not going to have to do that. We will not be writing any JavaScript today, I promise. I didn’t lie in my abstract.

00:05:45.960 Adding the call to subscribe and telling Pakyow that we want this view to keep up with its state is all we have to do. Let’s go back to our browser and test this out. When we reload the pages, notice they’re out of sync. I’m going to reload one client, and all of a sudden, the counters start counting together.

00:06:17.400 You can even see the blue counter go down to one and then come back up to two as the connection is lost and then reestablished with that second client. And we didn’t have to write any JavaScript; all we did was tell Pakyow what we wanted it to do, and the framework figured out how to make it happen.

00:06:46.530 So, our comments still aren't really working well, so let’s fix that next. We're going to go back to our routes and look for the code responsible for rendering comments. We’ll add a call to subscribe just like we did with the statistics.

00:07:07.389 Then we come back to the browser, reload the changes, and if everything works properly, we’re going to start to see comments show up. And in fact, we do! If one client adds a comment, any other connected client is going to see it immediately on their screen.

00:07:35.190 We can go the other way too if we want, and again we didn’t write any JavaScript to do this. This is all happening in a server-driven environment. You might have heard of Node.js; we embrace Node.js. I kind of like to think that maybe this is what they were going for when they built Node: a misunderstanding, like someone had a cold and said Node.js, and someone interpreted it as Node.js and went off and built that.

00:08:03.190 I don't know; just a theory. I want to be clear that I like JavaScript. I enjoy writing JavaScript! What I don’t like as a developer is having to write code that’s largely unnecessary, and I think a lot of the JavaScript that we write today is completely unnecessary.

00:08:52.100 So if we’re not writing client-side JavaScript, how is this working? That’s a fair question. You might think, well, we’re transpiling our view-rendering code, which is written in Ruby, and we’re transplanting that into JavaScript and sending it down to the client. No, we’re not doing that; I don’t think that’s a very good idea.

00:09:06.890 At the end of the day, this approach requires you to build an application in a particular way. It doesn’t give you the freedom to build a traditional server-side application. So then you might think that when the state changes, we’re doing a re-render on the server, building up the new HTML, and sending that down to the clients, replacing the DOM or even part of the DOM. No, we’re not doing that either.

00:09:47.040 I think this is probably a better solution than the first because at least the server is always doing what it knows to do, and that is to render HTML and send that down in a server-driven environment. What I dislike, and why we avoided this in Pakyow’s implementation, is that inevitably you replace more of the DOM than you need to.

00:10:02.170 We were really interested in performing the fewest number of DOM transformations necessary to render the new state. So how does this even work? It seems like we’ve kind of exhausted all our potential options here. What we haven’t mentioned is that Pakyow works in a different way. To explain this, we need to build up some context about how it all started, the initial problems it was built to solve, etc.

00:10:54.430 So welcome to 2011! This is when we first started working on getting a real release of Pakyow out the door. It’s been around for a long time; the first release was in August of 2011.

00:11:07.640 At the time, I was working for MetaBond, which is a consulting agency that I started back in 2007. We built web-based apps for all sorts of customers. We were working in Rails mostly at the time; we’re still around today and do some Rails development. Over the four years or so that we had been in business, we had developed some unique processes—at least I think they were pretty unique at the time.

00:12:16.220 We called this view-first development. Our priority with view-first development was to get something in front of project stakeholders that they could see, touch, and use in a browser as early in the project as we could.

00:12:34.200 When we started a project, we would meet with project stakeholders to determine what we were building. We’d figure out the feature set, prioritize it, and start with the highest priority feature.

00:12:43.580 The kinds of applications we built mostly involved user interfaces, so that’s what we wanted to start with, as it was the tangible thing we could play with together with stakeholders. This allowed us to discuss the more abstract conceptual details of the feature.

00:13:17.640 The way we wanted to work was to create a useful prototype by building just the view. We didn’t want to build a prototype application; we only wanted to write the HTML views, the things presented in the browser, and start there. We could deliver that in a day or two and iterate on it with the project stakeholders.

00:13:42.400 We could throw it out and start over if we wanted to since we hadn’t invested much time. Once the prototype was nailed down, we wanted to build the backend on top. This implied a consistent forward movement. We didn’t want to throw out previous work; we wanted to build on everything that had happened previously.

00:14:09.240 The third goal of view-first development was to be able to revert back to the prototype at any point. We weren’t building things in a waterfall fashion; we weren't prototyping the entire application upfront and then integrating it into the application before shipping it. We were doing this one feature at a time.

00:14:23.809 What we found with technologies such as Rails, and most server-side frameworks, was that we’d build our prototype in HTML ERB templates that contain only HTML; they didn’t have the Ruby code. When it came time to integrate the feature into our application, we would sprinkle in the Ruby code, and that would destroy our prototype.

00:14:52.450 We couldn’t just see the raw HTML without our Ruby code sprinkles, which meant when we started building the second feature, we could see its prototype, but we couldn’t view it in the context of the original feature's prototype. This made it difficult to work through this with stakeholders.

00:15:18.199 So we did the only logical thing we could think of at the time—we built our own framework to solve this specific problem. When we set out to build Pakyow, we really had this one goal: to identify a pattern that I and a few others had noticed in every web technology we had used since starting back in the early 2000s.

00:15:42.790 That pattern was the presentation side of web applications, which combined two things that we thought were really separate and should be treated separately: view structure and rendering logic.

00:16:07.910 If we look at an ERB template here, we’re iterating over a set of posts and rendering an article for each post while putting the value of the body in the paragraph tag. There are two things represented here: the structure of the view, which is the HTML, and the logic to render it, which is the Ruby code.

00:16:34.480 This ERB template has some semantic meaning, and we can see that our <article> tag represents a blog post or post object in this case, because we’re creating one for every post in our dataset. We can also see that the <p> tag represents the body, as that’s the value we’re sticking into the body tag.

00:16:58.700 So this ERB template has some meaning that gets lost when you build it because you end up with just the HTML, so we set out to try to save that. What you see at the bottom is a Pakyow view; it’s the same thing as the top. You’ll notice that there’s no Ruby code in this view, but there are two attributes: a `data-scope` and a `data-prop`.

00:17:25.650 In Pakyow terminology, a `data-scope` represents a data type, so here we’re saying that this article represents a post in our system. The paragraph tag nested within the post represents an attribute of it—specifically, the body of its parent scope or post.

00:17:53.050 That's all the view is in Pakyow. We can stop here. The logic to render it looks something like this.

00:18:19.910 This is just an example; there are many different ways that you can do view rendering in Pakyow, but this is a simple one. You’ll notice that this doesn’t really look like we’re rendering HTML; there are no DOM transformations that we’re needing to express. Instead, we’re expressing our rendering logic in the context of the semantics represented by the view.

00:18:52.780 So we’re not saying, 'Find the <article> tag in the view and render that.' We’re saying, 'Find me the node, whatever it is, that represents the post in my presentation and apply this dataset to it.'

00:19:15.940 Now, ‘apply’ is not magical; it’s pretty simple to understand. Let's look at a couple of examples. If we have a hash representing a single post object that we want to present and we apply that to our template, ‘apply’ is going to do two steps.

00:19:52.180 First, it makes the view template match the data structure we’re applying to it. Here, there’s no change required because our template already represents a single post, so all ‘apply’ has to do is map the body value right into its place in the template.

00:20:29.289 We’re left with a result looking at a more complex example with two posts. Now ‘apply’ has something to do.

00:20:42.490 Our template only represents one post, but we’re trying to present two, so it creates a duplicate and can simply map the values into the view. That’s all view rendering is in Pakyow. This allowed us to achieve our goals.

00:21:07.300 We could prototype by writing the HTML, then add the backend code without having to change our views at all. We could turn the view rendering part off and be left with our prototype.

00:21:42.590 We built applications this way for a few months, and we quite enjoyed it. I had this nagging feeling in my mind that I couldn’t shake. Keep in mind, this was around 2012. Single page apps were becoming more popular as client-side frameworks.

00:22:01.330 I couldn’t shake the feeling that we had something to offer people who wanted more than just static views presented in a browser but really wanted a dynamic user experience. I thought we might have a unique solution to this, stemming from one idea.

00:22:30.240 This is our first rendering example, and you’ll notice that our rendered view still has the semantics that the template has. We still know what every node represents in this rendered view. So we can actually use that previous render as the starting point for the next one. If we add another post to our state, we don’t have to start with the original template.

00:23:09.200 We can start with the first render and build on top of that, resulting in the same outcome. This idea suggests that rendering is additive. The code to render the initial view may be enough to also render changes to an existing view.

00:23:51.190 In Pakyow, view rendering occurs on the server and is presented in the browser. That means what’s presented in the browser retains those semantics—we know exactly what it represents in our application.

00:24:27.740 So we already have that piece in our template to work from. All we need now is to get the transformations to the client. Theoretically, with some magic, we can make the client update itself.

00:24:57.440 Let’s walk through a full example of this. The client requests the server for the initial view; the server responds with a full HTML response displayed in the browser. Subsequent updates originate from the server as well, so the server tells the client how it can update itself. This happens with something called UI transformations.

00:25:40.180 The way we build these transformations is by executing the same rendering code we used for the initial render, but instead of rendering HTML, we record the transformations that need to occur.

00:26:07.540 So we’re not transpiling that code into JavaScript; we’re just creating a set of instructions for the rendering that should reflect the new state. At this point, it's just data—a large data structure. We can serialize that.

00:26:46.640 That’s what we do; we serialize it as a JSON object and send it down a WebSocket to any client concerned with the state change. There’s a client-side library that is part of Pakyow called Ring, and Ring is what receives these transformations and does something with them.

00:27:23.130 Ring has the template and the transformations, and all it has to do is step through the transformations to make the necessary updates to reflect them in the view. This happens through something called the view transformation API.

00:27:52.800 Because we’ve separated our view structure from our rendering logic, we have a nice set of methods to express our view rendering in Ruby code. We implement that on the server through a class called Presenter. Presenters are responsible for the initial HTML rendering, while Ring also has a client-side implementation of the same API.

00:28:51.210 When Ring receives these UI transformations, they’re expressed in this common API. Ring can then iterate over it, calling the right functions on the client side, and it will make the browser reflect the new state.

00:29:45.760 So Pakyow falls somewhere between a traditional and a client-side framework. It’s server-driven; it’s a server-side framework, just like Rails, where we receive full HTML responses, but once rendered, we get this client-side behavior.

00:30:06.560 We don’t have to move anything over to the client—none of Pakyow’s code has to move to the client. All of our code stays Ruby and remains server-side.

00:30:33.230 Thinking about how we might do this in other server-driven frameworks, we inevitably adopt some sort of hybrid approach. Let’s use Rails as an example. If we care about building our real-time blog in Rails, we might do the initial rendering on the server, allowing us to send the full HTML response back.

00:30:54.320 We might then use some client-side tool, maybe Ember.js or Backbone, or we could even use Action Cable for some components that are responsible for making client-side state changes as needed to maintain the dynamic view.

00:31:36.880 But then we have two problems: we have two codebases operating in two different contexts, both concerning how to render our state. Some code runs on the server, and some on the client. This creates a web of complexity; the further down the rabbit hole you go, the more complex it becomes.

00:32:00.860 I think this is why people began building single-page applications (SPAs). They seem conceptually simpler to understand. We started off building server-side applications where we only managed state on the server.

00:32:34.810 Now we’ve got this hybrid approach, and we have to manage two codebases that both take care of state. So, the logical response is to move everything to the client and handle it there. On the surface, that seems like a good idea, and for some applications, it might be.

00:33:15.330 But I genuinely believe that single-page apps break the web. That’s a bold statement, but I really believe it. If you look back at the history of the web, it started off as server-driven.

00:34:00.960 It comprised a series of hyperlinked documents provided by a server, and you received exactly what you wanted back in response to your request. With a single-page app model, however, if I request a blog post, what I get back is an empty skeleton of an HTML file with JavaScript included.

00:34:56.360 Hopefully, I can load the JavaScript—the CDN is not down, or something—and if the JavaScript can be fetched and executed, it makes another round trip to the server to get the data through our API, and then finally, we can start rendering. This is a departure from how we’ve built applications for decades.

00:35:40.360 This isn’t how the web works! Every successful technology built on the web builds on what came before, and it feels genuinely shameful to throw out the server-side applications we’ve been creating for so long.

00:36:08.030 So in Pakyow, we implement real-time as an enhancement. Thinking back to our example application—the one we first demoed in the browser—initially, it looked like just a static view.

00:36:29.170 That’s all it was—just like any other back-end driven application, like a Rails app or a PHP app. It was a static thing that didn’t change in front of you. And what we did was go into the back-end code and tell the framework that we wanted it to be subscribed to state changes.

00:37:15.290 That’s all we did, and the framework added that capability into our application, but it did so as an enhancement—a layer on top. The application continues to work even if real-time doesn’t.

00:37:38.360 Let’s say, for example, someone visits the application without JavaScript. No problem! The first response is a fully rendered HTML page, and we can present it. That’s not too likely nowadays—no one disables JavaScript anymore—but what if your CDN is down and we can’t fetch your JavaScript? That happens often; maybe it’s a common problem.

00:38:06.740 Regardless, we still have a rendered view we can present in the browser. I think this reflects the true spirit of the web: real-time as an enhancement on top of how we’ve been building applications for decades.

00:38:51.640 Lastly, I love Ruby, and I don’t mind JavaScript. But given the choice, you know which one I’m going to pick.

00:39:00.740 I love the language, and that’s what drew me to Rails back in 2005 when I first started hearing about this phenomenal framework. I thought, 'Oh my goodness, look how we can build applications! This is awesome!'

00:39:51.900 I had been stuck in PHP for so long, and I had lost my soul. Rails has a lot to do with my rejuvenation; it's a fantastic tool that allows us to build things in a much simpler way with good patterns and processes.

00:40:23.790 But at the heart of Rails is Ruby, and that’s what I want to build my applications on. I believe, as developers, we need options for how we build things.

00:41:02.660 I don’t like this migration towards the client, and even on the server, having to write JavaScript to create our applications. That just seems like a shame. So, where’s this project going?

00:41:35.590 This real-time functionality has been out for about six months. It took about two years of work to figure out how to make it work, but it works. There are production applications—some of them we've built, and some of them we have not—that use UI and Pakyow real-time to provide these dynamic views for people in the browser.

00:42:09.290 We feel like we’ve solved the hard problems; we've got all of that figured out, and now we need to improve the framework because if you’ve ever tried to use Pakyow, the documentation is not great, and the APIs aren’t exactly perfect.

00:42:40.850 And those are the things we want to fix. We really have a vision where Pakyow makes it possible for people to build applications in a different way—building server-side applications that feel like modern client-side apps.

00:43:12.510 We want to bring that to people, but more than that, we want this to be the easiest way—easier than a client-side framework. We want someone to be able to pick this up, build these applications, and do so with much less effort.

00:43:34.960 That’s really the focus. We hope to have more updates later this year; we’re not really putting a date on it. We’d love your help! As I mentioned, there’s a small community around this, and we're passionate about this project.

00:43:56.930 Thank you! You kind of have to be passionate to work on this for five years. I love building apps this way. You can read more about Pakyow on our website; we have guides and documentation.

00:44:18.640 You can build a real-time chat app—not in five minutes but in about half an hour, promise! We walk you through how to do that and deploy to Heroku. We’re on GitHub; give us a star, and follow along with development.

00:44:45.830 There are open issues at pakyow/pakyow. I also wanted to mention that we’ve started doing something with our issues; we label some of them as starter issues.

00:45:03.150 We’re interested in pulling people from the community into some of these because Pakyow has such a unique approach to things. Many people don’t know where to start and lack the confidence to begin. So we went ahead and tagged some tasks that require only a couple of hours of context-buildup.

00:45:31.330 We promise you’ll be able to complete this work. If you want to find a place to jump in, look for a starter issue. You can follow Pakyow at @pakyow; you can follow me as well. I’d love to talk more about this—it's something I’m genuinely passionate about. Thank you very much!