Web API's with ERLANG a Ruby Dev's POV

Talks

Patrick Huesler

1 talk

#functional-programming

#ruby

Web API's with ERLANG a Ruby Dev's POV

by Patrick Huesler

In the talk titled 'Web APIs with Erlang: A Ruby Developer's Point of View,' Patrick Huesler discusses the transition from Ruby to Erlang, focusing on how to develop web services using Erlang and integrate them with Ruby tools. Drawing parallels between learning Erlang and his experience learning to surf, Huesler emphasizes that understanding a new programming language involves more than just syntax.

Key Points Discussed:
- Erlang Overview: Erlang is a functional language based on mathematical functions, compiled to bytecode (BEAM files) executed on a virtual machine, with a focus on concurrency and fault tolerance.
- Concurrency Management: Erlang’s actor model and message passing remove the issues of data corruption associated with shared data and simplify the management of CPU and I/O concurrency.
- Live Coding and Hot Code Swapping: Erlang allows developers to make changes in production without shutting down the system, highlighting its robustness in real-world applications.
- User Session Management: At MOG, the company Huesler works for, they tackled performance issues by managing user session data in Erlang instead of relying heavily on traditional databases, allowing for improved efficiency during high traffic.
- Development Tools: Huesler discusses tools like Rebar for application setup and dependencies management, and illustrates building a simple web server while drawing comparisons to known Ruby frameworks.
- Testing in Erlang: He mentions the standard testing framework, 'Unit', and integration testing with HTTP requests, alongside the challenges faced with stack traces and error management in dependencies.
- Deployment on Heroku: He explains how Heroku simplifies deploying Erlang applications, allowing developers to focus on coding without worrying about infrastructure complications.

Conclusions and Takeaways:
- Huesler encourages Ruby developers to explore Erlang, highlighting that it provides valuable lessons in concurrency patterns and design principles.
- Erlang’s architecture reinforces the development of reliable, fault-tolerant distributed systems, making it a powerful tool in backend development.

This talk from Aloha RubyConf 2012 serves as an informative introduction to Erlang for Ruby developers, focusing on practical integration and the value of understanding different programming paradigms.

00:00:14.710 Good morning everyone, and welcome to my talk titled 'More Than Syntax.' My name is Patrick Huesler.

00:00:24.320 You can find me on Twitter as @fueler. I've been told by a trainer that you should not give out your name on stage, but I think that's just rude. So I decided to introduce myself.

00:00:37.399 Originally, 'More Than Syntax' was the title of my talk, but on the way here, I changed my mind. Like a good startup, I pivoted my talk.

00:00:49.280 I changed the title to 'Web APIs with Erlang: A Ruby Developer's Point of View.' In this talk, I will discuss how you can write web services in Erlang and how to integrate your Ruby tools, automations, and deployments within your environment.

00:01:08.780 Over the last three months, my focus has been on achieving this integration.

00:01:21.170 Also, on a very important note from yesterday, Terence Lee is not a liar. He mentioned that he is looking forward to live coding in my talk, but that's not going to happen.

00:01:32.570 The reason is that I tend to struggle with coding while framing coherent sentences, and I would prefer to spare you all the embarrassment that could lead to.

00:01:45.469 So there isn't a liar; in fact, Terence is awesome, especially with his work on the Rails group. My topic is cool for Hawaii, I guess.

00:01:52.729 I grew up in Switzerland, in the mountains. I grew up around snow, so snowboarding, skiing, and ice hockey— you name it, I did it. There was even one incident where I went to kindergarten with my skis on.

00:02:04.440 I'm fairly proficient in snowboarding and understand how it works. This brings me to the topic of surfing. I have this slide so I can wear shorts on stage as a prop without feeling embarrassed.

00:02:19.209 Additionally, I wanted to learn surfing. I thought it would be easy, given my experience snowboarding.

00:02:31.110 I figured it would be a piece of cake, so I read a book that detailed every aspect of the sport. I thought, 'Well, that's great, I can do this!'

00:02:51.250 However, one day I went out surfing, wearing my board shorts. In Europe, most of the time you need a wetsuit because the water is too cold.

00:03:07.690 So, I did the surf check, looking for waves, currents, and where the lineup was, then I attempted to paddle out.

00:03:14.829 I fell off the board immediately. I didn't know how wobbly those boards could be, so I struggled to paddle.

00:03:26.290 The next day, at the surf spot, I tried again to get out, and I managed to work around my difficulties.

00:03:32.320 I planned to catch my first wave and felt confident that it would be as easy as snowboarding.

00:03:44.410 But this did not happen, and I ended up falling repeatedly for the whole afternoon.

00:03:51.450 The lesson here is that sometimes, just reading a book and banking on your previous knowledge doesn't translate well to a new skill.

00:03:57.720 This brings me to Erlang, a similar story to my experience with Ruby.

00:04:10.340 Just a quick check: who here has looked into Erlang?

00:04:19.230 And who here has written production code in Erlang?

00:04:28.520 So I consider myself an Erlang beginner; I've only been using it productively for a couple of months. I started learning it about four years ago, but never went beyond a simple chat server or 'Hello, World!' application.

00:04:42.180 This means that my perspective comes from that of a beginner, so there might be things in my talk that experienced Erlang developers might disagree with.

00:04:55.290 I will not cover an introduction to Erlang here, as others have done that much better.

00:05:02.700 My focus will be on the concepts and the tools I needed to combine my Rails and Ruby mindset with Erlang in a more original way, rather than trying to shoehorn Ruby practices into Erlang.

00:05:15.240 This will involve a lot of content and code, but I won't go into every single detail. I have everything uploaded to a GitHub repository and I deployed the application on Heroku.

00:05:27.510 To make sure we're on the same page, a quick Erlang refresher: Erlang is a functional language based on mathematical principles. Everything is a mathematical function, and it strives to avoid state and mutable data.

00:05:44.400 This approach is particularly useful for the kinds of things you build in Erlang.

00:05:50.130 Erlang is a compiled language that builds bytecode known as BEAM files and executes it via a virtual machine. This VM handles everything, including connections between nodes and clusters.

00:06:03.390 Additionally, Erlang OTP provides a set of common practices, patterns, and abstractions to aid in building robust applications in Erlang, like generic servers, state machines, and event handling.

00:06:19.020 Now, why Erlang? First and foremost, Erlang is highly concurrent—it was specifically built for that.

00:06:38.200 Managing CPU and I/O concurrency is straightforward in Erlang.

00:06:43.720 You don't need to worry about non-blocking I/O or event machines since they are inherently built into the language.

00:06:50.480 Erlang uses an actor model which employs message passing, ensuring that there is no shared data that could be corrupted. This avoids the issues of data corruption and the need for locks when accessing shared data.

00:07:08.200 You can see analogous behavior in the Ruby world with frameworks like Celluloid, which provides an actor-based approach, and it's particularly effective in JRuby and Rubinius.

00:07:21.250 Erlang is also designed for distributed systems, making networking almost trivial.

00:07:28.420 You can communicate with other machines and build service-oriented architectures seamlessly.

00:07:35.620 If you look at database systems like CouchDB or Riak, many are written in Erlang.

00:07:43.450 Another core attribute of Erlang is fault tolerance. Erlang systems are designed to run continuously and expect failure.

00:07:51.220 One of the main strategies is to let processes crash, allowing the infrastructure to restart them automatically when things go wrong.

00:08:04.870 This provides great strategies around what to do when something goes wrong and what gets restarted.

00:08:12.310 One of my favorite features is the ability to do live coding; this can be really beneficial when you're in a production environment.

00:08:27.220 For example, when you're running low on time and need to get things done, Erlang allows you to make changes without tearing everything down.

00:08:38.340 Erlang is a compiled language, yet it allows for hot code swapping.

00:08:43.570 This means that you don't have to shut down the entire virtual machine or system to apply updates.

00:08:52.080 It provides infrastructure for reloading code and modifying it on-the-fly.

00:09:00.150 You can attach remotely to every node in your Erlang cluster and operate similarly to an IRB within the context of your running virtual machine.

00:09:09.620 At MOG, the company I work for, we started using Erlang because we develop games on Facebook and iOS, such as Monster World.

00:09:29.360 A critical aspect of our game is sharing data among friends, where players can interact and help each other.

00:09:43.160 We face technical challenges with our backend due to high user volume—about 20 million users each month, with roughly 4 million users daily.

00:10:05.060 From our data, we observed peak traffic reaching 6,500 requests per second, which fluctuates based on user activity.

00:10:21.130 The real challenge lies in our database; it becomes a bottleneck, especially when handling write requests.

00:10:39.560 We've tried common solutions such as caching and adding read replicas, but these did not solve the writing issues.

00:10:54.300 Eventually, one of our engineers developed a new architecture where the user session information is held in memory instead of on the database.

00:11:06.469 The concept is to eliminate the database pain point altogether: when a player logs in, we create a worker to control their session, loading and saving the data from the cloud in the background.

00:11:29.500 The thought was if we lose a few minutes of data, it's usually not critical in games like this.

00:11:46.410 User sessions exist as separate processes in Erlang, allowing us to run many sessions efficiently on a single machine.

00:11:55.680 The Erlang VM manages these processes, and if one process encounters an error, it won't affect the others.

00:12:10.480 In Erlang, processes are lightweight, making it possible to manage a vast number of them concurrently on a single machine.

00:12:23.930 Next, I want to discuss environments and logging, similar to what you might be familiar with from Ruby, Rails, and Sinatra.

00:12:40.190 The first tool I often use for development is called Rebar, which is a blend of your Rails commands and RubyGems.

00:12:54.260 To start, you create a new application, which generates the necessary skeleton for your code.

00:13:06.190 Erlang has an inherent concept of applications, which allows you to fragment them into manageable parts or services.

00:13:16.990 Next, you will want to set up your dependencies.

00:13:30.350 For example, to access a MySQL database or utilize some specific functionality, you'll need to specify these dependencies.

00:13:44.400 Rebar automatically manages this for you and provides commands to fetch and compile the necessary dependencies.

00:13:55.230 The configuration file for your Erlang application might look something like this.

00:14:09.740 Although it may not be visually appealing, it represents a proper Erlang structure.

00:14:15.569 In this file, you specify the name of the application and its revision.

00:14:22.620 Then you can execute the command to fetch all your dependencies.

00:14:31.640 Next, we need a web server. There are several options in Erlang, including Cowboy, MochiWeb, and others.

00:14:45.540 At our company, we decided to build our own web server because we wanted more control over the request lifecycle.

00:15:02.480 We aimed for deeper insights into the requests to allow for better monitoring. This turned out to be critically important.

00:15:16.650 With minimal code, I even contributed to the creation of a web server in Erlang that is quite fast.

00:15:28.180 I think it's faster than everything I have encountered in Ruby so far.

00:15:40.260 I won’t delve into the details of modules and functions since I'm sure most of you are already familiar with them.

00:15:53.200 In Erlang, you define a callback module that acts like a base class in C, which implements specific functions that get called when requests come in.

00:16:05.410 As you can imagine, having multiple endpoints is vital, so I built a router that behaves somewhat like the routing in Rails.

00:16:20.720 This allows you to define paths and HTTP methods, associating them with specific handlers.

00:16:36.060 The design allows for vivid expression of routes, making it easy to understand request processing.

00:16:46.570 It took me a morning to write this and I felt proud of my accomplishment, though I soon learned from seasoned Erlang developers that I had done it wrong.

00:17:01.470 They advised me to use pattern matching, which is a powerful concept in Erlang.

00:17:16.050 Does anyone here have familiarity with pattern matching? This technique allows you to extract parts of data efficiently.

00:17:26.650 For instance, when defining a function, you can specify expected parameters using a pattern.

00:17:35.450 If Erlang cannot match the given pattern, it will search through the available function definitions to find a suitable match.

00:17:49.560 This gives you flexibility and allows you to write cleaner, more expressive code.

00:18:01.040 Now, let's talk about middleware. Who here likes Rack middleware? I am a big fan of abstraction in that context.

00:18:17.260 It allows you to separate reusable functionalities, such as logging or monitoring, into distinct components.

00:18:32.170 We implemented middleware in our Erlang applications and did so in a way that integrates neatly with the events that happen within the application.

00:18:46.490 There’s a multitude of events you can listen to, which helps you gather metrics and detect issues efficiently.

00:19:04.750 Now let’s shift focus to environments. In any well-rounded application, you typically have different environments for testing, staging, and production.

00:19:16.859 It's crucial that each of these environments has its own configuration.

00:19:31.150 After exploring various solutions in the Erlang ecosystem, I found that passing in configuration parameters via command line was the simplest approach.

00:19:42.480 Erlang supports property list configurations, which are essentially tuples denoting key-value pairs representing your settings.

00:19:55.910 This setup allows you to define various configurations, including application-specific parameters.

00:20:10.340 You can easily adjust logging settings based on user needs, enabling you to gather valuable data about system performance.

00:20:25.560 An important aspect of logging is that it aids in operations by highlighting slow queries and other anomalies.

00:20:38.040 In Erlang, the standard logging approach uses a logger class, which manages log levels and provides comprehensive logging solutions.

00:20:54.080 However, in practice, this generates excessive log data—sometimes you only need concise one-liners.

00:21:06.230 Overall, the complexity of parsing these logs with standard tools can become an obstacle for developers.

00:21:20.850 To address this issue, we explored a tool written by Basho, which offers simpler logging solutions.

00:21:34.930 This solution allows for easier configuration and output that's more manageable to read and parse.

00:21:51.980 Using Logger means you can adjust output to ensure compatibility with tools like syslog, making distributed logging straightforward.

00:22:08.320 A major annoyance in Erlang is managing library dependencies when trying to run applications.

00:22:24.140 Often, errors arise when a sub-dependency cannot be started, which requires a workaround to diagnose.

00:22:39.860 Elements like missing dependencies can create confusion, especially when managing external libraries.

00:22:53.210 While we’re on the subject of dependencies, how is the Erlang code rate? Is it appropriate for everyone?

00:23:06.900 Do keep up the feedback while we proceed with the rest of the content.

00:23:17.570 As we discuss tools for testing, I want to introduce 'Unit,' which is the standard testing framework used in Erlang.

00:23:31.500 I found it effective for basic testing needs, although I ran into an issue with stack traces not being available until more recent releases.

00:23:45.240 With the introduction of version 15, stack traces became accessible, which greatly improved debugging.

00:24:01.550 To create a functional test, you'll follow a typical setup routine similar to most testing frameworks. It's all straightforward.

00:24:17.850 You'll have options like certain assertions that can be called to validate your functionalities.

00:24:31.540 Then, for integration testing of our web services, we played with HTTP requests.

00:24:47.130 One of our team members introduced 'Dtest,' an extension focused on HTTP, which greatly simplifies the integration testing process.

00:25:02.370 With views crafted for the same requests as your application’s, it follows the intuitive structure set in unit tests.

00:25:15.640 Notably, you need to start your application before running HTTP requests against it to get valid responses.

00:25:30.780 Integration testing allows you to confirm your API’s responses without diving into the minutiae of internal components.

00:25:44.930 This framework also supports working with JSON out of the box, making it easier to handle common payloads.

00:25:59.290 Now let’s explore another powerful feature of Erlang: hot code reloading.

00:26:10.340 This feature allows you to save, compile, and start applications without needing to shut the system down.

00:26:23.889 You'll need to embed appropriate modules into your application structure to utilize this effectively.

00:26:41.680 While working with native extensions, however, ensure that you're cautious with code reloading as it does not always carry over seamlessly.

00:27:01.540 One improvement might involve syncing your external libraries to your reload implementation, ensuring stability.

00:27:16.360 Looking over to Heroku, it's an excellent platform I learned about recently.

00:27:32.200 Using Heroku is quite simple; it enables you to deploy applications seamlessly with minimal hassle.

00:27:48.510 Creating your application begins with the standard procedure— establishing a directory and linking it to Git.

00:28:02.480 With the help of buildpacks, you're no longer restricted to one language— you can create apps with various architectures.

00:28:16.970 Once your application is linked with the buildpacks, you're all set up.

00:28:31.120 The process of deploying your Erlang application is straightforward; you only need to provide certain commands.

00:28:48.350 Heroku compiles your project, manages dependencies, and brings everything online effortlessly.

00:29:02.400 You can simply curl your application and verify its functionality, thus confirming if everything was set up correctly.

00:29:15.720 In addition to launching, you have full access to monitor your app's performance.

00:29:28.000 Encouragingly, this is also the stage where you'll find all the codes I've been speaking about.

00:29:42.180 I plan to convert this project into more of an unveiling command, which would facilitate creating new web services.

00:29:58.680 I encourage each of you to venture into creating your first web service in Erlang, as there's much to learn from this powerful language.

00:30:15.040 By gaining insights into new paradigms and concurrency patterns, you can find fresh ways to enhance your Ruby skill set.

00:30:29.080 Overall, Erlang was created with a distinct purpose centered around building reliable, distributed systems that are resilient to failure.

00:30:41.950 Thank you for your attention.

Aloha RubyConf 2012