Talks
mruby: a Packaging Story Filled with Freedom
Summarized using AI
mruby: a Packaging Story Filled with Freedom
by Terence LeeIn the talk titled "mruby: a Packaging Story Filled with Freedom," Terence Lee discusses the challenges of building and distributing command-line applications in Ruby and introduces a solution through the lightweight, embeddable Ruby implementation called mruby.
Key Points:
- Introduction to Command-Line Applications: Terence begins by explaining command-line applications and their importance in interacting with the terminal. He contextualizes his background and experiences related to Ruby and command-line tools.
- Distribution Challenges: A significant problem with deploying Ruby command-line tools is the necessity of having Ruby installed on the user's system. Terence shares how Heroku initially used Ruby gems for the Toolbelt but faced issues with Ruby version discrepancies among users.
- Transition to Bundled Ruby Runtime: To mitigate distribution problems, Heroku decided to bundle Ruby with their Toolbelt, but Terence notes the inefficiencies this created during packaging and sharing across different platforms. He elaborates on how the adoption of Go for building CLI tools emerged as a solution to ease distribution and runtime concerns.
- Introduction to mruby: In 2012, Matz developed mruby to address these issues, making Ruby usable in embedded environments. Terence introduces his project, MRuby CLI, aimed at creating self-contained binaries which can be easily distributed without requiring a separate Ruby installation.
- Performance Advantages: He highlights the performance improvements offered by mruby, noting its faster boot times and minimal runtime overhead compared to traditional MRI Ruby applications. Demonstrations include the stark contrast between boot times of "Hello, World!" programs using MRI and MRuby CLI, showcasing MRuby’s efficiency.
- Cross-Compilation and Ease of Use: Terence emphasizes that MRuby CLI allows for cross-compilation across major operating systems, simplifying the build process and making it straightforward for developers to create and share applications. He describes the setup process and emphasizes the lightweight nature of the binaries produced.
- Comparison with Existing Solutions: Additionally, he discusses existing alternatives like Traveling Ruby and other emerging programming languages that also offer easy distribution, acknowledging their benefits and drawbacks.
- Final Insights: Terence encourages the Ruby community to explore MRuby and its ecosystem while acknowledging it is still evolving. He invites developers to contribute and helps them navigate the shift in packaging paradigms to enhance their command-line tools.
In conclusion, Terence Lee’s talk illustrates the potential of mruby as a transformative tool that allows Ruby developers to overcome longstanding distribution challenges for command-line applications, emphasizing community engagement and continued innovation in the Ruby ecosystem.
00:00:24.730
I'm here to talk to you about MRuby and packaging applications specifically for command-line applications. For those of you who are not familiar, command-line applications are the commands you run on the terminal when you type things, like 'ls' or 'grep' or any of those tools. If you've ever run 'rails new', then you've used a CLI app.
00:00:30.019
My name is Terence, and my Twitter handle is 'hone02'. I'm known for wearing blue hats. I come from the wonderful city of Austin, Texas, which I like to think is the taco capital. I was very excited to find out that Denver is getting Torchy's Tacos, one of my favorite taco joints. I've been here since last Saturday, hanging out with people in Denver. I got to go tutoring on Tuesday, which was super cool.
00:00:57.140
Some people who've never been to Austin are quite skeptical, like, "Are these tacos really that good?" For those of you who have never been to Austin, I can assure you that they are pretty amazing. Definitely check them out when they open. One of the really awesome things that Torchy's Tacos does is have a taco of the month. Every month they try to put something together that’s innovative and different.
00:01:23.270
Sometimes it’s thematic; like in October, they have the Scarecrow, which is this pumpkin-crusted chicken. This month in September, they have the Tipsy Chicken, which is super colorful, and I really want one, but I’m in Denver, and it’s not open yet. They also have really awesome queso. If you love queso, Austin has great queso, and I’m excited to share that with all of you.
00:01:54.079
When it opens, I also run a conference called 'Keep Austin Weird', which is happening in October. Hopefully, I’ll see some of you there. I know Zach is coming; he’s one of the organizers. It’s going to be awesome to have tacos and eat barbecue. For those of you who didn’t come out to the Ruby Karaoke event last night, we had a really good time, so thank you for setting that up.
00:02:26.630
Here's a picture of my scholar mentee Mimi and some classmates singing 'Killing Me Softly' on stage. It was a super fun night. In addition to that, I’ve done karaoke all around the world, and here’s a picture of Charlie Nutter, Tom, and Evo, some heroes of mine from the Ruby community, who also partake. If you’ve never done karaoke with Charlie Nutter, it’s quite an experience; his go-to song is a Shakira song, "Hips Don’t Lie." He’s really good at it.
00:03:03.260
It would probably be uncool to show pictures of other people doing karaoke, so here’s a picture of me and PJ singing 'Total Eclipse of the Heart' at Eurocamp. It’s a really good time, and I would recommend it if you’ve never been. Finally, I work at Heroku and I manage the Ruby experience there. If you’ve ever deployed a Ruby app to the platform, you’re using code that I write and maintain, along with Richard Schneemann.
00:03:55.150
If you are having problems or issues with that experience, then that is my fault, and you should come talk to me about it after my talk. But I’m not here to talk about any of that. I’m here to talk about building command-line applications in Ruby. To start, you really have to look at what it’s like to build command-line applications using MRI itself.
00:04:08.599
Ruby has a lot of really great tools that make it easy to build applications. You have libraries like Thor, which is used in Rails for the Rails command. There’s also GLI and several other networking libraries from Net::HTTP to Excon, Typhoeus, etc. There’s a huge ecosystem, so whatever you need, most of it has probably already been built, allowing you to focus on your business logic and plug it all in.
00:04:54.290
To understand where I’m coming from, I want to tell a story about the Heroku Toolbelt. For those of you who aren’t familiar, the Toolbelt is the command that you run on your terminal to create an app, interact with Heroku, manage add-ons, and configure your application. Our first iteration of it was a Ruby gem, so you could run 'gem install heroku', which I’m sure a lot of you are familiar with from other gems like Bundler or Rails.
00:05:23.720
Back then, we were a Ruby-only company, built by Ruby developers for Ruby developers to run Rails applications. Using a Ruby gem made a lot of sense; it removed a lot of extra work because we could leverage the fact that Ruby gems had a distribution system in packaging. If you’ve never created a gem before, it’s one of the easiest systems out there to do that, especially with Bundler’s 'gem' command.
00:06:00.640
However, it didn’t come without its problems. If you’re using anything with Ruby gems, you need to have Ruby installed, right? You need Ruby to actually run the Ruby gems, install them, and execute the actual code when you run the command. The other problem was that as a product company distributing a binary, you have no idea what version of Ruby users are running locally on their machines.
00:06:54.210
Apple was notorious for a long time on older versions of OS X, up until I think around version 10.7 for having Ruby 1.8.7 as the system Ruby. This meant that at Heroku, we had to support Ruby 1.8.7 both in syntax and for support tickets, because that’s what people were running on their machines. It was a nightmare to manage all these versions of Ruby; if we ever broke compatibility, we would have to revert the patch and fix it.
00:07:27.580
This wasn't a great long-term solution, so we decided to build the Heroku Toolbelt and move beyond the gem model. The big difference here was that it allowed us to bundle a Ruby runtime with the distribution, ensuring the version of Ruby we were running. We could build a specific version of Ruby, which alleviated many of those past problems.
00:08:01.570
However, we quickly found that packaging this up and sharing it was not very fun. In the early days when we started this, we actually had a Mac Mini under someone’s desk whose job was to package up the DMG for the OS X client we were distributing. There was basically this huge README for how to package everything up for Windows with all sorts of special licensing notes, and every time we had to do a release, we would have to run through those steps.
00:08:42.540
While this was helpful for targeting a specific version of Ruby, it was not the best solution. As we moved from being a Ruby-only company to supporting other languages like JavaScript, Java, and PHP, we realized that we couldn’t expect people to have Ruby installed and set up on their machines.
00:09:15.820
An employee of Heroku, Lake Gentry, started working in his free time on a project called HK, which was an implementation of the entire Heroku Toolbelt. He didn’t implement all the features, but he began with a core set of functionalities. One of the main motivators for that project was speed, which is a common focus at our company.
00:09:59.440
At that time, when you ran the command to convert something, all it did was read the version string in memory and then perform some disk operations on your machine to check what versions of plugins you were running. This didn’t hit any network calls, yet it took about 1.8 seconds when benchmarked on his machine. The HK version of that command took only 16 milliseconds.
00:10:25.540
So, you’re talking about a two orders of magnitude difference in speed. For CLI applications, having that responsiveness when running commands is noticeable for the end user. Since then, we’ve done a lot to optimize and improve performance. One of the biggest problems is the loading time in Ruby, which is not known for its speed.
00:10:57.930
When loading files at runtime, Ruby has to check the load path and find the files. As you add more things to the load path, it gets slower and slower to check them. You can imagine that as you start building out a complex command-line application, like your Rails app, there will be more files to manage. So, the challenge is to find a balance and make trade-offs for speed and maintainability.
00:11:31.800
One of the great features of the Go programming language is that it can produce a statically linked binary. This means you would have a single file that you can package up and give to someone, and they can run it without any other dependencies to install. Unlike Ruby applications, like Rails, which require a whole directory of files to work correctly, a binary allows you to simply distribute one single file.
00:12:20.880
When I say statically linked, I mean that there are no other dependencies on that binary that you need to install to get it working. You can just give someone this single file and let them run it, and it just works. This could be done across all major operating systems: Linux, Mac, and Windows.
00:12:57.290
Having a native binary for each platform makes it much easier to distribute and share with our customers. Heroku wasn’t the only company that started going through this process. Tools like Vagrant, which is an operations tool for setting up images, started off as a Ruby gem but stopped doing that and also moved to package up a runtime.
00:13:22.350
Now, they’ve transitioned completely to a Go CLI. This highlights a similar trend where developers, as their user base expands beyond just Ruby users, find it increasingly painful to deal with the challenges of packaging Ruby applications.
00:14:03.080
There are options out there; one of the ones I found was Traveling Ruby. This allows you to package up your Ruby application. It doesn't compile it down to a single binary, but you can package it up as a single tarball or package. They provide a precompiled Ruby binary, meaning you don’t have to do that work yourself depending on the operating system.
00:14:52.560
This means you could create platform-specific packages for Windows, Linux, or macOS. They also have a collection of native gems, like Nokogiri, which is commonly used. However, the downside to this approach is that it is not a single binary. It still unpacks a Ruby application and you’re limited to the gems that they precompiled unless you want to do that work yourself.
00:15:43.190
This can be a solid solution if you don’t want to rewrite things and can work with the existing setup, but moment you need to extend beyond those specific versions, it becomes a lot harder. While I was in Portugal last year, I chatted with Eric, who is now working on building alternatives in Crystal to handle this concern.
00:16:08.090
Besides Go, there are other languages that are emerging in this space which allow you to create compiled binaries that are easy to distribute. After looking at this challenge from a distance for a few years, I came to the conclusion that there wasn’t a great packaging story inside of MRI itself.
00:16:32.760
This was a little disappointing because I have seen various Ruby projects migrate away to other languages due to the challenges of packaging. We’re at a Ruby conference, and I’m sure the reason you’re here is that, like me, you enjoy building things in Ruby. There is something genuinely special about belonging to this community.
00:17:18.800
I have been involved in the Ruby community for about six years now, and there's so much I gained from it. I want to continue building the tools I need to get my job done without experiencing roadblocks that prevent me from doing what I love.
00:17:54.580
I set out to try to make packaging possible while still using Ruby, and together with Zachary Scott, we started a project called MRuby CLI. You can find it on GitHub under 'hone/mruby-cli'. We had a few design goals in mind, where ultimately you should be writing things in Ruby for the most part.
00:18:43.830
Here's the directory structure of the MRuby CLI app, which itself is an MRuby application. As you can see, it mainly contains Ruby files; approximately ninety percent of the code is written in Ruby. This means you’ll be working with familiar syntax and structures like you do in your everyday coding.
00:19:11.400
One of our key features is performance. As I found while working at Heroku, having a CLI that takes a long time to start up is significant. Runtime performance is essential, however, for CLI applications, boot time and startup performance become even more crucial.
00:19:47.700
To demonstrate this, I often take 'Hello, World!' applications as examples of assessing performance. This is applicable because it showcases boot time and how fast it is to initiate and exit an application. Many of us have written a simple 'Hello, World!' program at some point. On my old laptop, running it on MRI took about 41 milliseconds, which is totally bearable.
00:20:22.740
However, when we implemented this with MRuby CLI, we got it down to just 3 milliseconds. The base boot-up time is an order of magnitude faster, which gives us a lot more headroom during runtime. A major contributing factor to this is that in MRuby, your Ruby files are compiled into clean bytecode.
00:21:07.640
Instead of loading files at runtime, all your Ruby files are available in memory, allowing you to access them instantly without the delays associated with loading during execution.
00:21:35.840
Another significant aspect of MRuby is the ease of distribution. Unlike Rails applications that rely on advanced Capistrano scripts for deployment, when installing something on a user’s machine, I want a single file they can take and run without requiring extra setup. One of our goals was to allow developers to cross-compile for major operating systems easily.
00:22:25.480
So out of the box for the core library, we've been able to cross-compile for 64-bit Linux, Apple's 64-bit, and 32-bit versions of Windows. To give you context, when we checked the file size on OS X, the binary was just 421 KB. That's pretty reasonable when you're including a whole runtime and VM.
00:23:05.080
To facilitate this easy distribution process, we utilize Docker to streamline our development environment. Docker allows us to set up a fully contained environment with all the tools required to compile and run MRuby CLI. For those not familiar, Docker enables us to ship consistent development environments.
00:23:56.570
The workflow for generating a basic 'Hello, World!' application using MRuby CLI is simple. Once you have MRuby CLI set up in your path, you run a command to create an application structure, similar to how you’d do it in Rails. After that, you can generate the compiled binaries and run the application, displaying 'Hello, World!' on the screen.
00:24:38.190
Before we proceed to the generated files, let’s take a step back to look at MRuby itself, the technology we're building upon. MRuby was designed to be a lightweight version of Ruby, aimed at being embedded in other programming languages and systems, such as Arduino and other hardware devices.
00:25:23.190
It's lightweight and efficient, making it suitable for smaller devices and systems. This helps improve boot times due simply to its design philosophy. Competing languages like Lua are often used for game scripting and other lightweight applications.
00:26:04.150
An interesting use case is that MRuby is being utilized within the OpenWRT firmware for routers, allowing for the development of plugins and extensions. MRuby was created by Matz, who has spoken about it at RubyConf, emphasizing its potential as a viable, lightweight Ruby implementation.
00:27:24.260
So, what are the key differences between MRuby and MRI? Since MRuby is built to be run on various architectures, it doesn't have built-in operating system libraries. This lightweight design means it lacks file socket I/O, threading, and other OS-specific features.
00:28:01.120
The syntax of MRuby is a subset of Ruby 2.1, but you won’t find some features, such as keyword arguments. However, MRuby retains the core functionalities of Ruby, including blocks, metaprogramming, and literals.
00:28:44.940
To execute MRuby code, you can write a standard Ruby script, just as you do with MRI. You can run the MRuby interpreter on a .rb file and execute it directly. It’s great for dynamic scripting or creating plugins.
00:29:30.340
There is also a bytecode compiler built into MRuby that allows you to take any .rb file intended for MRuby, compile it into bytecode, and run it with a specific flag. This reduces the amount of parsing and processing MRuby has to do at runtime, allowing for much faster performance.
00:30:20.700
Additionally, you can embed the bytecode within C files, generating a clean execution environment. MRuby CLI manages this for you by generating wrapper C code that handles the execution context for your Ruby code and provides an interface for you.
00:30:58.060
One requirement is having a function named `main` where you handle command line arguments. All your Ruby code will exist in the 'lib' folder. The build system for MRuby is also designed to be straightforward, utilizing Rake and MRI itself to perform builds.
00:31:39.900
We offer both debug and stripped versions of your executables to optimize the performance in production. Additionally, MRuby’s structure allows automatic cross-compilation for each supported platform, further streamlining the build process.
00:32:30.840
The build process generates a summary for every generated binary and includes the necessary metadata for your MRuby gem. You can specify dependencies similar to Ruby gems, but the process is simplified, involving fewer configurations.
00:33:05.550
You will have an 'MTest' implementation, designed to facilitate unit testing directly within MRuby. This mirrors Ruby’s testing libraries and supports integration tests to allow comprehensive testing practices for your applications.
00:34:08.140
However, there are challenges; for example, there is no standard library in MRuby like there is in MRI. If you need certain functionalities, you may have to find or create MRuby gems as many pre-built libraries from MRI won’t work.
00:34:50.270
Translating an existing Ruby gem to MRuby can be non-trivial, as many Ruby gems depend on the standard library. Nevertheless, this allows us to avoid the baggage associated with MRI and move towards a lightweight ecosystem.
00:35:41.060
That said, there's this exciting project called JRuby Launcher, which was designed to set up and boot JRuby applications. While they are transitioning their option-parsing features to use Ruby, such compatibility suggestions and ideas are welcomed.
00:36:27.020
Ultimately, MRuby is still an evolving project compared to Ruby itself. Ruby has recently celebrated its 20th birthday, while MRuby is around six years old. Matz is actively involved and responsive to contributions made via GitHub.
00:37:18.560
This is a good moment to dive into the MRuby ecosystem. If you check the GitHub repository under the releases tab, you’ll find binaries for different operating systems. You can download it and place it anywhere on your file system.
00:37:54.700
As long as it’s in your path, you can run it. I encourage you to give it a try, and get your hands dirty with it! Build a simple application to see how easily you can get started; it’s quite straightforward.
00:38:12.300
So let’s go build some things in Ruby! I’m excited to bring people back into the community who might have thought of leaving, and I'd love to see more people involved.
00:38:35.700
Thank you! Also, here’s my blue hat sticker. If you’d like one, I have some with me, so come talk to me and say hi!
00:39:01.420
I don't think we have time.
Explore all talks recorded at Rocky Mountain Ruby 2015
+5