Building CLI Apps for Everyone

Talks

Terence Lee

@hone

Zachary Scott

@hcaz

#command-line-interface-cli

#mruby

#ruby

#cross-platform-development

#packaging

#rubygems

#heroku

Building CLI Apps for Everyone

by Terence Lee and Zachary Scott

The video titled 'Building CLI Apps for Everyone' features Terence Lee and Zachary Scott discussing the challenges of creating command-line applications that are cross-platform and accessible to all users while utilizing Ruby. They introduce mruby as a solution to overcome packaging issues faced with the standard Ruby implementation (MRI). The presentation is structured into several key points:

Introduction to Ruby CLI Applications: The speakers highlight the importance of command-line tools and the challenges faced when they are used across different environments and operating systems.
Historical Context of Ruby Packaging: They discuss the evolution of Ruby packaging, referencing the Heroku tool belt's transition from a Ruby gem to a statically compiled package to solve distribution issues.
Challenges in Current Solutions: Various approaches like removing Ruby dependencies in favor of Go for improved performance and the difficulties faced by projects reliant on native bindings exemplify hurdles encountered by Ruby applications.
Advantages of mruby: Mruby is introduced as a smaller, embeddable version of Ruby ideal for building lightweight applications. It retains core Ruby functionalities while allowing for the creation of self-contained binaries, essential for cross-platform compatibility.
Building Applications with mbCLI: The mbCLI project, which allows developers to create Ruby-based command-line applications, is presented. The workflow involves using the mruby build system alongside tools like Docker to streamline the compilation process for various platforms.
Testing and Deployment: The importance of testing and deployment is emphasized, where the setup includes test frameworks and methods for publishing applications on platforms like GitHub.
Future Aspirations: The speakers express intentions to enhance their build system and to create a collection of gems tailored for building CLI applications using mruby.

In conclusion, the presentation advocates for the wider adoption of mruby to simplify CLI application development for Ruby developers, urging the community to explore and contribute to the mbCLI project, ultimately striving for an efficient cross-compiled ecosystem that can work seamlessly across major platforms.

00:00:04.600 Hello everyone, we're here to talk to you about building command-line applications for everyone.

00:00:10.000 I am not Yehuda, nor am I Godfree, but we are members of the Ruby community, and we're here to share our insights with you.

00:00:21.480 Today, we will discuss how to build command-line applications that work for everyone, while using Ruby.

00:00:58.519 Let's dive right into it.

00:01:08.159 Hi, I’m Terence. I go by h02 on Twitter. I wear blue hats and work at Heroku managing the Ruby experience.

00:01:14.920 That means when you push Ruby apps to the Heroku platform, that’s what I work on.

00:01:20.840 I'm originally from Austin, Texas, where we have great barbecue and amazing tacos.

00:01:26.720 Additionally, I'm one of the co-founders of the Ruby karaoke movement, and we're actually organizing a Ruby karaoke event tonight.

00:01:33.880 If you want to sing and haven't signed up yet, please go to the RubyKaigi events page to find the link and sign up.

00:01:45.680 I'm excited to see you all there! Today, we'll be discussing building command-line applications in Ruby.

00:01:52.079 This talk will be divided into three parts: firstly, we'll cover the history of packaging in Ruby, and why mruby is a fitting solution for these challenges.

00:02:06.000 Then, we'll discuss the work Z and I have done to create an excellent experience using mruby for building CLI apps.

00:02:21.760 Let's start by examining the packaging landscape in Ruby.

00:02:27.440 The Heroku tool belt provides an illustrative example to discuss this topic.

00:02:32.640 Initially, when we began six or seven years ago, Heroku was a Ruby gem, primarily serving Ruby customers.

00:02:41.159 This made it a good fit since it offered a native experience that Ruby users were familiar with, but it also meant we didn’t have to tackle packaging or distribution issues.

00:02:54.120 However, as we aimed to attract other types of users, such as Python developers, it became problematic.

00:03:06.480 Forcing others to install Ruby just to use Heroku was inconvenient.

00:03:12.319 Even for existing Ruby users, it created significant maintenance challenges, as we could not guarantee which Ruby versions our users had.

00:03:24.799 Consequently, we were often limited to supporting older versions of Ruby, which hindered our ability to introduce new features.

00:03:36.760 To solve this issue, we introduced the Heroku tool belt, bundling the Ruby gem with a statically compiled Ruby runtime.

00:03:47.480 This allowed us to deliver a package that could be built for all target platforms.

00:03:55.400 However, the process came with its challenges, requiring specialized machines for compiling different versions.

00:04:01.760 We had to use a Mac Mini for OSX, a separate Windows machine, and releasing became significantly harder.

00:04:08.599 To address these packaging problems, we explored other technologies.

00:04:19.759 One of my coworkers started rewriting the entire experience using Go, citing performance reasons as a significant factor.

00:04:37.120 For example, running a command to display the Heroku version took approximately 1.8 seconds in C Ruby, while it took only 16 milliseconds in Go.

00:04:48.240 So, the performance gains from switching technologies were substantial.

00:04:55.760 Moreover, Go provided a cross-compiling toolchain that eliminated our need for numerous physical machines.

00:05:01.080 This meant we could generate a single binary to distribute to all customers, a much more efficient solution.

00:05:07.560 This trend isn't unique to Heroku; companies like HashiCorp also transitioned many tools written in Ruby, like Vagrant.

00:05:12.800 While Vagrant is still written in Ruby, other tools they develop are being re-written in Go.

00:05:20.000 Another significant project was Shoes, designed to build desktop applications across multiple operating systems using native bindings.

00:05:34.440 This project, however, was challenging to maintain, leading the team to rewrite it in JRuby.

00:05:41.520 The maintainers noted that packaging issues were primarily what made the project's C code difficult to manage.

00:05:50.080 More recently, the Traveling Ruby project emerged, which was built by the creators of Phusion Passenger.

00:06:06.840 They aimed to package a precompiled Ruby runtime along with the application code and third-party extensions.

00:06:15.800 However, the issue with this approach was that third-party extensions had to be precompiled for each platform.

00:06:25.720 Moreover, when security fixes were necessary, the maintenance burden was significant.

00:06:32.760 This led to many solutions falling short.

00:06:39.920 Our goal is to continue creating Ruby applications without letting the technical limitations of packaging in MRI hold us back.

00:06:50.080 Today, we’ll discuss how mruby can help us solve these problems and why it’s a viable option.

00:07:02.080 The advantage of mruby is that it’s a smaller, embeddable Ruby that has been designed with a low memory footprint in mind.

00:07:09.120 This means we do not have operating system-level features like file or socket functionalities, or thread support.

00:07:17.639 Still, we can utilize the core of Ruby for building command-line applications.

00:07:24.639 You can still use procs, blocks, and other Ruby features to create versatile applications.

00:07:30.840 Even though mruby might not have as rich a standard library as CRuby, it still retains the essence of Ruby.

00:07:39.840 Interestingly, some projects have already employed mruby in production environments.

00:07:51.120 One example is the project Engine X, which enables extending it with Ruby.

00:08:00.320 We've recently implemented an Ember project called Dashboard, which runs on top of Engine X using mruby.

00:08:11.760 This is part of our largest web property, and it's currently running Ruby in a production environment.

00:08:24.160 Another area where mruby is gaining traction is in IoT devices, where resource constraints demand lightweight solutions.

00:08:36.480 In one case, a project called OpenWrt allows embedding mruby in routers for Internet access.

00:08:46.639 Users can set up configurations using a custom interface that determines which binaries are compiled.

00:08:59.280 All of this is achievable through using the mruby gem's build system.

00:09:04.760 Even though the core of mruby is small, the mrb gem facilitates access to numerous libraries.

00:09:18.960 The MB gem specification file is fundamental for any mruby application.

00:09:26.160 You can define dependencies much like you do with Ruby's gem specification.

00:09:35.479 There are essentially three sources to pull from for mrb gems.

00:09:50.520 The first source is the core mrb gems folder, which includes essential libraries.

00:09:58.200 Secondly, you can use the mgem source, which acts like a repository for publicly published mrb gems.

00:10:06.239 Lastly, you can fork projects or use GitHub directly to pull in any dependencies, even if they aren't published.

00:10:19.840 For example, we forked the mruby-o project while waiting for a Pull Request to be merged.

00:10:27.560 This allowed us to statically compile libraries that were critical to our project.

00:10:39.920 Today, we will delve into what’s behind the construction of these applications.

00:10:53.840 During this conference, we heard a talk by K SAS about adding a bytecode ahead-of-time compiler to MRI.

00:11:08.960 However, with mruby, we've already implemented something similar.

00:11:24.960 Mruby allows us to compile Ruby code into a binary format suitable for embedding.

00:11:39.360 Using the mrbc tool comes in handy to convert Ruby files into binary, enhancing performance.

00:11:50.960 This binary format can then be interpreted by the mruby interpreter.

00:12:03.240 You can embed this compiled binary code into a C program, which allows for a seamless integration.

00:12:16.560 After compiling mruby, you receive a build summary encapsulating all included gems.

00:12:23.360 However, there are limitations in this approach, especially concerning the embedding design.

00:12:34.480 Since mruby aims to remain small, various existing Ruby gems may require porting to function correctly.

00:12:46.320 We faced challenges with libraries such as op-parse that were not readily compatible.

00:12:56.000 Additionally, some frameworks did not work smoothly, leading to slower iterations during development.

00:13:05.440 Cross compiling for multiple targets also necessitates configuring the native extensions appropriately.

00:13:17.080 Recently, we added two flags to mruby’s build system to enable cross-compilation targeting.

00:13:27.639 This result was necessary to allow regular expression extensions to work in our Windows builds.

00:13:38.320 We navigated these configuration hurdles to ensure better user experiences when building applications.

00:13:47.880 That led us to create the mbCLI project, which Terence will discuss more in-depth.

00:14:03.200 One of our main design goals in the mbCLI project was to enable developers to write primarily in Ruby.

00:14:16.480 The mbCLI application itself is structured like any Ruby application, organized with multiple Ruby files.

00:14:31.520 Furthermore, just like the Heroku experience showed us, performance is indeed crucial for command-line applications.

00:14:42.680 Users expect quick interactions, so startup time must be optimized.

00:14:57.480 For instance, the simple hello world example takes approximately 41 milliseconds to run in MRI.

00:15:07.920 However, using the mbCLI we can drop that runtime to just 3 milliseconds, illustrating a significant performance difference.

00:15:18.239 Another major asset provided by the mruby ecosystem is the ability to create a single binary that runs across all major platforms.

00:15:28.960 In our case, compiling solutions for Linux, OSX, and Windows was essential.

00:15:41.040 We also aimed to keep file sizes manageable; for instance, the OSX mbCLI binary is only 421 KB.

00:15:52.679 Our goal is to build a solid platform for the Ruby ecosystem that developers can leverage to easily create command-line applications.

00:16:01.440 For adoption to succeed, we want to simplify the setup process, which is where Zach will take over and demonstrate creating a hello world application.

00:16:13.680 To get started with mbCLI, the first step is visiting the project page on GitHub to access the releases.

00:16:31.520 You will find various packages available for each architecture needed to run mbCLI.

00:16:40.640 As an example, while using a Mac, I downloaded the x86_64 Apple Darwin package, extracted it, and added the binary to my system path.

00:16:55.360 From there, I was able to output the version of mbCLI, confirming that it works as expected.

00:17:06.120 Additionally, the command can print help information, appearing just like any other command-line application.

00:17:20.480 We also have a setup command to generate applications for mbCLI.

00:17:34.240 All you need to do is pass the name of your application, much like using the 'rails new' command.

00:17:46.320 Once that’s done, a list of files gets created, and we will review each file's purpose.

00:18:02.240 Among those files, the build configuration file orchestrates how the mruby compiler functions.

00:18:14.480 We have various target configurations defined for compiling on native Linux hosts, OSX, and Windows.

00:18:26.080 Specifically, each target tells Ruby itself how to compile the application.

00:18:41.120 Included in this file is a gem configuration method, specifying which gems belong to your application.

00:18:56.200 In the gem specification for your application, you'll define dependencies required to run your code.

00:19:10.720 Most critical is the 'bins' option, which directs mruby to generate a binary from your application.

00:19:17.120 Alongside this, a bootstrapper for your command-line app is generated, which is just a C file.

00:19:32.520 Although you may not need to read the bootstrapper, it sets up the environment for command-line arguments.

00:19:45.840 Additionally, we generate a skeleton for the main method that needs to be defined globally.

00:20:00.200 The testing structure includes two types of tests: bin tests and unit tests.

00:20:12.720 The bin tests provide an integration testing interface, while unit tests evaluate the implementation of your code.

00:20:27.520 To run your tests, you also get a rake file, which provides a range of useful tasks for compiling and running your tests.

00:20:41.120 This rake file works best when executed inside a Docker container designed to manage the build process.

00:20:56.520 The workflow utilizes Docker to streamline support and maintenance as we build for multiple platforms.

00:21:08.560 Essentially, with the Dockerfile and docker-compose.yml, we create an environment to run all necessary commands.

00:21:24.320 By using these tools, you can compile your projects with the necessary dependencies and cross-compilation support.

00:21:40.080 The first command to execute after setting everything up is 'docker-compose run compile'.

00:21:53.600 This command builds the application while mounting your local volume and creating binaries for all targeted platforms.

00:22:06.480 Once everything compiles successfully, a summary is printed, outlining the binaries and included gems.

00:22:19.320 When you're ready to use the binaries, you can locate them in the build directory.

00:22:31.920 You can find corresponding entries for your native operating system that can be executed.

00:22:45.440 After everything has properly compiled, you can initiate tests with 'docker-compose run tests', which runs both unit and integration tests.

00:22:59.200 This enables you to iterate and refine your development before releasing.

00:23:08.960 When you feel satisfied with your CLI application, you can publish the files in the M build directory as needed.

00:23:21.440 For example, for mbCLI, we typically publish it on the GitHub releases page.

00:23:42.000 We're excited about our journey so far; there’s a lot of work ahead to improve the process further.

00:23:51.520 One project I’m eager to develop is enhancing our build system to create a more flexible and maintainable environment.

00:24:04.320 Another goal involves curating a collection of gems specifically tailored for building command-line applications in mruby.

00:24:21.320 We want to draw inspiration from Rails, which did a great job at offering reliable platforms and curated gems.

00:24:36.160 We plan to start by focusing on commonly needed features, like option parsing.

00:24:52.200 Our aim is to create a cross-compiled ecosystem that works seamlessly across all major platforms.

00:25:07.760 We welcome everyone to participate! Please install mbCLI and explore it; get Docker configured if you haven't.

00:25:23.000 We encourage you to create your own applications and share them with us.

00:25:32.880 Please don’t hesitate to tweet at us; we’re excited to see what you build in Ruby.