00:00:00.000
Hello, welcome to the talk on building maintainable command-line tools with mruby. I'm Eric Hodel.
00:00:06.690
In this presentation, I will discuss working with mruby and mruby-cli.
00:00:12.230
I am a Ruby committer, a RubyGems committer, and the author of many gems.
00:00:18.359
Currently, I’m on vacation from these responsibilities. Instead of working on Ruby in my free time, I've been spending a lot of time making shelves like these.
00:00:25.500
This particular one is still a work in progress, and I'm trying to fill in that little gap there.
00:00:31.740
I work for Fastly, a real-time content delivery network. This is our website, featuring Gordo, one of our amazing Fastly office dogs.
00:00:38.940
Fastly supports open source, including hosting the Ruby Lang website and all the Ruby gems, including their downloads.
00:00:45.860
We also have a Japanese Twitter account at Fastly Japan, so you can follow us there.
00:00:51.930
In part one, I will cover a bit of the background on what mruby is and what mruby-cli is useful for.
00:00:59.570
My exploration of mruby and mruby-cli started with a web API wrapper.
00:01:05.339
We were building a new product at Fastly that involved new APIs, and I wanted an easy way to interact with them.
00:01:12.150
This would allow me to demonstrate that the APIs are usable.
00:01:20.009
So, I thought, why not choose mruby to write the wrapper?
00:01:27.180
When we want to release something with CRuby, the user has to install the right version of CRuby.
00:01:32.970
This is necessary for them to use the specific features they want. In addition, they also have to install any gems that they depend on.
00:01:39.810
Then, they need to maintain those gems and inform others to update them as long as they are supporting the tool.
00:01:45.780
This process can be daunting and involves a lot of work for users.
00:01:52.380
mruby is very similar to CRuby, which allows us to start with a prototype in CRuby, port that code over to mruby, and then maintain it with less work for the users.
00:01:59.640
So, what makes mruby more suitable for this type of work?
00:02:05.729
mruby is a lightweight Ruby implementation, meaning you only need to use the parts of mruby that you require.
00:02:13.230
This helps keep your project smaller and more manageable.
00:02:21.390
It is also designed for embedding, which means you can run Ruby on low-resource hardware.
00:02:27.019
This includes devices such as Arduino or Lego Mindstorms, as well as robots and set-top boxes.
00:02:34.080
Moreover, you can embed it in other software, which is much easier than trying to embed CRuby.
00:02:41.879
Similarly, mruby has libraries linked to gems, but it ships with a limited standard library.
00:02:48.750
You can use gems to build your standard library to resemble that of CRuby.
00:02:56.250
There are also standalone mruby libraries, some of which are C extensions. For instance, there's a C wrapper for the curl tool called mruby-curl.
00:03:01.340
Given there have been many talks about mruby, I recommend checking them out if you're interested.
00:03:07.739
At this year’s conference, there were three other talks about mruby, including one tomorrow. You can find recordings of all these talks on the Ruby Kaigi website.
00:03:14.640
Since mruby is compiled, unlike CRuby, you might wonder how to get started with it.
00:03:20.540
The first step is to edit your build configuration.
00:03:25.970
Unlike CRuby, where you can load any gem at any time, with mruby, you must select which pieces to use in advance.
00:03:31.310
You also choose which mruby gems to use and specify which compiler to use, especially for cross-compiling.
00:03:36.930
mruby-cli utilizes this setup to build for multiple platforms, and you must indicate the library paths to link any C libraries.
00:03:44.069
In fact, you can compile many builds for various platforms in a single step.
00:03:49.319
If you are writing an mruby gem, you will fill out a mruby gem specification, which goes into the mrb file.
00:03:55.760
mruby CLI applications also utilize the mruby gem specification, allowing you to easily specify dependencies, executable names, and test files.
00:04:00.810
The mruby gem specification is where the metadata for your gem resides, including dependencies and binary specifications.
00:04:09.660
The metadata consists of the gem's name, version, license, and other relevant details.
00:04:17.010
So once you have these files configured, you run rake to build the binaries, executables, and libraries.
00:04:25.169
Upon completion, you will find all the executables in the bin directory.
00:04:31.740
With mruby-cli, your CLI team will have a built-in mruby executable, which you can use to write simple scripts or perform little tests.
00:04:37.830
There will also be an mruby test executable, which is the test runner for your unit tests.
00:04:45.150
The ultimate goal of mruby CLI is to allow Rubyists to build single-file executables in a familiar environment.
00:04:50.700
Last year, Terrence Lee and Zachary Scott gave a talk about mruby CLI, showcasing the basics of the tool and how it works.
00:04:56.820
I encourage you to check that out for a more extensive overview.
00:05:03.600
The basic description of mruby CLI is that you can write Ruby and release software on Linux, OS X, and Windows.
00:05:09.360
The typical development loop when using mruby CLI begins by writing some mruby code.
00:05:14.580
Then you compile the code you’ve written, run the tests, and revisit the coding process if your tests don't pass.
00:05:21.360
Once all your tests are passing, you can release your executables.
00:05:26.640
When I first learned about mruby and mruby CLI, I anticipated an easy transition.
00:05:34.400
However, I knew that some challenges might arise due to the differences in Ruby implementations.
00:05:39.420
To begin with, mruby is not equal to CRuby; the implementation doesn’t support everything you would do in CRuby.
00:05:46.110
The first issue I encountered was related to the modular standard library.
00:05:52.140
Core classes in Ruby, such as String, Array, and Hash, are very limited in mruby.
00:05:59.430
They only support the minimum number of methods you would need.
00:06:05.610
To get the full CRuby experience, you need to add extension gems, such as mruby-array-ext.
00:06:12.690
When you develop your own mruby gem, it can be easy to forget to add these dependencies.
00:06:18.210
You might be using the features without realizing the gem isn't included, leading to errors when you try to use it elsewhere.
00:06:24.330
Additionally, mruby has a smaller language feature set compared to CRuby.
00:06:30.510
Outside of the preloaded library, there are no predefined globals such as load path.
00:06:35.700
Everything is compiled, so none of the globals inherited from Perl or shell scripts exist.
00:06:40.950
Some libraries you may wish to port over to mruby might depend on these globals, which necessitates workarounds.
00:06:47.790
I also encountered difficulties with your document support when porting a library from CRuby.
00:06:54.120
While I haven't reproduced the bug yet to file a good report on it, there are limitations.
00:07:01.020
While keyword arguments can be used to call methods, you can't define methods that automatically check for them, unlike CRuby.
00:07:07.290
Additionally, some gsub replacements don't function in mruby, so you may need to adjust your regular expressions for them to work.
00:07:12.990
The backtraces in mruby have appeared less useful than those in CRuby.
00:07:19.920
There is an issue where the backtrace line number may sometimes be off by one or two lines.
00:07:25.680
As a result, when exceptions occur, you must be particularly careful in verifying the source of the error.
00:07:31.980
It felt like I had to do much more exploring in the C sources than I did with CRuby projects when an exception is raised.
00:07:39.210
However, I’m not sure if that stems from unfamiliarity with mruby or another reason.
00:07:46.100
There are many pure Ruby libraries already written in CRuby that you might wish to use in mruby to simplify your work.
00:07:55.100
Overall, porting from CRuby to mruby is relatively straightforward, but you must be aware of the restricted syntax.
00:08:02.870
Unfortunately, porting tests is much more difficult due to differences in the testing frameworks.
00:08:09.890
The gems for mruby feel reminiscent of Ruby 1.6 libraries, largely because fewer people have actively contributed to build up mruby libraries.
00:08:17.290
As a result, you'll often have to spend time hunting for a library that suits your use case or submit patches to get libraries to work.
00:08:25.100
So far, I've submitted patches to the guruma regular expression library, the curl wrapper, and the HTTP client.
00:08:30.950
Additionally, I’ve made multiple patches to mruby-cli.
00:08:37.100
The last challenge I want to discuss for mruby is getting build information from it.
00:08:42.000
Since mruby is compiled, it needs to link in C libraries to compile successfully.
00:08:49.779
With mruby-cli, you often need to cross-compile these C libraries with the correct C compiler, which can be complex.
00:08:55.130
It's challenging to obtain information on which compiler to use for successful cross-compilation from the build system.
00:09:05.100
I believe improving this aspect will make mruby's build system more flexible, but I haven't attempted to fix it just yet.
00:09:12.980
I like mruby CLI; it was a new tool for me, and it took some time to learn the best usages.
00:09:19.500
Several larger challenges arose with it, the first being Docker.
00:09:27.550
The mruby CLI build system is built on Docker, which allows it to cross-compile for Linux, OS X, and Windows right out of the box.
00:09:34.840
However, I found Docker rather clumsy to use for this purpose.
00:09:40.640
Typically, Docker is used to run long-lived services, while mruby CLI only needs the container running for the duration of compilation or testing.
00:09:47.180
Yet, despite its limitations, Docker appears to be the best option since there’s not much that needs to be done to get started.
00:09:53.320
To initiate a build for your command line tool, all you need to do is run 'docker-compose run compile'.
00:09:59.890
However, this command is quite lengthy and cumbersome to type, even for someone familiar with their shell history.
00:10:06.000
Additionally, debugging compile issues from outside the container can be challenging.
00:10:12.820
You have to start a shell inside Docker, determine the commands being run, and then execute those to reproduce any errors.
00:10:18.530
When working in a Docker shell, it's harder to access your usual development environment and your shell history is lost.
00:10:24.360
As a result, fixing problems becomes more complicated.
00:10:32.260
The primary challenge I faced with mruby CLI was the build system.
00:10:39.930
Most issues stemmed from the way tasks are organized.
00:10:47.130
Currently, the tasks for building your mruby CLI executable and the mruby library are mixed together within a single rake command.
00:10:53.670
This setup complicates the addition of your own custom tasks and their appropriate execution.
00:11:00.520
For example, before compiling mruby-curl, the mruby C library must be built and installed across all platforms.
00:11:07.410
However, achieving this by default is quite challenging.
00:11:14.050
To address these challenges, I made improvements to the build tasks.
00:11:20.680
The first improvement was separating the building of mruby from other Docker tasks.
00:11:27.090
This allows me to cross-compile C libraries at the correct time.
00:11:34.170
The second enhancement involved creating a rake file that exists outside Docker.
00:11:40.300
This file starts Docker for me and helps perform additional tasks that do not require Docker and are easier to debug.
00:11:46.500
The restructured build system runs rake three times.
00:11:53.160
First, the outer tasks execute tasks that do not require Docker, which then starts Docker for the inner tasks.
00:11:59.220
The inner tasks perform any necessary cross-compilation and set up before the mruby build process initiates.
00:12:05.250
The outer tasks facilitate downloads of mruby, iterative compilation, and set up global tests.
00:12:15.080
When it comes time to release, all the tasks are executed in a familiar development environment without extensive Docker debugging.
00:12:21.190
The inner tasks that run within Docker cross-compile the specified C libraries and then call Rake again to compile mruby and run tests.
00:12:27.780
Further benefits stem from these changes, as it is now simpler to incorporate hooks for custom tasks.
00:12:32.720
There are defined phases of the build, making it straightforward to add additional functionalities.
00:12:39.690
I can also speed up test execution by compiling only for the host platform during the testing phase.
00:12:46.350
There’s generally no need to build for every platform since I can run my tests on the host's Docker platform.
00:12:54.080
You will likely need a C library as a dependency in your mruby gem project.
00:13:01.070
In that case, there are two options available for including C libraries.
00:13:08.640
The first is static linking, which embeds the external library inside your executable, maintaining a single file executable.
00:13:15.110
The advantage here is that you don't need to provide instructions for installing the correct C libraries to use your tool.
00:13:23.780
However, the downside is that if any security vulnerability is found in that C library, you may need to re-release the gem.
00:13:29.740
Static linking also increases the size of your executable.
00:13:37.160
The second option involves dynamic linking, making vulnerability management easier.
00:13:44.660
If there’s an exploit, the user should update the C library, and all applications using it would be updated automatically.
00:13:50.220
This means less responsibility falls on you as the CLI author.
00:13:57.470
However, the downside is that you lose the single-file nature of your tool and must provide installation instructions.
00:14:03.940
Another challenge I faced with mruby CLI was cross-compiling the C libraries.
00:14:11.380
mruby CLI can cross-compile executables for six different platforms.
00:14:18.210
This means the C library dependencies need to be compiled for each of those platforms as well.
00:14:25.000
For example, for libcurl, you can't simply install the libcurl development package inside Docker.
00:14:31.030
Doing so would only work for your host system and not for macOS or Windows.
00:14:37.820
Instead, you have to download the source, unpack it, and then cross-compile it using the correct cross-compiler for every platform.
00:14:44.650
After that setup, you will configure mruby to use the different cross-compiled libraries you have built.
00:14:51.390
To assist with automation, I created a small tool that automates cross-compiling dependencies.
00:14:58.280
You can configure the library to cross-compile by providing a few parameters, such as its name and the source directory.
00:15:04.440
The release name is used to create the source directory, as libraries do not always use consistent naming.
00:15:10.670
You can also set a URL to fetch the source from and configure flags, like disabling OpenSSL if it's not needed.
00:15:16.670
In part two, I will discuss the origins and structure of the command line tool.
00:15:23.700
My exploration of mruby CLI began with a tool for our customer support team to configure the new service we were developing.
00:15:30.640
Our service was, and still is, API-only, meaning there’s no web UI.
00:15:38.180
The service has a complicated setup, so I wanted to automate common use cases to reduce errors.
00:15:44.440
I also wanted a tool that our customer support team could send to customers to ease the burden of supporting the service.
00:15:52.790
The solution to these problems was writing a tool that automates setup, validates user inputs, and minimizes errors.
00:15:59.290
This tool would also provide design feedback on our APIs, fostering improvements.
00:16:05.890
Before exploring mruby CLI, I started with prototypes in CRuby.
00:16:13.400
The initial prototype was a single-file script that primarily served as documentation.
00:16:20.310
It was easy for me to write, and customer support could modify it for their needs.
00:16:26.840
However, it lacked flexibility to handle all their API interactions.
00:16:32.880
As customer support's need grew, I split that script into multiple ones, each with its specific purpose.
00:16:39.420
For instance, I created scripts to list options, perform initial setups, and enable or disable features.
00:16:46.490
These changes created a more flexible architecture and were well-documented for future modifications.
00:16:52.420
Finally, I extracted a couple of classes for reuse, including a basic implementation for our JSON API and one for argument parsing.
00:16:59.080
These modular components streamlined the use of commonly shared arguments across many commands.
00:17:06.230
What lessons did I learn from these prototypes?
00:17:14.060
First, documentation within the tool is imperative.
00:17:20.290
We had not yet budgeted time for service API documentation, so the command-line tool documentation proved useful.
00:17:26.970
It turned out that roughly half the lines in the scripts were comments, enabling our support staff to understand what the tool did.
00:17:33.030
They could then suggest improvements that would make their experiences better.
00:17:40.020
Keeping the scripts simple increased overall understanding. None of the scripts required external dependencies, making installation easier.
00:17:46.350
Most scripts were small with well-defined purposes, and large scripts were acceptable for automating significant actions.
00:17:54.560
However, managing these scripts became quite challenging.
00:18:02.360
Our development environments ran an older Ruby version that lacked keyword arguments, making scripts harder to read.
00:18:09.660
Customer service would also write their tools based on the existing documentation, leading to multiple independent versions.
00:18:16.190
This made support difficult for both the teams.
00:18:23.490
Due to these challenges, I started investigating a switch to mruby CLI to reduce the maintenance burden while keeping quality high.
00:18:29.570
I decided to build my command-line tool using a pattern from RubyGems.
00:18:36.230
The subcommands for my CLI are implemented as subclasses of Subcommand, similar to those in RubyGems.
00:18:42.640
Both utilize OptionParser to parse command line arguments.
00:18:50.120
In the API, there are two methods; a setup method for defining the command line options and an execute method for running the command.
00:18:57.190
For example, if we have a command to list conditions, the setup method would require a service and version, while the execute method fetches and displays them.
00:19:04.320
To implement this in mruby, I ported OptionParser from the CRuby standard library.
00:19:10.350
I chose this since it was familiar, well-documented, and offered features such as automatic type checking and argument completion.
00:19:16.030
This reduces the amount of code I need to handle to verify arguments.
00:19:23.780
Additionally, OptionParser provides shell completion as a standard feature.
00:19:29.920
For API requests, I evaluated several HTTP libraries and chose mruby-curl, as it seemed the most mature.
00:19:36.230
This library supports both HTTP and HTTPS, essential for our production systems.
00:19:43.390
It enables persistent connections, optimizing the speed of HTTP requests.
00:19:50.440
The only downside is the libcurl dependency, which must be cross-compiled and linked.
00:19:56.460
As for testing, two mruby gems stood out: mruby-test and mruby-mtest.
00:20:02.380
mruby-mtest provides a library for assertions, while mruby-test runs all test cases.
00:20:09.040
Although using these gems is simple, setting up the test cases has been a pain.
00:20:15.990
Testing became the most frustrating aspect of building the command line tool.
00:20:22.510
Testing is crucial for maintaining quality software; however, I faced many adjustments due to the unfamiliar environment.
00:20:28.100
The mruby-mtest gem operates similarly to MiniTest found in CRuby.
00:20:34.150
You create a subclass of the test class, write test methods, and ensure to run the test suite at the end.
00:20:40.600
There's no at_exit hook in place, so you need to ensure a final test run.
00:20:47.170
In mruby, tests aren't executed directly using myrb; you must use the mruby-test gem, which provides a test runner.
00:20:53.040
Unlike CRuby, where each test file operates under a single Ruby VM, each test file in mruby gets its own Ruby VM.
00:21:00.170
This arrangement gives complete isolation between various test files.
00:21:06.580
The design of the test runner poses some complications.
00:21:13.100
Creating a global test helper requires preloading test setup in your mruby gem specification.
00:21:19.410
I didn't discover this until recent days, and it has hindered getting things right.
00:21:26.570
Moreover, there are complications with Docker; mruby CLI uses Docker Compose.
00:21:32.950
However, most documentation I found focuses on the standalone Docker command.
00:21:38.330
It took time to figure out how to set up DNS records so the tests could communicate with local development services.
00:21:45.990
Although I would have preferred to mock these services out, that would mean writing all the setups in mruby.
00:21:53.400
This would create significant trade-offs.
00:22:00.270
Lastly, running tests in mruby CLI is notably slow, as it compiles all platforms beforehand.
00:22:07.070
This makes the feedback loop lengthy when implementing small changes.
00:22:14.000
To counter the global test setup issue, I preferred to conduct much of the setup in Ruby before running any tests.
00:22:21.500
I then utilized the improved build infrastructure outside Docker, where it's easier to work.
00:22:28.310
As I recently learned about test preloading, I passed necessary data from global setup into tests using environment variables.
00:22:34.600
To enhance test startup speed, I adjusted rake tasks to build only for the host platform, which the tests utilize.
00:22:41.080
This was beneficial since it reduced the compilation requirement to one platform instead of seven.
00:22:49.080
mruby CLI supports bin tests or integration tests. Since I prefer strong unit tests, I haven't explored the integration testing as much.
00:22:56.800
For integration tests, you would run your command and assert the output it returns.
00:23:04.570
Approaching the end, there are improvements I'd like to see in mruby and mruby CLI.
00:23:10.010
Regarding mruby CLI, I plan to contribute the enhancements I've made to the build system.
00:23:16.250
I’ve already had discussions with authors about these improvements, and the maintainers have seemed positive.
00:23:23.360
Documentation for mruby CLI could be improved, as it’s currently unclear where to start or what potential troubles you may run into.
00:23:30.570
Moreover, I'd like to make upgrading to newer versions of mruby CLI easier.
00:23:38.720
As things stand, you take your changes in your rake file and port them to the new version, which becomes increasingly complicated.
00:23:45.170
For mruby, I would like to see more accurate backtraces; for now, you need to keep close attention to the line numbers.
00:23:51.330
This focus on accuracy enables you to effectively track errors.
00:23:58.390
I also wish to separate loading the build configuration from generating rake tasks, simplifying integration with the build system.
00:24:05.280
Finally, I’d like to see keyword arguments in method definitions.
00:24:09.700
mruby's C API is quite simplistic in its argument checking difficulties.
00:24:17.050
As a result, I'm unsure how keyword arguments will integrate into its current framework.
00:24:24.400
Working with mruby has also highlighted aspects I'd like to see improved in CRuby.
00:24:31.780
One improvement is with the C API function mrb_get_args, found in mruby's C API.
00:24:39.280
This function extracts method arguments from Ruby method calls into a C function for mruby.
00:24:46.230
The function lets you specify types for every argument and performs automatic type checking and conversion.
00:24:52.540
This functionality makes it easy to write a C method in mruby.
00:25:00.460
The equivalent in CRuby is rb_scan_args, which is less expressive, allowing only for checking the number of arguments.
00:25:08.560
Finding information on test preload was challenging; I read the documentation several times.
00:25:15.850
It wasn't until after that I realized how to leverage the preload functionality.
00:25:22.290
I think more thorough documentation accompanied by examples would clarify these concepts.
00:25:29.000
Moreover, there should be some automation for setup, possibly by designating a specific filename to auto-load.
00:25:35.780
Unfortunately, all tests run every time without any options for filtering by test name or file.
00:25:42.830
If you're focusing on changes causing many tests, you must manually scan through all of them.
00:25:50.410
Implementing a name filtering option would be tremendously helpful.
00:25:57.570
Thank you! Are there any questions?
00:26:13.000
As to the idea of opening a browser using mruby, you would need to link to a C library for a browser.
00:26:20.360
This way, you could control it through mruby.
00:26:27.120
Earlier in the day, Cohesion presented a GTK WebKit binding using GObject.
00:26:32.720
You could probably link mruby to GObject, gaining access to those functionalities.
00:26:41.800
So yes, that's feasible. Be sure to check out his talk for more ideas on accomplishing that.
00:26:50.150
The C APIs in mruby and CRuby have many similarities, so there is minimal learning curve.
00:26:56.560
Thank you for your informative presentation.
00:27:02.260
My question is, what is the most important advantage of mruby over rivals like Go?
00:27:07.920
What sets mruby CLI apart from Go or other tools?
00:27:14.270
For me as a Rubyist, I know Ruby well; learning a new language could delay development.
00:27:20.780
With mruby CLI, I can train my team to code efficiently without needing extensive education in another language.
00:27:28.750
Does that make sense?
00:27:34.780
Thank you!
00:27:41.740
Are there any additional questions?
00:27:49.470
Could you elaborate on the debugging difficulties you mentioned in relation to stack traces?
00:27:55.360
Yes, my stack traces show inaccuracies despite debug mode not being on.
00:28:01.160
The line numbers are often off by a couple of lines.
00:28:07.600
I’m not certain if debug mode helps; sometimes it could be a simple typo causing the confusion.
00:28:12.780
The discrepancy leads to significant time spent returning to check previous lines.
00:28:18.480
I will certainly research the debug options after the session concludes.
00:28:25.240
Thank you for bringing that to my attention.
00:28:32.550
Additionally, I should perform minimal reproduction tests to locate the issues.
00:28:39.960
Have you had experience using testing in a CI environment?
00:28:45.220
I have successfully set up mruby CLI with Travis CI, which uses Docker to execute rake tests.
00:28:51.030
At Fastly, we also utilize a proprietary gem, which requires setting up a deploy key for Travis.
00:28:58.500
Otherwise, the tests run smoothly in Travis without issues.
00:29:05.000
Thank you again for your kind attention!