Talks
Placing Things into Other Things
Summarized using AI
Placing Things into Other Things
by Charles LowellIn the presentation titled "Placing Things into Other Things" by Charles Lowell at LoneStarRuby Conf 2013, the speaker delves into the complexities of developing command line interfaces (CLIs) and proposes solutions to common issues faced by developers in this domain. The talk emphasizes the importance of enhancing the developer experience, particularly through improved CLI tools.
Key points discussed include:
- Challenges of CLI Output: The speaker highlights the common problems related to managing output to the terminal, where direct writing to standard output can lead to clashes, particularly in larger applications.
- Output Formatting: There are issues in sharing formatting code across various output generators, which can make applications complex and hard to manage. An example given involves a CLI created with Thor, emphasizing the need for a tailored help display.
- Integration of External Streams: Integrating output from other applications can complicate how information is presented, particularly when external output doesn't match the application’s style.
- Progress Indicators: Highlighting the difficulty of providing user feedback during long operations, the need for efficient and clear feedback mechanisms is discussed with reference to progress bars.
- Decoding and Validation of Inputs: The speaker points out the challenges in transforming strings into meaningful objects, advocating for a structured approach to decoding and validation.
- Plugin System Necessity: As CLIs grow, implementing a robust plugin system becomes necessary to manage additional functionalities. This helps break down complex applications into manageable parts.
- Proposed Solutions: Solutions mentioned include using middleware to streamline output management, implementing templates for better organization of output, and structuring input through forms which helps define expected input and ensures validations.
- Adaptation of Web Principles: The speaker compares command line tools to web applications, advocating for the adoption of established frameworks and paradigms to enhance CLI development.
- Introduction of MVCLI: The speaker introduces MVCLI, a project on GitHub, as a framework that organizes CLI development and offers extensive plug-in capabilities.
Overall, the presentation concludes with a reinforcement of treating command line applications as complete entities rather than simple scripts, emphasizing the importance of established structures and methodologies to facilitate their development to enhance user and developer experiences.
00:00:15.599
All right, well, I'll go ahead and get started. I'm a little bit nervous this morning.
00:00:21.359
A little bit more so than normal because I'm actually going to start out with a confession.
00:00:27.599
Don't worry, it doesn't involve viruses or anything like that.
00:00:33.280
Actually, I’m not going to talk about what I told you I was going to talk about.
00:00:38.640
I'm going to talk about something else.
00:00:45.040
I think that what I've been working on in the last couple of months has greater potential to be useful to you and to make your life easier, which is what I sincerely hope.
00:00:51.840
So, what I'm actually going to talk to you about is command line interfaces. You might be thinking, 'Oh my goodness, why talk about command line interfaces when there are so many great things to discuss?' But as a developer, the primary way you experience your world is through streams of text.
00:01:04.720
So, I think we can do a lot better than what we're currently doing. Yes, I am in fact going to talk about CLIs.
00:01:10.640
And to drive the point home, I’ll illustrate why it’s worth discussing.
00:01:18.080
As a company focused on crafting human experiences, the developer experience is very important to us. It’s why we take CLI tools very seriously and write quite a few of them.
00:01:24.799
The space is very crowded with option parsing tools: Slop, Trollop, Mixlib, Optitron, Rake, Thor, GLI, and Option Parser.
00:01:29.040
It’s fun to name them all! There are probably 20 that I haven’t even mentioned. Each has been written by people with strong opinions about the problems they need solved.
00:01:41.360
I have used many of these tools over the past five to six years, and they've all given me this great feeling of winning when everything is going right.
00:01:49.120
But unfortunately, when writing command line applications, I often hit a wall.
00:01:55.120
I'm going to talk about what that wall is and what happens to make the experience difficult for us.
00:02:01.600
The first issue is output. This is probably the biggest bugbear of CLI development for me—the one that causes the most irrational fear and loathing: how we manage output to the terminal.
00:02:08.399
The basic problem is that everybody writes straight to standard output everywhere in their code. This is something you see a lot.
00:02:15.280
Sometimes this approach is appropriate for small systems, but as your system grows, more and more code contends for the single resource of standard output.
00:02:23.840
It’s a global variable, and it’s problematic for all the reasons that singletons and global variables are typically frowned upon.
00:02:29.040
This is a typical anti-pattern. So, how do you share calls to puts?
00:02:34.640
The next thing is sharing formatting code in a command line application. This issue arises when you have several actions generating output that share common attributes.
00:02:41.519
You want to substitute the varying elements, while keeping others static.
00:02:49.120
For example, a command line interface we created with Thor required a tailored help display.
00:02:55.120
We ended up writing a class and a method that made imperative calls to standard output, incorporating formatting. It was complex and not very clear.
00:03:04.800
The other problem is integrating other streams into your output. At some point, your command line application will want to call out to other applications and integrate their output.
00:03:10.800
An example of this is with Heroku build packs, which have strict style guides for output.
00:03:16.000
But I would argue that formatting should be the responsibility of the output renderer, not the builds.
00:03:22.000
Embedding control characters can also be tricky when integrating external streams. This leads to headaches.
00:03:30.000
For instance, Bundler's output has issues with clarity due to random new lines and spacing.
00:03:36.800
When viewing this output, I should instantly recognize where Bundler's contributions are rather than having to decipher them.
00:03:43.200
Progress bars are another area where complexity arises. I love the RubyGems interface when it shows progress, but maintaining user feedback when blocking for over 600 milliseconds is crucial.
00:03:50.319
However, achieving this can be tricky since the progress bar might end up writing to standard output separately.
00:03:57.600
Another constant complication is decoding and validation.
00:04:05.200
Inputs come as strings, the arguments are strings, option names are strings, and so are the option values.
00:04:12.080
But we want entities that we can derive knowledge from, which makes it difficult when we’re stuck with strings.
00:04:18.000
You end up needing to front-load the extraction of information to make handling them easier.
00:04:24.960
For example, a string might represent a path, but I want a Pathname object instead.
00:04:31.840
Similarly, an IP address string should be decoded into an IPAddress object.
00:04:36.960
Even an email address input should map to an ActiveRecord object.
00:04:42.080
Decoding and validation need to occur in tandem; you can't pursue one without the other.
00:04:49.199
This is why distinguishing them is essential, because decoding just verifies the representation while validation ensures usability.
00:04:56.000
In English, this can be illustrated by well-formed sentences that make no sense.
00:05:02.560
The same holds true for objects; even if they’re validly structured, they can fail to provide meaningful information.
00:05:08.720
This applies in many contexts, especially for pathnames.
00:05:14.640
Once I have a valid Pathname object, my next questions involve file existence and permission to write.
00:05:22.000
I feel that this output-input imbalance is not just limited to command-line applications; the Ruby ecosystem reflects this as well.
00:05:28.400
While we have many tools for output, we lack effective, orderly, repeatable methods for converting strings back into usable objects.
00:05:36.959
This is a concern because the closer we get to knowledge about our data, the smarter our programs can become.
00:05:43.760
However, objects are smart, and utilizing smart objects leads to the development of efficient programs.
00:05:51.040
Any command line application will eventually need a plugin system as it grows more complex.
00:05:57.679
When a program expands, it must be broken down and distributed into smaller, manageable components.
00:06:03.440
This concept applies to command line applications as well, as we see in tools like RubyGems, Vagrant, and Heroku.
00:06:09.760
The need for plugins becomes evident when considering the extensibility of the application.
00:06:15.840
But implementing a plugin system can be a daunting task because of the effort involved.
00:06:21.840
However, it’s crucial to use a system that provides this infrastructure for free.
00:06:29.760
There is a lot to discuss, and I haven't touched on many little issues that bother me.
00:06:35.759
I will pivot my discussion to solutions since what's the point of discussing problems without proposing answers?
00:06:42.799
I cannot stress enough that option parsing DSLs should not be confused with application frameworks.
00:06:49.760
Though this distinction may seem obvious, it’s crucial to keep in mind when evaluating option parsing libraries.
00:06:56.240
Ultimately, the challenges of command line applications extend far beyond converting strings into hashes.
00:07:03.680
So, fast forward to a couple of months ago — we were hired to create a command-line tool for Rackspace.
00:07:11.440
The tool aims to make Rackspace's technology more accessible to developers.
00:07:17.200
We are building a CLI with vast functionality, containing many commands and library components.
00:07:24.240
It will inevitably require extensive features and plugins.
00:07:31.599
To tackle these mentioned challenges, we need to identify and address them specifically.
00:07:38.000
The first solution for output is to contextualize the streams.
00:07:45.440
We must create a structure that prevents uncontrolled access to standard output.
00:07:53.040
We also want to ensure any code can generate output, but in a controlled manner.
00:07:58.800
One effective approach is introducing middleware.
00:08:05.680
Middleware is a pattern where an application consists of small components sharing a uniform interface.
00:08:12.080
This pattern is familiar to many of you, likely from using Rails.
00:08:20.240
But why would middleware be beneficial for a command line application?
00:08:26.560
The answer lies in adapting the essence of the Unix process model for CLI.
00:08:33.760
A command consists of a list of arguments, input, output to a logging stream, and environment variables.
00:08:40.800
This data structure allows you to move information from one middleware to another.
00:08:48.080
The essence of middleware is simple: you call the method that handles commands, manipulate the inputs, and yield control.”
00:08:54.560
Each middleware component manages all streams and controls downstream outputs.
00:09:02.240
For example, imagine a middleware that substitutes emojis for output characters.
00:09:10.560
It wraps the command's output, without interfering with the command's internal logic.
00:09:17.599
Templates are another essential element I want to touch upon. They are not just a trend; they have real value.
00:09:25.920
In CLI contexts, templates can help with text layout while maintaining clarity.
00:09:31.920
For example, let’s revisit the help output example from earlier.
00:09:39.040
Implementing this using a template can often provide clearer intent.
00:09:47.440
Moreover, we wanted our templates to support streaming, allowing data bytes to flow through the middleware stack to standard output.
00:09:54.320
This allows us to manage outputs in a coherent way while hiding the complexity of streaming from the generating components.
00:10:01.920
For instance, we can pass control to a progress bar within a template, maintaining a clean flow of data.
00:10:08.720
Templates facilitate organization and flexibility, so they seamlessly work in CLIs.
00:10:15.920
Regarding decoding and validation, we encountered challenges but settled on introducing a new object.
00:10:24.080
We effectively introduced the concept of forms in command line applications.
00:10:30.240
The idea of 'form' here holds a theoretical aspect, denoting the ideal structure of inputs for actions.
00:10:37.280
To illustrate, here's the input form for the 'create' action in our CLI.
00:10:44.160
We list the expected inputs, types, defaults, and decoding procedures in a structured way.
00:10:51.680
Once you set up these input forms, you can derive your option parser easily.
00:10:57.920
No matter what parser you decide to use, whether it's Trollop or Slop, this structure is the foundation.
00:11:05.280
However, let's take a moment to reflect on the current state—we have templating, validations, and middleware functioning together.
00:11:12.600
Are we inadvertently reinventing the wheel? Is this not like Rails?
00:11:18.720
Well, in a way, yes, and that is perfectly fine.
00:11:25.520
The principles we apply to command line invocations share characteristics with web requests.
00:11:32.720
For instance, standard input can correlate with request bodies, while output streams to output streams.
00:11:39.680
It’s logical to adopt these patterns as they help us tackle similar challenges in diverse contexts.
00:11:45.920
To illustrate this concept, consider the notion of convergent evolution.
00:11:53.200
Different species, like dolphins and sharks, independently adapt to their environments and develop similar traits.
00:12:00.000
Likewise, command line tools and web applications evolve to meet related needs.
00:12:06.240
So, let’s embrace these effective structures and strategies for our CLIs.
00:12:14.720
We have created something called MVCLI that encapsulates these organized patterns for command line applications.
00:12:20.800
It’s on my GitHub for public use, and it offers a broad array of robust functionalities.
00:12:27.920
We continue to enhance its plug-in architecture to ensure comprehensive usability.
00:12:34.080
Remember, command line applications should be treated as complete applications, not merely scripts.
00:12:41.200
It’s crucial to lean on established frameworks for support, just like we do with Rails.
00:12:48.080
The core takeaway is that a command isn't just a class or object; it's a process governed by business rules.
00:12:54.320
And I will start treating it that way.
Explore all talks recorded at LoneStarRuby Conf 2013
+21