The Shell Hater's Handbook

Shell Scripting

Ryan Tomayko

1 talk

#programming-languages

#shell-scripting

#developer-tools

#automation

#command-line-interface-cli

#ruby-programming

The Shell Hater's Handbook

by Ryan Tomayko

In the talk titled The Shell Hater's Handbook, Ryan Tomayko discusses the complexities and perceived flaws of the Unix shell while highlighting its potential for productivity in programming. He opens by establishing that, despite being a common source of disdain among developers, particularly in the Ruby community, the Unix shell holds a vital place in many developers' toolchains. Tomayko emphasizes that the shell is not just for interactive command-line tasks but functions as a programming language that, when mastered, can significantly streamline workflows.

Key Points Discussed:
- Two Modes of Shell Use: Tomayko differentiates between the interactive command-line use of the shell and its application as a programming language, which is often undervalued and misunderstood.
- Importance of Understanding Shell for Tools: He points to tools like RVM (Ruby Version Manager) and Git, explaining that extensive knowledge of shell programming underpins their functionality and effectiveness.
- Special-Purpose Language: Tomayko explains the concept that the shell is a special-purpose language, designed for assembling commands rather than functioning like general-purpose programming languages such as Ruby or Python.
- Syntax Quirks: He critiques shell programming syntax, noting frustrations with constructs like conditional statements and the peculiarities of command use.
- Documentation Challenges: A major theme is the difficulty of finding and understanding shell documentation, contributing to the misconceptions and learning hurdles developers face when trying to master shell scripting.
- Power of Pipelines: Tomayko demonstrates the power of pipelines using a practical example of processing an ebook to analyze word frequency. He illustrates how shell pipelines allow combining commands to manipulate data effectively, showcasing the shell’s potential for automation.
- Evolution and Standardization: He discusses the historical evolution of shell programming and the establishment of POSIX compliance aimed at promoting consistency across different shell implementations, though documentation for these standards can still be hard to find and navigate.

Conclusion: Tomayko concludes by acknowledging the unique power of shell programming as a tool for developers, particularly through its use of pipelines for automating tasks, while also shedding light on the significant barriers posed by its syntax and documentation.

Understanding how to effectively leverage the shell can lead to increased productivity, allowing developers to automate complex tasks efficiently. He encourages the audience to embrace the quirks of shell programming for the substantial benefits it can offer in their development workflows.

00:00:10.440 Okay, so I'm writing to Mako, and this is The Shell Hater's Handbook. Josh, go me! We're going to have some incredible AV here today, and I figured I'd take advantage of it to do a monster with 20-foot ASCII art. I think it turned out pretty good.

00:00:31.060 I'm here today to talk about the Unix shell, which may seem kind of strange at a Ruby conference. But there was no shell conference. When I think about the technologies, the languages that I use, and get the most value out of, they might not be the ones that are the most interesting or the ones that I want to learn, but the ones that I actually use the most to produce results—the Unix shell ranks up there strangely high. In fact, I would say that after Ruby and maybe JavaScript, I probably use shell more than any other language.

00:01:06.340 Yeah, we’re shell haters, remember? That's a little bit disturbing for me, but also really interesting. I started thinking about it, and you rarely see people talking about shell programming, but it's a very significant part of my programming toolchain. I thought it would be interesting to talk about. I'm not a sysadmin; I guess I'm an application developer. I work on Sinatra, Rack, and some other Ruby projects that Josh mentioned.

00:01:23.890 I work at GitHub, where I do product work in front-end development, and I don't typically use shell to deliver products. However, I couldn't imagine delivering them at all without it, or at least not as quickly as I can with shell. I think that’s interesting and worth discussing because shell programming is very much misunderstood.

00:01:50.409 Today, I want to hone in a little more on what I want to talk about. The shell has kind of two modes of operation. One is the interactive command-line, which I’m sure you’re all familiar with. You enter a command, and if you're lucky, something useful runs. If you're not lucky, it can mean deleting all your files or shutting down the internet. In the Ruby world, this interactive shell is pretty well-known and widely used. Essentially, you can't do Ruby development without some usage of the interactive shell.

00:02:19.480 However, that’s not what I want to talk about today. The shell is also a programming language, which I think is used much less frequently, even though I find it really interesting. However, it can be hard to approach, and there are fundamental learning challenges associated with understanding shell programming. Has anybody heard of this project called RVM? It came onto the scene about a year ago and really ignited the Ruby world. It’s pretty much assumed that you use RVM now; everybody’s doing it. If you haven’t checked it out, please do! A lot of the reasons that RVM is able to perform the things it does is because it embraces shell and uses shell scripting extensively. You simply can’t leverage RVM’s features without a solid understanding of shell programming.

00:02:56.500 Another project that uses shell extensively, which a lot of people don’t know, is Git. Initially, it was primarily a series of shell scripts with some core utilities written in C. It began as a single command and was built up from there. The way that shell has defined Git's evolution has greatly influenced the number of commands Git has today.

00:03:41.480 A lot of people, when Git first came out, looked at it and saw all this shell code. They thought it was a joke, like you couldn’t actually use it to accomplish anything. But of course, here we are today, and we’ll be doing a little bit of that with shell programming today. I think it's essential to address some of the issues with shell. I believe one of the keys to understanding shell is recognizing that it's not a general-purpose language.

00:04:39.450 If you look at shell like a general-purpose language, such as Ruby, Perl, Python, or C, and evaluate it that way, you’re going to get poor results. Shell is actually a special-purpose language designed for assembling commands. I want to illustrate this concept with an example, so let’s look at a simple hello world shell script.

00:05:31.420 This script takes a single argument and echoes back 'Hello' plus whatever name you input. If you input 'Hello World,' then it tells you how cliché you are. It’s a simple piece of shell code, but let’s take a moment to critique it. Syntactically, Ruby is visually appealing, while shell code can feel strange. For example, the use of square brackets in conditionals seems odd to me, and the extra 'then' feels unnecessary.

00:07:12.850 Additionally, the line ending of conditionals using 'fi' is just weird. These inconsistencies can feel frustrating, making you feel like you're programming in a way that feels dirty and gross. Yet, if you can overlook these superficial syntax differences, you can express similar logic as in Ruby, such as basic conditionals. That’s an important takeaway.

00:07:50.420 Now let’s make it a little more complicated. Suppose the boss checks out this hello program and is upset because it says 'Hello World,' calling it cliché. He’s steaming mad and wants that message removed from the script. Instead of deleting it outright, you add a conditional to check if the name is 'World,' and if the current user is not the boss, show the cliché message; otherwise, let them say 'Hello World' all day.

00:09:24.430 Again, we run into syntax issues. The syntax for 'and' is different, and you can only use ‘&’ once; it should really be a double ampersand. The various conditionals and syntax quirks can be frustrating, especially if you are trying to learn by looking at existing code and reasoning through it based on your experience with other programming languages.

00:10:43.700 Let’s say we find a bug where if the script is called without a name, it doesn't provide a helpful error message. So, we add an else condition to handle this case properly and also check if the name length is zero using the '-z' option. However, it's another one of those strange syntax things that make you question if you're learning correctly.

00:11:57.330 Now let’s say the boss wants to restrict who can be greeted with 'Hello.' You check the /etc/passwd file to verify that whoever is being greeted is a valid user. This opens up a can of frustration again, since now you're introducing yet another variation of 'and,' which becomes increasingly confusing.

00:12:52.610 At this point, you have two options: you can give up on the language altogether or try to explore the documentation. I’ll admit that for a long time, I chose the first option and thought rewriting this in Ruby would be quicker than trying to understand how shell works. However, if you decide to check the documentation, you might find it highly unhelpful.

00:13:41.729 For instance, when consulting the manual on your Mac, you end up with pages filled with commands and built-ins that don’t really help you at all. Often, learning about the shell feels overwhelming. If you do manage to find the help built into Bash, you may come upon a brief synopsis that doesn’t clarify much of anything about the language. This can lead to more misinformation or confusion.

00:14:59.260 So, we have this fundamental takeaway of shell programming: it’s a special-purpose language designed to assemble commands in various ways. By running commands based on their exit statuses, shell creates a unique layer of interaction. The shell syntax may give the impression of being similar to general-purpose languages, but under the hood, it operates differently.

00:15:55.500 Another interesting point is that the shell has commands built in, like 'test' and 'false,' which are special programs internally executed. This allows for faster handling of certain tasks, as utilizing built-in versus external commands can improve performance, but this can lead to confusion.

00:17:26.410 Functions within the shell act similarly to commands. They take arguments and behave just like commands in terms of semantics. You can have conditional blocks, loops, and pipelines functioning just like regular shell script programming—allowing for flexible programming techniques.

00:18:50.460 The title of today’s talk is *The Shell Hater’s Handbook*, and while I initially wanted to focus on the flaws of shell programming, I also want to acknowledge that there’s something powerful about it, particularly pipelines. Pipelines allow you to take data, manipulate it, and process it efficiently using a series of commands.

00:19:44.470 For example, let’s say we have an ebook of Jonathan Swift's *A Modest Proposal*, and we’re interested in finding out the word frequency. By grabbing the URL and using a sequence of commands with pipelines, we can break down the output step by step. We begin by extracting the text and replacing non-alphabetical characters with new lines. This approach helps condense the text to a more manageable format.

00:20:55.400 Then we can sort the output, helping us visualize each word independently. As we progress, we can easily introduce new commands into the pipeline to filter out empty lines or count the frequency of each word. By continuously iterating through each command, we refine our process and reach our final goal, making our overall workflow smoother and more efficient.

00:22:16.090 The pipelines can eventually be developed into fully functional shell scripts that can help with data management tasks, such as logging files and database analysis. As the complexity of the commands grows, we can create more intricate scripts and understand the power of shell programming.

00:23:16.610 However, one of the biggest challenges shell programmers face is the struggle with documentation. The quality of shell documentation can often be poor, leading to confusion. This is a real issue because the available references are not always up to date or applicable. It’s important to find reliable resources to aid in understanding shell programming.

00:24:10.780 For instance, early shells evolved methodically from Ken Thompson's original Unix shell through various improvements. The Bourne shell, for example, became the first shell capable of running robust programming. For a while, specialized shell languages were used, and many programs emerged from this kind of development. However, as time went on, new shells began to innovate, introducing features that substantially improved user interaction.

00:25:44.600 Many modern shells like Bash and Zsh have become the default in most systems, incorporating a vast range of features. While this advancement has made using shell programming easier in certain respects, it has also complicated the learning process, making it harder for new users to grasp the fundamental concepts of shell programming without getting overwhelmed.

00:26:54.300 The sheer size of the Bash man page compared to other shell versions creates additional hurdles for learners. When considering various shells, it is crucial to keep track of the differences between features and the associated programming aspects. The increased complexity makes mastering shell programming practices more challenging for users.

00:28:27.410 Yet, there's a movement toward standardization, with POSIX compliance serving as a guiding reference for many shells today. This was an important development because it aimed to unify the varying implementations across systems into a more harmonious whole. The goal was to ensure users would have consistent experiences regardless of which system they were operating on.

00:30:00.050 Nonetheless, finding comprehensive and understandable POSIX shell documentation remains a challenge. Online resources can be disorganized and frustrating to navigate. However, there are valuable sites and cheat sheets that can help you understand the essentials of shell programming without overwhelming you with unnecessary details.

00:31:11.470 Thank you! That's my talk.

00:31:23.800 Thank you, Ryan! We could do one question?

00:31:46.040 There are actually a lot of things, but I didn’t touch on string manipulation and other aspects due to time constraints. During my first rehearsal, my talk ran for about an hour and 45 minutes, and I had to trim a lot of really good material. There is a lot to cover in shell programming, including critical aspects like SHEBANG that I couldn’t discuss due to time.

GoGaRuCo 2010