00:00:08.560
Hello everybody, my name is Thorsten Ball. I'm a software developer from Germany. I program in Ruby, Go, and sometimes in Lisp. On the left is my name on GitHub and on the right is my name on Twitter. I work for a company called Fling in Germany as a Rails developer, and we do dynamic ride-sharing as a web application. The name of this talk is "Unicorn Unix Magic Tricks," which is quite a mouthful, and actually a weird title. Allow me to explain.
00:00:32.770
Unicorn is a web server written in Ruby for Rack and Rails applications. When I first encountered Unicorn, it seemed like magic to me. Unicorn had all these amazing features, like a master-worker architecture, where you had one master process and multiple worker processes. It had hot reloading, which allows Unicorn to reload a new version of your application while the old one is still running, enabling it to serve all requests while the new version boots up. This felt like magic to me.
00:02:14.530
Unicorn also had several interesting signals you could send, such as the TTY signal to increase the number of workers, the USR2 signal for hot reloading, the HUP signal to reload the configuration, and the QUIT signal for a graceful shutdown. Additionally, Unicorn featured preloading, which allowed it to preload my Rails application—taking around 10 seconds to load—into memory and spawn new worker processes in approximately 50 milliseconds. All of this seemed magical to me at first, but as I explored Whisper's source code, I discovered that it all relied on Unix.
00:02:40.960
Many of you know Unix from the user's perspective—you use the shell, command line, pipes, redirection, and so on. However, there is also a developer's side of Unix, which you can leverage in your programs. In this talk, we're going to look at how Unicorn employs Unix to build its exceptional features. We'll explore some basic Unix tricks and principles, see how they work, how we can use them, and how Unicorn utilizes them.
00:05:05.750
The first Unix trick I want to talk about is the 'fork' function. Every process on your Unix system, except the first one, is created by a call to fork, which is a system call documented in section 2 of the Unix manual. System calls represent the API of the kernel. When you call fork, the kernel splits your process into two: a parent process and a child process, where the child is nearly an exact copy of the parent, including the data, stack, heap, environment, user ID, and current working directory.
00:07:40.680
Using fork in Ruby is straightforward; we use the 'fork' method. In an example, after calling fork, we have two processes: a parent and a child process. The parent must wait for the child to finish execution to avoid becoming a zombie process—this is a technical term in Unix. The parent prints something after the child finishes, and if you run this, you can see that the output of the child process matches the parent process ID. With this single fork call, you have created two processes.
00:08:09.480
Unicorn leverages fork in a similar manner. Its method called 'spawn_missing_workers' creates each worker in a loop by calling fork multiple times. The parent process takes the returned child process ID and saves it, while the child enters a worker loop. After this call, you'll find 16 worker processes doing their tasks while the parent process handles work concurrently.
00:09:03.660
Now, let's discuss another Unix trick: pipes. You have probably used pipes before, where the output of one command becomes the input of another. However, we can also use pipes outside the shell with the 'pipe' system call, which gives us two file descriptors—one for reading and one for writing. File descriptors are numbers pointing to a file entry in the kernel. Since child processes inherit these file descriptors, pipes become an excellent means of communication between processes.
00:10:22.320
In Ruby, we start by calling IO.pipe, which eventually invokes the pipe system call and provides us with read and write ends. Once we close the read end in the child process, we can write a message to the write end and close it after sending. The parent process needs to wait for the child to exit and closes the write end. Then, it reads the message from the child process. This approach allows us to communicate effectively between two processes. Unicorn employs pipes extensively to manage its worker processes.
00:11:54.340
For instance, when you configure Unicorn to use 16 worker processes, it opens a separate pipe for every connection between one worker process and the master process. Thus, there will be 16 pipes through which the master communicates with the workers. Unicorn also implements a self-pipe used by the master process to communicate with itself. An interesting application of pipes arises when you start Unicorn as a daemon process, meaning it runs in the background and isn't attached to the terminal.
00:15:00.820
To detach from the terminal correctly, you must call fork twice, creating a grandchild. Unicorn utilizes a pipe to communicate with the grandchild. Once the grandchild sends a message indicating that it is fully booted up and ready to go, the master process will exit and detach from the terminal. This use of pipes helps synchronize the booting of a daemon process.
00:17:18.700
Next, let's discuss a basic Unix principle: sockets. Sockets are essential for networking in a Unix system. They represent connections, and there are various types including TCP, UDP, SCTP, and raw sockets. In Unix, everything is treated as a file, meaning that sockets also behave like files. As such, child processes inherit sockets just as they do file descriptors.
00:20:35.800
For web servers, like Unicorn, you must go beyond simply calling socket system calls. The basic Unix networking socket lifecycle involves creating a socket with the socket system call, binding it, and then listening for incoming connections. The listen call initializes the socket, effectively turning it into a passive server socket that accepts connections. Additionally, we must use the 'accept' method to get new connections, which blocks until a new connection is available.
00:23:07.140
However, blocking on a socket makes it challenging to manage multiple connections. This is where the select system call steps in. Select monitors file descriptors, returning when one or more are readable or writable, effectively allowing multiplexing. Using select, we can replace blocking accepts, creating a fully functional multi-process TCP server with relatively few lines of Ruby code.
00:25:36.590
Unicorn employs the same strategy, creating a listening socket and utilizing the select method to distribute connections among its workers. When Unicorn initializes, it establishes a listening socket and pipes for worker communication. Each worker calls IO.select to monitor the listening socket and pipes to ensure it doesn't miss messages from the master process.
00:28:30.110
Additionally, let's talk about signals. Many of you have probably used signals to kill processes. A signal is a software interrupt that gets delivered to a process through the kernel. Signals have various actions, such as quitting or cleaning up, and can be defined in the kernel. For example, if a process receives a quit signal, it can gracefully handle the signal by cleaning up resources instead of abruptly exiting.
00:30:22.150
In Ruby, you can easily handle signals using the 'trap' method, allowing you to execute specified actions upon receiving signals. However, Unicorn has its own extensive signal handling mechanism. It creates a self-pipe that captures signals, allowing it to maintain control while efficiently managing signals to handle state changes.
00:30:58.790
It does this by writing the name of the received signal to a queue. When the master process detects a signal in its main loop, it's awoken and acts on the signal accordingly, turning signals into a synchronous stream of events for straightforward processing. All of these components—forking, sockets, selecting, and signal handling—are fundamentally simple yet come together to give Unicorn its rich set of features.
00:33:04.290
Now, let's take a look at Unicorn's extraordinary features, starting with preloading. Preloading allows Unicorn to load the Rails application into memory once, enabling faster worker process startup. The master process sets up a lambda to handle loading, and when you instruct Unicorn to preload, it simply calls fork, and the child processes inherit the loaded memory. This innovative approach makes Unicorn’s scaling and performance improvements quite impressive.
00:34:37.550
Another fascinating feature provided by Unicorn is worker scaling through signals, enabling the addition or removal of worker processes seamlessly. The master process is responsible for managing the total worker count, creating new workers when needed or gracefully shutting down ones that are no longer necessary. This mechanism ensures Unicorn maintains optimal performance while being resource-efficient.
00:36:20.570
Lastly, hot reloading is a standout feature sometimes deemed as zero-downtime deployment. Unicorn achieves this by starting a new master process while the existing one continues to operate, allowing for deployment of new application versions without request interruption. This complex functionality is achieved using previously established mechanisms, such as signaling, forks, and pipes.
00:37:59.510
With hot reloading, the master process spawns a new child and, if successful, the new master can seamlessly take control of the same sockets as the old one, ensuring continuous service without service disruption. All of these enhancements exemplify how Unicorn has combined simple concepts into powerful features.
00:38:26.470
Before concluding, I'd like to emphasize the importance of understanding these concepts for application developers. Debugging Rails applications often occurs on Unix-based systems, so knowledge of this environment better prepares you to tackle production issues. Furthermore, understanding how processes communicate and how system calls operate helps in designing more efficient applications.
00:39:41.470
By knowing how these lower-level details influence your application, you can make better architectural decisions that align with performance requirements. Ultimately, learning about these principles has greatly benefited my own programming skills, enriching my work with Ruby and Rails due to a deeper understanding of their underlying systems.
00:41:18.000
I appreciate your attention and invite everyone to ask questions or delve into discussions about low-level programming, Unix, or Ruby. Thank you for listening.
00:43:20.590
Do we have any questions? Yes, the first part is about hot reloading. It raises a concern about whether killing the parent would leave the new process as a zombie process. Generally speaking, it is uncommon to kill your own parent process, which is something that can confuse the system. Most users leverage orchestration tools to manage old processes after the new ones are booted.
00:45:47.900
The second part of the question touched on whether multiple processes can share a socket. With modern Linux, and even FreeBSD, it's feasible to have separate processes use the same socket, although the socket must be appropriately configured to allow this interaction. You can also send file descriptors between processes, which opens additional avenues for inter-process communication.
00:48:20.000
Thank you all for your participation and insightful questions.