Distributed Systems

Summarized using AI

Workshop: All Aboard The Elixir Express!

Chris McCord • April 22, 2014 • Chicago, IL

In the workshop titled "All Aboard The Elixir Express!" presented by Chris McCord at RailsConf 2014, attendees are introduced to the essentials of the Elixir programming language. Emphasizing Elixir's advantages, McCord compares it to Erlang, highlighting its concurrency and fault-tolerance features. The workshop is structured as a guided tour through Elixir, starting with the fundamental concepts and gradually moving to more advanced topics, including:

  • Erlang's Background: McCord explains Erlang's development for telecom systems, focusing on its design for distributed, concurrent, and fault-tolerant systems.

  • Elixir’s Relationship to Erlang: Elixir is introduced as a modern language that runs on the Erlang VM, incorporating Erlang's strengths while adding features like metaprogramming, ease of use, and rapid deployment, contrasting its syntax with Ruby.

  • Language Features: Key features of Elixir are introduced, including immutability, the actor model of concurrency, and pattern matching. There is discussion on constructs such as tuples, lists, and terms.

  • Concurrency Model: McCord discusses how Elixir’s approach to concurrency is superior to that of Ruby. He explains how Elixir processes are lightweight and how message passing effectively eliminates shared mutable state, allowing developers to write safer concurrent applications.

  • Process Management and Supervision: The concept of processes in Elixir is explored in detail, including how they can be supervised to handle failures gracefully. McCord demonstrates how to send messages between processes and manage state effectively.

  • Use Cases and Real-World Applications: Throughout the presentation, McCord illustrates the practical implications of using Elixir over Ruby for certain applications, particularly those that require high concurrency and fault tolerance. He shares his experiences with Elixir while discussing a Twitter aggregator project that demonstrates these concepts.

The workshop concludes with a discussion of the advantages of adopting Elixir for future programming endeavors, particularly for those familiar with Rails and looking to explore concurrent and highly reliable software development.

Workshop: All Aboard The Elixir Express!
Chris McCord • April 22, 2014 • Chicago, IL

Elixir provides the joy and productivity of Ruby with the concurrency and fault-tolerance of Erlang. Together, we'll take a guided tour through the language, going from the very basics to macros and distributed programming. Along the way, we'll see how Elixir embraces concurrency and how we can construct self-healing programs that restart automatically on failure. Attendees should leave with a great head-start into Elixir, some minor language envy, and a strong desire to continue exploration.

Help us caption & translate this video!

http://amara.org/v/FGZr/

RailsConf 2014

00:00:16.600 Before we get started, I actually want to talk a little bit about what Erlang is. Does anyone here have experience with Erlang?
00:00:21.720 A few? Okay, so some of the topics we discuss should be pretty straightforward to you, while for everyone else, it might feel like programming 101.
00:00:28.080 When I was getting into Elixir, trying to think recursively in a functional language was really difficult at first, and I felt really stupid. But once it clicks, it will change the way you think about writing programs, especially coming from an object-oriented background.
00:00:39.040 We can't talk about Elixir without first discussing Erlang. Erlang is a language that has been somewhat obscure at least for me. I come from a computer science background, but I never heard of Erlang or really touched it in college.
00:00:44.719 However, it has been a nugget of innovation for the last 20 years and has largely been ignored by the programming community at large, especially the object-oriented programming community.
00:00:50.840 Erlang was developed by Ericsson to build telecommunication systems. It is particularly unique because in 1986, when it was developed, it was guided by a specific set of requirements that many languages today still don’t follow.
00:01:01.840 They needed to create something that was distributed at its core, allowing it to run on a network of telecom switches.
00:01:07.000 It had to be highly concurrent and highly fault tolerant. Even many languages today fail to meet the trifecta of these three requirements as a primary focus.
00:01:12.960 Some languages may be considered fault tolerant because they are stable, but Erlang has built-in mechanisms to automatically restart processes if something crashes.
00:01:18.560 Erlang's focus on fault tolerance is significant. For example, if you make a phone call today, there's about a 50% chance that it's running through an Erlang system.
00:01:25.479 Erlang is tried and true. Just to prove its fault tolerance, you may have noticed that your phone company never calls you to say something like,
00:01:32.079 'Tonight you can't make a phone call from 8:00 PM to 10:00 PM because we're going down for scheduled maintenance.' You never hear that.
00:01:37.360 Meanwhile, many of our web services frequently have downtime or scheduled maintenance. Erlang has mechanisms built-in that allow for hot code swapping, which enables zero downtime deployments.
00:01:43.960 You can literally upgrade code while the production system is live. You can run programs distributedly without any extra configuration.
00:01:50.200 You simply write code against one machine, and it can run on fifty seamlessly.
00:01:55.320 So let’s get started with Elixir. If you’re familiar with Clojure and how it relates to Java, Elixir relates to Erlang.
00:02:00.520 Elixir compiles to the Erlang virtual machine bytecode, allowing you to write Elixir code and call it from Erlang, and vice versa.
00:02:05.200 Elixir isn't trying to reinvent the wheel; it's trying to build on all the innovations that Erlang and the Erlang virtual machine have provided over the last 20 years.
00:02:11.159 One reason I really love Elixir is that it brings some awesome features to the table, such as metaprogramming.
00:02:16.560 Erlang doesn’t really have great mechanisms for macros or metaprogramming, and as Rubyists, many of us appreciate metaprogramming.
00:02:21.959 Elixir focuses on adding a polymorphic layer through protocols, so we're not writing object-oriented code, but we don’t lose polymorphism.
00:02:28.560 If you’re familiar with Clojure’s protocols, that is what inspired Elixir protocols.
00:02:34.840 Elixir also emphasizes ease of use, which is something many of us appreciate, coming from a Rails background.
00:02:40.519 Rails is all about being easy to use and providing really nice features. Erlang has a bit of a history of being challenging to get into.
00:02:46.720 This has made the learning curve much steeper.
00:02:52.079 Elixir focuses on ramping up quickly and making it easy for beginners to get started.
00:02:58.560 To get started, we need to touch on some terminology, then we’ll get into some code.
00:03:05.919 Just to clarify a couple of things: the term 'term' is used extensively in Elixir, and when we refer to an 'Elixir term,' it essentially means an element of any data type.
00:03:11.159 A 'literal' is just a value of a term.
00:03:16.560 When you hear the term 'binary,' it refers to a string of bits, which comes from Erlang.
00:03:21.959 In Elixir, we often use binaries to store UTF-8 encoded strings.
00:03:28.560 The term 'arity' refers to the number of arguments a function accepts.
00:03:34.840 If you understand these terms, we should be good to go as far as discussing the rest of the concepts.
00:03:40.519 One particularly cool aspect of Elixir is that everything is an expression.
00:03:46.720 This is something that, after I got into Ruby, I’d never really encountered before.
00:03:52.079 Once I got into this idea of everything being an expression, going back to a language that doesn't follow this concept feels limiting.
00:03:58.560 For example, here's an 'if' condition in Elixir.
00:04:05.320 We can store it as a result variable, and it works just like you would expect in Ruby.
00:04:11.720 It includes a 'do' here, which is essentially the only difference. If you were to write this in Ruby, it would be mostly comparable.
00:04:17.920 Some people look at Elixir and think it's just Ruby syntax on top of Erlang. Don’t be fooled!
00:04:24.440 It looks a bit like Ruby, but the internals—semantically—are a completely different experience.
00:04:30.600 One complaint I have about switching back and forth between Ruby and Elixir is that I constantly mix up whether I should include 'do' or leave it off.
00:04:36.960 Other than that, it's a great experience!
00:04:43.960 Elixir is an immutable language, so we must change how we write programs.
00:04:49.960 We're no longer able to write objects that hide and abstract mutable state. We need to write our programs using a data transformation approach—this can be difficult at first.
00:05:01.760 But hopefully today we can discuss this approach.
00:05:08.560 However, Elixir follows the actor model of concurrency.
00:05:14.920 If you're familiar with the syntax, parentheses in Elixir are optional, just like in Ruby.
00:05:22.960 So, to call a function, as long as it doesn't introduce ambiguity, it will be very similar to what you'd do in Ruby.
00:05:30.560 This has sparked much debate, similar to Ruby; already people dislike that parentheses are optional, but I love that they are.
00:05:36.320 Documentation in Elixir is a first-class feature. This is one of my favorite aspects.
00:05:41.720 You write documentation in Markdown, meaning you document your methods or functions in Markdown, then have access to it within IEx.
00:05:48.520 It’s like having IRB open; you can just type a function and get the formatted Markdown of the documentation right in your shell.
00:05:54.440 This feature is incredible. Also, there is coexistence with the Erlang ecosystem, which is a huge aspect.
00:06:01.120 Elixir isn't trying to reinvent everything. We can call an Erlang function in Elixir, similar to calling a Ruby method off a symbol. So, the way we utilize that would be to call the functions you need.
00:06:09.720 The syntax is a bit uglier if you're trying to call Elixir from Erlang, as you need to prefix it with an Elixir module.
00:06:15.760 However, calling it from Erlang to Elixir is super straightforward. You could call math.pi from the Erlang math module and would get Pi.
00:06:21.239 Someone might ask, 'Why would you call Elixir from Erlang?' Well, if I release an Elixir library that does something awesome and an Erlang person wants to use it, they have that option.
00:06:30.520 This essentially mirrors how you could drop in your project and utilize it. Awesome!
00:06:36.760 Before diving into some code, I want to talk a little bit about why I think Elixir is the future of my programming career.
00:06:44.520 As much as I love Ruby, how many people in the audience do threading in their Rails apps? Show of hands!
00:06:51.920 How many people do threading on MRI in their Rails apps? Awesome! Only one person?
00:06:57.120 The concurrency story in Ruby on Rails is really weak. It can be done, but it's a wild west and doesn't really work that well.
00:07:04.560 For instance, when building Rails applications at work, we often need to shove work into a background job and pull the server when trying to hit a third-party API, as we can't keep a connection open.
00:07:11.720 This leads us to achieve concurrency in Ruby; if we have a quad-core system, we run four processes of our Rails app.
00:07:16.520 As more cores are added to CPUs—like we're starting to see with 50 cores on the horizon—I don't believe we're going to stand up here.
00:07:22.080 We'd just say, 'Well, let's just run 50 Rails processes to achieve concurrency.' I think this landscape creates a problem that's inherent to Ruby's architecture.
00:07:28.520 Some people can get by using this approach, and it works well. But it starts to fall down when anything is highly concurrent.
00:07:35.960 If you try to parallelize work, it's almost impossible. We jump through hoops to run Sidekiq or Rescue workers, trying to coordinate them to complete tasks.
00:07:43.440 It's because we cannot properly do concurrency in Ruby in the way that we desire.
00:07:50.760 So, getting into Elixir, concurrency is essentially at the heart of everything we do and it eliminates threading.
00:07:59.040 You may think, 'Well, we could just use proper threading in Ruby or use JRuby for better threading.' However, that still isn't an ideal way to achieve concurrency.
00:08:03.720 You have to deal with mutexes and locks, and you have shared mutable state that makes reasoning about the code and writing for concurrency difficult.
00:08:09.120 So, with functional programming using immutable state, we can write safer and more concurrent programs that are simpler to reason about.
00:08:19.040 We'll later see how message passing works, and we use processes that are lightweight and not operating system processes.
00:08:25.200 You can run millions of Elixir processes on your machines, something you cannot do with operating system threads.
00:08:31.360 For those who are familiar with WhatsApp, they recently received a valuation of $20 billion, and they handled a million concurrent connections per server.
00:08:39.760 It gives you an idea of the language envy I experience coming from Ruby, especially considering that in Rails we can barely handle more than 15 concurrent requests without overwhelming our database connection pool.
00:08:49.200 All these observations lead me to consider where the Rails community needs to progress to effectively manage concurrency.
00:08:56.960 So, for those still getting set up, try to pair with someone working next to you.
00:09:03.760 We're starting out with basic concepts, but hopefully by the time you dive into some real code examples, you'll feel confident typing it in.
00:09:11.760 So, in Elixir, we work with a very small set of types: integer, float, atom—when we say 'atom,' think of it almost synonymously with symbol in Ruby.
00:09:18.399 A tuple is stored continuously in memory, similar to an array. Tuples store multiple items, while lists are linked lists.
00:09:25.040 A bit is essentially a process ID and is lightweight, essentially functioning as a unit of concurrency.
00:09:31.040 Using this small set of types, we perform all of our programming. There are additional layers that Elixir builds on top of this.
00:09:39.759 Instead of writing objects that encapsulate state, we use functions to mutate data structures.
00:09:46.720 It's conceptually a data-in, data-out approach built on a limited set of types that guide how we structure our programs.
00:09:54.120 Atoms behave like symbols in Ruby. To create an atom, we can use IEx; if we open IEx, we can declare an atom to see how it behaves.
00:10:01.920 The use of built-in functions helps us check types, allowing us to stay consistent and understand our data.
00:10:09.079 Another point to emphasize is that everything in Elixir is an expression. There's no need for an explicit return keyword, as the last evaluated expression is returned.
00:10:15.160 This creates a seamless flow of writing expressions, very similar to what you've experienced in Ruby.
00:10:21.919 Quickly, here's an example of an if condition that behaves just like you would expect in Ruby, but remember, 'do' is one additional difference in syntax.
00:10:29.479 We can compare it to Ruby and realize that Elixir introduces semantics like blocks that help maintain logical flow.
00:10:36.919 Elixir, as an immutable language, requires you to adopt a different mindset when compared to object-oriented languages.
00:10:43.919 You’ll no longer be hiding and abstracting mutable states through objects—this forces a paradigm shift to see your programs through a data transformation lens.
00:10:52.079 While that can be challenging at first, I hope we'll explore this together today and you will appreciate the benefits.
00:10:57.960 Elixir follows the actor model of concurrency, which is similar to other functional programming paradigms.
00:11:05.200 In Elixir, parentheses are optional, just like in Ruby, unless it introduces any ambiguity in the code.
00:11:10.560 This aspect has led to various debates similar to Ruby. There are myriad preferences, yet some developers love that parentheses are optional.
00:11:16.120 Documentation in Elixir is first-class and absolutely delightful!
00:11:24.479 You write documentation in Markdown, allowing you to document your methods in a rich format.
00:11:31.920 This documentation is wonderfully accessible within IEx, providing you formatted responses when you query your functions.
00:11:38.240 This direct accessibility is phenomenal for new users to get immediate context on functions, enhancing comprehension.
00:11:45.360 Furthermore, Elixir's coexistence with the Erlang ecosystem is another noteworthy feature.
00:11:51.440 Elixir is not reinventing the wheel; it calls Erlang functions directly, making development and interop between the two languages incredibly seamless.
00:11:57.760 And you just have to remember to prefix your calls with the appropriate module when working between the two.
00:12:04.760 As we gauge the myriad of features Elixir has, let’s embrace Delicacies such as polymorphism via protocols, which further enhances flexibility in our programs.
00:12:10.840 The focus on ease of use is paramount in Elixir, aimed at assisting newcomers to navigate development effectively.
00:12:16.240 We start by learning about foundational terms and usage before jumping into code examples, keeping it relatable.
00:12:20.400 Next, let’s discover how values like atoms, binaries, and terms work in Elixir.
00:12:25.760 We derive meaning from documentation and specific keywords necessary to work effectively.
00:12:31.040 Becoming acquainted with these terms will simplify future interactions and coding practices.
00:12:36.560 Understanding values and syntax through practice will unveil the logic and enhance your capabilities within Elixir.
00:12:43.760 From actively utilizing processes, defining modules, to writing code where persistence becomes important, we are embarking on a journey together.
00:12:48.840 Now, let's delve into the various structures like tuples and lists that make up the foundation of Elixir.
00:12:55.520 Tuples, which store multiple elements, and lists, which implore a linked list structure, are essential data types we’ll explore practically.
00:13:02.960 A string of bits refers to binaries; this notion aids in understanding how conversions work seamlessly without requiring explicit data state management.
00:13:10.280 Elixir’s seamless interfacing with its systems leads to better memory utilization, allowing efficient programming practices.
00:13:16.600 As we continue, various tools like IEx become our instruments, granting development efficiency as we sync code and documentation.
00:13:23.319 Understanding how to manage timeouts and failures with processes will yield a robust development experience as a whole.
00:13:29.560 As we emphasize the principles of functional programming, our perspective shifts to fit the Elixir paradigm.
00:13:36.560 We’ll work through examples to exemplify the routines and clarify complex notions surrounding the language.
00:13:42.799 In conclusion, our excursion into Elixir will cultivate a rich understanding of programming paradigms moving forward.
00:13:49.920 See, we’ve barely scratched the surface; there’s so much to learn and explore!
Explore all talks recorded at RailsConf 2014
+133