Metaprogramming For Generalists

Chris Salzberg

Chris Salzberg

#developer-experience-dx

Metaprogramming For Generalists

by Chris Salzberg

The video "Metaprogramming for Generalists" presented by Chris Salzberg at EuRuKo 2018 dives deep into the concept of metaprogramming in Ruby, emphasizing its significance for developers looking to write modular and extensible code. The talk is structured into two parts: the first part introduces a new perspective on metaprogramming, while the second part demonstrates practical applications through the development of generic software.

Key points discussed include:
- Definition of Metaprogramming: Salzberg discusses various definitions, highlighting the struggle within the programming community to agree on what metaprogramming truly means. The basic idea is described as 'writing code that writes code' and involves structures recognized as reflective programming.
- Relevance to Ruby: Ruby’s metaprogramming capabilities set it apart from other languages, underpinning many features such as human-like syntax and the creation of Domain-Specific Languages (DSLs).
- Critique of the Current Understanding: The speaker critiques the common view that metaprogramming is a niche concept, proposing that it’s central to Ruby's functionality and raises the question of why it is often neglected in application code.
- Role of Libraries and Gems: Libraries, or gems, play a crucial role in facilitating metaprogramming by generalizing common problems faced across applications. ActiveRecord is cited as a major example demonstrating extensive metaprogramming use.
- Concept of Attributes: Saltzberg illustrates his points with examples of attributes in Ruby classes, showcasing how metaprogramming can obscure complexity for ease of use in applications.
- Generic Software Development: Referencing Jeremy Evans’ ideas, Salzberg advocates for breaking down single APIs into smaller, more manageable components, easing the development of flexible software.
- Conclusion on Generalists: Emphasizing the importance of embracing metaprogramming, the talk concludes with a call for Ruby developers to become generalists—individuals who grasp core concepts to tackle broader programming challenges.

The overall takeaway is the power of metaprogramming in enhancing the Ruby ecosystem, allowing developers to handle more generic problems with flexibility and efficiency. The closing remarks stem from the notion that understanding metaprogramming can lead to improved community practices and software development.

00:01:32.280 There are some selfies being taken, but we really need to get started again, as we're on a tight schedule.

00:01:37.960 Remember, otherwise I will take away your lunch break. And everybody's like, 'What's lunch break?'

00:01:45.789 I need to take a picture with Mats. Okay, so we will have two talks, and they will be great.

00:01:58.920 Then there will be a lunch break, which means you need to go outside. I've also been told that people should come on time.

00:02:09.280 Schedule silence—I have no authority on this stage. Do I? No authority.

00:02:27.940 I love that! Yes, yes.

00:02:37.760 All right, we have to talk about the schedule. So, there will be a lunch break, as long as you cooperate with me.

00:02:44.360 If you don't, you know the consequences. I have been told there will be coffee and such, whatever 'such' means, but I'm guessing it includes food and snacks in the next break.

00:03:00.019 So, that's happening, and we need to do some technical stuff here. I'm just going to announce the next speaker right away to stay on schedule.

00:03:08.840 Breaks are for talking, while conference room attendance is for not talking and for listening. Again, thank you to my kind volunteers in the audience for the clapping, it works really well.

00:03:15.790 So, our next speaker is originally from Canada and now lives in Japan. He is the author of Mobility and an open-source contributor.

00:03:22.159 He is also very active in the Japanese Ruby community, and he will be teaching us all about metaprogramming, which is the machinery that drives Ruby.

00:03:29.030 We will be learning about writing modular, extensible code and tackling real hard problems. His goal is that we all walk away from here with a conceptual understanding to be more generalists.

00:03:36.290 Because, according to Chris, that's what we need more of in this community. So, give it up for Chris!

00:04:46.940 Look, okay, everybody, my name is Chris Salzberg. The title of my talk is 'Metaprogramming for Generalists.'

00:04:52.400 I apologize—it's the first morning of the first day, and we're starting with metaprogramming. But hey, don't worry, it'll be interesting!

00:04:57.950 Some quick points about me: my handle is 'ChillYama,' if you've seen that name around. I live in Tokyo, Japan.

00:05:05.810 The background picture is from my neighborhood. Don't be fooled; I'm not Japanese—in case that wasn't obvious. I'm Canadian, originally from Montreal.

00:05:11.660 I work at a company called Edgeago. We have a booth outside, so check us out.

00:05:18.080 In the open-source world, I am the author of a few gems, with Mobility being the most well-known, which is for translating model data.

00:05:24.230 I also write about things like the module builder pattern on my blog, Digimon.com. Okay, enough about me.

00:05:34.670 This is a two-part talk. In the first part, I'm going to present a new way of thinking about metaprogramming that I think will be more relevant and meaningful for solving generic problems.

00:05:40.250 In the second part, we're going to solve a generic problem and build what I'm going to refer to as generic software.

00:05:45.530 So, talking about generalist metaprogramming, it's Ruby's 25th birthday, and I think more than any other area, metaprogramming is where Ruby truly stands out among programming languages.

00:05:52.160 It's something that Ruby really embraces, and it underlies all the aspects of Ruby that Matz just mentioned—the human-like syntax, the DSLs, and the nice use of it.

00:06:00.220 However, we still have a hard time defining this term. Programmers, in general, struggle with this concept.

00:06:05.480 A basic definition states that if programming is writing code, then metaprogramming is writing code that writes code.

00:06:12.410 A second definition, from Wikipedia, describes it as a technique in which computer programs can treat other programs as data, also known as reflection.

00:06:19.130 It's a bit broader and also incorporates introspection, which is a way for a program to find out about itself or other programs.

00:06:26.000 By the end of the day, though, people can't seem to agree, and we tend to throw our hands up in the air and refer to it as a 'bag of tricks.'

00:06:31.520 This quote is from the book 'Metaprogramming Ruby,' which I highly recommend. What are those tricks? They include techniques such as monkey-patching, yield, self, and others.

00:06:37.250 In practice, however, we often use a different working definition that identifies certain Ruby methods as metaprogramming methods.

00:06:42.290 The classic one is 'evil', but also methods like 'define_method,' 'method_missing,' 'instance_eval,' and even 'send'—depending on where you draw your boundaries.

00:06:47.870 This is what I will call the 'what' view of metaprogramming. It describes these methods and how to use them.

00:06:53.510 However, I don’t want to focus on that today. If we zoom out on that view, we start to see a problem.

00:06:59.390 This is my concern: how we see metaprogramming as a community. We often view it as a niche, fringe topic.

00:07:05.099 It's often seen as scary, magical, or irrelevant to many of us. However, for another part of the community, which includes myself, this is not how we see it.

00:07:11.110 Metaprogramming is central to Ruby and what we do with it. I believe this divide exists because, as I said, the materials out there do not answer the fundamental question: What is metaprogramming for?

00:07:16.449 So that's what I want to tackle today. To do this, we're going to consider the role that metaprogramming plays in our ecosystem.

00:07:21.550 We start with fire. We put Ruby in the fire, and Matz is feeling the heat.

00:07:28.200 Down here, we have the Ruby committers building this language we love. Up here, we have the applications, where most of us spend our time.

00:07:35.219 We are building things that pay the bills for us, using Ruby. We understand how to use it, but we don't always know its internals or care about them—unless something goes wrong.

00:07:42.230 However, this picture is incomplete. I'm missing a layer here, and that layer is the libraries.

00:07:50.150 I will use the terms 'libraries' and 'gems' interchangeably here. Libraries fill the gaps that multiple applications encounter, which the programming language itself does not provide.

00:07:58.670 In Ruby, these libraries blend almost seamlessly into the programming language itself. ActiveSupport is a classic example of that.

00:08:05.360 So what does this have to do with metaprogramming? The point here is that metaprogramming overwhelmingly lives in this middle layer.

00:08:12.020 A classic example is ActiveRecord, where all your attribute methods and association methods are created using metaprogramming.

00:08:19.320 However, it is not just Rails or ActiveRecord; many gems use metaprogramming significantly. When you arrive at the application layer, you don't see all of this very clearly.

00:08:25.750 You might notice monkey-patching here and there, but you don’t often see a lot of metaprogramming in action.

00:08:32.610 This leads to an important question: Why?

00:08:39.050 Why is metaprogramming so prevalent in our libraries but virtually non-existent in our application code?

00:08:46.380 I think that this is an important question that hasn't really been addressed, and it might seem obvious, but the answer is more subtle.

00:08:53.720 To explore this, we need to consider another big word in my talk title: generalization.

00:09:00.650 Generalization is what libraries do; they generalize problems faced by more than one application into generic components that we can use in all our applications.

00:09:07.990 Here’s a definition from Wikipedia: Formulation of general concepts from specific instances by abstracting common properties.

00:09:14.620 We will dive into the details of abstracting common properties in the second part of this talk, but for now, I want to focus on this: What role does metaprogramming play in developing general concepts?

00:09:21.450 Because general concepts are what our libraries or gems implement for us.

00:09:28.720 To illustrate this, we need a general concept. I'm going to pick an example that I believe is relevant for everyone here: the concept of attributes.

00:09:36.440 To understand the role metaprogramming plays in the concept of attributes, let’s take a cue from genetics.

00:09:43.360 If there are any biologists in the room, I apologize in advance. In genetics, to understand what role a gene plays in a mouse's DNA, one technique is to conduct a knockout experiment.

00:09:50.490 You knock out the gene and then observe the differences between the mouse with the gene and the mouse without it.

00:09:57.890 So, let’s do the same with the concept of attributes by removing metaprogramming and observing the results.

00:10:05.150 This is our library example. The library gem has a single class called Model, which has an initializer that sets a blank hash of attributes.

00:10:12.720 It has a single class method called 'define_attribute,' which takes an attribute name and creates both a setter and a getter. The getter fetches the value for that key in the hash while the setter sets the value.

00:10:19.200 This is a very simple setup. Now, we’ll switch to the application perspective implementing this simple library.

00:10:25.030 We create a class called Talk, which inherits from Model, and we call 'define_attribute' twice for two attributes: title and abstract.

00:10:32.290 We create a talk, set the title, and retrieve it back. Similarly, we set the abstract and get it back. This is very simple stuff.

00:10:39.110 The key point here is that the abstraction is transparent. It's invisible. In our example, we use the word 'attribute' when defining the attributes, but once they are defined, we don't see that abstraction at work.

00:10:45.960 Now, let’s switch to the knockout case, which is without metaprogramming.

00:10:53.340 In this scenario, we have the same class Model; it initializes the same way with a blank hash, but we can no longer define methods dynamically.

00:10:59.050 So instead, the only thing we can do is define two instance methods: get_attribute and set_attribute.

00:11:06.150 They perform the same functions as before but require the name of the attribute to be passed in to fetch or set the value from the hash.

00:11:12.880 This forms our library example without metaprogramming. Now, if we check the application that implements this library, it again inherits from Model.

00:11:20.050 We still create a talk, but since we can't define the attributes dynamically, we allow users to set or get anything they want.

00:11:27.500 In this case, we create a talk and call 'set_attribute', passing the name of the attribute 'title' and its value, and we can get it back the same way.

00:11:34.380 I want to highlight that this example is really simple, but there are significant subtleties. The abstraction is now visible; the abstraction is literally the term 'attribute.'

00:11:41.970 When you want to set or get an attribute, you need to call 'set_attribute' or 'get_attribute'. The words 'set_attribute' and 'get_attribute' are right there.

00:11:48.910 As for the attribute names themselves, they become second-class citizens, as we're passing them as arguments to these methods.

00:11:55.940 Now, let's summarize the results of our little experiment between the application and library when comparing the cases with and without metaprogramming.

00:12:03.850 In our control case—with metaprogramming—in the application, the abstraction was invisible. We simply call attributes directly without thinking about the abstraction involved.

00:12:10.440 In this situation, we can say that the library speaks in the language of the domain, which is Talk, Title, and Abstract.

00:12:17.310 However, in the knockout case—without metaprogramming—the abstractions become visible. You have to specifically call out those attributes when interacting with them.

00:12:24.530 Here, the library speaks in the language of the abstraction, making it clear you have to work with the concept of attributes explicitly.

00:12:31.250 Now, turning to the library perspective, the comparison was simple. Our library was really basic in its implementation.

00:12:38.380 But in the knockout case without metaprogramming, the unknowns cannot refer to code; therefore, we must pass the name into the getter method, which then passes it to the hash.

00:12:45.540 In general, this makes the knockout case without metaprogramming more manageable to understand, given there is no hidden abstraction.

00:12:52.440 But if that's the case, why not eliminate metaprogramming entirely? Why would libraries and gems employ it when it arguably complicates things?

00:12:59.550 The context for this question isn't straightforward. You could implement something like ActiveRecord without metaprogramming, but it would look like a mess.

00:13:07.290 You would have to call 'get_attribute' and 'get_association' for every interaction, which is cumbersome and obscures the domain language.

00:13:14.520 For the application developer, these variables become known. You can define them without metaprogramming. However, for a library, they are essentially unknown.

00:13:21.250 This is where metaprogramming becomes invaluable for gem authors. It allows you to speak the language of the domain effectively.

00:13:27.800 Metaprogramming levels the playing field by enabling libraries to translate unknowns into code.

00:13:35.000 This perspective on metaprogramming is meaningful and actionable: it provides a working framework to understand it.

00:13:42.800 For instance, the unknowns at the library level deal with column names and tables, while the unknowns at the application level tend to be user data.

00:13:49.570 You, as an application developer, don't know what kind of user input will come through. You may not want to translate user input into code.

00:13:56.230 This clarifies the divide between applications and libraries: for libraries, metaprogramming serves as an enabler, while for applications, it can be viewed as an obstacle.

00:14:03.360 Moreover, we can see issues with our working definition of metaprogramming as a collection of methods.

00:14:10.000 Our assumption was that using any of these methods qualifies as metaprogramming, but I want to illustrate why this assumption can lead to misunderstandings.

00:14:16.890 Let's examine the use of 'eval': if we write 'eval foo,' our understanding of metaprogramming implies it is metaprogramming.

00:14:24.010 But what does this evaluate to? It simply calls the method for the variable 'foo' without translating unknowns into code.

00:14:31.100 'eval' may seem like metaprogramming, but it falls into a category of its own known as automatic programming.

00:14:37.560 Now consider the case when we append a string to 'foo': if we see 'foo = #{foo}', that still doesn’t change the situation.

00:14:44.150 The only way to make something unknown is to wrap it in a method; we name this method 'foo_gal'. It takes a string and appends it to foo.

00:14:51.920 This method still earns the title of metaprogramming as it is translating unknowns into code.

00:14:58.470 Using 'defined_writer' is similar to the second half of 'define_attribute.' Here, we create a method with 'defined_method'.

00:15:06.090 While its use of 'define_method' qualifies it as metaprogramming according to the original working definition, it does not actually translate unknowns into code.

00:15:12.650 Thus, we need a definition that differentiates the context of whether the attribute name is known or unknown.

00:15:20.125 For example, if we use 'auto_writer,' by passing a string or a symbol, those are generally known values.

00:15:27.240 In contrast, if we pass in an unknown value, it would accurately qualify as metaprogramming. So we can visualize these results in terms of our definitions.

00:15:34.510 The original 'what' definition encompasses numerous metaprogramming methods, and on the other side, the knockout does not.

00:15:41.140 As we introduce additional data points, we see conflicts, such as the eval example where it shouldn’t belong here.

00:15:48.300 Thus, we will establish a new definition grounded in the idea of 'how' instead of merely 'what'.

00:15:55.200 Therefore, we define metaprogramming based on its capacity to convert unknowns into code.

00:16:01.600 This revised definition will considerably enrich our understanding of metaprogramming.

00:16:09.720 Now we can explore how to apply this new understanding and how libraries can use metaprogramming to solve generic problems.

00:16:17.680 The second half of my talk is about generic software. This concept isn't my own; it comes from Jeremy Evans, the author of several gems including the ORM 'Sequel.'

00:16:25.660 In a talk from 2012 about the development of Sequel, he mentioned that one of the best ways to write flexible software is by writing generic software.

00:16:34.360 Instead of designing a single API that completely handles a specific case, you can create multiple APIs that handle smaller, more generic parts of that use case.

00:16:41.340 Then handling the entire case becomes simply a matter of gluing those pieces together.

00:16:47.840 Now, let's build an example to illustrate this concept—one about equality. In Ruby, there are several kinds of equality.

00:16:54.560 We will use double equals equality, which checks if two objects with similar content are equal.

00:17:01.630 For instance, two strings that have the same characters will be considered equal.

00:17:09.720 To provide an example, I didn't go to Sequel this time; instead, we are going to Rails.

00:17:17.780 After searching through Rails code, we found a test file with two classes intertwined with our topic.

00:17:24.900 The first class is called Address, where an 'add_reader' method takes street, city, and country.

00:17:32.360 In addition, there's an equals method that compares the street, city, and country attributes of two Address objects.

00:17:40.200 The other class, GPSLocation, captures latitude and longitude, with its own equality check.

00:17:46.830 Now, let’s align these two classes to extract similarities that will help us build our generic solution.

00:17:54.080 The differences are trivial, so we will focus on code that checks equality by comparing the attributes.

00:18:03.180 We can create an array of keys representing attributes in both classes and then utilize the all method to verify equality.

00:18:11.510 Next, we will create a method called 'equalize,' distinguishing keys as our unknowns across the classes.

00:18:18.410 The use of defined_method allows us to dynamic create methods based on our array of keys.

00:18:25.920 This, however, falls under the previous working definition of metaprogramming, as it converts unknowns into functionalities.

00:18:33.170 No change occurs aside from our extraction of commonality using the module called Equalizer.

00:18:40.870 We have successfully derived a gem that allows us to call equality checks for both classes through a unified, generic interface.

00:18:48.460 This method now provides a more efficient approach and makes our code reusable across different objects with minimal adjustments.

00:18:56.660 We can build on this abstraction further, creating methods for inspecting attributes, returning and checking hashes associated with attributes.

00:19:03.480 This modular approach allows for easy further development and enhancements to our applications, striking a balance of performance and readability.

00:19:09.860 Lastly, what I want to stress is that all of the concepts we've talked about fall under the broader umbrella of generic software.

00:19:17.290 Dry Equalizer is a gem built upon these principles, implicating that we have already implemented much of this work before.

00:19:23.860 In addition, taking a glance at the DryRB ecosystem demonstrates how these components harmoniously interact.

00:19:30.960 As we reach the end of this talk, let's return to the title, 'Metaprogramming for Generalists.'

00:19:37.360 The term generalist often refers to someone with broad knowledge across fields, but I defined it differently.

00:19:43.920 In my interpretation, a generalist is someone willing to dig deeply into core concepts that address various problems—like attributes and equality.

00:19:50.700 To summarize, we need more community members to become these generalists who can build and share generic software.

00:19:57.350 By solving larger, generic problems, we make specific instances easier to handle.

00:20:04.690 This is the magic of metaprogramming—the metaphor that forms the backbone of the Ruby ecosystem.

00:20:11.350 That's what makes Ruby the powerful language that it is.

00:20:19.200 Thank you!

EuRuKo 2018