To Clojure and back: writing and rewriting in Ruby

by Phill MV

In the talk titled "To Clojure and back: writing and rewriting in Ruby," presented by Phill MV at RubyConf 2016, the speaker recounts the journey of transitioning from Ruby to Clojure while building a software service at App Canary.

Key Points:

Act 1 - State of Grace: The initial excitement of utilizing Clojure, described as a "breath of fresh air," emerges from a desire to expand skills and move beyond the perceived stagnation of Ruby. The use of Datomic, a unique database, facilitated this transition.
Act 2 - The Fall: The challenges of adapting to a new ecosystem, where Clojure’s functional programming paradigm contrasts with Ruby's, are discussed. Issues like poor architectural decisions and difficulties in structuring apps lead to complications.
Act 3 - The Wilderness: Despite recognizing the risks of rewrites, the speaker reflects on the decision to rewrite their application due to frustrations with the initial architecture and ecosystem.
Act 4 - Reconciliation: The experience of re-immersing into Ruby after using Clojure allowed for significant development shifts. Key insights on applying immutability and creating value objects in Ruby emerge from this experience.

Significant Examples:

Clojure vs Ruby: Clojure’s immutability and functional programming principles streamline code and enhance predictability, contrasting with the complexities of mutable state in Ruby. An example discussed is the assurance around variable mutations in Clojure, enhancing code integrity.
The Rewrite Process: The final transition back to Ruby saw an increase in code lines but a decrease in complexity and improved performance by switching to PostgreSQL. The speaker notes that the rewrite was conducted more efficiently than the initial build.

Important Conclusions and Takeaways:

The journey underlines the value of embracing new technologies while recognizing the depth of knowledge gained from them. Although the immersion in Clojure was beneficial, the speaker advocates for appreciating the Ruby ecosystem and integrating lessons learned about immutability and functional programming.
A warning against hastily implemented distributed systems points to the advantages of monolithic architectures for simplified development processes.
The importance of developer satisfaction and the enjoyment of coding are paramount, emphasizing the need for balance between cleverness and fulfillment in software development.

00:00:16.720 Hi everyone, thank you so much for coming. The title of my talk is very evocative.

00:00:22.680 My name is Philip Mosa, but you can find me on various social media as Phil MV, as well as on Twitter.

00:00:28.640 I am one of the co-founders of Ruby.com, where we host the Ruby Security Advisory Database. If there's anyone in the audience who maintains gems, I would love to talk to you afterwards about how you handle security disclosures.

00:00:42.280 For my day job, I run a startup called App Canary. App Canary notifies you whenever you are running a vulnerable package in your servers or applications. We spend a lot of time thinking about whether you are running vulnerable software.

00:00:53.559 This talk is ultimately about a series of technical decisions we made while building the service at App Canary. To best understand our thought process, we should start by recounting the beginning of our incredible journey.

00:01:12.360 Back in 2012, I began a consultancy with a friend of mine, Max. We specialized in penetration tests, security code audits, MVPs, business automation, and advising teams on how to improve their software development processes.

00:01:29.000 One benefit of being a consultant is having a lot of control over your schedule. By November 2014, I had just spent two months working 12-hour days on the Toronto mayoral elections, and my co-founder Max had just enjoyed a magical summer at the Recurse Center.

00:01:40.920 As a result, I was really burnt out, while Max was looking for a challenge. We stumbled upon a market opportunity that could truly use our skills, and we decided to build a product.

00:01:53.680 We had some experience in this problem domain as we had previously built a free service called Gem Canary a few years earlier. To put it simply, an advisory may have many vulnerabilities, a vulnerability can exist in many packages, and each package can have multiple versions. We needed to keep track of all of this to inform users whether they were running a version that should not be in use.

00:02:22.840 As we looked around, Max suggested we use a system called Datomic. If you've never heard of Datomic, it's a fascinating key-value graph database where, instead of SQL, you write data logs, which is a kind of programming language.

00:02:35.239 On top of this, Datomic provides a free point-in-time database feature, meaning you can revert to any previous state of your data. Essentially, nothing ever gets deleted, making it a really cool tool to work with.

00:02:54.879 Given our struggles with the free service we built, we thought this technology might be really useful. Unfortunately, all the client libraries for Datomic had issues, but it worked remarkably well with Clojure, so we decided to use Clojure.

00:03:14.319 For those of you unfamiliar, Clojure is a functional programming language that features immutable data structures, providing you with asynchronous primitives for free. It runs on the JVM, which is touted to be web-scale.

00:03:39.519 Isn't it about time we all learned a Lisp? Shouldn't Lisp be this profound, enlightening experience that alters how you program for the rest of your life? Personally, I had been writing Ruby for five years, having built a dozen apps. All these reasons were compelling, but I was also driven by fear: was I stagnating?

00:04:06.480 When I started with Ruby, it represented cutting-edge web tech, but I worried it wasn't the same anymore. What happens to programmers at the age of 35? Do they vanish? I didn't want to spend the rest of my career writing JavaScript. The idea of investing in a new technology to expand my skills was appealing. So we thought, why not just go for it? We were going to run out of money in six months anyway.

00:04:32.280 We built it in Clojure, and it was a fantastic experience. For those already familiar with Clojure, I may be using the incorrect terminology, so please bear with me. Both Clojure and Ruby handle truthy values where things are either nil or false, and they both spend a lot of time dealing with hashes of symbols.

00:05:03.039 They both incorporate fun metaprogramming constructs. Now, why is Clojure appealing? The primary reason is functional programming. In Clojure, functions are the basic semantic building blocks. You're encouraged to encapsulate behavior into the smallest practical units.

00:05:21.360 In this example, we are filtering out all the even integers from an array by applying an even function. If you look closely, you might find it somewhat familiar. There's an old joke that Ruby is an acceptable Lisp, and a 2005 blog post supports this argument.

00:05:41.199 However, comparing the two, the Clojure code exhibits the more flexible behavior: the filter function operates without needing to know the specifics of the data structures. This shows the power of Clojure's functional capabilities—behavior can be combined without being tied to the properties of the numbers or items in the array.

00:06:19.520 When everything is a function, it often gives you the freedom to isolate and change behaviors without affecting the original structure. This greatly simplifies the development process. Another notable feature is immutability, which historically was quite costly until breakthroughs in 2002 made it feasible to include in common programming.

00:06:38.240 Immutability means that the state cannot change; any insert, update, or deletion results in a new structure. This is crucial since mutation can lead to complex behaviors in your code that can produce unforeseen bugs. In a language like Ruby, there's always uncertainty about whether an operation will mutate or copy an object.

00:07:24.919 For instance, if you have a list of strings and you assign it to another variable, the results of appending new values can be tricky to predict. Ruby developers deal with pass-by-reference or pass-by-value concepts, which complicates object mutations considerably.

00:07:51.040 The only way to ensure that objects are not modified is to create extensive workarounds in your code, which leads me to the conclusion that Ruby's mutability complicates things unnecessarily. In contrast, with immutability, the flow of state through your application becomes clearer and more predictable.

00:08:10.480 Take an example from an old app of ours: a route that processes a user and a database. The way my code is indented clearly shows what cannot affect the variables outside their lexical scope. The assurance that outer variables cannot be tampered with allows me to feel confident about code integrity.

00:08:39.840 On a side note, let me illustrate why Datomic is appealing. Let's say your boss comes up to you asking for a report based on last year's data. In traditional databases, depending on your design, it can be a nightmare to execute. With Datomic, I can grab the database’s previous state and rerun my report, and it works seamlessly.

00:08:59.920 Despite our enthusiasm for this new technology, I found myself reluctant to dive into user authentication and cookie management anew when we already had this experience building websites. So we built an API that communicates with Datomic, along with a Rails frontend for mundane tasks like user management and payment processing.

00:09:38.640 We began working on this in February 2015, and by May, to our astonishment, we got accepted into Y Combinator. This eased our financial concerns, allowing us to release our product into production by July. After moving backs to Toronto in October, we worked tirelessly to add features and address the actual demands of paying customers.

00:10:37.360 However, we soon encountered significant issues. We made poor architectural decisions, underestimated our domain's complexity, and faced separate deployments, introducing unnecessary overhead. We ended up spending an exorbitant amount of time fixing simple issues, resulting in a death by a thousand paper cuts.

00:11:30.960 To give you an idea, structuring large apps in Clojure can be less intuitive compared to Rails, which provides more hand-holding. While coding in Clojure is fun and often makes you feel clever, it can also result in code that is hard to read due to excessive expression and various subtleties of the language.

00:12:23.440 It also becomes challenging to pinpoint errors in stack traces when using anonymous functions, potentially leading to complicated debugging experiences. Documentation is often less accessible, and the JVM overhead complicates rapid development.

00:12:32.879 My application's slow boot time added more layers of frustration, as deep integration became necessary. The requirement of packaging all dependencies into one single object meant managing significant payloads, which created hurdles for minor fixes during deployments.

00:12:59.280 With all that being said, I have a natural fluency in Ruby that I unfortunately lacked in Clojure, leading to considerable frustration. A comment I found on Hacker News resonated with me, describing the Clojure ecosystem as user-hostile. It became clear that developer happiness is not a priority in that community.

00:13:46.960 Fast forward to May of this year; we recognized the need to refactor our application. After discussing it with friends, I was urged to consider a complete rewrite. This push became a pivotal moment, making it difficult to dismiss such an option any longer.

00:14:17.200 However, we knew that rewrites are fraught with risk. If successful, they often go unnoticed, but we were tired of fighting the ecosystem. We aimed to eliminate unnecessary components to streamline operations and refocus on what truly matters.

00:14:58.360 We built an API, and the process involved reviewing each step meticulously, resulting in complex workflows that ultimately made it impractical for our small team to manage efficiently. This led to unforeseen complications, with microservices creating a distributed system we hadn't intended, ultimately causing more work and less efficiency.

00:15:47.440 Dave McKinley, an early guide at Etsy, suggests focusing on a few hard things at once that truly drive a business. This lesson had a profound impact when realizing we were spending effort on building an object relational mapper for Datomic—something that offered little value to our clients.

00:16:31.880 Reducing risk is vital, especially in the startup world where every second and resource counts. As a small operation, we had the freedom to stall, which isn't possible for larger companies dealing with looming deadlines and customer pressures.

00:17:06.000 By mid-October this year, we had 8,300 lines of code in Clojure. The next day, after the rewrite, we had 8,700 lines of Ruby. The rewrite process took four months—a vast improvement when compared to the year and a half it took the first time when we were inexperienced.

00:17:29.080 While it was demoralizing to work on a system you cannot release, when the product finally shipped, it felt immensely relieving. Turning our application into one that leverages PostgreSQL's strengths made it more effective and performant. Generally, our application is now anywhere from 50% to 100% faster, a testament to the effort we put into the new data model.

00:18:29.600 Reflecting on the experience, I realized a sub-genre of technical blog posts yelling, 'Look, we rewrote our app from shiny new tech, and everything is faster!' If you didn't learn from your first attempt and improve performance, what was the purpose behind your efforts?

00:19:09.680 Returning to Ruby, I found that my approach had shifted dramatically during my time with Clojure. The immersion in functional programming crystallized thoughts I previously had about Ruby. The evolution in my understanding of Rails highlighted the importance of developing fat models and lean controllers.

00:19:44.560 I embraced the idea of creating value objects as well as applying immutability where possible. This practice of introducing value objects helps streamline the management of state within Ruby, ensuring that objects intended to represent states are immutable. By making this practice a habit, I found that my code became clearer and easier to manage.

00:20:57.519 Moreover, I've become far more aware of state flow through my applications. This also led me to create distinct manager objects that separate concerns, alleviating the role of controllers and models while maintaining clarity around object state.

00:21:43.680 To encapsulate my experience succinctly, I urge you all to dive into Clojure. While it may not be the most transcendental experience, the knowledge gained from the language can be beneficial. It's worth embracing its lessons even if my current focus veers toward Ruby.

00:22:20.039 When considering new tech, avoid building distributed systems as long as feasible. Embracing a monolithic architecture can simplify the development process significantly until your application demands scaling.

00:23:03.320 Before you reach for a new tool, exhaust use of those you already know well. These familiar tools can often save time and frustration when deadlines loom. Lastly, the experience I derive from writing code must be enjoyable and fulfilling, not solely the result of cleverness but genuinely rewarding.

00:23:43.629 In closing, cherish what you have within the Ruby ecosystem, and bring those values to any new communities you engage with. This is the journey and story of how we built a service that helps notify users about vulnerable software.

00:24:09.780 My name is Philip Mosa, and thank you for listening!