Migrating To Clojure. So Much Fn

Talks

Jan Stępień

@jstepien

#clojure

#migration

#software-architecture

#functional-programming

#ruby

#java-virtual-machine-jvm

#microservices

Migrating To Clojure. So Much Fn

by Jan Stępień

In this talk, Jan Stępień from stylefruits discusses the migration of their technology stack from Ruby on Rails to Clojure, highlighting the complexities involved in such a transition.

Key Points Discussed:
- Background of stylefruits: An online fashion and interior design hub that serves five million monthly visitors across several European countries, originally built entirely on Ruby on Rails.
- Initial Challenges: The company faced problems with its monolithic application built in Ruby, which resulted in poor separation of concerns and increasingly painful scaling issues.
- Introduction of Clojure: An engineer proposed using Clojure for a new project, which led to initial skepticism but eventually demonstrated Clojure's viability due to its successful deployment.
- Architecture Shift: The team implemented a feature freeze to rework the application architecture gradually. They initially aimed for a hybrid system, running Clojure alongside Ruby, which required tackling data access complexities.
- Key Concepts of Clojure: Jan emphasized the importance of immutability in Clojure, explaining how it simplifies data handling and application state management through the use of widgets—self-contained components with separated concerns.
- Template Management: The Enlive library was introduced as a solution to their previous templating issues, allowing team members to work in HTML without the complexity of templating languages.
- Development Benefits: Clojure's REPL over TCP/IP greatly enhances developer productivity, allowing real-time code changes and inspection directly in a live environment, something that greatly benefited their development process.
- Community Support: Jan praised the Clojure community for its professionalism and the availability of quality libraries, albeit noting that hiring Clojure developers can be challenging but attracts curious candidates eager to learn.
- Future Vision: He expressed aspirations to develop a more microservices-oriented architecture by continuing to refine their existing codebase and progressively decoupling functions from the legacy Ruby system.

Conclusions:
- The transition to Clojure has generally been rewarding for stylefruits, providing a distraction-free environment conducive to development. While challenges remain, including performance considerations and technical debt, the process has been manageable and positively transformative.
- Jan encourages other organizations to consider Clojure for similar migrations, citing its benefits in performance and development experience.

Throughout the talk, Jan stresses the importance of gradual migration, careful planning, and the upsides of having a supportive programming community. Questions about productivity comparison with Ruby and technical nuances reveal the ongoing learning involved in such migrations.

00:00:13.759 Good afternoon, ladies and gentlemen. My name is Jan Stępień, and I work in Munich, Germany, at a company named stylefruits. We are an online fashion and interior design hub operating in five European markets: Germany, France, the Netherlands, the UK, and Poland.

00:00:20.199 To give you a sense of scale, we're dealing with 10 million monthly unique sessions, half of which are in Germany. We used to be a purely Ruby on Rails-based company, but we've been gradually migrating to Clojure, and that's what I'd like to talk to you about today.

00:00:27.119 Let's start at the beginning. In 2008, the first version of stylefruits was built in Ruby on Rails 2.1. It took three months to complete. It's customary to compare software architecture to various dishes. We've heard about "spaghetti code" and "baklava-like architecture" with too many layers. I think it would be fair to compare this application, which took care of everything in the company, to a famous South German and Austrian delicacy called "leberkäs." Just like leberkäs, our application was monolithic with poor separation of concerns.

00:01:00.480 If you are more familiar with Polish cuisine, a good comparison would be "pierogi." This is how our code looked. It worked quite well, and scaling issues were solved through horizontal scaling and caching. This continued until 2011 when something interesting happened. One of our engineers decided to take a risky path and implemented a new site project in this exotic niche language called Clojure. This suggestion did not initially receive a warm welcome from the head of engineering, but the argument, "Look, it's 100 lines of code. Worst case, I can rewrite it in two days," settled the issue.

00:01:41.079 As a result, the first piece of Clojure code was deployed to production. In fact, it's still running somewhere out there since 2011. It experienced roughly 15 minutes of downtime, with 10 of those minutes occurring when the entire AWS Europe was down. This set an interesting precedent for us, demonstrating that Clojure is a viable solution, maybe not just an alternative.

00:02:07.720 Returning to our main application: features piled on top of features, layers built upon layers. Things weren't improving, and the fact is, all our time was dedicated solely to feature development. The company's future was uncertain, leading to the conclusion that adding features without any time for refactoring would only worsen the situation. Load times suffered terribly, and we needed something new—a solid alternative. The idea stemmed from the observation that our pages were built from independent parts, independent chunks that did not share data and could be rendered independently.

00:02:40.799 Based on this observation, the company came up with the idea of widgets—independent chunks upon which the entire app should be based. The next step was clear: implement a feature freeze and dedicate all engineering resources to rework the application as quickly as possible. This is where things did not go as planned. The partial rewrite gained acceptance from decision-makers on the condition that it would not interfere with daily development activities. Based on this constraint, a new architecture was conceived. The idea was that the new app would run side by side with the old one, sharing code responsible for data access and business logic, serving gradually more requests as we ported pages from the old to the new app.

00:04:14.760 Unfortunately, this approach did not work out too well. The team quickly discovered that the data access code was not very useful due to its coupling with the rest of the application and its poor state. The next step was to build a new independent library for this data access. We've heard about the repository pattern in the morning—it would serve as this repository for data access. However, the situation became a bit complicated because Ruby had to handle all requests due to the complexity of our routes, which required access to the database to determine which controller should render a particular request.

00:06:04.680 This forced us, at least for the time being, to have all requests served by the Ruby application. In one of the first middlewares, we had to decide which application should respond to each request and, if necessary, forward it to the new widget application. Work began in 2012, and early in the process, another bold idea emerged: rewriting everything in Clojure. We had a piece of code running that was performing well and we liked it. We even toyed with the idea of using a library for templating called "Enlive," which I will discuss shortly.

00:06:29.440 Interestingly, the Clojure application and this data access library written in Ruby now run on a single JVM where the interoperability between JRuby and Clojure is surprisingly seamless. They just work very well together, separated only by a thin layer of code responsible for translating data structures back and forth.

00:06:52.080 Now that I've explained how we ended up in this situation, I would like to share some observations and experiences from the entire process. I will begin with the concept of immutability. This may sound counterintuitive and not very impressive; after all, you can just freeze all values. But let me show you an example: in Clojure, we can define a vector—"[1 2 3]"—and print it. Using the "conj" function, we can add "4" to the vector, resulting in "[1 2 3 4]." The original vector remains unchanged.

00:07:29.680 Now you know everything there is to know about immutability. Every single data structure and value in Clojure, with some minor exceptions, is immutable. This may not seem critical at first, but as I aim to demonstrate, it was one of the game-changers for us. Now, let's talk about those widgets I mentioned earlier. They were the cornerstone upon which the entire application was built. Essentially, widgets consist of two functions. In this hash map, you can see a simplified widget: a fetch function responsible for gathering the data necessary to render a page based on the request, environment, and a view function, which is fed with the data from the fetch function.

00:09:20.400 The view function then returns an abstract representation of the HTML of the resulting widget. This separation of concerns is vital—the fetch function interacts with data sources like databases or Redis, while the view function is supposed to be pure, returning the same output for given input arguments without altering the state of the application. This immutability simplifies reasoning about the behavior of the entire widget-based system. We do not need to worry about how data is shared between different widgets, as they cannot change anything. If they can't change it, they can't break anything.

00:10:58.400 The rendering pipeline also benefits significantly from this approach. For instance, if we apply the "map" function over a collection of widgets, we can get rendered widgets as a result. We parallelized the entire rendering process simply by replacing "map" with "pmap," a parallel version of "map" that executes in a multi-threaded environment. As a result, our pages were rendered effortlessly in parallel, which was fantastic.

00:12:14.440 Now, let's discuss templating, which was a major pain point for us. Our previous templates were huge and reflective of years of technical debt, leading us to search for something more solid. The Enlive library I mentioned was intended to address this issue. Enlive provides an abstract representation of HTML, structured as a simple hash with nested vectors. It also offers functions to transform the representation and a CSS-based selector language for simplification.

00:12:37.640 The critical point is that our templates are stored as pure HTML, without traces of a templating language or DSL. People at stylefruits responsible for front-end development are working in their natural habitat—HTML and CSS. During the application startup, Enlive translates the HTML files into the Clojure data structures, so programmers only deal with simple data structures and operate on them. The widgets fill in these data structures with information from the database, applying interpolation strings before rendering the whole page as a string to send to the user.

00:13:34.960 This approach works very well, as we've found it easier to manage. When we look at programming languages, we typically think of those available almost everywhere nowadays. The Read-Eval-Print Loop (REPL) has been a feature in Ruby for a long time, with tools like IRB and Pry. Python has also had similar options in the browser, with JavaScript REPL available in developer tools.

00:14:05.680 What makes Clojure special is "nREPL," which provides a REPL over TCP/IP. All our development machines—our application instances—expose an open nREPL port. You can connect your REPL session or editor directly to it. This setup allows you to edit code within the application environment without affecting the rest of the JVM, providing full immersion in the virtual machine's state.

00:14:35.360 You can inspect the state, modify functions at a granular level, and even load the entire test suite without needing to restart. There is no waiting time; it creates a distraction-free environment. An interesting side note is the availability of the REPL on production machines. If a problem arises, we can execute a few lines in production to quickly address issues without affecting the rest of the application. However, we try to avoid doing this unless necessary.

00:15:26.239 In terms of performance, the JVM gives us solid results. However, when performance is lacking, we have tools at our disposal to tackle issues. All the profilers and heap analyzers originally developed for Java work seamlessly with Clojure. Though the stack traces produced by Clojure may require some practice to interpret, with time you will overcome that.

00:16:08.239 One of the most wonderful aspects we've encountered is the Clojure community itself, which boasts a high level of professionalism. The quality of discussions on forums or pull requests is commendable. There's respect for backward compatibility and semantic versioning, along with a collection of focused libraries solving specific problems. Although Ruby has a broader ecosystem of libraries, Clojure offers a good selection, ensuring we have access to drivers for various databases, enhancing our problem-solving capabilities.

00:17:04.480 That said, it hasn't all been sunshine and roses. We made plenty of mistakes along the way. The application I'm talking about was our first large-scale Clojure project, and we were still not using many idiomatic practices. We were steadily eradicating various code smells, but much work remains to be done.

00:17:51.700 An interesting side effect of our migration to Clojure has been the impact on hiring. Hiring skilled software developers is challenging, and using an exotic language can compound that difficulty. However, we found that advertising closure skills in job postings helped attract open-minded candidates, eager to learn and explore new ideas. We now have an interesting mix of people in the company, bringing diverse experiences from enterprise banking, Python development, and Ruby backgrounds.

00:19:05.799 Our overarching vision was to separate these widgets into a network of microservices—each being a standalone microservice handling a single widget. This is still an aspiration, and while we have made significant improvements, we continue to clean and refine the existing codebase, getting rid of inherited code smells. Overall, we made a good decision by moving to Clojure, gaining a distraction-free development environment, robust tools, and great community support.

00:20:43.880 Currently, no new projects are being initiated in Ruby; everything is being started in Clojure. The legacy application is still operational and servicing many requests, so we're far from finishing the entire migration. However, we are gradually decoupling responsibilities from the old system, such as backend data processing, which is now in a separate Clojure-based network pipeline.

00:21:37.560 In summary, I encourage you to consider migrating to Clojure—it has been a rewarding experience for us. I have touched on many significant topics and aspects of our journey. If you have further questions or areas of interest, feel free to reach out. Thank you all very much for your attention.

00:23:08.960 Questions?

00:24:02.040 I just have a short question. You mentioned at the end about the pipeline of small Clojure apps. Are they all communicating over HTTP, or do they run in one JVM?

00:24:09.720 We mostly use AMQP on top of RabbitMQ for these purposes. I don't believe we have anything communicating internally over HTTP; it's predominantly AMQP.

00:24:19.800 Can you compare productivity? How many features can you deliver using the current approach compared to the previous one?

00:25:32.760 I think it's unfair to compare since it would contrast a large, monstrous legacy app with productivity in a brand new application without technical debt.

00:25:38.080 You transitioned Ruby into essentially a JVM environment, right?

00:26:15.600 If I recall correctly, about a year and a half ago we ran 50 or 60 Ruby servers to manage the load, and now we’re operating roughly 12 JVM instances on the same AWS boxes. Meanwhile, traffic has doubled, and those machines now serve the bulk of the request load.

00:26:46.560 I think this transition is a win in terms of performance, and from a maintenance perspective, deployment has changed slightly.

00:27:00.040 Both Ruby and Clojure files are packaged in the same archive; we build a unified jar that contains everything.

00:27:11.560 When we launch this jar, it initiates the Jetty server, and that is the extent of our setup.

00:27:21.760 When you first considered splitting the whole page into widgets, did you consider generating HTML within the browser?

00:27:31.560 If I recall correctly, that solution was unacceptable for SEO purposes, but I might not be entirely up to date with current capabilities.

00:27:50.360 Given the significant differences in programming languages, how does it feel to switch between Ruby and Clojure, especially since they require different thinking styles?

00:28:17.840 Although there are indeed profound differences, both languages are dynamically typed. While Ruby emphasizes strong object-oriented principles, it also incorporates many functional elements.

00:28:56.160 For newcomers, switching from Ruby to Clojure can be challenging at first due to the functional programming idioms, but code reviews significantly help with this transition. If you observe our contemporary Ruby code, you'll notice a strong influence of functional programming.

00:29:43.840 Regarding your experience writing code in both languages, what was the main reason behind your decision to switch from Ruby to Clojure? Was the Ruby codebase just not capable of scaling efficiently?

00:30:57.760 I believe if the code had been recovered sooner or if the rescue team had been more competent, it could still be using Rails. Ultimately, we were tired of the limitations, and Clojure allows for simpler modeling in our thinking.

00:31:43.560 In terms of hiring, you mentioned some drastic improvements in your candidate filtering process after introducing Clojure skills in job advertisements compared to when you only stated Ruby requirements.

00:32:35.200 Comparison might not be entirely valid since we didn't have Clojure advertisements prior to it.

00:33:03.640 I still appreciate Ruby. If I find myself needing to solve a problem in a quick way, Ruby will almost certainly be in my top three choices.

00:33:26.960 The syntax of Clojure can also be daunting at first, especially with all the parentheses. How do you cope with this complexity given its Lisp nature?

00:34:01.480 In short, I don't notice parentheses anymore. It's just blocks of code once you’re accustomed to it. And with the aid of suitable editors like Vim or Emacs equipped with plugins, it becomes trivial.

00:34:35.480 I've even noticed that people find it easier to read with multispectral parentheses coloring. With practice, you won't find it challenging to learn.

00:35:05.200 To clarify further, when switching languages, you might find you need to adjust, but, ultimately, with the right strategies and mindset, you can transition smoothly.

00:35:32.960 Thank you everyone for your time and all insightful questions. If there are any more inquiries or if you'd like to continue this conversation, I'm available!

Jan Stępień

@jstepien

wroclove.rb 2014