00:00:13.759
Good afternoon, ladies and gentlemen. My name is Jan Stępień, and I work in Munich, Germany, at a company named stylefruits. We are an online fashion and interior design hub operating in five European markets: Germany, France, the Netherlands, the UK, and Poland.
00:00:20.199
To give you a sense of scale, we're dealing with 10 million monthly unique sessions, half of which are in Germany. We used to be a purely Ruby on Rails-based company, but we've been gradually migrating to Clojure, and that's what I'd like to talk to you about today.
00:00:27.119
Let's start at the beginning. In 2008, the first version of stylefruits was built in Ruby on Rails 2.1. It took three months to complete. It's customary to compare software architecture to various dishes. We've heard about "spaghetti code" and "baklava-like architecture" with too many layers. I think it would be fair to compare this application, which took care of everything in the company, to a famous South German and Austrian delicacy called "leberkäs." Just like leberkäs, our application was monolithic with poor separation of concerns.
00:01:00.480
If you are more familiar with Polish cuisine, a good comparison would be "pierogi." This is how our code looked. It worked quite well, and scaling issues were solved through horizontal scaling and caching. This continued until 2011 when something interesting happened. One of our engineers decided to take a risky path and implemented a new site project in this exotic niche language called Clojure. This suggestion did not initially receive a warm welcome from the head of engineering, but the argument, "Look, it's 100 lines of code. Worst case, I can rewrite it in two days," settled the issue.
00:01:41.079
As a result, the first piece of Clojure code was deployed to production. In fact, it's still running somewhere out there since 2011. It experienced roughly 15 minutes of downtime, with 10 of those minutes occurring when the entire AWS Europe was down. This set an interesting precedent for us, demonstrating that Clojure is a viable solution, maybe not just an alternative.
00:02:07.720
Returning to our main application: features piled on top of features, layers built upon layers. Things weren't improving, and the fact is, all our time was dedicated solely to feature development. The company's future was uncertain, leading to the conclusion that adding features without any time for refactoring would only worsen the situation. Load times suffered terribly, and we needed something new—a solid alternative. The idea stemmed from the observation that our pages were built from independent parts, independent chunks that did not share data and could be rendered independently.
00:02:40.799
Based on this observation, the company came up with the idea of widgets—independent chunks upon which the entire app should be based. The next step was clear: implement a feature freeze and dedicate all engineering resources to rework the application as quickly as possible. This is where things did not go as planned. The partial rewrite gained acceptance from decision-makers on the condition that it would not interfere with daily development activities. Based on this constraint, a new architecture was conceived. The idea was that the new app would run side by side with the old one, sharing code responsible for data access and business logic, serving gradually more requests as we ported pages from the old to the new app.
00:04:14.760
Unfortunately, this approach did not work out too well. The team quickly discovered that the data access code was not very useful due to its coupling with the rest of the application and its poor state. The next step was to build a new independent library for this data access. We've heard about the repository pattern in the morning—it would serve as this repository for data access. However, the situation became a bit complicated because Ruby had to handle all requests due to the complexity of our routes, which required access to the database to determine which controller should render a particular request.
00:06:04.680
This forced us, at least for the time being, to have all requests served by the Ruby application. In one of the first middlewares, we had to decide which application should respond to each request and, if necessary, forward it to the new widget application. Work began in 2012, and early in the process, another bold idea emerged: rewriting everything in Clojure. We had a piece of code running that was performing well and we liked it. We even toyed with the idea of using a library for templating called "Enlive," which I will discuss shortly.
00:06:29.440
Interestingly, the Clojure application and this data access library written in Ruby now run on a single JVM where the interoperability between JRuby and Clojure is surprisingly seamless. They just work very well together, separated only by a thin layer of code responsible for translating data structures back and forth.
00:06:52.080
Now that I've explained how we ended up in this situation, I would like to share some observations and experiences from the entire process. I will begin with the concept of immutability. This may sound counterintuitive and not very impressive; after all, you can just freeze all values. But let me show you an example: in Clojure, we can define a vector—"[1 2 3]"—and print it. Using the "conj" function, we can add "4" to the vector, resulting in "[1 2 3 4]." The original vector remains unchanged.
00:07:29.680
Now you know everything there is to know about immutability. Every single data structure and value in Clojure, with some minor exceptions, is immutable. This may not seem critical at first, but as I aim to demonstrate, it was one of the game-changers for us. Now, let's talk about those widgets I mentioned earlier. They were the cornerstone upon which the entire application was built. Essentially, widgets consist of two functions. In this hash map, you can see a simplified widget: a fetch function responsible for gathering the data necessary to render a page based on the request, environment, and a view function, which is fed with the data from the fetch function.
00:09:20.400
The view function then returns an abstract representation of the HTML of the resulting widget. This separation of concerns is vital—the fetch function interacts with data sources like databases or Redis, while the view function is supposed to be pure, returning the same output for given input arguments without altering the state of the application. This immutability simplifies reasoning about the behavior of the entire widget-based system. We do not need to worry about how data is shared between different widgets, as they cannot change anything. If they can't change it, they can't break anything.
00:10:58.400
The rendering pipeline also benefits significantly from this approach. For instance, if we apply the "map" function over a collection of widgets, we can get rendered widgets as a result. We parallelized the entire rendering process simply by replacing "map" with "pmap," a parallel version of "map" that executes in a multi-threaded environment. As a result, our pages were rendered effortlessly in parallel, which was fantastic.
00:12:14.440
Now, let's discuss templating, which was a major pain point for us. Our previous templates were huge and reflective of years of technical debt, leading us to search for something more solid. The Enlive library I mentioned was intended to address this issue. Enlive provides an abstract representation of HTML, structured as a simple hash with nested vectors. It also offers functions to transform the representation and a CSS-based selector language for simplification.
00:12:37.640
The critical point is that our templates are stored as pure HTML, without traces of a templating language or DSL. People at stylefruits responsible for front-end development are working in their natural habitat—HTML and CSS. During the application startup, Enlive translates the HTML files into the Clojure data structures, so programmers only deal with simple data structures and operate on them. The widgets fill in these data structures with information from the database, applying interpolation strings before rendering the whole page as a string to send to the user.
00:13:34.960
This approach works very well, as we've found it easier to manage. When we look at programming languages, we typically think of those available almost everywhere nowadays. The Read-Eval-Print Loop (REPL) has been a feature in Ruby for a long time, with tools like IRB and Pry. Python has also had similar options in the browser, with JavaScript REPL available in developer tools.
00:14:05.680
What makes Clojure special is "nREPL," which provides a REPL over TCP/IP. All our development machines—our application instances—expose an open nREPL port. You can connect your REPL session or editor directly to it. This setup allows you to edit code within the application environment without affecting the rest of the JVM, providing full immersion in the virtual machine's state.
00:14:35.360
You can inspect the state, modify functions at a granular level, and even load the entire test suite without needing to restart. There is no waiting time; it creates a distraction-free environment. An interesting side note is the availability of the REPL on production machines. If a problem arises, we can execute a few lines in production to quickly address issues without affecting the rest of the application. However, we try to avoid doing this unless necessary.
00:15:26.239
In terms of performance, the JVM gives us solid results. However, when performance is lacking, we have tools at our disposal to tackle issues. All the profilers and heap analyzers originally developed for Java work seamlessly with Clojure. Though the stack traces produced by Clojure may require some practice to interpret, with time you will overcome that.
00:16:08.239
One of the most wonderful aspects we've encountered is the Clojure community itself, which boasts a high level of professionalism. The quality of discussions on forums or pull requests is commendable. There's respect for backward compatibility and semantic versioning, along with a collection of focused libraries solving specific problems. Although Ruby has a broader ecosystem of libraries, Clojure offers a good selection, ensuring we have access to drivers for various databases, enhancing our problem-solving capabilities.
00:17:04.480
That said, it hasn't all been sunshine and roses. We made plenty of mistakes along the way. The application I'm talking about was our first large-scale Clojure project, and we were still not using many idiomatic practices. We were steadily eradicating various code smells, but much work remains to be done.
00:17:51.700
An interesting side effect of our migration to Clojure has been the impact on hiring. Hiring skilled software developers is challenging, and using an exotic language can compound that difficulty. However, we found that advertising closure skills in job postings helped attract open-minded candidates, eager to learn and explore new ideas. We now have an interesting mix of people in the company, bringing diverse experiences from enterprise banking, Python development, and Ruby backgrounds.
00:19:05.799
Our overarching vision was to separate these widgets into a network of microservices—each being a standalone microservice handling a single widget. This is still an aspiration, and while we have made significant improvements, we continue to clean and refine the existing codebase, getting rid of inherited code smells. Overall, we made a good decision by moving to Clojure, gaining a distraction-free development environment, robust tools, and great community support.
00:20:43.880
Currently, no new projects are being initiated in Ruby; everything is being started in Clojure. The legacy application is still operational and servicing many requests, so we're far from finishing the entire migration. However, we are gradually decoupling responsibilities from the old system, such as backend data processing, which is now in a separate Clojure-based network pipeline.
00:21:37.560
In summary, I encourage you to consider migrating to Clojure—it has been a rewarding experience for us. I have touched on many significant topics and aspects of our journey. If you have further questions or areas of interest, feel free to reach out. Thank you all very much for your attention.
00:23:08.960
Questions?
00:24:02.040
I just have a short question. You mentioned at the end about the pipeline of small Clojure apps. Are they all communicating over HTTP, or do they run in one JVM?
00:24:09.720
We mostly use AMQP on top of RabbitMQ for these purposes. I don't believe we have anything communicating internally over HTTP; it's predominantly AMQP.
00:24:19.800
Can you compare productivity? How many features can you deliver using the current approach compared to the previous one?
00:25:32.760
I think it's unfair to compare since it would contrast a large, monstrous legacy app with productivity in a brand new application without technical debt.
00:25:38.080
You transitioned Ruby into essentially a JVM environment, right?
00:26:15.600
If I recall correctly, about a year and a half ago we ran 50 or 60 Ruby servers to manage the load, and now we’re operating roughly 12 JVM instances on the same AWS boxes. Meanwhile, traffic has doubled, and those machines now serve the bulk of the request load.
00:26:46.560
I think this transition is a win in terms of performance, and from a maintenance perspective, deployment has changed slightly.
00:27:00.040
Both Ruby and Clojure files are packaged in the same archive; we build a unified jar that contains everything.
00:27:11.560
When we launch this jar, it initiates the Jetty server, and that is the extent of our setup.
00:27:21.760
When you first considered splitting the whole page into widgets, did you consider generating HTML within the browser?
00:27:31.560
If I recall correctly, that solution was unacceptable for SEO purposes, but I might not be entirely up to date with current capabilities.
00:27:50.360
Given the significant differences in programming languages, how does it feel to switch between Ruby and Clojure, especially since they require different thinking styles?
00:28:17.840
Although there are indeed profound differences, both languages are dynamically typed. While Ruby emphasizes strong object-oriented principles, it also incorporates many functional elements.
00:28:56.160
For newcomers, switching from Ruby to Clojure can be challenging at first due to the functional programming idioms, but code reviews significantly help with this transition. If you observe our contemporary Ruby code, you'll notice a strong influence of functional programming.
00:29:43.840
Regarding your experience writing code in both languages, what was the main reason behind your decision to switch from Ruby to Clojure? Was the Ruby codebase just not capable of scaling efficiently?
00:30:57.760
I believe if the code had been recovered sooner or if the rescue team had been more competent, it could still be using Rails. Ultimately, we were tired of the limitations, and Clojure allows for simpler modeling in our thinking.
00:31:43.560
In terms of hiring, you mentioned some drastic improvements in your candidate filtering process after introducing Clojure skills in job advertisements compared to when you only stated Ruby requirements.
00:32:35.200
Comparison might not be entirely valid since we didn't have Clojure advertisements prior to it.
00:33:03.640
I still appreciate Ruby. If I find myself needing to solve a problem in a quick way, Ruby will almost certainly be in my top three choices.
00:33:26.960
The syntax of Clojure can also be daunting at first, especially with all the parentheses. How do you cope with this complexity given its Lisp nature?
00:34:01.480
In short, I don't notice parentheses anymore. It's just blocks of code once you’re accustomed to it. And with the aid of suitable editors like Vim or Emacs equipped with plugins, it becomes trivial.
00:34:35.480
I've even noticed that people find it easier to read with multispectral parentheses coloring. With practice, you won't find it challenging to learn.
00:35:05.200
To clarify further, when switching languages, you might find you need to adjust, but, ultimately, with the right strategies and mindset, you can transition smoothly.
00:35:32.960
Thank you everyone for your time and all insightful questions. If there are any more inquiries or if you'd like to continue this conversation, I'm available!