2,000 Engineers, 2 Million Lines of Code: The History of a Rails Monolith

Summarized using AI

2,000 Engineers, 2 Million Lines of Code: The History of a Rails Monolith

Cristian Planas and Anatoly Mikhaylov • May 31, 2024 • Verona, Italy • Talk

Overview

In the presentation titled "2,000 Engineers, 2 Million Lines of Code: The History of a Rails Monolith," Cristian Planas, a Senior Staff Engineer at Zendesk, shares insights from his decade-long experience working with a Rails-based monolithic application at Zendesk. The talk reflects on the evolution of Ruby on Rails within the company over 15 years, addressing how to successfully scale a tech stack as a startup grows to a leading business.

Key Points

  • Rails as a Foundation:

    • Zendesk was founded on Ruby on Rails and continues to utilize it as a core component of its technology stack.
    • The presentation draws parallels between the narrative of Romeo and Juliet and the evolution of the Rails community, emphasizing the drama and dynamics within the community over the years.
  • Evolution of Monoliths:

    • The talk explores the changing perceptions of monoliths in tech, from being perceived negatively to a trend towards appreciation again.
    • Planas discusses his experiences with continuous optimization and the transition from a monolithic architecture to one including microservices and event-driven patterns, adapting as the company acquired new services and technologies.
  • Technology Stacks and New Language Adoption:

    • As Zendesk has evolved, the company has developed a 'tech menu' for adopting new technologies, balancing the strong Ruby on Rails base with other languages like Java and Scala for specific services.
    • Much of the technology adoption at Zendesk has come from acquisitions, which introduce new languages and paradigms into the existing infrastructure.
  • Testing Strategies:

    • Testing is heavily emphasized within the organization, with over 55,000 tests run every time changes are made, ensuring code reliability as the application grows.
    • The transition to automated tests and the continuous integration processes are key to maintaining healthy software delivery practices.
  • Challenges of Scaling and Upgrading:

    • Planas highlights the difficulties faced during upgrades of the Rails framework, sharing anecdotes about the evolution of the codebase and the need to eliminate legacy practices such as extensive metaprogramming.
    • The complex interdependencies within a large legacy codebase often lead to significant issues during upgrades, illustrating the importance of proactive maintenance and clear ownership of code segments.

Conclusion

The talk concludes with reflections on the importance of adapting to change, learning from the past, and fostering a collaborative community within software engineering to keep pace with evolving technologies. Cristian Planas emphasizes that while building on a legacy codebase offers challenges, it also enriches the experience of engineers as they devise solutions to complex problems in a dynamic industry.

2,000 Engineers, 2 Million Lines of Code: The History of a Rails Monolith
Cristian Planas and Anatoly Mikhaylov • May 31, 2024 • Verona, Italy • Talk

2000 engineers, 2 millions lines of code: the history of a Rails monolith

Rails is the best framework for building your startup. But what happens when the startup becomes a leading business? How do companies that have Rails at the heart of its stack manage growth? How do you maintain a growing application for 15 years in a constantly changing environment? In this Cristian Planas, Senior Staff Engineer at Zendesk, will share with you his 10 years of experience in a company that has succeeded by keeping Rails in its core. He will guide you through the life of a Rails-centered organization, that scaled from zero to hundreds of millions of users.

---

rubyday 2024 is the 11th edition of the Italian Ruby conference, organized by GrUSP,
The event is international, and all sessions will be in English.
📍 Verona | 📅 May 21, 2024

Join the next edition
🔗 www.rubyday.it

---

rubyday is organized by GrUSP.
We organize events, conferences and informal meetings involving Italian and international professionals.
We aim to make the ecosystem of the Italian world of web development better both in terms of skills and opportunities by creating greater awareness through comparison and sharing.

Subscribe to our newsletter:
✉️ [www.grusp.org/en/newsletter](http://www.grusp.org/en/newsletter)

 Follow us
 Website https://www.grusp.org/en/
 LinkedIn https://www.linkedin.com/company/grusp
 Twitter https://twitter.com/grusp
 Instagram https://www.instagram.com/grusp_
 Facebook https://www.facebook.com/GrUSP

rubyday 2024

00:00:00.880 um so hello R uh as she said he's an Natoli I'm Christian we work at zendesk
00:00:07.439 UH zenes is a pretty big company that was founded and still to today uses rails uh and we're going to talk about
00:00:14.120 the history of the last 15 years of it um I think it's actually very nice that
00:00:19.279 we are talking here in Verona because I see a certain relationship of the city or of the most famous story about the
00:00:24.760 city with the real Community you know again this talk is going to be about the history of rails and history of RA
00:00:31.800 community and you will be thinking what are you talking about how is this related okay spoiler alert so if someone
00:00:38.800 doesn't know what Romeo and Juliet is about you you can leave the the room now I mean you're 400 years old 4 400 years
00:00:45.960 late but well let's say Romeo and Julet is the story of these two families the montos and the capuletos that they have
00:00:52.480 fighted for generations and then two teenagers of each family fall in love and this kind of triggers a huge drama
00:01:15.640 no it's maybe the biggest drama in history so how is this crazy drama
00:01:21.560 related to a ruby Community I mean think again about that how is drama related to
00:01:27.680 the ru community I mean come on I did a recopilation of
00:01:32.840 Dramas apparently there was a website called ruid drama.com that that put all
00:01:37.920 of them and you see only you can stop the Ruby drama millions of unicorns die each year due to un necessary drama in
00:01:43.640 the Ruby Community I mean don't get me wrong I I love drama huge entertainment
00:01:48.960 value like for a while I tried to be a Scala developer that was terrible times and I went to conference in Scala and
00:01:55.960 you see like the typical speaker in those conferences looks like this you know it seems like
00:02:02.119 AI generated while our our typical speakers in a Ru conference look like
00:02:17.920 but I mean others we some of us know how to address to impress but he is a bit
00:02:23.360 like this okay so introducing the talk uh this there is a reason why I wanted to
00:02:29.120 talk about the passage of time it's a bit of a special moment in my life uh I am originally from Barcelona but I mov
00:02:35.400 abroad for 10 years first to Denmark then to the US and I I moved back
00:02:40.599 recently I also made my 10th anniversary working for zenas and finally this kind
00:02:46.080 of a ner thing but I watching this TV show called feren does anybody know what
00:02:51.159 is this okay so so basically friend is this typical anime in Award of elves
00:02:56.760 Warriors wizards but the main idea that they use there is the passage of time
00:03:01.800 because friend is an immortal being and and they play a lot with the idea this person has seen it all before moreover
00:03:08.280 in my opinion uh in Fr they use magic as something that can be read as technology
00:03:13.840 or at least you can see the progress of magic that follows similar patterns to the progress of Technology this actually
00:03:20.080 not something super new my favorite fantasy novel uh a wizard of SE by UL
00:03:25.760 lein that's that's that's incredible yeah I always read it in the same way so in in the magic in this world what was
00:03:33.040 the what the following way if you know the true name of things you have po over
00:03:38.360 that thing I always think that it's about it's like object in the programming you create a perfect
00:03:43.480 abstraction that is impossible then you you you gain power over over over those objects we all work it with terrible
00:03:49.920 abstractions and we know the difference no I mean again all these ideas are not mine arer has a very famous quote about
00:03:56.599 the relationship between uh Magic and technology and it's L advanced technology is indistinguishable from
00:04:02.920 Magic but well going back to time and and Fen You need to think that in this
00:04:08.000 world uh it's very structure like the show is very structur around flashbacks so you see here when magic was something
00:04:14.680 very very rare then you see her one her years before where there was a spell that was considered incredible and then
00:04:20.759 you see here in the present that that same spell is now the basic the the base of everything it's something similar to
00:04:26.440 H that happens in technology that something amazing 10 years ago ago now it's completely normal more thing for to
00:04:33.520 and me it was natural for us to identify with his character like frin is an elf
00:04:38.759 that has Liv over a thousand years old uh she's the that makes her the possessor of forbidden forgotten magical
00:04:46.039 knowledge and I would say that we are equally mythical me we we are Engineers
00:04:51.400 that have worked in the same company without changing for 10 years that makes us the possessor of
00:04:57.360 Forgotten forbidden magical knowledge and we are here to share it uh we will
00:05:04.360 start the discussion talking about the monolith both how the monolith evolve at zendesk because U zes was originally a
00:05:12.199 pure rails monolith created in 2007 but also about the concept of monolith I've seen a lot of things changes like I
00:05:19.000 started working with Ruby late 2000s early 2010s um I was was way younger
00:05:25.080 then but you know the 2010s were weird times you know we like all kind of
00:05:30.440 strange stuff uh and and then in the late 2010s the idea of monolith became
00:05:36.039 not cool anymore I remember going to an interview and when I said yes I think monolith makes sense in the context of
00:05:41.759 startup they look at me like go away you know uh now kind of the concept of Mor
00:05:47.360 is trendy again or at least not as bad as it used to be for me this is not a
00:05:52.560 surprise I personally believe that a lot of uh debates that we have in in Tech uh they they behave like pendulums
00:05:59.039 something that was was completely mainstream 10 years ago then becomes terrible and then goes back to a more
00:06:05.080 balanced position uh understand here I don't mean that you can say whatever and nothing
00:06:10.919 nothing matters there are debates that uh that uh suggest a very dangerous
00:06:17.000 pendulum for example uh six seven eight years ago I think uh relational
00:06:22.120 databases were there were at this trend like no SQL which makes sense in certain context but there was a lot of people
00:06:27.280 saying that we shouldn't use relational databases at all and SQL was was kind of dead or in the way out well that was
00:06:33.880 that wasn't the case eventually and if you think that this was a uh spicy take
00:06:40.039 I have one way more spicy I actually think that the boom of microservices in the last 10 years is a zero interest
00:06:47.160 rate phenomena and by this I mean tech companies had so much money that uh they
00:06:52.440 went in a lot of cases for pre preemptive optimization I am not the only one that
00:06:57.800 thinks this actually uh one or two months after I did this presentation the first time dhh put this this tweet which
00:07:04.639 I kind of agree maybe I wouldn't WR it that way but it's the HH um but I mean I
00:07:11.160 can I I've seen this because uh I remember a friend of mine that was talking me about his new startup he just
00:07:17.080 joined and he was describing to
00:07:25.720 me he was describing to me this incredibly complex architect Ure I couldn't understand but I was thinking
00:07:31.840 well I'm happy for you because this seems like very popular said no no we have 10 customers it was like okay man I
00:07:37.280 don't know moreover uh now that unfortunately like uh you know there is less money in
00:07:43.520 the market when companies need to reduce amount of Engineers having to support uh
00:07:49.120 2,000 Services is going to be very very very very hard like this from a newsletter I found where some company
00:07:54.879 complained about this say this I do believe that microservices may sense in
00:08:00.240 in a lot of us cases I have a very different opinion from let's say El musk that called microservices blw I mean I
00:08:08.400 find this hilarious but I I wouldn't be go that far uh my opinion is more similar to Jason Warner the C of GitHub
00:08:15.319 he believe that there is like a spectrum a lot of times when you are a startup or you create an application that needs to
00:08:21.720 try to find market field let's call it uh you need to start with something that you can evolve very fast and then slowly
00:08:28.639 break it down into to different Services actually with this we go back to zendesk because internally at zes we
00:08:35.080 had a lot of debates about that but somehow what we end up doing is kind of like that we we broke some parts of the
00:08:42.479 of the monolith into services but I still think that if you are someone on boarding into Z in4 you will feel that
00:08:50.080 it's similar to a monolith it's pretty monolith uh said this we had different
00:08:55.760 Visions during all these 15 years originally our our first Morton had this idea of I'm going to
00:09:02.160 create a monolith in rails but then I'm going to extract part of the logic so it's perfectly modular and then in the
00:09:08.640 future I will be we will be able to spin new rails applications that have a lot in
00:09:13.800 common if you check the the gem file of the monolith desk we have a lot of gems like that and actually not we don't
00:09:20.800 create applications like this anymore but we have a few applications that do that they share code through
00:09:27.120 gems then we move to what I call the service ERA this came a lot because we
00:09:32.959 acquir companies and we wanted them to share data across the the ecosystem for
00:09:38.399 example we acquire weire a chat company we wanted to share accounts it didn't make sense for us that this the chat
00:09:45.079 product calls the support product to to ask about accounts so we created an account service and finally these days
00:09:51.920 we are doing more of an event driven architecture so if something happens in the monolith it generates a gafka
00:09:57.399 message that's consumed by other services and this triggers multiple actions but going back to the monolith
00:10:04.160 our way to to grow it now is experimenting more with mization uh we are playing a bit around with pwork that
00:10:10.720 is a gem created by Shopify that helps in in this regard and I would be but I wanted to
00:10:16.920 comment a little bit on the front end what happened in the last 15 years in the in the front end side of creating
00:10:22.600 applications well again uh I joined the the the community in 2010 which was
00:10:28.440 strange times and at that time is it's when it started
00:10:34.240 one of the most violent conflicts in human history which I call the great JavaScript Wars I
00:10:40.720 mean I mean if you if you remember if you you were active in those times and
00:10:45.880 you entering Hacker News Chan is that every week there was a new a new JavaScript framework this still happens
00:10:52.160 today but the difference is that then all of them were somehow getting traction it was very hard to follow like
00:10:58.399 at Z weu used a lot this three uh backbone Ember and react uh and I'm sure
00:11:04.200 that somewhere we use also angular and other stuff but but this one very very used I mean there is something about
00:11:09.639 JavaScript Engineers that they seem to love creating new Frameworks we have something called CJs KJS you don't know it for sure
00:11:17.120 because the real is that someone called cart nobody at Zer remembers his last name anymore created his own
00:11:23.079 framework and is undocumented of course and we are building on top of CJs it's
00:11:29.000 something fascinating um but still it seems that in the last years we kind of agreed to
00:11:34.760 go on the r way yeah there is there are other Frameworks but seems like react is kind of the king of the hill at the moment and I like react I I use react at
00:11:42.240 work I use specifically react native a lot for my own applications but sometimes I feel that the this victory
00:11:47.920 of react is just because people got too tired you know it was just just hard to follow to follow with the JavaScript uh
00:11:56.000 ecosystem the next part I would like to talk about deployment uh we don't have much time and this is like a very long
00:12:01.480 long story but uh let's say that when uh Z was started we went using private
00:12:07.399 server servers like engine yard Rackspace and with each of them we kind
00:12:12.720 of went over the limit that what they were expecting uh it it actually at that
00:12:18.040 point it just made sense to go uh for own metal I think at that point we were
00:12:23.560 kind of a little bit pioneering pioneering this like we didn't have the right tools So eventually we moved to AWS and we've been in in AWS uh since
00:12:31.800 then if you want to know more about this uh we have a series of of articles about that with a with an interview with the
00:12:38.240 first C of zenas that can illuminate that better than me the next part yes I know that this is
00:12:44.240 a ruby conference but I wanted to to talk about how do you adopt new technologies in a big
00:12:49.880 company again we are a ruby shop more or less but rub is very important for us
00:12:56.800 but then of course you you start thinking how do you adapt new technologies into the stack well when you're a startup it's very easy a lot of
00:13:03.440 times you can be even one engineer that decides to hey let's use K yes or let's use something and you talk with your
00:13:10.399 manager and then that's it but the problem is that when you're a company that has around a thousand a th000 Engineers that that doesn't scale that
00:13:17.639 becomes foolish um it's the the cognitive load of having to support hundreds of languages is just too much
00:13:24.720 so we created something called the tech menu so the idea of the tech menu is that hey you want to add I don't know
00:13:31.839 Elixir to to the to to the systems that we use at sendes to the pring languages great do a proposal we have an
00:13:38.600 architecture team that decides which in which path we should go and if it's accepted that's it you can create applications on
00:13:44.760 Elixir uh it's funny that I use l here as an example because the I think French have a word for that like C CBR like
00:13:51.560 famous case uh was about which language should we use for creating Services uh I
00:13:57.120 was in team Elixir that's why it came to my mind and I kind of lost so we we don't build
00:14:02.399 services in using alexir at zesk we use two languages jav and Scala one thing
00:14:08.920 that was very fascinating is that both were accepted because two very significant parts of the company opposed
00:14:14.600 a bit they they both defended them um and for a while it was completely accepted and equally defended to use
00:14:21.360 Scala and Java uh I will tell you that as 2024 uh we kind of prefer to uh create
00:14:28.759 service in Java and interestingly the reason is that it was very hard for us to hire people in Scala uh to train
00:14:35.360 people in Scala it just became very hard to maintain our Scala systems so we just prefer Java these
00:14:41.199 days but actually the the main way that we adopt new technologies that desk is be Acquisitions you can say whatever but
00:14:48.000 if the com you can say whatever about python for example but if you if the COO
00:14:53.079 once a company and this company is using python python python enters the stack
00:14:58.199 for as are important around half of our products came be Acquisitions and I will just ex
00:15:04.920 exemplify this with the our first acquisition that was a company in Singapore called zopim zopim is a chat
00:15:11.399 company and the product now what this is called zenes chat as I was putting example they were using Python and
00:15:18.320 python is a zes language but from a software engineering perspective I think the most interesting part what I call
00:15:24.720 domain Fusion we are a customer support company we have the idea of ticket of user of ticket is the the main Center a
00:15:32.079 chat company has a concept of chat I guess conversation whatever it's completely different
00:15:37.880 domains and then suddenly all the chats in this company need to become ticket somehow because they are now not a chat
00:15:44.240 a neutral chat company they are part of a a customer support product so like in St Trek think they're called as zcs you
00:15:51.160 know that you need to absorb and make two one out of two so again what is a ticket what is a chat now we have a
00:15:57.959 protocol things and this voice that phone calls are also part of this customer support uh flow I mean yeah
00:16:05.120 this is a product problem because it's very important from a pro perspective that your customers feel like they have
00:16:10.440 a unified experience but it obviously has theep consequences on the technical side now
00:16:16.199 some something that was modeled for years as a conversation needs to be part of a ticket eventually what I felt is
00:16:22.800 that what we needed is uh shared features uh common features into very different objects I would like in
00:16:29.759 interfaces and the final solution is that everything eventually becomes a ticket for us so any object created but
00:16:36.680 by an application calls the monolith uh originally via an API call now through
00:16:42.759 Kafka messages and generates the basic slate for everything that is the ticket here okay and after this kind of PR
00:16:50.040 conversation an here will tell you reality is about performance test test test hear
00:16:56.319 me so I had a very boring 30 slides and then I talk to badar and now my slides
00:17:01.959 are not boring anymore I hope all
00:17:07.439 right this is the photo I have taken probably U I took probably maybe two years
00:17:13.760 ago has anybody uh been to Dublin about Dublin weather okay quite a few people so we
00:17:21.160 believe that we don't have any summer and if you want to have summer you you come to Italy right well this is what I was thinking
00:17:28.319 and then I I checked Wikipedia and and this is what I found how many Sun hours we have per
00:17:34.320 year in Verona versus Dublin I was shocked I was shocked so D supposed to have zero hours
00:17:43.400 no it's not it's not zero and why am i showing this slide is the
00:17:51.760 system uh we think about a large
00:17:56.960 system it's very similar to where better we think it's something one but
00:18:02.039 it's something else it's completely something else and now attention B has
00:18:07.520 to smile and everybody has to
00:18:13.120 say I love it okay so this is the photo I took uh 7 years ago in
00:18:21.240 Dublin with three three days of snow uh no like no traffic No Buses no flights
00:18:27.960 nothing so the system we work with at zendesk is
00:18:34.520 similar it's unpredictable very unpredictable this the photo from San
00:18:39.559 Francisco where we have a headquarter it's supposed to be sunny but it's not so again it's a a
00:18:46.679 challenging weather and sometimes the system makes its own decision for you so you just you
00:18:53.400 you don't ask and and system behaves one way but you can't explain
00:19:00.000 and next slide I stole from Christian I think that was my only funny slide and I made it even more
00:19:06.760 funnier all happy systems are alike each unhappy system is unhappy in a own
00:19:13.640 way yeah I have to take a 50 seconds pause but I don't have too much time for
00:19:20.120 that uh yeah it's like it's very it's yeah okay I'll keep it because I don't
00:19:26.919 have time as then transition from AWS uh from data center to the cloud in 2016 I
00:19:34.679 believe 17 yeah that's when we started the process I think we ended in 2018 I think 2018 when we finished and if you
00:19:41.720 ask what was the uh the biggest challenge with zesk
00:19:46.840 uh it was the database so database performance was the reason why did it take
00:19:52.760 longer and why we had many problems than we expected and does anybody you know I'm
00:19:59.120 the law I'm the law so if your system is stateless you have no data it scales
00:20:06.120 forever it scales like well but does anybody have no data in your system no
00:20:11.760 no hands one hand no hands so as soon as you have small data
00:20:17.360 set performance degrades but as as soon as you have large data set like we have at zesk you hit the moment of contention
00:20:25.320 and now I should have badar slide I don't know why I don't have it
00:20:30.559 here I think I have it a couple slides later yes
00:20:36.280 uh Happy large systems have one thing in common and this is the data this is
00:20:42.679 Happy databases happy performance databases uh
00:20:48.600 there's no large system which is safe reliable past with a unperformed
00:20:56.280 database and performance database have one thing in common
00:21:02.240 and you have to pick two out of three okay you either have consistent run time
00:21:08.080 and complex queries but small data set you have complex queries large data
00:21:13.720 set but forget about
00:21:19.120 performance that's it okay thank you and this is exactly what happened to
00:21:25.919 zes this is our past data set was small query run time was
00:21:33.240 relatively consistent and we could do anything we wanted anything absolutely anything the future
00:21:41.799 actually present is little bit different uh query around time took the back seat
00:21:48.520 because data set took over the future is different the future is no complex
00:21:56.960 queries whatsoever uh large data set will dominate and performance of the
00:22:02.440 database and API and points will be consistent back to Christian yeah and
00:22:09.720 after this that actually is what you should keep in mind because what I say is mainly mainly stories um um we'll
00:22:16.200 talk a little bit about testing and reliability like how we use test zendesk to keep our hundreds of thousand of customers uh
00:22:24.440 happy but before that I would like to comment a little bit on a on a typical debate in the Ruby community
00:22:30.840 that is tests versus types so in the 2010 like the proposal of Ruby was that you don't need types you have test and
00:22:37.480 you need test anyway so you don't need types uh with the time in the late 20110 or maybe more more more recently I think
00:22:44.880 in the community there been a bit debate about that I think Ruby 3 supports type checking uh natively we also have svet
00:22:51.480 that is the G that adds type checking uh so there's a debate here um I will tell you at zesk in this debate we are EX L
00:22:58.799 on the side of tests and for test test for us are incredibly important because
00:23:04.279 we have a lot of code around 1.6 million lines of code and we have a absur amount of test
00:23:11.320 how many test do you think we have like unit test I'm not counting uh i counting uh feature test or integration ones say
00:23:20.880 numbers no no how many test un need test
00:23:27.159 um yeah yeah okay so well close 55,000
00:23:58.840 uh this is his real Avatar in the slack let me merch or I will cry like literally a few months ago we were on
00:24:04.520 boarding a new Junior engineer and I was waiting for him to complete the feature say hey Christian I'm sorry I couldn't
00:24:10.039 match there is this guy that is telling me that my test are wrong and I cannot match this and I was like oh you met
00:24:16.960 Ryan um some of this you may know Ryan because he's also the Creator and maintainer of mini test and actually
00:24:24.240 this brings me to the second comment that most Junior engineers make at the desk why mini test and by why mini test
00:24:30.919 they mean why not a respect which is I think the most popular test framework in in the Ruby Community well there's a
00:24:37.919 good reason we do have a lot of test and to make it even worse we have we run uh
00:24:44.000 all the all the test Suite twice every time there is a push later will explain why so this means that every time that
00:24:50.120 someone does get push to GitHub we're running 11,572
00:24:56.240 deaths uh so the advantage of me test against our spec is that smaller less magic and it's way faster I told with
00:25:03.279 Ryan and he told me that other big companies using using rails like uh Shopify also prefer mini test for this
00:25:09.520 reason in fact when we try to modify mini test we try to make it less less
00:25:14.840 magic less faster we care about that still despite that we have pain points in our testing
00:25:21.200 pipeline they are flaky test a flaky test for us can be terrible for productivity because again in imagine uh
00:25:28.960 this is like a a real Snapshot from from our PRS there is like
00:25:34.000 120 I know how it's called checks being run each check is around 2,000 test so
00:25:39.880 if you one test is flaky you need to rerun it this means that you need to rerun 2,000 test that they can take
00:25:45.240 around 20 minutes that's pretty painful moreover now we're in a much better world because I don't know if you
00:25:51.679 remember but until three years ago you couldn't rerun one check in GitHub actions you need to run all of them so
00:25:58.679 this means if you have one flaky test this means run over 100,000 test that was very painful and was a very happy
00:26:05.480 day when when you have released that um but say this I would say that the true secret for uh for uh reliability for us
00:26:12.440 is ownership everything needs to have an owner why well we have a lot of
00:26:17.960 code and we have a lot of committers I checked the the monolith and there's been 2,459 different accounts of with
00:26:26.320 that uh push code into into my in domain and okay you may be thinking well this application has been around for 17 years
00:26:32.919 of course most probably lately there's way less people well the last year 570
00:26:39.000 different people so a lot of people pushing code and if you don't have pretty clear
00:26:45.039 processes you're going to end up in a fight it's going to be very hard to know who did what who uh had responsibility
00:26:51.960 of fixing certain things so we have a rule that you all all files are owned by
00:26:58.240 at least one team uh like this it makes it way easier if something breaks to find who who
00:27:03.760 needs to fix it or who needs to work on that is super important but still of course I couldn't have a talk about uh
00:27:11.480 history of rails without explaining what happens when you upgrade rails because again zenes is kind of ancient uh what
00:27:19.120 started in rails one actually there is a there is an a story about that uh like uh the original founder of of of zenes
00:27:26.679 Mikel uh used to work with dhh before before dhh created rail so they know each
00:27:32.559 other uh we used tests for upgrading rails how do we do that well remember I
00:27:39.039 told you that we run all the test twice all so what we do is like of course we run all the test with the current
00:27:44.760 version of rails and then we run them again with the verion that we move want to move next the idea for that is that
00:27:52.320 if it's red that's optional that's fine but once a test becomes green we do not
00:27:57.679 accept that this test goes back to red again like this we because imagine like
00:28:02.919 you are trying to upgrade the version you're fixing problems but someone introduces something new that isn't
00:28:08.440 compatible you will never end like this we we can we can uh guarantee that uh we
00:28:13.679 will have dations I think they're called um still we have had historically pain
00:28:19.360 points on the on when raing rails and I think this something can be pretty interesting for you because again I've
00:28:25.399 been around since 2010 at the time the there was something that was incredibly popular in this community that was not
00:28:32.279 popular at all in other programming communities and it's one reason why I started doing Ruby and the next slide
00:28:40.399 hurts my heart because I love the book but this is the the reason meta programming I literally fell in love
00:28:47.240 like I literally have the book signed by by Paulo Pera like with know like he's my idol but oh my God metam is such a
00:28:54.600 pain when you go try to upgrade we had until rails 5 we had our or Code full
00:28:59.919 full full of monkey Poes some ghost methods like we had gems that we were
00:29:05.080 using the main branch but then we had modified versions of the gem for us it was terrible eventually it doesn't scale
00:29:10.360 it became just too the cognitive load is just too hard still we move for a lot of
00:29:16.320 years we were keeping this app until it got to a point of upating to rails 5 basically when we try toate ra five all
00:29:22.679 test broke like it was nothing worked so at that point we decided to go hard mode
00:29:28.640 and remove all the metaprogramming or as much meta programing as possible from the from our
00:29:35.240 app moving from rails four to five took two years and a half of and with people
00:29:40.880 dedicated to it uh this is like the last year of of
00:29:46.519 progression amount of Errors failures how how much it went and you'll be thinking what were the specific issues
00:29:52.279 that were crazier where well we had our own Branch basically the most important things were ISS ues that were uh
00:29:59.240 distributed across the whole the whole code base for example stronger parameters we were using a a fork of
00:30:06.000 stronger parameters and when uh when we arrive at five we decide this doesn't
00:30:11.399 make sense we don't want to be maintaining a stronger parameters forever our own version let's go back to rails the rails way this was hard still
00:30:20.080 the most famous issue that uh we had and it's something that the community got a bit angry at the time was dirty
00:30:27.080 attributes that is what actually a thing with ra 52 I don't know if you remember this but the idea is that so you have an
00:30:33.919 object in ra 51 you change the the an attribute and you save it you do user do
00:30:41.279 object do change it will return true in ra 52 was decided that if the change has
00:30:46.320 been committed change the method change should return false we're using a lot of
00:30:51.720 logic based on that and actually this was not even like oh there's an exception now it's just that it just
00:30:57.519 returns different value uh so this actually I was talking with Ryan about this and he is very
00:31:03.639 strong believer that this should this should have a minor change and should have been in real six but it it was
00:31:09.279 really really problematic for us I mean this thing alone added six months to the upgrade and there is a reason for this
00:31:14.600 this seems crazy but imagine that you have an application that is being used by 500 Engineers every year that means
00:31:20.919 around at least 100 teams you need to communicate with 100 teams tell please could you check that those hooks that
00:31:27.360 are broken fix them everybody have their own road maps there's communication issues you discover that some endpoints
00:31:33.320 don't have an owner um it's really hard to change the good news is that since then
00:31:39.279 it has been way easier and now more or less we keep up with the latest upgrades still maybe the lesson I want you to
00:31:45.279 take is what I call the first mover disadvantage so if you have there is something that you want to do and nobody
00:31:51.519 else has done it before there's going to be a price that you're going to have to pay one example uh sharding we've been
00:31:58.279 using sharding for a long time since probably rails to and we have our own gem our own stuff um recently well
00:32:05.960 recently no maybe five five years ago um that's probably before the pandemic uh
00:32:11.240 rail six came with sharding like uh out of the box we're still using our own
00:32:16.880 version of sharding and we want to move to to to main uh to the main branch of
00:32:23.000 rails of sharding so my advice after all this would be to join the party essentially
00:32:30.080 don't develop something by yourself closeing your little room or your little you know private repo if you find a
00:32:36.480 problem and you think there is a solution like mainstream create an issue talk with people maybe you will find
00:32:42.679 that someone is already working that solution and you can join or maybe something new and you can become a r
00:32:48.120 contributor what I don't recommend is just maintain your private solution for years because eventually there's going
00:32:54.000 to be another solution that will be public and it will be maintained by a lot of people and eventually that solution is going to
00:32:59.760 be better than yours and you're going to have to adopt it okay we are we are getting to the end
00:33:07.880 um this presentation is a bit of a sequel of the previous one that we did uh last year we were talking about uh
00:33:14.159 performance uh scaling an application but from a from a performance
00:33:19.960 perspective uh the things that iong with people I realized that not being to scale an engineer organization not being
00:33:26.279 to scale processes is way more problematic I mean in the end performance problems are kind of
00:33:32.600 easy to the de so if application is too slow it doesn't respond well to have a huge problem but uh the slowness of an
00:33:39.000 application is kind of obvious up to a certain point with processes organizations is completely different
00:33:45.559 point like even what how a a wood organization looks is kind of an object
00:33:51.200 of debate it's not very clear to me so then I start to think okay how how how
00:33:56.360 do I imagine this ideal organization how do I imagine this this ideal process at this point I start to think about the
00:34:02.399 past and you know all the things that i' I've seen at zesk or in different companies and seeing well maybe I can
00:34:09.960 take things out of what I've learned and this you know when you you've been around in a in a in a company or in a
00:34:16.599 place you feel a like Gandalf like oh I have this secret knowledge that I can share with people know I have power no
00:34:23.000 but then you start to realize and I saw this a lot that pass can make you also like s man you know so someone corrupted
00:34:28.679 like because the pass can become this little thing that talks to you say this is not the way to go I mean in 2005 you
00:34:34.760 develop an application with completely different par completely different paradigms it's such complicated and
00:34:40.760 there is an anti pattern I discovered this the other day called the Frozen caveman about this it's like Oh My My
00:34:47.560 paradig my patterns have worked for me forever so they will work like this and
00:34:52.599 it doesn't happen particularly it doesn't happen in Tech so what's the right attitude here
00:34:58.440 well I think that you need to learn the past but take some take some distance iine like a Bart you know like someone
00:35:03.800 that comes here and explains your story and you just need to pay attention maybe think oh the story was nice but I didn't
00:35:09.720 learn anything out of it or maybe say well I can apply that so you should know the past and take your decisions it's a
00:35:16.760 bit like like in Dune I don't know this is not very clear in the in the in the movie but it's more clear in the in the
00:35:23.640 in the in the novels so in the novels he's always able to see the future
00:35:28.800 but then only if you've seen the movie you remember that scene he drinks some kind of water all that and then in that moment He's Able also to see the past
00:35:35.800 and just when he's able to see the past he's able to see way way way farther into the
00:35:41.000 future um I think that if you are able to do that you will be able to to H the
00:35:46.079 Ruby Community much more and maybe you can continue doing uh presentations like this in in 10 years thank you
00:36:00.520 this was incredible and I I I recently joined a company that has some of those
00:36:07.520 problems it's uh I can't relate um do we have any questions one
00:36:14.640 over there and one over there and another one over there so shall we go
00:36:20.319 left to right um I can share my that's okay
00:36:31.040 hello um I'm curious to know uh how long does it take to run the entire test
00:36:38.440 suit and just to confirm are there 60
00:36:43.599 instances running in parallel of the and and also what do you use to parallelize
00:36:49.720 that that's interesting I I will have to check it runs less than 20 minutes uh
00:36:54.800 and because we're running all in parallel of course if not it would be it be impossible what do do you remember what we use for running the
00:37:04.160 test yeah I can check if you give me the contact I can tell
00:37:12.800 you check later after
00:37:18.200 yeah uh sorry I'll come back uh where was the over there and you right come
00:37:26.119 okay thank you uh you mentioned using local
00:37:33.240 gems to break some of your app complexity I guess one question that is did you have a dependency structure or
00:37:39.720 all those gems that we saw in that screenshot required by the main app and second about uh packwork sounds like
00:37:45.839 you're using some of that as well curious about your take on the different approaches yeah so uh actually good
00:37:52.480 point I mean uh um historically like the the idea of using private gems you know
00:37:57.560 like you take that uh eventually didn't work uh because it is what you say we didn't have like clear dependency maps
00:38:05.160 and right now some of those gems are so strongly entangled with the monolith that you cannot use them anymore like
00:38:11.960 that's uh yeah uh we have some exceptions and this because we created pretty early a pretty successful
00:38:18.200 application and in this case it's very clear that you cannot create PRS that entangle too much the the gem with the
00:38:26.280 with the month uh on pwor we actually just payment at this point I I very strongly believe in it because because I
00:38:33.079 believe in Mor but you need to be well organized Mor like sometimes people says oh Mor are like that and it's not like
00:38:39.359 that it's just a spaghetti monit you know uh that the fact that you have Mor doesn't mean that you can you know have
00:38:45.000 a complete mess up uh map of
00:38:51.359 dependencies and one more question here yes probably it's a silly question you
00:38:56.480 said that you test twice for every PR right I suppose the twice is one just as you file it to to start the review and
00:39:04.359 one is a presubmit check just before it get the merg or is something else no it's something else so it runs in
00:39:10.640 parallel they both runs but it's not exactly the same PR in the way that one run with rails the current version of
00:39:17.079 rails and the other with ra rails 7.1 for example so and the idea is that um
00:39:23.599 if uh once a test is green we don't allow it to go back to red because a lot of times our rails team says well I just
00:39:30.520 fix this but now there is a new hole so they wouldn't end it wouldn't do this but when you don't change release if you
00:39:36.000 just have a PR with no release change what release with you're using the same
00:39:41.640 ra version you're just changing something else without changing the rails version no no like like it's so
00:39:46.680 one run is with Rail 7even and the with rail 71 but the thing is that to make ra 71
00:39:52.760 completely green it takes time what we don't accept is test that are fine are are green in real 71 to break them
00:39:59.800 anymore so at any point in time you always have a current version and the next yes version okay sorry I missed
00:40:06.040 that I thought it was like a one exception event not at all events okay got it sorry MH
00:40:11.720 yeah last question I have a lot of questions but
00:40:19.079 let me ask one uh more about the how you organize the code because you're starting with rails 1.0 right so so a
00:40:26.560 couple of one question it's in tests let's say for the test what folders do you have how do you organize
00:40:33.440 them and how do you connect them with the code because when you want to change the code you want to change the test
00:40:39.079 right yeah in the monolith in in the r of the monolith we only have unit test and those corresp it's like test and
00:40:45.200 they they correspond to the file uh both for feature test integration test we actually have two
00:40:51.319 completely different projects one for the API and the other for the browser and they run completely we use Jenkins
00:40:56.760 and they are run completely you know a completely different way okay
00:41:04.520 yeah sure so still about the organizing code so are you using how do you
00:41:10.319 organize again about ra monolyth right so how do you organize your code let's a
00:41:16.040 bit about code design let say are you using name spaces so I'm not talking of P pwor I know how it works but the one
00:41:21.960 that is not there right are you namespacing things um are you yes
00:41:27.319 putting everything in up models or I don't know what how how is Oran we use quite a lot of name spacing like part
00:41:33.119 except even models are name space at sometimes but uh we use a lot of name spacing for anything related like for
00:41:39.240 example ticket again I was saying that everything is a ticket ticket itself like the model it's probably 2,000 lines
00:41:45.960 of code but we have a lot of modules that are included into it and of course they are they are Nam space under ticket
00:41:52.400 and where is this ticket in which folder do you have uh it's a app models ticket
00:41:57.440 uh so you have app models ticket RV and then you will have a folder called app models ticket yeah yeah that so yeah
00:42:03.480 ticket is a is a yeah I get it thank you thank you very very much enjoy the
00:42:57.359 oh
Explore all talks recorded at rubyday 2024
+1