Microservices
How Rails fosters a diverse and competitive tech ecosystem in the era of big tech

Summarized using AI

How Rails fosters a diverse and competitive tech ecosystem in the era of big tech

Jordan Trevino • April 24, 2023 • Atlanta, GA

In the video titled "How Rails fosters a diverse and competitive tech ecosystem in the era of big tech," speaker Jordan Trevino discusses the impact of Rails technology on fostering a diverse tech ecosystem, particularly in contrast to patterns popularized by big tech companies. Trevino argues against reliance on big tech architecture, suggesting that it can jeopardize smaller and medium-sized companies due to increased resource demands and complexity. Instead, he advocates for Rails’ organizational fit and its supportive nature for innovation and diversity.

Key Points Discussed:
- Historical Context: Trevino draws an analogy from the light bulb industry to illustrate how companies historically managed innovations through planned obsolescence, contrasting this with the Rails community’s ethos of sharing knowledge and fostering each other's success.
- Critical Views on Big Tech Patterns: Two primary big tech patterns, microservices and single-page applications (SPAs), are evaluated for their fit with smaller organizations, suggesting they demand higher resource levels and can lead to development complexities.
- Economics of Diverse Structures: Trevino stresses the importance of diverse organizational structures as a means to combat autocratic power, providing employees with more choices and preserving essential human values.
- Tech Patterns' Impact: A survey from the Rails community highlighted that most developers still use monoliths and many opt for Rails-first practices, promoting a culture of resource efficiency and collaborative success rather than fragmentation.
- Case Study: Trevino shares a case study of a client who transitioned from a Rails backend with a SPA frontend to a microservices architecture. The company faced significant challenges due to increased complexity and development delays, ultimately leading to the conclusion that reverting to Rails' monolithic structure improved development velocity and cost efficiency.
- Market Power Concerns: The talk wraps up with a critical look at the monopolistic practices of big tech companies and how their scale affects industry standards and competition, warning against adopting their patterns without considering the unique needs of smaller businesses.

In conclusion, Trevino posits that while trends from big tech can seem attractive, Rails' frameworks and patterns are more conducive to creating successful, innovative, and diverse tech ecosystems, particularly for smaller developers and companies looking to maintain agility and lower resource demands.

How Rails fosters a diverse and competitive tech ecosystem in the era of big tech
Jordan Trevino • April 24, 2023 • Atlanta, GA

How Rails fosters a diverse and competitive tech ecosystem in the era of big tech by Jordan Trevino

Join us for a discussion on how Rails technology patterns and practices help foster a more diverse and competitive tech ecosystem amidst the proliferation of patterns popularized by big tech. We develop an economic and organizational critique of such approaches as microservices and single-page apps.

RailsConf 2023

00:00:18.619 Welcome! We're going to talk about Rails in an era of big tech.
00:00:23.820 First, I want to get started by talking about light bulbs. In the early 1920s, light bulb engineers in Europe and the U.S. made great technical advances; they were able to produce light bulbs that lasted over 2,500 hours.
00:00:29.640 However, the owners of these companies noticed that as their technology improved and the lifespan of their light bulbs increased, their sales began to decline. They made less money. This was counterintuitive because you would expect that technological innovations should lead to greater profits.
00:00:42.660 The owners, however, didn’t think that was how it should work. They were on the verge of inventing planned obsolescence. In 1925, they established the Phoebus cartel, an organization of the leading light bulb producers. They literally tested the lifespan of light bulbs and fined companies for exceeding a lifespan of 1,000 hours per bulb.
00:01:00.960 This limit was far below the average their companies had achieved before then. By enforcing this limit, they were able to double their sales. But how long do you think this cartel lasted? Only 14 years, until 1939. Initially, they had plans to sustain the cartel until 1955, but the war disrupted their collaboration.
00:01:15.119 You might be wondering what this has to do with Rails. In her keynote, Eileen spoke about how much purpose she found in making the framework better and giving back to the community. This is because the core of Rails is about enabling innovation and the success of others.
00:01:28.619 This stands in stark contrast to the light bulb cartel. Rails is particularly beneficial for smaller companies that are just starting out and for developers early in their careers. The Rails community provides societal benefits by organizing industry developers and companies to utilize innovations to improve the world.
00:01:39.660 Unlike the light bulb cartel of the 20th century, we do not collaborate to plan obsolescence or sabotage our competitors. Instead, we share knowledge and help each other succeed.
00:01:53.880 But can the same be said for the big tech giants? As consumers, we often overlook their free-use model with products like Google and Facebook. As developers, we also experience the tech patterns they popularize. In this talk, I will urge you to be cautious of these patterns and to consider a few key arguments.
00:02:10.879 The first argument is that big tech architecture can jeopardize a lot of companies, especially smaller and medium-sized ones. A corollary to this is that organizational tech fit truly matters. Often, we tend to adopt patterns based on their popularity or computational elegance, but this isn't always a good fit for our organizations.
00:02:27.900 Next, I will discuss some novel dangers associated with big tech's market power. While monopolies have existed for centuries with an antitrust legacy in the U.S. and around the world, in the past 20 years, there has been a new kind of market power that utilizes cybernetics and digital tools.
00:02:39.660 This is something that we, as an industry, need to be aware of, as we have been participating in it to some degree. Finally, I will argue that Rails patterns promote greater diversity because they are friendly toward smaller companies and enable smaller teams to compete in ways that would otherwise be impossible.
00:02:53.820 About me: I'm Jordan Trevino, the founder of Telos Labs, and I am a software developer. However, I wasn’t always a Rails developer; I previously worked as a strategy consultant for large pharmaceutical companies, launching cancer therapies. This experience taught me to take a broader view that encompasses business, operations, and culture.
00:03:06.000 I am also an enthusiast of democracy and organizational diversity. As you'll be able to tell, I’m from Mexico. While Mexico is not a resource-poor country, the welfare of its population is relatively poor due to crony capitalism and a legacy of colonialism.
00:03:19.260 I believe that the U.S. is at risk of becoming like Mexico, rather than the other way around.
00:03:23.820 Now, as a first-time RailsConf speaker, I am very excited.
00:03:30.300 Talking about diversity, we typically hear this term in the context of demographics. However, I want to expand the definition. I believe that diverse organizational structures defend against autocratic power and provide employees with choices regarding where to work and collaborate.
00:03:36.300 This diversity preserves essential human values, traits, and attributes. The image you are looking at displays a vibrant, lush ecosystem, which emphasizes that diversity is indeed a tremendous asset that helps organizations survive.
00:03:44.400 Monocultures, or having a single crop, are detrimental to resilience when faced with pests or ecological threats. Therefore, diversity is crucial for allowing people to find the right company for them.
00:03:56.400 Today, we are facing significant challenges such as climate change, inequality, and political polarization. In this big tech era, these issues seem to be worsening because businesses as usual offer no solutions. This business-as-usual approach could have catastrophic consequences for our society.
00:04:12.420 We need to nurture diverse attributes that big tech giants or traditional profit-seeking corporations may not cultivate effectively. Organizational alternatives could include non-profits, worker cooperatives, public utilities, community-funded organizations, or even platforms like Wikipedia, which operates differently in the digital space.
00:04:26.640 Let’s get underway with our discussion by first addressing the decisions we face daily when writing web software.
00:04:34.200 Typically, our careers start on the right-hand side, focusing on features. As we gain more experience and as projects become complex, we move to consider broader architectural strategies.
00:04:44.640 Rails encompasses all these areas. However, there are other popular practices in the community that we must discuss, especially microservices and single-page applications.
00:04:56.820 There are two approaches to these questions. The first is computational thinking, which developers are comfortable with; this is what is taught in computer science curriculums.
00:05:10.440 Computational thinking focuses narrowly on the task at hand, using linear and reductive problem-solving methods. While it works for computer-based problems, it is limited and reductive. It does not consider factors outside that narrow, linear perspective.
00:05:32.220 Therefore, it should be complemented by systems thinking. Systems thinking takes a more expansive, multidisciplinary approach, allowing us to address customer needs in diverse and effective ways.
00:05:45.480 Without systems thinking, we may fall into the trap of premature optimization or be enchanted by computational elegance that may not fit our organization or solve our issues.
00:06:01.380 Now, let's talk about tech patterns and the buzz around them and how we relate to them in this community. About two-thirds of developers work in small to mid-tech companies, and of course, we love Rails.
00:06:13.280 It is powerful for small to mid-tech firms, including larger companies like Shopify and GitHub. However, the big tech companies like Google, Facebook, etc., are categorically different.
00:06:27.600 These companies operate at a completely different scale and have different organizational characteristics. However, they are so powerful in society that they have tremendous influence in tech media, effectively defining patterns that become popular in our Rails community.
00:06:41.220 While we may use Rails, we do not always apply the Rails-first patterns. Instead, we may opt for trending practices based on the allure of their perceived benefits.
00:06:56.460 In a survey of the Rails community from 2022, we see that most Rails developers are still primarily working on monoliths.
00:07:07.020 About a third are experimenting with microservices or service hybrids. Now, I want a show of hands: who here is working in either a microservice or a microservice hybrid environment?
00:07:19.740 Yes, it seems to be maybe a third. On the other hand, single-page apps are much more popular. I think it might be controversial to consider them as big tech patterns.
00:07:31.020 We see that about 40 to 50 percent of Rails projects are using some sort of single-page app front end. In this regard, I found the usage of Stimulus and Hotwire to be surprisingly high.
00:07:43.300 Again, let’s have a show of hands: who here is using a single-page app front end for their main project? It looks like about 40 or 50 percent.
00:07:55.620 Now that we’ve talked about some common architectural choices and how they may influence Rails-first patterns, let's examine how these patterns impact our organizations.
00:08:07.500 In addition to tech patterns, we face business constraints like budget and resources. External market constraints also affect how quickly we must develop certain features.
00:08:21.520 These constraints are typically beyond our control as developers. However, one variable we can influence is the technology we choose in this environment.
00:08:34.380 Think of a metaphor: trying to reach escape velocity. The resources we have available are like fuel, while market constraints represent the Earth's gravitational pull.
00:08:44.520 Both the fuel and gravitational forces are out of our control, but we control how we assemble our technical team to boost our company’s chances of success.
00:08:59.859 A lack of resources compared to need can hinder success. The choices we make about technology significantly influence our resource needs.
00:09:14.640 We can visualize this with an abstraction, where the left side shows resource levels (high vs low) and the bottom axis indicates product development velocity.
00:09:26.520 When there’s a high need for product development velocity, it typically indicates the company is a startup aiming for product-market fit.
00:09:38.880 Conversely, established companies like Google can afford a more static product velocity. The key point I want to highlight is that big tech patterns require higher levels of resourcing, no matter what level of product development velocity you are at.
00:09:52.080 These patterns demand more people, skill sets, servers, and structured development teams (front-end or back-end), while Rails typically requires less overall resources.
00:10:07.500 The main advantage of Rails is tied to its doctrine of convention over configuration and integrated front-end patterns through a monolithic approach.
00:10:20.940 Throughout this presentation, I will present arguments about how single-page app approaches and microservices are big tech patterns that may not be as useful as the tech news would like you to believe.
00:10:33.060 Let’s consider what happens if your organization adopts big tech patterns but experiences an unexpected resource constraint. Suppose your company’s valuations or funding opportunities change.
00:10:45.780 If your organization has a required level of development velocity that is beyond what it can now produce, you may find yourself in what I call the death zone.
00:10:57.300 In this area, the technology you depended on is no longer sufficient. Let’s examine the differences in this scenario from a big tech perspective versus a Rails-based company.
00:11:10.440 This perspective assumes that Rails-first patterns genuinely require a lower level of resources. If this assumption holds true, then a Rails-based organization can achieve greater velocity within the same resource constraints.
00:11:24.300 This effectively reduces the size of the death zone, thereby increasing the chances of the company's success. However, it’s important to consider how often we evaluate these dynamics.
00:11:37.080 For example, do we consider how resourcing or market conditions may change when influencing tech decisions at our companies? This is typically outside the domain of developers who get caught up in computational thinking.
00:11:50.880 I argue that Rails already incorporates this approach, and we benefit when we sustain these considerations.
00:12:05.220 Now, let's look at a case study. At Teleslabs, we recently assisted a client that built its product with a Rails back end and a single-page app front end.
00:12:17.640 This company had raised hundreds of millions of dollars and had around 500 employees with a technology team of about 50 to 75 individuals.
00:12:30.840 After about six years, their tech team grew frustrated with their stack. They encountered issues with their monolith that they felt they couldn't or didn't want to address.
00:12:44.220 During this time, there was significant team turnover alongside the popularization of single-page apps and the notion of 'JavaScript everywhere,' which pushed the team to think that migrating away from Rails was the ideal solution.
00:12:57.840 They then decided to transition to a microservices architecture. In retrospect, we can name this approach a "distributed ball of mud." The Rails API was on one side, a single-page app at the top, and they ended up creating nearly 20 microservices running on Node.
00:13:09.840 The single-page app unfortunately comprised over 300,000 lines of code, which connected to these various services. They faced duplicative data transformation logic, resulting in many interdependencies across the services, which rendered it ineffective.
00:13:25.620 Their problems manifested in extremely slow development timelines, making it difficult to estimate features accurately. Despite estimates from the development team, it often took three times longer than anticipated.
00:13:39.240 Reliability issues arose due to the complex DevOps situation, including maintaining 20 microservices across multiple environments. Every outage was akin to a murder mystery, as they attempted to identify the malfunctioning service.
00:13:55.500 From a systems thinking perspective, experts in microservices had exited the company, yet they required significant product enhancements for growth. The company’s cash burn rate was high, and they were at risk of running out of funds.
00:14:10.020 Their customer base centered around enterprise clients numbering in the thousands—scaling to millions of users was unfeasible for them. So why did they choose this pattern?
00:14:23.220 The client was looking to accelerate development velocity, requiring us to build new processes that would allow for accurate feature estimates while also delivering a significant new reporting feature set in under six months. Thus, it was a complicated and urgent challenge.
00:14:37.620 Our first step was to conduct comprehensive learning sessions, working through technology audits and assessing requisite processes for the new feature. It was essential to engage with team leaders and consider the system perspective.
00:14:52.680 As we delved deeper, it became clear that the team wouldn't be able to deliver the new functionalities using the existing patterns due to their complexity. The experts who had built these patterns had left, making it challenging to navigate such a complicated domain.
00:15:09.900 It was evident to me that we needed to transform the organization's way of working, focusing on two main areas: first, how should we handle the complex single-page app that had been part of their structure from the beginning?
00:15:21.840 Second, how should we handle the data layer, which led to the options of either business as usual or a transition to a newer framework? Perhaps the challenge arose simply because they had chosen the wrong single-page app framework.
00:15:33.240 Conversely, they thought we might suggest revamping their Node services in line with their customer needs. Other teams were already working on overhauling those Node services, reflecting a crisis situation.
00:15:46.860 Referring back to the diagram from earlier, the green dot on the big tech pattern could represent where the company was performing. They had a specific level of high resources and lower product development velocity.
00:16:01.920 Now recognizing their need to improve velocity, they faced dwindling resources due to a high turnover rate and financial constraints. The only way forward, in my assessment, was to leap from the blue line to the red line.
00:16:15.840 If we could transition this client to a Rails-based solution, we could enable them to enhance their product velocity at a lower cost, highlighting Rails’s amazing power.
00:16:30.960 Rails not only yields more cost-effective outcomes than big tech patterns, but it can provide significantly higher product velocity for equivalent resources—at least that was my belief at the time.
00:16:47.520 Our recommendations centered on two primary strategies: transitioning to a server-rendered front end and restoring the monolithic structure, both of which were rather controversial from the client’s viewpoint.
00:17:02.280 The most contentious piece involved shifting to a server-side rendered front end. The previous diagram comes from Sam Stevenson’s presentation at RailsConf 2016 and illustrates the complexity inherent in a conventional single-page app configuration.
00:17:15.000 While it demonstrates two applications interacting—the initial load of an HTML page followed by a JavaScript request to a JSON API—multiple points of complexity arise.
00:17:28.620 In contrast, the multi-page app/server-side rendered pattern simplifies the structure, loading a straightforward HTML page initially before making requests to obtain new content.
00:17:41.520 I supported this shift, believing it would lead to simpler execution flows, making it easier for this company to utilize full-stack patterns instead of separating development teams.
00:17:57.000 This streamlining would minimize costs and resources by eliminating front-end state concerns while maintaining the swift user experience they desired.
00:18:11.760 Moreover, having strong domain models allowed for a clearer development approach. We initially thought Turbo was the best route; however, it didn’t have Internet Explorer support, which was a requirement for the client.
00:18:25.560 Thus, Turbo Links became our focus. The challenge was showing new content rendered from a monolith in a complex single-page application environment.
00:18:39.300 The eventual solution we implemented involved integrating an iframe to present the new server-side rendered front end into the single-page app. Upon clicking a link for the new feature, the content would render in a conventional Rails style.
00:18:53.820 This rendered content would be framed as a single-page app thanks to Turbo Links. Additionally, we could scope authentication to the user at the Rails side by passing their token as a query parameter.
00:19:09.540 However, authentication posed challenges, as did instances where the iframe had to communicate back to the parent single-page app.
00:19:22.680 To address this, we utilized postMessage communication alongside a library called Penpal, which facilitated message exchanges between the two environments.
00:19:35.460 Another technical hurdle involved the rapid provision of new reporting functionality, relying on data access from 12 different services. Comparing the previous single-page app with the new monolithic architecture illustrates this challenge.
00:19:49.920 On the left, we see a classic monolith with a single database structure. On the right, the new microservices pattern had multiple services, each managing its own databases.
00:20:02.520 As we contemplated our options, we debated whether to connect to the API layer or bypass it. The first option, while straightforward, complicated our Rails monolith.
00:20:15.180 The goal was to maintain the advantages of a classic Rails application while accessing various services. We couldn’t consolidate everything into a single database; instead, the solution involved integrating with multiple databases.
00:20:29.160 Restoring a traditional Rails monolith while managing multiple databases allowed us to tap into ActiveRecord's power.
00:20:42.420 A significant aspect of recreating a Rails monolith from a myriad of microservices is the chance to instill a domain-driven design, which can improve overall coherence in application construction.
00:20:55.740 A common challenge in microservices is generated complexity, especially if the services maintain bizarre naming conventions that don’t correlate with their utility.
00:21:09.060 By employing a Rails-driven methodology and creating understandable model names, we enhanced clarity not only for the current reporting feature but also for future feature development.
00:21:25.200 Ensuring that our approach allowed for easy access to the necessary domain information through ActiveRecord was key, despite managing multiple databases.
00:21:38.760 When comparing operational benefits of integrated patterns through full-stack development, we recognized that single-page applications commonly necessitate division among development teams, creating communication obstacles.
00:21:52.620 This fragmentation complicates resources management and is prone to errors. Full-stack development mitigates this complexity, ultimately improving project velocity.
00:22:06.420 This discovery, coupled with our previous efforts, afforded us the ability to create Rails teams, addressing the no-longer-feasible separate development team model.
00:22:20.460 Eventually, the company successfully went public. It was a remarkable instance showcasing the strengths offered by Rails.
00:22:34.020 Elaborating further, let’s tackle the concept of big tech and its implications. With concentration among significant players like Google or Meta, the enormity between them and a typical large Rails user, such as Shopify, is evident.
00:22:48.420 The disparity in lifetime company profits demonstrates the scale difference, where big tech giants operate with completely different operational characteristics.
00:23:02.580 These large companies have billions of users and hundreds of thousands of employees, with their sizeable cash reserves minimizing financial constraints. Instead, their main hurdles often relate to regulation, internal communications, or economic limitations.
00:23:16.260 Given their internal structures, managing operations efficiently becomes exceedingly challenging. Nevertheless, we analyze these companies and tend to assume that their success can be attributed to their adopted practices.
00:23:30.060 This assumption posits that if we utilize microservices or single-page apps, success is guaranteed. Contrarily, the earlier-mentioned resource constraints can jeopardize smaller companies.
00:23:44.460 Observing Google's growth trajectory illuminates that their embracing of microservices was a product of their existing success rather than a determinant of it.
00:23:58.740 In actuality, many Rails developers work in small to medium tech firms, typically comprising fewer than 50 employees, and even flourishing Rails companies tend to have far fewer than 100,000 staff.
00:24:11.940 It's crucial to understand market power's implications, as monopolies have existed for centuries, shaping economies and affecting supplier dynamics.
00:24:26.520 Yet, new dangers arise from these massive digital platforms, where the control over perception is significant. Social media giants may not provide identical content experiences for users, raising questions about trust.
00:24:38.520 Such technological practices primarily serve to modify consumer behavior for ad sales, but they also pose threats related to political control or economic dominance.
00:24:53.520 The danger extends into a realm termed techno-feudalism—where products are perceived as free, yet the underlying ownership dynamics mirror exploitative historical systems.
00:25:07.380 The primary take-home from this is that when engaging with big tech, companies must understand that these patterns may not fit their needs. Big tech patterns are inherently correlated with a scale that doesn’t always apply to smaller entities.
00:25:21.120 In closing, while the adoption of big tech patterns is tempting, embracing Rails-first practices is beneficial to your organization as a general approach.
00:25:35.280 Recognizing the tech architecture that you choose to implement must be aligned with your organization’s needs will be crucial for success.
00:25:49.620 This alignment is critical as you progress in your career, whether by leading teams or a technology group. As developers, focusing on computational thinking and trending tech can distract from the bigger picture.
00:26:03.000 Instead, prioritize understanding the organizational dynamics within which you operate, as Rails fosters the community's success. Thank you for helping make the Rails community such a fantastic place.
Explore all talks recorded at RailsConf 2023
+81