Microservices
Minecart - A story of Ruby at a growing company

Summarized using AI

Minecart - A story of Ruby at a growing company

Matt Wilson • January 27, 2020 • Earth

In the presentation titled Minecart - A Story of Ruby at a Growing Company, Matt Wilson, an engineer at Square, discusses the integration of Ruby applications within the Java service infrastructure as Square transitioned towards a service-oriented architecture (SOA). The key themes and insights from the talk illustrate the versatility of JRuby and the techniques involved in achieving a seamless workflow between Ruby and Java environments.

Key Points Discussed:

  • Rise of JRuby: Wilson begins by highlighting the advantages of JRuby, such as mature garbage collection and the capability to utilize Java libraries, emphasizing that it is a platform that can interpret Ruby applications effectively despite some challenges like slow startup times.

  • Square's Transition to SOA: The talk outlines Square's journey from monolithic applications to a more modular service-oriented approach, stressing the need for a common framework that facilitates collaboration among engineers without conflicting changes.

  • Service Communication Challenges: Wilson elaborates on overcoming the challenges of service communication, specifically the construction of a message bus and the nuances involved in implementing features across Ruby and Java, including maintaining code parity and dealing with race conditions.

  • Integration with Major Projects: The narrative recounts experiences from two major projects, Wallet and Starbucks, illustrating the learning curves around decision sequencing and effective communication between teams. The Starbucks project notably increased the number of services Square handled, signifying a clear expansion.

  • Minecart Project Overview: The central part of the presentation introduces the Minecart project, which was designed to provide a thin API layer that harmonizes Ruby development with the existing Java infrastructure, achieving interoperability while preserving best practices in Ruby.

  • Infrastructure Management: Wilson details the various components that make communication between services robust, including mutual SSL for security, protocol buffers for API definitions, and a custom protocol, Sake, which streamlines request management and improves error handling.

  • Optimizing Development and Production: The ultimate goal of Minecart is to create a cohesive environment where developers can focus on adding value without being encumbered by the complexities of intertwined services.

Conclusion and Takeaways:
- The presentation emphasizes the importance of optimizing a production environment for developers, ensuring they can communicate effectively and build applications that are compatible across technology stacks. By leveraging JRuby and planning appropriately, engineering teams can thrive in a multifaceted environment, as demonstrated by the success of Minecart and other projects at Square.
- Wilson concludes with his belief in the scalability of the framework, allowing for effortless integration and development of new services, with the potential for anyone at Square to innovate seamlessly in the integrated system.

Minecart - A story of Ruby at a growing company
Matt Wilson • January 27, 2020 • Earth

This is the story of Ruby at Square. As Square grew, we invested heavily in a service oriented architecture (SOA) and mainly used Java as our language of choice for new service development. Recently we finished a project, named Minecart, to deeply tie Ruby applications into our Java service infrastructure. To accomplish this we had to dive into the bowels of JRuby. To date, Square is roughly around 50/50 Java services vs Ruby services and everyone enjoys the benefits of a standard ecosystem. I will share many of our learnings along with a couple of practical examples and pitfalls for integrating Ruby into a custom Java framework.

Help us caption & translate this video!

http://amara.org/v/FG7J/

Rocky Mountain Ruby 2013

00:00:30.519 Hello everyone, I'm Matt Wilson, and I'm an engineer at Square. So, real quick, how many people have heard of someone deploying to JRuby in production?
00:00:36.320 Alright, that's smaller than I thought—about 45% of the audience. How many people actually deploy to JRuby in production?
00:00:43.200 Okay, that's like three percent of the audience. So, I love JRuby. JRuby has mature garbage collection and native threading.
00:00:49.840 It has the promise of fine-tuning slow parts of your application, and you can use well-known crypto libraries like Bouncy Castle without having to shell out to the command line.
00:01:05.000 Now, JRuby is not a ploy to get Java into your stack. I think they have not done a great job branding it. JRuby is simply a platform that can interpret Ruby, and it does that well.
00:01:16.840 There are a couple of problems with JRuby. One is startup time, and the second is that sometimes you will get Java stack traces. But I believe Charles can fix this, and I believe in Charles.
00:01:31.040 Let's look at what we're going to talk about. I will share many of our learnings along with a couple of practical examples and pitfalls for integrating Ruby into our custom Java framework.
00:01:42.600 To begin, there is a lot you have to do to convert from a typed language to something not typed. Can everyone see my pointer? I'm just kidding. I wrote that presentation—it was really boring.
00:01:55.159 We have a lot of technical details on our blog at corner.squareup.com if you're interested in how this was done. But what was really fascinating about the talk as I developed it is that Square's Ruby stack leverages our production environment completely.
00:02:06.520 So, what is a production environment, and why couldn't Ruby leverage it before? I'm going to talk a little bit about that today. What I'd like you to take away from this talk is, first of all, that JRuby is awesome.
00:02:25.879 The JVM is something you should consider deploying to or using, even if you never intend to drop down and write Java, Clojure, or other JVM languages. As you split out into a service-oriented architecture, it's critical that you think about building a common framework that all your apps are deployed on top of.
00:02:52.159 This gives you the ability to design and engineer the production environment. For those of you more polyglots, I'll show you an example of a very large Java framework that many of our Java engineers worked on. It's a good framework that allows us to seamlessly integrate Ruby on top in a way that any Ruby engineer off the street can come in and develop Ruby using all the familiar toolkits without understanding what's happening underneath.
00:03:31.599 Now, let's give a brief history of how Square began. I got very lucky. I joined Square early on and worked on a project called Wallet, which is a second application from Square.
00:03:50.160 The idea was you could walk into a store, and by proximity, we could detect when you were close to a merchant. If you had given authorization for that merchant to charge you, your name and picture would show up on the register.
00:04:06.120 You could walk up, order a coffee, and say, 'Charge Matt,' take your coffee, and leave without touching any technology. It changed how people interacted, and it was a really cool project to work on.
00:04:27.000 How this was done was we took four people out of one of the four teams, placed them in a room, and began developing. It was a three-month sprint to add the ability to store credit cards at Square, publish merchant locations for customers to find, and allow customers to search for these merchants.
00:04:43.880 The reason we were able to achieve this was due to highly optimized communication within the team; we came up with a custom vision for where we wanted to go and made decisions autonomously. After three months, we emerged, and the company supported us for another month to push it out, and it launched successfully.
00:05:02.479 Later, we faced a common challenge that many growing companies experience: how do we depart from our monolith?
00:05:07.600 I want to emphasize that monolithic apps are great. Service-oriented architecture is a people solution; it is not a code solution or an availability solution. It's a way for many engineers to collaborate without affecting each other.
00:05:32.520 As we approached this transition, one of the key challenges you must solve is your message bus. So what is a message bus? It defines how data transitions or moves through your system.
00:05:50.040 One implementation is feeds. Feeds are basically a pagination API where you page through data via an append-only list. We wrote it in Java first, and it turns out there are some nuances around sharding because you want to horizontally scale out your consumers.
00:06:06.720 We found additional complexities around publishing, using auto-increment, and dealing with race conditions in the database. There was a significant amount of code involved, but we got it working in Java, and Java can consume from our Ruby services. Now we had to get it working in Ruby, but there were challenges.
00:06:45.000 Our implementation was based on ActiveRecord, and it was daunting. We had a lot of code to maintain, so many would argue it's not idiomatic Ruby.
00:06:58.560 The truth is, it is challenging to maintain feature parity across languages. Whether you have language one and language two, you must choose a primary to implement first and then copy over the class structure. If there’s a security vulnerability or a new feature, you need to make that change in both languages.
00:07:27.000 I don't believe there is a mythical team that can effectively leverage two stacks or build two stacks with feature equivalents over time.
00:07:47.000 But it worked! Ruby was finally able to communicate with Java, and this was essentially the only infrastructure we had in our Java stack at that point. We continued building new Ruby and Java services, but then...
00:08:07.680 Starbucks came along, which was a completely different animal. From day four, we had 50 engineers working on this project. Every time you swipe a card at a Starbucks location, it goes through Square's network. Because of the work I did on Wallet, I got to architect the Wallet integration with Starbucks.
00:08:45.240 This experience taught me about decision sequencing. What decisions needed to be made today? What could we find room to experiment with? And what decisions could we postpone until later?
00:09:03.600 A couple of engineers collaborated to define how services would communicate within the production environment. They established how we would maintain these services, provide necessary SLAs, and expose our APIs both internally and externally.
00:09:31.679 At that time, we were in a single data center with about six apps in production. After three months, we had 15 services in production across two data centers, actively processing card transactions. This challenge significantly advanced our infrastructure.
00:10:06.560 What made the Starbucks project successful was that we optimized for communication. By defining how systems would function in production, we created a framework that all teams could work within iteratively. We had a clear goal to strive toward.
00:10:32.560 This developed very differently compared to how the Wallet project evolved, but it is amazing that both development styles can coexist. Some might think they conflict, but they truly do not; day-to-day agility is how we develop at Square.
00:10:57.600 The question then became: how do we leverage our production stack? We built out technology that made Square robust, especially while other companies aimed for exceedingly high availability.
00:11:23.440 One option was to build a Ruby service container and keep it in feature parity with the Java service container, as I previously described. Alternatively, we could allow Ruby to access some resources in the service container and communicate through a Java API.
00:11:55.640 Another idea was to overlay Rails on the service container and communicate with resources via a Java API. However, I found it most appealing to create a thin API layer that adapted the concepts in the service container to be more idiomatic Ruby.
00:12:34.080 I believed this approach could succeed and remain maintainable since the service container was already optimized for application developers. It offered a few key concepts we could represent in a project we called Minecart.
00:12:56.760 Both sides would develop independently while maintaining a stable API. Now, when I refer to service containers and stability, what does that mean? I like this picture because it reminds me of something Steve Jobs once said: software is like scaffolding.
00:13:21.760 By using Objective C and other native APIs, you can start at a higher level and build a taller building. If you start without any scaffolding, you can only build so high before it falls over. Excitingly, the production environment introduces a new layer of optimization.
00:13:46.840 During development, you often build projects, ship them, and several months later, a team may decide to create something new. However, this often leads to losing out on wisdom gained from the initial project.
00:14:07.440 Each new project may not integrate well with the previous ones, leading to confusion. In contrast, what you want in your production environment is complexity managed like a city. You want all road sizes to be uniform, power grids to supply buildings uniformly, and common fixtures across the design.
00:14:21.879 This uniformity enables infrastructure engineers to balance trade-offs between short-term and long-term goals, allowing them to predict the success of projects without running into unforeseen complications.
00:14:44.480 Regarding scaffolding, production scaffolding like Heroku and App Engine offer particular APIs for monitoring service performance. Meanwhile, environments like EC2 or Rackspace give you more control without caring about the deployed code.
00:15:01.960 We also have frameworks optimized for development environments, such as Rails or Java Play, which are not necessarily designed for production but perform well in it nonetheless.
00:15:32.320 Common features found within our framework include consistent packaging of dependencies for uniform deployment, an application lifecycle managing traffic, and rolling deployments to avoid losing requests.
00:15:44.760 Additionally, we require trust and security measures to maintain the integrity of sensitive information, monitoring for individual nodes and overall performance, and systems for job management and messages to facilitate seamless data flow.
00:16:17.040 We also implement health checks to ensure all downstream dependencies function properly, alongside distributed tracing and common logging to pinpoint errors when they occur.
00:16:43.200 The ultimate goal of all this work is to enable application engineers to focus on creating value for customers. As an application engineer myself, the most essential part is having an idea, building it, and shipping it without burdensome maintenance.
00:17:07.680 When we optimize for the production environment, we elevate the conversation back to the application's needs. For example, if I wanted a feature that charged a wallet credit card, I could sit down with security and application engineers to discuss the necessary security protocols.
00:17:29.840 With a common system, the security engineers can intuitively understand where the service fits into our infrastructure, leading to much more efficient collaboration and increased chances of success for complex products.
00:17:58.880 Unfortunately, optimizing for the production environment can sometimes conflict with development needs. That was the goal of Minecart: to create a system that maximizes the advantages of both.
00:18:22.080 Now, let's explore what Ruby code at Square looks like. This is a standard Rails controller, and here’s two special methods: kitchen and employee. These methods serve as APIs to communicate with other services.
00:18:39.960 We have a remote kitchen service and a remote employee service. We require login, and there’s a curious current user token used for tracking user activity without having to pass around actual user objects with behaviors.
00:19:07.440 In our distributed network, a system called Multipass handles user authentication, exchanging a session for a user. It uses the session token to retrieve the user token, allowing seamless interaction throughout the system.
00:19:21.680 Next, we call kitchen delivery orders to get the order list from the corresponding response. However, there's a specific syntax meaning, here: that we’re performing this request synchronously.
00:19:36.880 To comprehend what's occurring in this request, we should briefly explore the story of service-to-service interaction within our project.
00:19:58.840 There are three components crucial to our connection. The first is mutual SSL, which defines our trusting security mechanism. Then there's protocol buffers for defining the API, and finally, we have a custom protocol for service-to-service communication called Sake.
00:20:29.760 Starting with mutual SSL: it works by allowing service A to talk to service B, which then checks its access control list to ensure service A is permitted to establish a connection.
00:20:47.960 However, if service B tries to talk to service C but isn't authorized, that connection gets blocked. This creates a secure, granular framework for communications between services.
00:21:04.960 Once the connection is established, we examine our API. Protocol buffers function as value objects, providing a snapshot of an object at a specific moment, facilitating easy data transmission.
00:21:30.040 We define our services by saying when we call 'show' with a driver ID, to provide a corresponding driver response. Additionally, we can specify properties within the API definition.
00:21:56.240 What actually facilitates data movement is our protocol called Sake, a standard socket that manages requests and responses using message IDs, ensuring requestors receive the correct responses.
00:22:17.280 The Sake protocol also has this unique concept of a side channel that continues to persist throughout requests. When service A makes a request to service B, which in turn requests from service C, the side channel remains active.
00:22:55.320 This allows for exception chaining. If service C encounters an error, service B will relay the same error back to service A. Thus, if you're alerted regarding service A's issues, you can swiftly identify service C as the source.
00:23:23.320 Now that we're sufficiently prepared, let’s see what happens when we make our request to the Pizza Kitchen service. If the Pizza Kitchen service is unavailable, it automatically redirects to another service.
00:23:45.000 You can loop through all the orders and integrate the driver names, as the data is stored across separate services. Notably, we avoid waiting on any requests, as this script will use promises to access future data as needed.
00:24:05.240 This returns a promise, which provides access to future data. At the time of instantiation, it waits for the request to conclude and returns the requested value.
00:24:20.560 All we have to do is render a simple HTML template for the order list, and it integrates smoothly. It's designed in such a way that a developer accustomed to standard Ruby can easily adapt to our RPC API and develop significant products.
00:24:37.840 This system, although complex, does not complicate the development story. Regarding the database, we're still utilizing a connection pool from our service container while adhering to the SQL ORM practices we expect.
00:25:10.360 Jobs operate simply, akin to how you'd typically expect, while load balancing is distributed across the cluster for RPC. You can opt for synchronous, asynchronous, or callback styles and set properties about those connections.
00:25:50.680 We also accommodate RPC calls on Rails controllers originating from web requests. If another service invokes a request, we offer a Rails-like controller that translates a protocol buffer into a Ruby API resembling Rails.
00:26:07.960 What's remarkable is that no one externally can discern whether they're interfacing with a Ruby or Java application, enabling full flexibility.
00:26:29.840 Lastly, regarding the message bus, while substantial amounts of code underpin this system, it doesn't lead to excessive Ruby code—just a single implementation.
00:26:47.200 The developers of our feeds infrastructure can depend on the service container's job system while providing flexibility to Ruby developers, letting them use tools like Sidekiq or Rescue according to their requirements.
00:27:12.680 This fulfillment carries an anticlimactic feel, but the payoff is the emergence of a new service, the Global Name Service, aimed at facilitating service discovery.
00:27:29.560 In environments where redundancy is crucial, services typically interact with resources through a load balancer, which identifies out-of-service nodes. However, once you expand to multiple data centers, the common practice is to manage routing directly in the application.
00:28:02.560 GNS resolves this challenge by integrating into the application lifecycle, allowing an app during startup to request service connections, which GNS fulfills without delaying operations.
00:28:25.840 As clusters of clients bear the load balancing logic in software, they preferentially route traffic to the local data center while handling failures. Upon signaling readiness, GNS seamlessly begins directing traffic without hassle.
00:28:38.480 The question becomes: what does an application engineer need to do to access this functionality? Essentially, they only need to bundle update Minecart.
00:29:07.320 This reflects the promise of an integrated world that Minecart aims to deliver. The ultimate goal of Minecart is to provide a consistent production story that ensures seamless communication among services.
00:29:26.080 Currently, we have approximately 150 services registered, with around 110 deployed. The concept is that anyone can come in, have an idea, build it, and see it working through development and production.
00:29:46.000 This reduces time spent worrying about dependencies, allowing application engineers to focus on shipping remarkable products. During our last hack week, a team of three managed to develop two new services and integrate them with seven existing services without disrupting others.
00:29:59.000 I brought along Daniel Nyman and Matthew Todd, who played vital roles in bringing this vision to reality. They would be delighted to discuss their experiences if you're interested.
00:30:09.320 I would also love to talk with you about it, and with that, are there any questions?
00:30:19.720 Thank you!
Explore all talks recorded at Rocky Mountain Ruby 2013
+13