Future-Proofing Your 3rd Party Integrations

by Jeffrey Matthias

This presentation by Jeffrey Matias at RubyConf 2014 focuses on best practices for future-proofing third-party integrations in software development. Matias, who has experience in refactoring legacy code and managing third-party interactions, provides a collection of practices that emphasize separation, predictability, and replaceability in interacting with third-party services.

Key Points:

Separation of Logic: It is crucial to keep third-party service implementation separate from internal application logic. This separation enables developers to maintain flexibility and avoid getting tied to external service implementations.
Predictable Behavior: Establish predictable interactions with third-party services through comprehensive testing and clear testing patterns. This predictability allows developers to confidently deploy their code.
Replaceability: The ability to seamlessly swap out or upgrade third-party services and APIs is vital. This flexibility is facilitated by well-defined adapters and data transfer objects (DTOs).
Adapter Pattern: An adapter serves as a gateway to third-party systems. Developers should create a private adapter gem to encapsulate and manage third-party interactions, preventing the need to manage multiple repositories. Existing third-party SDKs can still be leveraged within the adapter.
Common Vocabulary: Clearly defining terminology used across the development team is essential for effective communication. Matias emphasizes that having consistent terms helps avoid confusion when discussing integrations and services.
Data Transfer Objects (DTOs): DTOs are used to codify contracts that specify the required parameters for third-party services, enhancing clarity and ensuring correct interactions.
Testing Practices: Testing should ensure that both integration and unit tests accurately reflect the behavior of third-party services. Setup and teardown processes are critical, especially in maintaining a clean test environment. Using sandbox environments can prevent data corruption.
Using Stubs Effectively: Stubs can accelerate the development process, but it is essential to ensure they accurately reflect the behavior of real third-party services. Regular updates and maintenance of these stubs are necessary to prevent discrepancies.

Conclusion:

Matias underscores the importance of maintaining clear boundaries between third-party services and company logic, utilizing adapters and DTOs for flexibility and replaceability, and having a rigorous testing approach. By following these practices, developers can achieve a structured, maintainable system that simplifies future changes to third-party services. In doing so, teams can significantly enhance their capacity to adapt to changing requirements or services in the future.

00:00:18.199 I'm Jeffrey Matias. I am a developer at SRID, and this is a presentation on a collection of practices about future-proofing your third-party services.

00:00:24.920 I say "a collection of practices" because I'm not going to tell you they're all best practices. They're working pretty well for us.

00:00:31.000 I hope you walk out of here today feeling like you've gained at least one insight that you can implement. Hopefully, you'll find more than that, but if we can aim for that, we'll be great.

00:00:36.760 So, I work on a team whose job is to refactor legacy code. We have this monolithic PHP app, and our job is to manage our third-party interactions. We're going back to refactor the way we're interacting with those services.

00:00:47.440 We're pulling them out of the PHP app and rewriting them in a Ruby codebase. Since we get to start from scratch, we can decide how we want to do these things.

00:01:01.960 Some of the work you will see today comes from my experience with Pivotal Labs about a year ago, and we've built some of our own structures on top of the concepts they provided. We've created a solid path for handling third-party services.

00:01:14.640 We've implemented this approach with several existing APIs, and so far, it's worked out pretty well.

00:01:26.720 The first thing you need to consider when refactoring or building third-party implementations is what you want to achieve with them.

00:01:39.960 The first goal is to keep the way third parties handle their code separate from the way you implement your own ideas. There should not be much correlation or crossover between the two.

00:01:53.000 As soon as you buy into their implementation, you lose the flexibility that we as developers really value.

00:02:05.000 The next part is making your third-party behavior predictable. How do we achieve predictability in code? We can accomplish this through testing and the implementation of testing patterns, which I will discuss further.

00:02:16.720 Ultimately, if you can be confident that things will work before you deploy, you have won.

00:02:21.800 Lastly, focus on making your third-party service replaceable. I understand that many of you work for companies that are third parties. While being replaceable is essential, it doesn't just mean you want to replace a third-party service.

00:02:34.840 It also pertains to being able to replace APIs. If a company upgrades their API or introduces new features, maintaining the ability to swap out that implementation or API seamlessly is crucial without needing to delve into your existing code.

00:02:53.200 To illustrate this, consider how I'm working diligently to demonstrate my degree in art here. This diagram represents our billing app, the adapter layer, and the third-party system.

00:03:05.360 Notice that I refer to a third party and place the internet as a dividing line. This distinction is important because it doesn't solely refer to systems that exist across the internet; it encompasses any third-party integrations, including service-oriented architectures.

00:03:25.840 As an example, we have a database abstraction. Love it or leave it, it’s a part of our lives for now, but many do not appreciate it and wish to keep their code flexible.

00:03:33.400 Flexibility is necessary for reimplementing that database abstraction without untangling the spaghetti code we often end up with.

00:03:39.799 It's worth noting that our dependencies generally run in one direction. Third parties rarely know about your application, and you want to maintain that isolation.

00:03:45.760 Now, back to the adapter level, the most crucial aspect to understand is that the pay-me adapter is merely a gateway. You likely anticipated that, as I jumped to the next slide.

00:04:06.439 The adapter is a standard Ruby gem that you might generate using 'bundle gem [adapter_name]'. However, you won't push it to a public repository; instead, you'll keep it within your project. In your gem file, you'll specifically point to the relative path from your app's root, naming it there.

00:04:50.320 Fortunately, gem files allow you to make such references. Ideally, you’re already familiar with this pattern elsewhere in your code. It’s critical to keep the implementation of the third party private as it details how we manage interactions with it. You want to avoid dealing with multiple repositories.

00:05:09.960 Despite suggesting using a gem, many third-party services already have gems available. It's not to say that you cannot use them; you simply want to wrap their execution within your own gem.

00:05:30.319 If there is a third-party SDK available, leverage it. There's no reason to reinvent the wheel, particularly if the SDK was developed by someone from that company who understands the intricacies.

00:05:55.280 Our gem essentially serves as a series of service level objects. These objects are a straightforward path through the functionalities and do not depend upon one another unless absolutely necessary.

00:06:07.080 They communicate with our system using our internal logic and translate that logic into the language suitable for the third-party system.

00:06:14.120 When we initialize our application, which in our case is a Rails app, we hand over these individual services. Throughout the codebase, we refer to the configuration instance using a singleton method.

00:06:27.639 As a result, other parts of the codebase remain unaware that we are interacting with the pay-me service or the Stripe service. The initializer is the only place in the main application where these service names appear.

00:06:46.280 This is crucial because it supports the flexibility needed in your codebase. One of the most important practices applicable to any project is defining your vocabulary.

00:07:11.720 Getting everyone on the same page and ensuring that they describe the same things in a consistent language is essential. It can be quite challenging to gather product owners, stakeholders, and developers in the same room to discuss terminology.

00:07:40.000 It may surprise you how the same word can have different meanings for various people.

00:07:51.759 Therefore, taking a step back and defining your language is the first step. If you don't have access to others, try your best to define terms as a team or as a developer.

00:08:03.280 Coordinating a group-wide buy-in is preferable, but even individual clarity is crucial.

00:08:22.760 Using common language aids in keeping concepts from third-party services outside of how your company handles its business.

00:08:39.800 For instance, when we refer to a package in our billing provider, that translates into a product rate plan.

00:08:45.760 When we mention an add-on in our terminology, it similarly means a product rate plan. Fortunately, talking about a coupon translates effortlessly into product rate plan charges, making it clear in conversations.

00:09:04.920 For us, using terms like package, add-on, and coupon helps avoid convoluted nested objects that exist in our billing provider. Our services handle these translations.

00:09:18.920 For example, we have a package service that speaks the language of product rate plans when communicating with our billing provider. We also have an add-on service that does the same, albeit in a different context.

00:09:38.240 While the API calls may be similar, the nuances don’t bleed into the higher levels of our stack.

00:09:52.480 I've mentioned maintaining this separation through gems; we implement data transfer objects (DTOs) as well. DTOs serve as codified contracts indicating the parameters necessary for each service.

00:10:10.640 Instead of relying on a cryptic options hash, DTOs let us express exactly what the service requires. Additionally, we have abstract services that define method signatures.

00:10:25.680 If a method is not implemented, it will simply raise an error, prompting developers to remember the requisite functions.

00:10:37.680 While this may add some overhead, it codifies the interface. Essentially, between DTOs and abstract services, we establish a clear definition for implementing a new service.

00:10:52.040 Unfortunately, human developers sometimes forget these standards, but we have a solution I'll elaborate on shortly.

00:11:04.440 Looking back at the graphic I shared before, we have our adapter—a framework for our billing application and connected third-party service. The DTOs and account services exist in a separate gem we call the adapter interface gem.

00:11:30.480 This defines how you interact with services, ensuring that as long as you meet the interface, you can swap in new services without issues.

00:11:47.760 While the adapter interface is utilized by the billing application itself in its gem file, to access DTOs for proper instantiation, the pay-me adapter must also implement it since its services inherit from these abstract classes.

00:12:02.720 We test drive DTOs because they contain logic, while the abstract services are thin and focus solely on raising errors. I believe firmly in thorough testing, but we also shouldn't overtest.

00:12:28.440 Returning to the topic of maintaining abstract services, we have a gem that matches everything through testing, specifically a shared context that verifies the public interfaces of classes.

00:12:43.600 This ensures that the public instance and methods of your classes reflect their inherited definitions, and it alerts you when you’re passing in incorrect argument numbers.

00:13:00.000 Though concepts of interfaces may conflict with Ruby's DNA, we are self-aware in that we have termed our gem 'uptyped', playing on the concept of 'uptight'.

00:13:16.640 Clearly, this is not something one would normally advocate widely, but for our situation, it helps maintain robust documentation, making it easier to replace third-party services.

00:13:38.440 One essential practice is observing the four components of testing. Some may think there are only three, but we have setup, expectations, the kickoff, and teardown.

00:13:55.520 Your approach may vary based on whether you are testing for return objects or side effects—this may switch the order.

00:14:12.920 Your teardown process becomes significantly more critical, especially when interacting with third-party services.

00:14:27.000 Next, always avoid assuming that your test data exists within the third-party system. Even if you logged into the UI and added data for testing, that is not a repeatable approach.

00:14:46.240 Make it a part of your testing framework to call the third party to ensure that test data is present. If available, use a sandbox to prevent corruption of customer data.

00:15:04.120 This allows you to engage in testing more safely.

00:15:19.760 To illustrate, let’s take a closer look at our payment environment. We refer to this as an environment pattern that enables easy tracking and creation methods.

00:15:47.360 When you check the initializer of our test environment, you will notice trackers that maintain IDs for things we create. There’s an 'ensure' block to handle cleanup.

00:16:05.720 In our user creation process, we utilize the pay-me user gem and keep track of created IDs. Once our tests conclude, we return to clean up the data within the third-party service.

00:16:22.800 This means once you build that environment, cleanup occurs automatically, requiring no extra thought, which is pivotal for maintaining a pristine third-party system.

00:16:44.120 Unlike some legacy codebases that result in unwanted dozens of accounts created during tests, we maintain a respectful approach.

00:17:02.960 Regularly hitting your third-party services, especially for test driving, can be slow. To speed things up during development, consider implementing stubs to act as your service.

00:17:20.840 There are two main approaches: create a fake service that mimics your API responses or record VCR sessions that log all HTTP interactions in a YAML file.

00:17:34.280 In unit tests, we call our services to interact with the third-party or the stub, as this is the objective of the service.

00:17:54.080 However, because HTTP calls are complex to stub without employing tools like FakeWeb, which might unnecessarily complicate your testing framework, you need to be cautious.

00:18:32.440 Our testing environment showcases an instance of our pay-me service. We initiate that within a scoped variable, apply our creation methods, and kick it off.

00:18:56.640 By the time the test concludes, we revert back to the clean environment, ensuring that if any issues arise during testing, we are more likely to invoke that cleaning procedure afterward.

00:19:10.440 This same approach applies to our integration specifications, but we ensure that instead of referencing a specific adapter, we inject dependencies during our initialization process.

00:19:38.720 At the integration level, we may have built wrappers using SRID language during these tests. Regardless of how you’ve constructed your environment, it should maintain that the application doesn’t recognize which services it interacts with.

00:20:03.840 It's important to avoid the risk of any implementation details leaking through.

00:20:19.440 Here’s a reminder, when you choose to stub your services, you must sustain the stubs effectively.

00:20:38.000 If you’re using a fake service, maintain a routine to verify its behavior aligns with the actual third-party service.

00:20:56.720 If utilizing VCR, remember to delete and re-record cassettes regularly.

00:21:09.480 The underlying goal is that your tests need to successfully hit the actual third-party integrations.

00:21:36.000 Thus, your tests need to encompass both creating data and subsequently tearing everything down.

00:21:50.640 If you are working in continuous integration, setting it up so that tests bypass fakes and interact with the live service can be incredibly beneficial.

00:22:04.400 It can allow you to catch potential issues earlier by running tests against actual APIs instead of relying solely on stubs.

00:22:27.600 We eventually wrote code that functioned appropriately, but it wasn’t part of the published API, which led to complications when the service altered its code.

00:22:51.440 Continuous integration could have provided early alerts for these problems, allowing for prompt corrective actions.

00:23:14.080 Reflecting on our goals for third-party implementations, one was to keep their logic separate from our own, ensuring the pay-me adapter retains a distinct separation.

00:23:29.440 The next goal is to maintain predictable third-party behavior, which relates back to comprehensive testing and establishing where that should occur.

00:23:50.360 Regular usage of stubs in tests without compromising the real service ensures everything works correctly.

00:24:03.600 Finally, consider the replaceability of services, which is where DTOs and abstract services play an essential role.

00:24:13.920 With these elements clearly defined, they provide concise documentation for the necessary methods for implementing or replacing services.

00:24:24.840 Moreover, DTOs dictate what the required data structure should look like, providing clarity.

00:24:45.760 Furthermore, this structure enhances flexibility, allowing you to shift and adapt operations accordingly.

00:25:01.680 We recognized, for example, the strong integration with our billing provider often limited our capabilities.

00:25:14.440 With prior implementations often setting rigid processes, we’ve found that delegating third-party implementation into an adapter allows greater operational flexibility.

00:25:29.760 It permits new product additions without concerns over existing implementations.

00:25:49.800 Portability is another key aspect your services must provide, especially if you need to shift towards a producer-consumer model in the future.

00:26:06.000 The groundwork for any necessary reimplementation resides within the well-defined interfaces.

00:26:13.440 Establishing your adapters and DTOs creates a framework to facilitate that transition.

00:26:28.000 Above all, focus on determining your internal vocabulary. Communication issues can often stem from differing interpretations of the same terms.

00:26:45.880 Next, create the adapter interface to define context and usage clearly. This abstraction removes muddying of implementation details and keeps discussion on higher-level concepts.

00:27:01.920 Once you create the adapter, your gem should implement the interface and manage the structured elements.

00:27:15.840 It is essential to maintain a pristine test environment to uphold standards.

00:27:24.720 Providing stubs helps accelerate testing to maintain a focus on development.

00:27:36.960 Lastly, ensure proper management of test data—removal and upkeep are critical to avoiding unnecessary sprawl.

00:27:52.160 I would like to thank my team, the portal team, and my big brother Carl for encouraging my programming journey. If anyone has questions, feel free to reach out.

00:28:07.680 If you're working on these topics as well, my colleagues are here to help with inquiries.

00:28:29.440 Lastly, thank you all for your attention.