Talks
Future-Proofing Your 3rd Party Integrations
Summarized using AI
Future-Proofing Your 3rd Party Integrations
by Jeffrey MatthiasThis presentation by Jeffrey Matias at RubyConf 2014 focuses on best practices for future-proofing third-party integrations in software development. Matias, who has experience in refactoring legacy code and managing third-party interactions, provides a collection of practices that emphasize separation, predictability, and replaceability in interacting with third-party services.
Key Points:
- Separation of Logic: It is crucial to keep third-party service implementation separate from internal application logic. This separation enables developers to maintain flexibility and avoid getting tied to external service implementations.
- Predictable Behavior: Establish predictable interactions with third-party services through comprehensive testing and clear testing patterns. This predictability allows developers to confidently deploy their code.
- Replaceability: The ability to seamlessly swap out or upgrade third-party services and APIs is vital. This flexibility is facilitated by well-defined adapters and data transfer objects (DTOs).
- Adapter Pattern: An adapter serves as a gateway to third-party systems. Developers should create a private adapter gem to encapsulate and manage third-party interactions, preventing the need to manage multiple repositories. Existing third-party SDKs can still be leveraged within the adapter.
- Common Vocabulary: Clearly defining terminology used across the development team is essential for effective communication. Matias emphasizes that having consistent terms helps avoid confusion when discussing integrations and services.
- Data Transfer Objects (DTOs): DTOs are used to codify contracts that specify the required parameters for third-party services, enhancing clarity and ensuring correct interactions.
- Testing Practices: Testing should ensure that both integration and unit tests accurately reflect the behavior of third-party services. Setup and teardown processes are critical, especially in maintaining a clean test environment. Using sandbox environments can prevent data corruption.
- Using Stubs Effectively: Stubs can accelerate the development process, but it is essential to ensure they accurately reflect the behavior of real third-party services. Regular updates and maintenance of these stubs are necessary to prevent discrepancies.
Conclusion:
Matias underscores the importance of maintaining clear boundaries between third-party services and company logic, utilizing adapters and DTOs for flexibility and replaceability, and having a rigorous testing approach. By following these practices, developers can achieve a structured, maintainable system that simplifies future changes to third-party services. In doing so, teams can significantly enhance their capacity to adapt to changing requirements or services in the future.
00:00:18.199
I'm Jeffrey Matias. I am a developer at SRID, and this is a presentation on a collection of practices about future-proofing your third-party services.
00:00:24.920
I say "a collection of practices" because I'm not going to tell you they're all best practices. They're working pretty well for us.
00:00:31.000
I hope you walk out of here today feeling like you've gained at least one insight that you can implement. Hopefully, you'll find more than that, but if we can aim for that, we'll be great.
00:00:36.760
So, I work on a team whose job is to refactor legacy code. We have this monolithic PHP app, and our job is to manage our third-party interactions. We're going back to refactor the way we're interacting with those services.
00:00:47.440
We're pulling them out of the PHP app and rewriting them in a Ruby codebase. Since we get to start from scratch, we can decide how we want to do these things.
00:01:01.960
Some of the work you will see today comes from my experience with Pivotal Labs about a year ago, and we've built some of our own structures on top of the concepts they provided. We've created a solid path for handling third-party services.
00:01:14.640
We've implemented this approach with several existing APIs, and so far, it's worked out pretty well.
00:01:26.720
The first thing you need to consider when refactoring or building third-party implementations is what you want to achieve with them.
00:01:39.960
The first goal is to keep the way third parties handle their code separate from the way you implement your own ideas. There should not be much correlation or crossover between the two.
00:01:53.000
As soon as you buy into their implementation, you lose the flexibility that we as developers really value.
00:02:05.000
The next part is making your third-party behavior predictable. How do we achieve predictability in code? We can accomplish this through testing and the implementation of testing patterns, which I will discuss further.
00:02:16.720
Ultimately, if you can be confident that things will work before you deploy, you have won.
00:02:21.800
Lastly, focus on making your third-party service replaceable. I understand that many of you work for companies that are third parties. While being replaceable is essential, it doesn't just mean you want to replace a third-party service.
00:02:34.840
It also pertains to being able to replace APIs. If a company upgrades their API or introduces new features, maintaining the ability to swap out that implementation or API seamlessly is crucial without needing to delve into your existing code.
00:02:53.200
To illustrate this, consider how I'm working diligently to demonstrate my degree in art here. This diagram represents our billing app, the adapter layer, and the third-party system.
00:03:05.360
Notice that I refer to a third party and place the internet as a dividing line. This distinction is important because it doesn't solely refer to systems that exist across the internet; it encompasses any third-party integrations, including service-oriented architectures.
00:03:25.840
As an example, we have a database abstraction. Love it or leave it, it’s a part of our lives for now, but many do not appreciate it and wish to keep their code flexible.
00:03:33.400
Flexibility is necessary for reimplementing that database abstraction without untangling the spaghetti code we often end up with.
00:03:39.799
It's worth noting that our dependencies generally run in one direction. Third parties rarely know about your application, and you want to maintain that isolation.
00:03:45.760
Now, back to the adapter level, the most crucial aspect to understand is that the pay-me adapter is merely a gateway. You likely anticipated that, as I jumped to the next slide.
00:04:06.439
The adapter is a standard Ruby gem that you might generate using 'bundle gem [adapter_name]'. However, you won't push it to a public repository; instead, you'll keep it within your project. In your gem file, you'll specifically point to the relative path from your app's root, naming it there.
00:04:50.320
Fortunately, gem files allow you to make such references. Ideally, you’re already familiar with this pattern elsewhere in your code. It’s critical to keep the implementation of the third party private as it details how we manage interactions with it. You want to avoid dealing with multiple repositories.
00:05:09.960
Despite suggesting using a gem, many third-party services already have gems available. It's not to say that you cannot use them; you simply want to wrap their execution within your own gem.
00:05:30.319
If there is a third-party SDK available, leverage it. There's no reason to reinvent the wheel, particularly if the SDK was developed by someone from that company who understands the intricacies.
00:05:55.280
Our gem essentially serves as a series of service level objects. These objects are a straightforward path through the functionalities and do not depend upon one another unless absolutely necessary.
00:06:07.080
They communicate with our system using our internal logic and translate that logic into the language suitable for the third-party system.
00:06:14.120
When we initialize our application, which in our case is a Rails app, we hand over these individual services. Throughout the codebase, we refer to the configuration instance using a singleton method.
00:06:27.639
As a result, other parts of the codebase remain unaware that we are interacting with the pay-me service or the Stripe service. The initializer is the only place in the main application where these service names appear.
00:06:46.280
This is crucial because it supports the flexibility needed in your codebase. One of the most important practices applicable to any project is defining your vocabulary.
00:07:11.720
Getting everyone on the same page and ensuring that they describe the same things in a consistent language is essential. It can be quite challenging to gather product owners, stakeholders, and developers in the same room to discuss terminology.
00:07:40.000
It may surprise you how the same word can have different meanings for various people.
00:07:51.759
Therefore, taking a step back and defining your language is the first step. If you don't have access to others, try your best to define terms as a team or as a developer.
00:08:03.280
Coordinating a group-wide buy-in is preferable, but even individual clarity is crucial.
00:08:22.760
Using common language aids in keeping concepts from third-party services outside of how your company handles its business.
00:08:39.800
For instance, when we refer to a package in our billing provider, that translates into a product rate plan.
00:08:45.760
When we mention an add-on in our terminology, it similarly means a product rate plan. Fortunately, talking about a coupon translates effortlessly into product rate plan charges, making it clear in conversations.
00:09:04.920
For us, using terms like package, add-on, and coupon helps avoid convoluted nested objects that exist in our billing provider. Our services handle these translations.
00:09:18.920
For example, we have a package service that speaks the language of product rate plans when communicating with our billing provider. We also have an add-on service that does the same, albeit in a different context.
00:09:38.240
While the API calls may be similar, the nuances don’t bleed into the higher levels of our stack.
00:09:52.480
I've mentioned maintaining this separation through gems; we implement data transfer objects (DTOs) as well. DTOs serve as codified contracts indicating the parameters necessary for each service.
00:10:10.640
Instead of relying on a cryptic options hash, DTOs let us express exactly what the service requires. Additionally, we have abstract services that define method signatures.
00:10:25.680
If a method is not implemented, it will simply raise an error, prompting developers to remember the requisite functions.
00:10:37.680
While this may add some overhead, it codifies the interface. Essentially, between DTOs and abstract services, we establish a clear definition for implementing a new service.
00:10:52.040
Unfortunately, human developers sometimes forget these standards, but we have a solution I'll elaborate on shortly.
00:11:04.440
Looking back at the graphic I shared before, we have our adapter—a framework for our billing application and connected third-party service. The DTOs and account services exist in a separate gem we call the adapter interface gem.
00:11:30.480
This defines how you interact with services, ensuring that as long as you meet the interface, you can swap in new services without issues.
00:11:47.760
While the adapter interface is utilized by the billing application itself in its gem file, to access DTOs for proper instantiation, the pay-me adapter must also implement it since its services inherit from these abstract classes.
00:12:02.720
We test drive DTOs because they contain logic, while the abstract services are thin and focus solely on raising errors. I believe firmly in thorough testing, but we also shouldn't overtest.
00:12:28.440
Returning to the topic of maintaining abstract services, we have a gem that matches everything through testing, specifically a shared context that verifies the public interfaces of classes.
00:12:43.600
This ensures that the public instance and methods of your classes reflect their inherited definitions, and it alerts you when you’re passing in incorrect argument numbers.
00:13:00.000
Though concepts of interfaces may conflict with Ruby's DNA, we are self-aware in that we have termed our gem 'uptyped', playing on the concept of 'uptight'.
00:13:16.640
Clearly, this is not something one would normally advocate widely, but for our situation, it helps maintain robust documentation, making it easier to replace third-party services.
00:13:38.440
One essential practice is observing the four components of testing. Some may think there are only three, but we have setup, expectations, the kickoff, and teardown.
00:13:55.520
Your approach may vary based on whether you are testing for return objects or side effects—this may switch the order.
00:14:12.920
Your teardown process becomes significantly more critical, especially when interacting with third-party services.
00:14:27.000
Next, always avoid assuming that your test data exists within the third-party system. Even if you logged into the UI and added data for testing, that is not a repeatable approach.
00:14:46.240
Make it a part of your testing framework to call the third party to ensure that test data is present. If available, use a sandbox to prevent corruption of customer data.
00:15:04.120
This allows you to engage in testing more safely.
00:15:19.760
To illustrate, let’s take a closer look at our payment environment. We refer to this as an environment pattern that enables easy tracking and creation methods.
00:15:47.360
When you check the initializer of our test environment, you will notice trackers that maintain IDs for things we create. There’s an 'ensure' block to handle cleanup.
00:16:05.720
In our user creation process, we utilize the pay-me user gem and keep track of created IDs. Once our tests conclude, we return to clean up the data within the third-party service.
00:16:22.800
This means once you build that environment, cleanup occurs automatically, requiring no extra thought, which is pivotal for maintaining a pristine third-party system.
00:16:44.120
Unlike some legacy codebases that result in unwanted dozens of accounts created during tests, we maintain a respectful approach.
00:17:02.960
Regularly hitting your third-party services, especially for test driving, can be slow. To speed things up during development, consider implementing stubs to act as your service.
00:17:20.840
There are two main approaches: create a fake service that mimics your API responses or record VCR sessions that log all HTTP interactions in a YAML file.
00:17:34.280
In unit tests, we call our services to interact with the third-party or the stub, as this is the objective of the service.
00:17:54.080
However, because HTTP calls are complex to stub without employing tools like FakeWeb, which might unnecessarily complicate your testing framework, you need to be cautious.
00:18:32.440
Our testing environment showcases an instance of our pay-me service. We initiate that within a scoped variable, apply our creation methods, and kick it off.
00:18:56.640
By the time the test concludes, we revert back to the clean environment, ensuring that if any issues arise during testing, we are more likely to invoke that cleaning procedure afterward.
00:19:10.440
This same approach applies to our integration specifications, but we ensure that instead of referencing a specific adapter, we inject dependencies during our initialization process.
00:19:38.720
At the integration level, we may have built wrappers using SRID language during these tests. Regardless of how you’ve constructed your environment, it should maintain that the application doesn’t recognize which services it interacts with.
00:20:03.840
It's important to avoid the risk of any implementation details leaking through.
00:20:19.440
Here’s a reminder, when you choose to stub your services, you must sustain the stubs effectively.
00:20:38.000
If you’re using a fake service, maintain a routine to verify its behavior aligns with the actual third-party service.
00:20:56.720
If utilizing VCR, remember to delete and re-record cassettes regularly.
00:21:09.480
The underlying goal is that your tests need to successfully hit the actual third-party integrations.
00:21:36.000
Thus, your tests need to encompass both creating data and subsequently tearing everything down.
00:21:50.640
If you are working in continuous integration, setting it up so that tests bypass fakes and interact with the live service can be incredibly beneficial.
00:22:04.400
It can allow you to catch potential issues earlier by running tests against actual APIs instead of relying solely on stubs.
00:22:27.600
We eventually wrote code that functioned appropriately, but it wasn’t part of the published API, which led to complications when the service altered its code.
00:22:51.440
Continuous integration could have provided early alerts for these problems, allowing for prompt corrective actions.
00:23:14.080
Reflecting on our goals for third-party implementations, one was to keep their logic separate from our own, ensuring the pay-me adapter retains a distinct separation.
00:23:29.440
The next goal is to maintain predictable third-party behavior, which relates back to comprehensive testing and establishing where that should occur.
00:23:50.360
Regular usage of stubs in tests without compromising the real service ensures everything works correctly.
00:24:03.600
Finally, consider the replaceability of services, which is where DTOs and abstract services play an essential role.
00:24:13.920
With these elements clearly defined, they provide concise documentation for the necessary methods for implementing or replacing services.
00:24:24.840
Moreover, DTOs dictate what the required data structure should look like, providing clarity.
00:24:45.760
Furthermore, this structure enhances flexibility, allowing you to shift and adapt operations accordingly.
00:25:01.680
We recognized, for example, the strong integration with our billing provider often limited our capabilities.
00:25:14.440
With prior implementations often setting rigid processes, we’ve found that delegating third-party implementation into an adapter allows greater operational flexibility.
00:25:29.760
It permits new product additions without concerns over existing implementations.
00:25:49.800
Portability is another key aspect your services must provide, especially if you need to shift towards a producer-consumer model in the future.
00:26:06.000
The groundwork for any necessary reimplementation resides within the well-defined interfaces.
00:26:13.440
Establishing your adapters and DTOs creates a framework to facilitate that transition.
00:26:28.000
Above all, focus on determining your internal vocabulary. Communication issues can often stem from differing interpretations of the same terms.
00:26:45.880
Next, create the adapter interface to define context and usage clearly. This abstraction removes muddying of implementation details and keeps discussion on higher-level concepts.
00:27:01.920
Once you create the adapter, your gem should implement the interface and manage the structured elements.
00:27:15.840
It is essential to maintain a pristine test environment to uphold standards.
00:27:24.720
Providing stubs helps accelerate testing to maintain a focus on development.
00:27:36.960
Lastly, ensure proper management of test data—removal and upkeep are critical to avoiding unnecessary sprawl.
00:27:52.160
I would like to thank my team, the portal team, and my big brother Carl for encouraging my programming journey. If anyone has questions, feel free to reach out.
00:28:07.680
If you're working on these topics as well, my colleagues are here to help with inquiries.
00:28:29.440
Lastly, thank you all for your attention.
Explore all talks recorded at RubyConf 2014
+73