00:00:14.340
Bonjour Paris.rb, welcome to my talk, "I Am Altering the Deal." Pray that I do not alter it further. I am absolutely honored to be speaking in front of you all today, and I cannot thank the organizers enough for this wonderful conference.
00:00:22.390
My name is Brittany Martin, and I am the lead web developer for the Pittsburgh Cultural Trust. Since 1984, the Pittsburgh Cultural Trust has been working hard to revolutionize downtown Pittsburgh, Pennsylvania, in the United States, to create a flourishing arts community.
00:00:30.910
Day by day, I'm a very proud Rubyist. I work on four different Ruby on Rails applications at the Trust. Speaking of Ruby, I am also the host of the 5x5 Ruby on Rails podcast, where we discuss Ruby, open source, and the programming profession.
00:00:40.720
I absolutely adore being the host of the podcast; I get to talk to a lot of my programming heroes. However, my favorite part of hosting is bringing on guests who have never been on a podcast before. I am always on the lookout for new guests, so if you have anything interesting to say about Ruby, Rails, or open source, please come see me; I would love to book you onto the show.
00:00:58.420
Additionally, if you have any feedback about the podcast, I would be more than happy to hear it. As was mentioned, I am the president of the Little Steel Roller Derby League based in Youngstown, Ohio. Essentially, roller derby is a fast-paced, full-contact sport that I absolutely adore. It has brought many wonderful friendships into my life, and I even found the love of my life through it.
00:01:22.899
So, what brings me here today? Today, I'm here to discuss a deal—a deal that could happen to any one of us. This involves your third-party vendor deciding to make a breaking API change that will affect you and your application. We will discuss the unspoken agreement that you have with your API vendor and what to do when a breaking change occurs. Fortunately, we have some amazing tools in our Ruby community that can help.
00:01:54.160
Before we proceed, I want you to take a moment to think about whether you have full control of your application. Chances are that you don’t. It's estimated that about 70% of applications thought to be entirely internal and hand-written actually rely on third-party dependencies.
00:02:06.190
It's standard practice to use third-party APIs and dependencies in order to make our applications grow. This is essential, as we need to utilize these resources to expedite our coding processes. However, the real question is: do those API vendors always behave as expected?
00:02:26.140
APIs, like web applications, go through changes, revisions, and versions. There are best practices to extend or replace an API, and if you're using a third-party vendor, you would expect them to follow these protocols. While it is great to expect that your API will version and change, you must also have a strategy in place for what you will do if your third party API undergoes a radical change.
00:02:44.530
It's essential to prepare for the possibility that if an API makes a breaking change without your account, your application might break. When your users access your application and find that it is broken, they will blame you. They are often unaware of your third-party ecosystem, and in most cases, you want to avoid personal catastrophe.
00:03:07.310
It is crucial to learn from others' experiences, especially since our community has faced several breaking API changes. Allow me to share some examples that are relevant to where I am standing today.
00:03:11.140
On September 14th, 2019, strong customer authentication came into effect in Europe, radically changing the way people buy and sell online. More than 300 million European consumers now need to confirm their identity for the majority of their online purchases, requiring methods such as a password, phone, or fingerprint.
00:03:28.740
If you did not implement SCA in your company overnight, all of your transactions would fail, whether you are a retailer or a ride-sharing company. Stripe was able to accommodate this change and launched a new API called the Payments Intents API, which was SCA ready and provided all available authentication methods.
00:03:42.220
Unfortunately, this was an absolute radical change for anyone using the Stripe API. If you didn’t do the necessary work, and I know many of us had to, all of your payments would suddenly fail. Another significant change we faced at the Trust occurred in 2018, when Instagram decided to revoke their public API. It seemed to happen overnight, resulting in many applications that relied on it either slowing down or ceasing to function altogether.
00:04:17.710
Instagram made this decision to shut off their public API to attract new businesses and increase revenue. They introduced the Graph API, which required an Instagram business account and became the only way to collect data and target hashtags.
00:04:36.360
Unfortunately, Instagram did not provide much warning; they shut off the API prematurely, leaving many developers scrambling.
00:04:50.180
The last API I want to discuss is very close to my heart. At the Trust, I work daily on a multi-tenant Ruby on Rails application that is connected to third-party arts management software with a custom CMS. This third-party API is essential; it powers not only our website but also our phone center and box office.
00:05:16.920
When I was hired at the Trust four and a half years ago, my boss informed me that the third-party API was built in SOAP. They were going to transition to a REST API, and one of my major responsibilities as the lead web developer was to facilitate that change.
00:05:36.540
I was told the REST API was not ready at that time, so I prepared myself to make the transition. Three years later, the third-party API finally rolled out the REST client, accompanied by a deprecation warning that the SOAP client would be discontinued.
00:06:05.019
To my horror, as I began making those changes from SOAP to REST, we encountered performance issues; the REST API simply did not work effectively. This led us into a vicious cycle of rollback after rollback, where I'd implement a new feature, test it in development, and deploy it to production, only to find it was broken due to some unforeseen issue.
00:06:31.170
The documentation was terrible, and the performance was so poor that we had to revert to the old code. We knew that we were going to lose the SOAP API, which was inevitable, despite how poorly the REST API functioned. As a team, we needed to strategize on re-architecting our dependency to accommodate these changes.
00:07:05.239
Fortunately, we found some comfort in knowing that many other companies were experiencing the same issues since this is the primary third-party API for the arts industry. This brings me to the title of my talk: "Altered Deal." When the third-party API alters the deal, it is not a time to pray; rather, it's time to think critically about how to minimize downtime and manage the costs associated with these changes.
00:07:27.360
Whenever an API change is announced, it's essential to have a strategy in place within your department to track and ticket those changes. This way, developers can address them promptly. I believe that this role should ideally be filled by the product manager or engineering manager to ensure accountability.
00:07:46.360
You should maintain a comprehensive overview of your ecosystem of third-party APIs, ensuring that changes and versions are meticulously tracked. Additionally, you want to ensure that your sprint planning accommodates these changes, so you have a change log of all affected APIs that you can ticket and manage properly.
00:08:09.970
When a third-party API makes a change, it’s important to involve as many people as possible to monitor for potential issues. This is where I like to bring in customer support. Even if you think an API change is minor, make sure to notify customer support so they can track any changes.
00:08:30.620
I may not be the first person to mention this at the conference, and I certainly won't be the last, but implementing automated testing is crucial. The challenge I faced when learning about testing was that I was told to only test internal code, while the responsibility of testing third-party APIs rested with them.
00:08:50.740
What happened, however, was that many of these third-party vendors lacked proper test coverage. As a result, they would release new versions without adequate assurance, putting the onus on me to catch these changes through my tests. We had to revise our entire test suite to incorporate tests for their APIs as well.
00:09:14.650
This highlights a crucial point: while it is indeed your responsibility to test your own code, your uptime is also dependent on external APIs. You may need to adjust tests to cover their code as well.
00:09:37.270
The challenge for us was that automating testing against these third-party APIs proved difficult. For typical scenarios in our CRM, it would often require multiple API calls to mimic a single interaction.
00:09:58.310
The beauty of testing an API, as opposed to the front end, is that it operates over HTTP and provides the same JSON responses and error messages regardless of where the requests originate.
00:10:06.300
Even if everything seems to be functioning correctly on the surface and your tests are passing, you must keep an eye on the third-party API’s performance. Performance testing is essential whenever you introduce changes to a third-party API.
00:10:31.390
More importantly, anytime there is a change in a third-party API, you should conduct load testing. This ensures that your system can handle the stress of concurrent users accessing the application at the same time.
00:10:45.790
I personally prefer using tools like JMeter or Flood.io for load testing, and it's vital to incorporate performance testing as well before new updates are released.
00:11:19.110
I suggest conducting load testing at least quarterly. We found this practice to be crucial when troubleshooting issues with our internal API.
00:11:47.240
We are fortunate to have a robust Ruby community that offers valuable tools to help with these changes. Three tools that I want to highlight today are Scientist, VCR, and Ruby's internal benchmarking tool.
00:12:11.320
In 2016, GitHub faced challenges with their sizeable Rails codebase when implementing changes confidently. To resolve these issues, they adopted a concept known as branch by abstraction, which is an architectural pattern designed to help with large-scale changes.
00:12:47.710
By inserting an abstraction layer around the code to be changed, GitHub was able to delegate to the existing code initially. Once the new code was in place, they could simply switch the execution over to the new implementation without disrupting functionality.
00:13:13.270
This process introduced a new tool called the Scientist gem, which allows you to conduct experiments while detecting edge cases within a complex codebase. It creates a lightweight abstraction, enabling you to transition smoothly between the legacy and the new code.
00:13:35.440
For instance, you can wrap a block around new behavior that you want to implement, and it returns whatever the block returns. Scientist determines whether to execute the legacy code or the new, refactored code and compares the results.
00:14:12.320
Another gem I want to mention is VCR, which records HTTP interactions and lets you create fast, deterministic, and accurate tests. It captures API responses and 'records' them into a 'cassette tape,' allowing you to run those tests without hitting the external API each time.
00:14:29.670
When you run your tests, if it matches a recorded request, it will use the stored response instead of making a new HTTP call, which is particularly useful for testing in offline situations.
00:15:02.260
To check for changes in a third-party API, I often delete all of my cassette tapes and rerun my test suite. If failures arise that I didn’t cause in my codebase, I know there is an issue with the third-party API.
00:15:26.030
Finally, Ruby has a built-in benchmarking tool, which is essential for comparing different implementations. This tool was particularly useful when we transitioned from SOAP to REST.
00:15:42.020
For example, I can call both the SOAP API and the REST API in a benchmarking block, counting how many times I execute each to measure performance.
00:16:01.410
This allows me to identify discrepancies in response times. For instance, executing the SOAP API a hundred times took 3.5 seconds, while the REST API took 26.4 seconds, determining that REST was significantly slower.
00:16:19.800
Such performance differences are crucial for us as a multi-tenant ecommerce application, making it vital to communicate any unacceptable changes to our third-party vendors.
00:16:39.920
As developers who are armed with these tools, we still experience a cycle of grief when we learn that dependencies are changing. We go through denial, thinking that perhaps our vendor will reverse changes.
00:17:08.610
Next is anger, where we feel personally attacked when something breaks. It's essential to remember that third-party vendors are also developers and are trying to solve their own challenges.
00:17:30.240
Bargaining may ensue as we negotiate with vendors to prolong functionalities, and eventually depression arises as we realize how much work lies ahead to maintain parity.
00:17:49.640
However, there can be a silver lining where we find opportunities to reduce technical debt. As we transition from SOAP to REST, we might discover unused API calls that allow us to clean up our codebase.
00:18:16.510
Often, eliminating unnecessary code can streamline our applications and benefit future developers. In some cases, you might find that breaking up with your third-party API is the best route if their API changes are too drastic.
00:18:43.160
Now may be the right time to evaluate available alternatives. As a reminder, negative thoughts can ruin your passion for development. Embrace the changes ahead and rely on your available tools.
00:19:05.230
Change is inevitable; welcome it, especially when the deal has been altered. Thank you.