DIY (Do-it-Yourself) Testing

Talks

Emily Stolfo

8 talks

DIY (Do-it-Yourself) Testing

by Emily Stolfo

In the presentation titled "DIY (Do-it-Yourself) Testing" at RubyKaigi 2015, Emily Stolfo discusses the development of a custom testing framework used by the MongoDB drivers team. The talk begins with Stolfo's personal connection to Berlin's DIY culture, setting the stage for an exploration of testing in software development. Key points include:

Overall, the session highlights the value of DIY approaches in software testing, advocating for tailored solutions that drive better consistency and quality in software development.

00:00:03.440 Well, I'm so impressed I'm almost speechless. Hi, I'm Emily, and I'm going to attempt to talk to you about do-it-yourself testing through a haze of severe jet lag and maybe too much sake from last night.

00:00:10.410 Let me just adjust this a little bit. We lovingly refer to do-it-yourself activities in Berlin as DIY Berlin. If you've been there, you'll know it’s a city that really loves crafts, using your hands, and artisanal activities. It's quite appropriate that I live in Berlin, while talking about do-it-yourself testing.

00:00:30.000 How many people, before I get started, are users of MongoDB? Just out of curiosity. Well, not that many. Okay, you've heard of MongoDB, right? It's an open-source document database. The documents it saves can be described somewhat like JSON documents, but we refer to the data type as BSON.

00:00:52.350 It's loosely binary JSON; it's not exactly JSON, but it’s similar. The data structure consists of keys and values, but they are rich documents. This means you can have embedded documents and arrays to store values within your documents.

00:01:11.549 As I think I mentioned, I’m a Ruby engineer on the MongoDB drivers team. I'm employed by MongoDB, the company behind this open-source database, to work on the Ruby project. I'm based in Berlin, but I'm originally from New York City, where MongoDB is based.

00:01:42.899 You may know my code through the following gems: BSON, which does the serialization and deserialization from Ruby hashes to BSON documents; the MongoDB Ruby driver itself, which we rewrote over the last year and a half, and now it's on version 2.0; and Mongoid, which is the Active Record replacement for using MongoDB with web frameworks, usually Rails. Most of our users on Mongoid use Rails. Finally, there's the BSON extension, which makes an older version of the BSON gem faster, and Kerberos extension, which is less commonly used.

00:02:23.020 So, that’s a little bit about the gems I maintain. I work as part of the drivers team at MongoDB. We have about 22 people spread around the world, and collectively, we maintain 10 different drivers for different languages.

00:02:59.579 As you can see from this list, these languages are very diverse; they are not all static or dynamic, and we all program in quite different ways. MongoDB is known for its effective user experience, and part of that success comes from offering drivers in each major programming language very early on.

00:03:08.050 This allowed us to gain quick popularity, even when the database was quite immature at the time. What that means is that drivers are the interface to the database; you cannot use the database without a driver. The driver handles serialization, knows the wire protocol, and knows the format of messages to send to the database server and how to interpret the messages coming back.

00:03:40.720 Each of these drivers essentially performs the same function but in different languages. As I mentioned, we have 10 official drivers, but there are more in the community. We're the team paid directly by MongoDB to work on these official drivers.

00:04:03.940 The reason we got adoption quickly was that people were building drivers in their language communities, which facilitated the use of MongoDB. Many of the drivers started as open-source projects. Over time, as these drivers gained popularity, we would notice them and sometimes decide to hire the developers to work officially on the project.

00:04:38.830 The reality is that all these drivers grew pretty organically, leading to some inconsistencies in features and behavior, especially when dealing with changes in a cluster. When you have a MongoDB database, you typically don't just have a single server; you will have a replica set or a sharded cluster, which consists of multiple servers.

00:05:09.849 As you can imagine, the driver needs to adapt to changes in your cluster state—detect when a certain server goes down and adjust its view of the cluster accordingly. Each driver needs to behave consistently; however, our initial lack of specifications led to a situation where the drivers had different and sometimes confusing interfaces to the server.

00:05:52.440 This inconsistency meant that it was difficult for support personnel to provide accurate answers regarding driver features across different languages. So, MongoDB grew as a company, hiring more people and gaining traction over the past few years. About two years ago, we recognized the urgent need for consistency among drivers, both in terms of behavior and documentation.

00:06:29.210 As part of this effort, some drivers rewrote their code entirely. For instance, the Ruby team completely overhauled our driver to create version 2.0. Other teams, like Python, made significant refactors. Along with these rewrites, we created specifications governing how the drivers should behave.

00:07:01.499 These specifications guided the design of each driver, clarifying what we expected. As MongoDB matured, we noticed that larger companies, like banks or government agencies, often had different projects within their organizations using different languages, leading to the need for engineers to switch drivers.

00:07:50.810 This situation highlighted the importance of our specifications: they allowed us to redesign our drivers and facilitate a more unified approach across the diverse languages we cater to. As a result, we all released our drivers over the past year, with the Ruby driver launching in March version 2.0.

00:08:29.960 I'm proud to think of us now as a coherent bouquet of flowers, all of which are the same kind, maybe just in different colors. So why am I emphasizing specifications? Firstly, as a team, we were delighted to have a common document to reference that described our design, helping to foster understanding.

00:08:56.010 In the past, there was much mystery surrounding how each driver's implementation worked, but the specifications have clarified that. They are all accessible in our GitHub repository, and we take pride in sharing this with the community to increase transparency regarding how our drivers operate.

00:09:43.760 One specific specification we first wrote is called the Server Discovery and Monitoring Specification, abbreviated as SDAM. This specification outlines the logic needed to create a highly available application using MongoDB. It defines the algorithms and interpretations required when pinging servers to assess their statuses.

00:10:08.880 When the driver pings each server and receives state information back, it then interprets that data to maintain its own view of the replica cluster state. This understanding is critical because it informs the direction of operations—whether a write, read, or administrative task, like building an index.

00:10:54.300 Writing this specification was significant, particularly as this was often the hardest section of code within drivers, prone to bugs and misunderstandings. This hard-to-navigate space was where our initial inconsistencies were most pronounced.

00:11:30.210 My colleague Jesse Davis, who was maintaining the Python driver at the time, led the effort to write this specification. In addition to understanding the core functionalities of the drivers, this document provides a good overview of what we intended each driver to achieve.

00:12:18.270 With this specification in place, the next significant challenge was ensuring compliance across all drivers. We needed to determine how to validate that each driver adhered to these specifications, especially considering the diversity of languages involved.

00:12:41.710 We recognized that we needed both unit tests and integration tests. Unit tests alone weren’t sufficient, as they could only confirm that given responses from the server are handled properly. We also required integration tests to account for the multitude of states that a collection of servers could be in.

00:13:05.430 Consequently, we began building our own test language, test format, and test framework for each driver. The tests themselves would be shared among the 10 languages, an approach we refer to as 'do-it-yourself testing.' This experience may help others think outside the box when it comes to their own needs.

00:13:58.340 Our testing process began with defining a format—we chose YAML—and establishing a structure for tests to describe how they should be formatted. This approach required that we maintain consistency across hundreds of tests.

00:14:37.130 We opted for YAML because it describes data statically. We required something that could be interpreted by all 10 different languages while still translating into actions. Most languages already support YAML parsing, which eliminated extra steps.

00:15:18.270 Our testing harness was designed to read the YAML file and interpret it as a test, which would then return pass or fail results. This reusable harness allowed us to avoid rewriting code for every new set of test cases.

00:15:57.190 We found a significant improvement in our workflow because as changes were made to the specifications document, we could couple those with changes in the YAML files. This meant that if alterations were made in the textual specifications, the YAML files could signal to the drivers about those changes immediately.

00:16:37.940 Given that MongoDB is constantly evolving, our specifications are living documents that sometimes need to be adjusted as development progresses. We needed to create both unit tests and integration tests.

00:17:01.160 Integration tests posed a more complex challenge—recreating the specific scenarios where every server within a cluster is in defined specific states and manipulating the cluster effectively during testing.

00:17:38.510 A solution we found was to build a service called MongoDB Orchestration that assists with setting up real MongoDB processes for testing. The orchestration is essentially a puppeteer that can create a wide range of MongoDB clusters.

00:18:05.640 Let me give you a bit more detail about MongoDB Orchestration. It's a Python-based service you can find on GitHub, which interacts with the MongoDB processes via a RESTful API. Essentially, it allows you to define how you want to manipulate clusters.

00:18:43.480 This service operates differently from our product called Automation, which is used for production setups, having checks in place to enforce best practices. The purpose of Orchestration, on the other hand, is to allow for flexibility and even fault injection during testing.

00:19:20.640 With it, you can start various cluster types—replica sets or sharded clusters—merely by sending specific JSON documents as requests. It makes setting up environments for testing incredibly straightforward.

00:20:02.470 In practice, you define configurations as a Ruby hash and make HTTP calls to the orchestration service. Different parameters map directly to command-line flags you would normally use to start MongoDB processes, enabling a high degree of flexibility.

00:20:45.750 What's particularly noteworthy is that you can start a replica set with members running on different MongoDB binaries, allowing for what we term multi-version testing.

00:21:32.950 I previously prepared a screencast demonstrating how to use MongoDB Orchestration, starting with launching the service, which will create a cluster based on your requests. This allows you to run tests without having to worry about service realism, as the orchestrations block until the setup is complete.

00:22:16.700 I will be demonstrating various requests to manipulate the primary server and evaluate how the driver behaves under those circumstances. The benefits of using MongoDB Orchestration culminate in the ability to develop reproducible tests across different configurations of your clusters.

00:22:59.540 Integration tests are especially crucial as they involve real-time interactions, ensuring that the driver can adequately handle real-world scenarios. The project has definitely improved our testing capabilities, thanks to the scalability and flexibility this service offers.

00:23:42.320 As we conclude, I encourage you to consider building your own personalized testing framework based on our experience, particularly regarding unit and integration tests. Think through how your tests are defined, what format they should use, and how best to ensure compliance with project specifications.

00:24:19.680 One of our key takeaways is the importance of a generic test runner that can adapt according to configuration changes. It’s invaluable to share knowledge across teams and ensure that all members are aligned with project goals.

00:24:56.520 Ultimately, I want to highlight how proud I am of the team's effort in the last few years to standardize our processes, write clearer specifications, and build robust testing frameworks. We’re excited to share this with the community in hopes that it spurs innovations elsewhere.

00:25:31.290 I have included some resources and links to repositories with relevant information. Feel free to connect with me via Twitter. I'm happy to answer questions about MongoDB or anything else you might have in mind.

00:26:06.220 Thank you very much!

RubyKaigi 2015