Devly, a multi-service development environment

Eric Hodel

Eric Hodel

@drbrain

Ezekiel Templin

@ezkl

#developer-experience-dx

#continuous-integration-ci

Devly, a multi-service development environment

by Eric Hodel and Ezekiel Templin

The video presents Devly, a multi-service development environment developed at Fastly, aimed at addressing the complexities faced by engineers as companies scale and work with diverse programming languages and workflows. Eric Hodel and Ezekiel Templin explain the evolution of Fastly’s API and their experiences with development environments, which led to the creation of Devly. Key themes discussed include the challenges of maintaining reliable and efficient development environments amid increasing software complexity, the necessity for effective collaboration among teams, and the ideal traits that development tools should encompass.

Key Points:
- Background: Devly supports developers dealing with multiple systems, languages, and an expanding team size at Fastly.
- Original Challenges: Early development environments were suitable for small teams but became unreliable as the size and complexity of systems grew. Growing dependency on diverse systems led to frustrations in maintaining consistent workflows across teams.
- Understanding Developers' Needs: Through observations and discussions, the team identified essential characteristics for effective development environments, including reliability, accessibility, maintainability, and reproducibility.
- Introduction of Devly: Devly was designed to build images from repositories to manage service containers. It allows developers to run services in isolated environments and supports easy changes, collaborative efforts, and integration of various programming languages.
- Core Features of Devly: Devly automatically configures services and allows for parallel container starting, environment variable setups, and collaborative image handling across the organization.
- Setup and Use: The presentation includes demonstrations on the setup process, executing commands, viewing logs, and updating services. It illustrates the simplicity and efficiency Devly aims to provide, making development tasks manageable and minimizing downtime.
- Importance of Documentation: Emphasis is placed on thorough documentation to assist users in using Devly effectively, which encourages community engagement and eases onboarding for new developers.

Conclusion: The video concludes by highlighting the importance of community support and documentation in making Devly a beneficial tool for developing applications across diverse services and teams. The approach taken in the development of Devly is presented as a model for building effective engineering tools in a collaborative environment.

00:00:02.060 Hello, and welcome to Devly, a multi-service development environment. I'm Eric Hodel, an engineer in the Engineering Experience department at Fastly.

00:00:05.700 Today, I'll be describing Devly to you. I've written lots of Ruby code, some of which you use every day.

00:00:11.880 My co-presenter, Ezekiel Templin, has been writing software of various kinds for over 20 years. We currently work within Fastly's Engineering Experience organization, focusing on improving Fastly's internal engineering tools and processes.

00:00:19.470 Fastly is a content delivery network and edge cloud provider that serves traffic for GitHub, New Relic, Spotify, and many other popular websites and services. Additionally, Fastly provides free services for all Ruby and RubyGems downloads, as well as for many other open-source projects.

00:00:50.610 We have servers located worldwide, which handle more than 14 trillion internet requests each month. This accounts for over 10% of all internet traffic today.

00:01:03.840 Today, we'd like to discuss a problem that we believe impacts organizations of all sizes. To help illustrate this problem, we want to tell you about the evolution of Fastly's API.

00:01:29.751 This is a rough approximation of the service architecture that backed the Fastly API around 2012. The original Fastly development environment consisted of a copy of each component of the Fastly API running on each engineer's laptop.

00:01:55.719 Soon after, the early team decided that virtual machines should be employed to provide a degree of operational uniformity and parity between development and production. In the early days, all engineering work was being done by a very small group of people, allowing changes to systems to be easily introduced and distributed through source control, enabling rapid development and deployment.

00:02:34.310 The small size of the company allowed for focused discussions, making decisions easy to communicate. As the company grew and became successful, new opportunities opened up to expand the business by adding additional functionality to our API.

00:02:57.209 While adding new functionalities, we also added new supporting systems. As software engineers, whenever we introduced new functionality and dependencies to our systems, we also introduced complexity. We do not imply that complexity is necessarily a bad thing; rather, it's an unavoidable side effect of growth.

00:03:18.959 Another complicating factor arises from our desire to use the right tool for the job, leading to many services being written in entirely different languages within very different workflows.

00:03:26.850 Despite the increase in the number of languages and services, our development environment remained largely unchanged. The gaps between each group's development workflows also grew, which became increasingly problematic as our engineering department doubled in size every six months.

00:03:51.194 Our original development environment then became increasingly unreliable, and the established processes to communicate changes broke down. Maintaining any single engineer's environment became more time-consuming, leading to unnecessary conflict.

00:04:11.280 With engineers working on everything from code that runs in the Linux kernel to code that runs in the browser, the needs of each team in these different areas are dramatically different.

00:04:27.509 Our original development environment couldn't meet the needs of one team without compromising the needs of another. This growth continued without regard for the challenges of scaling software, which is inherently complicated, with many moving pieces.

00:04:33.239 As an industry, we've established and continue to improve strategies that can help us direct our time and energy toward effective development, largely due to our ability to observe software systems in isolation.

00:05:03.280 Organizations, on the other hand, are far more complex and much harder to observe systematically. However, by reflecting on our experiences and listening to our coworkers, we identified common themes and frustrations.

00:05:42.060 Here are some summaries of conversations that we have had regarding the experiences of employees in our company: "Here’s your laptop, we'll see you in two weeks when your development environment is running." Or "Does anyone know why the API gateway crashes in a loop? I updated my development environment and now nothing works!"

00:06:05.220 Another example: "I can't do my work today because I need to rebuild my development environment." Does anyone in the room have experience with this kind of issue, too? Quite a few of you do!

00:06:29.960 This is a common but unsustainable problem, growing increasingly untenable over time. Moreover, our friends and coworkers are becoming frustrated, which is hampering our overall productivity.

00:06:51.570 During the same period, many new tools like Docker Compose and Minikube were released, yet none met all of our developer needs. Through observation, research, and extensive discussions with our colleagues and peers, we identified important themes that contribute to an effective development environment.

00:07:16.210 These themes encapsulate desirable traits for developer-focused productivity tools. First and foremost, a development environment must be reliable—developers should be able to run a small number of commands to get what they need up and running without needing to know how every system works.

00:07:52.610 It's also crucial for developers to easily see the local health of systems they depend on; time spent rebuilding a development environment should be negligible.

00:08:25.580 The development environment must be accessible; engineers need to be allowed and encouraged to maintain their development environments collaboratively. Developers should be empowered to build and test new changes across systems owned by different teams seamlessly.

00:09:01.940 A development environment that spans multiple teams and workflows must be maintainable by the community of users. Managing changes in source control should illuminate past and present ownership, even if there are many components involved.

00:09:36.120 The structure and conventions required to manage these changes should be encouraged through documentation and feedback loops, rather than enforced by gatekeepers. We want to create a development environment that allows for running discrete services together in composable units.

00:10:09.270 Furthermore, it should be easy to try out new supporting systems. Finally, the development environment must be reproducible; we need the ability to apply the last known good state of all systems effectively.

00:10:32.520 Source control and similar mechanisms must help us understand how we achieved this good state. We should be able to leverage existing tools like Git, RubyGems, Perl, pip, and Go modules.

00:10:56.270 Throughout the rest of this talk, we hope to show you how we've started addressing the needs of our coworkers at Fastly by applying these themes to a tool we've been developing together for the past year, which we call Devly.

00:11:17.960 To tell you more about Devly, I'll hand things off to my friend, close collaborator, and lead engineer on the Devly project, Eric Hodel.

00:11:36.110 Thank you, Zeke. I will discuss Devly and some of its components and features. Devly is designed for developers and builds images from repositories. It uses those images to manage containers and facilitates communication within and across teams of developers.

00:12:05.350 Devly is distributed for macOS and Linux, providing a standalone executable built with Ruby Packer and offering packages for macOS and Debian. Devly helps configure all of your services and aids in building images from your repositories using Docker files.

00:12:25.410 You can configure these images to run as services and even run groups of services together as part of a rack. Each image contains the necessary files to run a service.

00:12:43.890 For instance, the audit log image uses Ruby and includes our application code, which requires libraries like Rails, Sidekiq, and a JSON parser. The JSON parser relies on a C library, which we install via the OS package system.

00:13:16.290 In our repository, there’s a Dockerfile with the instructions for building this image. These images can contain applications in any language. For example, our Stats service, written in Go, has an image that includes a Go binary compiled from its application code.

00:13:50.620 Similarly, the web application our customers use is written in Ember and contains a copy of the application code ready to run. We share all these images across teams by uploading and downloading them from Google Container Registry, ensuring we're consistently using images built from the latest source code.

00:14:20.800 A Devly service is a runtime configuration for an image. For instance, we've created the Audit Log service using the Audit Log image. The service runs a command because the Audit Log service provides an API for managing event data.

00:14:53.920 It runs a Rails server to provide the HTTP interface for events, which must be accessible to other services for reading and writing event data. To facilitate communication, we expose port 8888.

00:15:20.900 If you use a framework like Rails that supports live development, you can mount your repository on top of the files in the image. This allows you to work in your preferred editor from your preferred OS.

00:15:46.200 You can change a file on your host OS and see the changes reflected in your browser. The service runs the Audit Log API, and we also have background jobs in Sidekiq to help process our logs.

00:16:07.480 To handle these background tasks, we can separate the service dedicated to running Sidekiq jobs, allowing these jobs to access the same models and databases used by our application.

00:16:40.260 Thus, we have the Audit Workers service, running the Sidekiq command instead of the Rails server command, while the Audit Log service runs the Rails server. This separation allows us to manage development more effectively.

00:17:05.570 Next, we’ll create additional services for our applications, such as the Authentication API, the Configuration API, and their respective databases. When working on the Configuration API, we don’t want to start up unnecessary services.

00:17:29.300 For the Authentication API, we can create a rack specifically for developing the Configuration API, containing only the services it needs, including a MySQL database, the Audit Log and Configuration API services.

00:17:51.560 To access services running in the rack for development, we can expose ports for a few services to the host OS. This setup allows us to connect to these ports from the browser.

00:18:07.390 We can also set environment variables and mount different files to change a service’s behavior. Additionally, Devly allows the configuration of multiple racks.

00:18:29.960 The Authentication team needs to work on its services, including the Postgres database and the Authentication API, while the Configuration team works with the same Audit Log service.

00:18:56.760 When we start these racks, we use independent containers to run their services, enabling each team to have different configurations and software versions that don’t conflict with one another.

00:19:15.920 For instance, we could start both racks simultaneously to isolate bugs impacting multiple services using common configurations, making sharing and work across teams easier.

00:19:41.150 The configuration for the images and services—along with their respective racks—are stored in the Devly library repository at Fastly, allowing any developer to contribute changes and discuss proposed alterations with their respective service teams.

00:20:10.270 All teams relying on the Audit Log service will need to collaborate whenever updates or changes are proposed, facilitating communication and improving service maintainability.

00:20:35.280 Now that we have discussed the components of Devly and their functionality, let's move on to some demonstrations of its common development tasks from the perspective of developers across various teams.

00:20:59.460 We'll start by setting up Devly as a first-time user. We run 'devly setup' and provide the repository to pull from the Devly library. This process downloads the Devly library along with other repositories required for our services.

00:21:24.570 The setup also includes checking our Docker version and Google Cloud SDK version. The command tries to fix any potential issues or provides messages to guide user resolutions.

00:21:51.860 Setting up usually takes just a few minutes to fetch necessary repositories and carry out essential checks. Upon completion, we can run 'devly info' to view available racks and services.

00:22:22.500 This command will yield a list of racks and services in our Devly library. Information about a specific rack, including its services, can also be retrieved easily.

00:22:48.890 Once we complete the Devly setup, we can start a rack and carry out basic development tasks, such as viewing logs and making small changes. Using the 'devly up' command starts the rack.

00:23:15.680 If certain images are not yet downloaded from the registry, we utilize 'devly pull' to fetch them. Once all images are downloaded to Docker on our host system, Devly creates a network to isolate the rack and starts all the containers.

00:23:38.920 When containers aren’t dependent on one another, Devly can start them in parallel, expediting the startup process. We can check the status of our racks and services by running 'devly status'.

00:24:01.560 After confirming that everything is functioning correctly, we can view the logs for the rack by executing the command 'devly logs.', allowing for insight into live log data.

00:24:29.660 Once we're sure the rack is operational, we can navigate to the Configuration service through the browser, verifying that the service is accessible on its designated port.

00:24:50.390 After ensuring our rack is set up and running smoothly, we can make changes to the main page of our application using our preferred editor. For instance, adding text to provide information about the service.

00:25:08.880 Once we save our changes, reloading the Configuration API will show the new content has been successfully loaded.

00:25:38.660 After making sure that our changes were successful, we can shut down the rack by using 'devly down', which stops all active containers that were running.

00:26:06.270 When working within our team, we’ll continuously push changes to our repository. When collaborating across teams using Devly, other teams will submit new images for their services as they have developed new features.

00:26:29.460 For example, the audit team has updated the audit log image to include a source field for events. First, we can verify if the source field is present by loading the Audit Logs service in the browser. If the source field is absent, it indicates that we are still running an outdated image.

00:27:00.390 To utilize this new feature, we’ll need to pull the latest image. If the currently running Audit Log service is outdated, we can use the 'devly restart' command to replace the existing Audit Log service with an updated one.

00:27:28.840 Once we check the browser again, we should see that the source field has appeared. With the Audit Logs service now running the latest image, we can proceed to update our services to leverage this new source field.

00:27:50.780 So far, we’ve mostly worked from outside of the container. Sometimes, though, we need to run commands from inside the container where our dependencies are loaded.

00:28:12.300 For better clarity, let's say we’re back on the Audit Log team. We’re tasked with adding a source column to our database, which requires running a migration we've just created.

00:28:31.560 To do this, we’ll need to execute the migration command from within the Audit Log service. We can start a Bash shell inside the container to explore and execute our tasks.

00:29:03.640 After navigating to the appropriate directory, we check our setup and then run the migrations. Once we verify that the migration is successful, we can check back in the browser to see if the new source column appears.

00:29:30.240 With the migration complete, we can run tasks more efficiently and ensure that everyone remembers how to update the database consistently.

00:29:47.150 In practice, we aim to document the steps and commands that every engineer can utilize as a clear reference. By streamlining our process, we can enhance our workflows.

00:30:11.270 As we build the community around Devly, we focus on keeping our documentation thorough and accessible. We acknowledge suggestions and put forth effort to resolve any reported issues quickly.

00:30:43.370 This promotes stability, making it essential to document everything—particularly getting started guides, workflows, and any administrative tasks that need clarity.

00:31:11.810 Documentation can help reduce friction and ensures all users feel empowered to contribute updates whenever necessary. As we have seen, automation of our release processes saves time and helps maintain consistent operations.

00:31:41.080 In conclusion, building a supportive and engaged community is essential for the collective success of tools like Devly. Thank you for your time, and we hope you find Devly beneficial as you develop your applications.

RubyKaigi 2018