Containers
Containerizing Local Development... Is It Worth it?
Containerizing Local Development... Is It Worth it?
by Tony DrakeIn his talk "Containerizing Local Development... Is It Worth it?" at RubyConf 2019, Tony Drake explores the use and implications of containerization in local development environments. He addresses the growing complexity in development setups as teams and companies expand, particularly highlighting issues that arise when multiple applications or microservices demand different dependencies. Key points discussed include:
Current State of Development: The typical local development environment often becomes complicated with various tools like Homebrew, Ruby version managers, and databases. As applications evolve into microservices, this complexity can lead to dependency challenges, like managing different Ruby or Postgres versions.
Introduction to Docker: The presentation focuses on Docker and Docker Compose as tools for local development configuration rather than deployment tutorials. Drake emphasizes that while Docker simplifies development for many, it is not universally beneficial.
Scenarios for Containerization: Three primary scenarios for utilizing containers were illustrated:
- Siloed Applications: Each Ruby app operates independently with no shared resources.
- Interconnected Applications: Multiple Ruby applications that need to interact, which may include microservices communicating with each other.
- Gem Development: Discusses the challenges of containerizing development for Ruby gems.
Dockerfile Configuration: Drake explains how to create a Dockerfile that supports a Ruby application, emphasizing the importance of binding to IP addresses and specifying version tags in order to match production environments closely.
Docker Compose for Multiple Services: He walks through setting up a Docker Compose file that manages multiple services, such as web applications, databases, and Redis. The advantages of using volumes and environment variables for real-time updates during development are highlighted.
Hybrid Approach: An alternative method discussed is using Docker solely for external services, like databases, allowing Ruby code to run natively. This can ease dependency management without overcomplicating local setups.
Challenges and Considerations: Drake candidly addresses potential downsides of using containerization, such as resource consumption (RAM and hard disk space) and learning curves for new developers. He concludes that while containerization can introduce speed limitations, especially for smaller projects, it significantly benefits larger applications needing structured dependency management.
Ultimately, the presentation emphasizes that while containerizing local development has its challenges, it can provide a streamlined solution for managing multiple applications, helping teams align their development environments closely with production.
00:00:11.940
All right, last talk of the conference and we are gathered here today to talk about containers.
00:00:18.130
My name is Tony; I'm from Indiana where we have corn and every two years we can muster a decent professional sports team.
00:00:25.990
This is the current state of local development. It may not look as complicated to many of you now, but as your team grows and your company expands, it'll start to look much more complex.
00:00:33.400
If you're on a Mac, like many of us are, we likely have Homebrew installed, a Ruby version manager in place, a database up and running, and at least one app.
00:00:39.010
As your company evolves, that single app will probably turn into multiple apps, or worse, a microservice architecture, where one of those apps might become legacy and you won't bother upgrading Ruby on it.
00:00:50.949
You then find yourself stuck with an older version of Ruby while everything else starts upgrading, or worse, one app insists on using Postgres 12, but everything else is on Postgres 10.
00:01:03.940
You can really only have one version of Postgres installed. So, how do you manage that?
00:01:09.850
Well, containers are trending right now in DevOps and deployment, but let's explore whether they're worth using for development.
00:01:22.090
Some ground rules for this talk: we'll use Docker Community Edition for all of these setups and Docker Compose for configuration.
00:01:35.590
Although Kubernetes is likely the future of orchestration, it's much more complicated than Docker Compose.
00:01:42.040
For local development, Docker Compose works out pretty well.
00:01:47.770
Keep in mind, I'm not an expert; I've only been working with containers for about two years.
00:01:53.470
With everything in development, there's no one correct setup.
00:02:00.970
When I refer to a Ruby app, I'm typically talking about a web app—it could be Rails, Sinatra, or another framework—essentially something that handles web requests.
00:02:07.210
And while it doesn't have to be, if your production app is already containerized by DevOps, you already have a significant head start.
00:02:14.709
You can leverage what they've done for you and simply add any additional components you need.
00:02:25.709
Fair warning, this is not a tutorial on how to use Docker, but rather on how Docker can be configured to work better for local development.
00:02:32.430
We'll go through three primary scenarios to demonstrate how you can utilize containerization for your local workflow.
00:02:40.200
These scenarios are backed by companies already using these setups or personal experience.
00:02:51.659
The first scenario involves having one app or multiple Ruby apps that are entirely siloed.
00:02:57.269
They do not share resources, they don't interact, and they operate separately with individual databases.
00:03:03.090
Next, we have a more complex setup where multiple Ruby applications do communicate with each other. This could involve microservices or several monolithic applications needing to interoperate.
00:03:15.150
In this case, they may share a database or have separate ones, and finally, the third scenario involves standard Ruby work.
00:03:21.870
Perhaps you're a gem author, and we'll evaluate whether it's worth containerizing the development process just to work on gems.
00:03:35.910
To kick things off, you'll need an app. For this example, let's pretend it's a highly complicated multi-million dollar application.
00:03:43.919
Though realistically, it's just a Sinatra app with a single endpoint.
00:03:49.829
We'll require Postgres, Redis, and the ImageMagick binary since we want to perform image manipulation.
00:03:55.109
For convenience, let's assume all configuration variables are managed with environment variables.
00:04:02.129
The only actual change needed for an existing application to be dockerized is that it should bind to the IP 0.0.0.0 instead of localhost.
00:04:07.260
This allows the internal container networking to bind more easily, since they can't see localhost.
00:04:13.019
For Sinatra, if you're using Rails, set it with the -b flag to bind to an IP address of your choice instead of localhost.
00:04:20.519
Next, we'll need a Dockerfile for our application.
00:04:25.889
We're going to build a Dockerfile to support our app. Every Dockerfile has a base image that you're starting from; in this case, we want to use the Ruby stock image from Docker Hub, specifically, the 2.6 tag with Debian stretch.
00:04:39.690
The benefit of version tagging is that you can closely match what's running in production.
00:04:46.050
If you're running Ubuntu, choose the Ubuntu container; if you're using a different Ruby version, make sure to specify that exact version.
00:04:52.230
A major advantage of this approach is that when you want to update Ruby, you simply change that tag, rebuild your containers, and suddenly your local development runs on the new Ruby version.
00:05:04.800
You don't have to wait for RVM to propagate updates or compile Ruby from scratch again.
00:05:10.950
Next, expose any ports necessary from your web app; in this case, for Sinatra, we expose port 4567.
00:05:17.130
You may also set custom environment variables that will be embedded in the container.
00:05:24.480
In this specific case, I'm overriding the default location where our gems are installed, setting it to /bundle within the Docker volume.
00:05:31.710
This is not a necessity, but I prefer it for local use so I can easily access the gem code for debugging.
00:05:37.530
Without this, they install in /usr/local/bundle, a path that's quite cumbersome to remember.
00:05:43.380
Next, we'll install any external dependencies that we need to run our application.
00:05:49.230
In this case, we're adding the Postgres repository using apt-get and installing the client libraries required for the PG gem, along with the ImageMagick binary.
00:06:02.280
These commands are analogous to any you would run in a terminal on a standard Linux distribution.
00:06:10.800
If you've previously used Vagrant, this is akin to the Vagrantfile, which outlines step-by-step how to build the Vagrant box.
00:06:17.400
It's crucial to keep the number of commands minimal.
00:06:21.030
Notice that the slashes at the end are intentional, indicating that all of this is one command.
00:06:27.510
Every line in the Dockerfile corresponds to a separate snapshot of the state of the container.
00:06:40.500
If a line changes later on, previous unchanged lines will be reused during a rebuild.
00:06:47.040
Next, we'll add a work directory, a key difference from a production container: this specifies where the app code will reside. Many tutorials will place it in /app, but you can choose a different location if you prefer.
00:07:07.830
We'll physically copy the Gemfile and Gemfile.lock into the volume to ensure those files exist, allowing us to run 'bundle install' to install all gems into the Docker volume.
00:07:15.810
At the point we specify the work directory, the app code isn't necessarily present in the container.
00:07:24.960
That's why we need to move the Gemfile inside the container.
00:07:32.060
Finally, every container needs a command to run by default. I typically use 'RUN bash' just to get things started, but we can override this later.
00:07:39.000
Alternatively, you might specify an entry point with a shell script to execute multiple commands.
00:07:44.640
For instance, if you want gem installation to occur after the container starts, you could place a 'bundle install' in there.
00:07:51.350
Here’s the completed Dockerfile: it's the minimal necessary configuration to run our Ruby application. Yes, we need Postgres and Redis, but they're not running within our container.
00:07:59.100
By design, each container should do one thing well; in this case, run our Ruby application.
00:08:05.740
Now, we're dealing with multiple containers since we need both a database and a Redis container.
00:08:11.100
That's managed via the Docker Compose file, which tells Docker how to handle these different containers.
00:08:18.170
In the services section, we start with the web service, which is our web container. We'll specify that all the code it needs is housed in the current directory, essentially at the root of your app code.
00:08:28.690
Typically, for local development, developers might place it in the root of their GitHub repository.
00:08:37.050
If you have DevOps, they often maintain a separate repository for their Docker files, or embed them in your code repository.
00:08:44.150
Next, we want to mount two volumes against our web service. Using '.' at the beginning specifies the current directory, which contains all our code.
00:08:52.410
We mount it to /app and give it a delegated tag. This is another change from what you would see in production.
00:08:59.050
This instructs Docker to ensure any local changes sync directly with /app in the container, allowing for real-time code changes without needing to rebuild the entire container.
00:09:07.490
Next, we override the default command. Here, we simply specify 'ruby app.rb' to initiate Sinatra.
00:09:14.210
We then declare which ports we want to expose from Docker's network to our local network.
00:09:20.310
In our case, we'll expose Docker's port 4567, allowing access through localhost:4567.
00:09:27.210
Now, we add two more services: database and Redis. We specify the default Postgres 12 and Redis containers.
00:09:35.580
In the Docker Compose file, we can specify the logging driver to none, which effectively turns off logging.
00:09:44.470
You can remove these lines if you need debugging, but they can clutter your logs.
00:09:50.520
We also introduce an environment section in our web service. These are custom environment variables exposed only to the container.
00:09:58.740
This allows us to modify or add variables without needing to rebuild the entire container.
00:10:03.400
For our mock application, we’ll expose the database URL and the Redis URL.
00:10:11.050
Lastly, we include a 'depends on' section to ensure the database and Redis services start up alongside the web service.
00:10:17.110
Here’s our complete Docker Compose file, which outlines the various services involved with our application and how to expose and link them.
00:10:25.869
As for workflow, the first step after adding these files is to run 'docker compose build,' which will execute all commands in the Dockerfile and pull down the necessary containers.
00:10:40.420
You can run this command repeatedly with no changes if nothing in the Dockerfile has been modified.
00:10:52.420
If you decide to adjust the Dockerfile, the first change point is reflected in the web container.
00:11:05.290
Next, you'll want to start everything up with 'docker compose up,' which will launch all the services.
00:11:20.150
You'll see various logs in your terminal; if you wish to start a specific service, provide its name with the command.
00:11:30.110
Keep in mind the dependencies we outlined earlier.
00:11:38.050
To turn everything off, you can hit Ctrl+C or run 'docker compose down' in a separate terminal window.
00:11:44.570
But that’s just the basics; we need to engage with our code beyond merely executing it.
00:11:50.300
Essentially, we need to run rake tests or drop into a console for debugging.
00:11:55.910
The flow for that involves using 'docker compose exec', where you specify a service name along with the command to run.
00:12:01.250
For instance, if you want to list all rake tasks, run 'docker compose exec web rake -T'.
00:12:10.520
Similarly, you can access IRB, the Rails console, or invoke 'bash' to troubleshoot within the container directly.
00:12:17.400
This is essentially the workflow when using containers locally. To summarize, we change into our app directory and run 'docker compose up'.
00:12:30.360
You'll see that DB, Redis, and the web container all start successfully.
00:12:38.050
Sinatra will listen on the designated port, and accessing localhost:4567 will connect you to your endpoint.
00:12:45.790
Congratulations! You've successfully containerized a local app from scratch.
00:12:54.460
Now, let's consider whether all of this effort is truly worthwhile.
00:13:03.980
We've discussed three scenarios, the first being about multiple Ruby applications that are completely independent from one another.
00:13:10.580
The setup I find most effective here involves creating a distinct Dockerfile and Docker Compose file for each repository.
00:13:17.120
You manage your app dependencies by navigating into each app directory and running it individually.
00:13:29.510
This is beneficial for side projects or small teams, especially when you only have a single app.
00:13:35.240
However, if you have multiple apps, this approach fosters clear separation of context.
00:13:41.670
If I want to switch from app 1 to app 2, I simply stop app 1, change directories to app 2, and start it.
00:13:47.810
The apps won't interfere with one another, and since this is essentially virtualization, they won't compete for laptop resources.
00:13:56.080
But I should note, Docker does consume a fair amount of RAM.
00:14:02.050
An alternative setup involves using Docker solely for external dependencies, like the database.
00:14:09.240
This is a hybrid approach, allowing your Ruby code to run natively on your machine while utilizing Docker only for database services.
00:14:15.460
This method reduces the hassle of managing Postgres upgrades.
00:14:22.750
To upgrade Postgres, you simply dump the existing database, remove the old Postgres installation, install the new one, and restore from a backup.
00:14:30.050
With Docker, you just change the tag in the Compose file, remove the containers, bring up the new one, and restore your dev database.
00:14:38.560
Also, every team can work on their respective apps without impacting each other's setups.
00:14:44.460
This hybrid approach also enables experimentation with different database versions or data stores.
00:14:50.470
You can pull down containers to test with different configurations without the need for local installations.
00:14:57.320
The workflow remains similar; you can specify just the database and Redis containers, then manually execute whatever command you need to start your local server.
00:15:04.280
The general consensus is that using Docker with all other dependencies is worth it.
00:15:11.150
It's particularly beneficial for context switching, especially with multiple projects.
00:15:17.560
Personally, I maintain four different Ruby applications, which makes context switching much easier. I can shift between them without concerns.
00:15:25.160
You won’t worry about a conflict between a MySQL and a Postgres installation because everything's contained within Docker.
00:15:32.920
A company called Lessonly in Indiana explicitly uses this Docker configuration. It expedites onboarding.
00:15:41.170
Their onboarding steps are simplified: install Ruby, install gems, and install Docker.
00:15:47.200
Simply run one command to get your database set up—no need to deal with Homebrew, Postgres versioning, or complex environment URL configurations.
00:15:53.300
This method saves you from managing Docker volumes manually, streamlining your workflow.
00:16:01.800
Okay, now let’s consider a scenario involving a client-facing application using Sidekiq for background jobs.
00:16:09.560
That application has a Redis instance, and your company develops a second application that utilizes Elasticsearch.
00:16:17.860
That app also contains its own Redis instance and potentially shares its main database.
00:16:25.690
Now with the addition of apps three, four, and five, each may involve varying dependencies and altering configurations.
00:16:34.360
In a large-scale environment, effective communication is paramount.
00:16:41.420
By deploying a system allowing teams to manage different containers per their requirements, you minimize dependency clashes.
00:16:49.780
One possible framework would be to have every team checking out code into a unified directory.
00:16:57.000
Let’s say the code resides in '/home/your_user/work/projects/app_1, app_2...', and so forth.
00:17:05.260
Then, introduce a bootstrap repository holding all the necessary Docker and Docker Compose files.
00:17:12.080
The bootstrap repository orchestrates all the services required by the applications, ensuring they work smoothly.
00:17:20.750
This makes for a neat, organized setup and aids in seamless teamwork.
00:17:31.200
A more condensed version of the Docker Compose file can manage dependencies efficiently.
00:17:40.200
If all developers agree to use Postgres 12, they only need to check out the necessary services available.
00:17:48.030
Share one database cluster to manage multiple database services, which keeps data organized.
00:17:57.560
Docker supports mounting volumes effectively so that multiple apps can work on distinct namespaces of the database.
00:18:06.100
This setup ensures that each app can interact with the necessary databases without conflicting with one another.
00:18:14.860
Each team thus focuses solely on the apps they need, running 'docker-compose up' without worrying about unrelated services.
00:18:22.880
The development flow becomes much simpler.
00:18:31.800
New developers check out just the bootstrap and the apps they’re interested in, reducing unnecessary complexity.
00:18:41.580
In doing so, another team could make updates independently without disrupting others.
00:18:51.840
However, it does call for synchronized communication among teams. For example, if a major Postgres upgrade is on the horizon...
00:19:01.460
the bootstrap app maintainer simply pulls a new request, updates the tag, which is followed by an announcement to all teams.
00:19:12.540
Everyone then rebuilds their containers accordingly.
00:19:20.930
No one needs to worry about the intricacies of integrating their applications with the latest changes.
00:19:28.520
So, is this structure worth it? Absolutely, especially for companies managing multiple applications.
00:19:39.510
It's particularly beneficial to manage numerous apps without requiring direct configurations from all developers.
00:19:46.049
SpringBuck is one example of an Indiana company leveraging this structure.
00:19:54.380
They've adeptly scaled their environment with five applications ringing alongside three databases.
00:20:01.930
All new devs simply sync with the bootstrap repository and follow a few explicit steps to start their setup.
00:20:10.830
This leads to less confusion and a more productive environment for everyone.
00:20:20.210
However, there is a caveat. What if you're developing gems?
00:20:29.710
Spoiler alert: this can be challenging. I've tried maintaining a few gems, and this is what I came up with.
00:20:36.960
You have a Dockerfile and a Docker Compose file set up in a root directory, while keeping your gem code above these directories.
00:20:43.560
The goal is to have one standard Dockerfile that essentially just executes Ruby.
00:20:50.310
This setup allows all directories to mount as volumes.
00:20:58.020
It leads to oversimplified Dockerfiles that handle a multitude of tasks, further complicating the actual work.
00:21:05.460
Using the default container may necessitate complexities that gem tasks sometimes present.
00:21:10.920
This might involve setting up Ruby versions but is clumsy at best.
00:21:17.550
If you want to run your tests against every database, you can indeed create corresponding entries but requires more setup effort.
00:21:27.320
This also enables you to run all your external dependencies in one network.
00:21:35.600
But I found myself overwhelmed while exploring this for gems—it's not worth it for standard Ruby developers.
00:21:44.250
Instead, I recommend using the first scenario as a template, which keeps the original gems isolated to easy work.
00:21:51.270
This way, external contributors can pull the repository and easily set everything up they require.
00:22:02.260
There are certainly pros shared across these examples.
00:22:08.650
They simplify bootstrapping: you avoid applying installations manually and can construct your development environment quickly.
00:22:16.430
It's critical to have your development environment closely resemble production; discrepancies can introduce costly errors.
00:22:24.270
Dependency management becomes easier as well, with applications utilizing Postgres, and upgrades seem virtually effortless.
00:22:32.310
The ultimate workaround is to completely rebuild the container if issues arise, relieving cumbersome environmental factors.
00:22:40.100
Nonetheless, there are some cons to containers: running everything on a VM does introduce some speed limitations.
00:22:48.090
If you’re running heavy computations, expect to wait a little longer compared to a native install.
00:22:55.880
Also, using linters like Rubocop might be slow, since those run in the Docker container, rather than directly.
00:23:02.650
You might still need to install Ruby locally, depending on your workflow.
00:23:10.770
Furthermore, Mac OS presents its own difficulties—especially regarding file system performance.
00:23:17.570
Changes made on your Mac may not always reflect in Docker containers immediately.
00:23:25.180
Some junior developers face a learning curve since they're not familiar with Docker.
00:23:33.490
They need to remember that the code runs in a container and requires consideration from the outside in.
00:23:41.760
Docker consumes considerable resources, like RAM and hard drive space.
00:23:48.739
Frequent Docker image builds can consume space quickly, necessitating a clean-up process.
00:23:56.290
Overall, while local containers may slow down development, they can benefit larger applications and teams significantly.
00:24:02.840
Still, for smaller projects, it may be best to evaluate requirements and adjust based on the intended use.
00:24:10.890
It’s a significant change in workflow, matching towards more orchestrated systems.
00:24:17.760
Ultimately, if your growth projections suggest more apps on the horizon, using Docker could simplify the future development process.
00:24:26.000
So, there you have it: my perspective on whether local containerization is worthwhile.
00:24:32.990
You may have already seen some slides or materials supporting the use of containers.
00:24:39.640
Thank you for spending your time here at the conference, and I hope you learned something beneficial today.
00:24:46.710
Any questions? I have about a minute or so. Thank you!
