Developer Experience (DX)

Summarized using AI

Building Confidence on a Docker Devbox

Tobias Meyer • May 05, 2018 • Hamburg, Germany

The video titled Building Confidence on a Docker Devbox, presented by Tobias Meyer at Ruby Unconf 2018, discusses the advantages of using Docker for developing and testing applications in a cohesive environment. Meyer highlights the challenges faced in modern development due to the increasing complexity and diversity of tech stacks within organizations, making local setups cumbersome. In particular, he emphasizes the need for developers to build confidence in their code by running entire systems on their local machines, and proposes a solution through Docker and Docker Compose.

Key Points Covered:

- Introduction to the Problem:

- Increased services, languages, and diverse stacks lead to difficulties in local environment setup.

- Confidence in running code and infrastructure is crucial for developers.

- Docker as a Solution:

- Docker allows running applications in isolated containers, simplifying the development process.

- Docker Compose helps manage multiple interconnected containers easily.

- Setting Up a Development Box:

- Importance of making the development environment act like a local staging system.

- The need for an efficient DNS setup to handle domain names for services in the local environment.

- Using Proxies and Certificates:

- Implementing NGINX Proxy to manage services under subdomains and improve routing.

- Using Let's Encrypt or self-signed certificates to secure HTTP connections.

- Integrating Services:

- Sharing and reusing existing Docker images to simplify setup, ensuring consistent configurations across environments.

- Available tools for database migrations and handling SSL communication among services.

- Managing Code Changes:

- Strategies to run and manage local code changes inside Docker containers and optimize the development process.

- Suggestions on organizing Docker Compose files to manage multiple services efficiently.

- Best Practices:

- Consistency in tooling and setup across team members.

- Keeping up-to-date with Docker and network settings to avoid friction in development.

Conclusion and Takeaways:

Meyer concludes with the importance of setting up a development environment that mirrors production. This not only builds developer confidence but also streamlines the overall development and testing processes within organizations. He encourages collaboration and sharing of best practices among developers in utilizing Docker effectively. He also invites the audience to reach out for further discussion on related topics.

Overall, the presentation provides insights into leveraging Docker to create an efficient and reliable development environment, showcasing its potential to improve productivity and code confidence.

Building Confidence on a Docker Devbox
Tobias Meyer • May 05, 2018 • Hamburg, Germany

Ruby Unconf 2018

00:00:16.630 Thank you.
00:00:18.680 So, no questions is not the case. You can ask me questions afterwards, and we can discuss them.
00:00:23.090 I said in the presentation at the beginning, welcome! Thank you for showing up to my talk, the first part of the day.
00:00:28.430 First of all, in my job, I think what I'm going to talk about is exciting. I tried to adjust some of the slides to convey the right message. So, it should work.
00:00:36.550 If you have any questions about how we will proceed, please feel free to ask.
00:00:39.690 Also, I would like to know about your experiences with the topic I'm talking about, so I would appreciate it if you could raise your hand.
00:00:48.980 It looks good, so I can keep this part short, I think.
00:01:01.880 So what is this about in general? We are getting more services, more languages, and more diversity in our stacks. There are a lot of different versions of software and many teams working on the same product.
00:01:07.270 Especially in big companies, but also small companies tend to split their systems up and distribute them. However, it's hard, in a setup like this, to run the whole system on your local machine.
00:01:19.320 The problem arises that you don't really have a way to build confidence in what you are holding, especially if you work on infrastructure or back end.
00:01:27.450 When I build it, I want to see it running. Of course, there are tests that should be running, code reviews, and many other things that you should do to build confidence in your code.
00:01:33.349 However, there's nothing like actually seeing it run the way it should and with all of the web services. It's sometimes hard to set up the whole system on your local development machine.
00:01:40.200 So the idea, based on my experience, is to use Docker. Docker and Docker Compose allow you to configure your services to run in containers.
00:01:49.790 These containers have a small operating system. It's not really the whole operating system, as the kernel is reused from the host system, but you have your database or Alpine on top.
00:02:00.090 On top of that, you can install packages or dependencies for your application, and then you can run your application inside this container.
00:02:10.530 Docker Compose allows you to compose these different containers and define how they should be started, what their environment should be, and you can link files into the container.
00:02:20.040 You can even link them together so they can communicate with each other. These services can be the services you build yourself, but also databases, caches, or anything else you might need.
00:02:27.210 The requirements for such a development box should be that it acts like a local staging environment and is usable during development.
00:02:35.640 I want the changes I make to be reflected in this running system as fast as possible. It should be short, easy to bootstrap, and easy to use.
00:02:42.510 We don't want to be bogged down by details while using containers. It should be something you can just pack and play.
00:02:50.830 My experience with a system like this comes from working at Jindo, where we have created a setup using Docker and Docker Compose to run our entire system for quite some time.
00:03:02.600 We have had quite a good experience. We even use it during our continuous integration process to run integration tests.
00:03:11.950 We spin up the whole system during a pull request, run our tests against it, and it automatically shuts down after that.
00:03:21.020 With our new product, Olson & Photos, we are in the process of setting it up as well. The feeling of gaining confidence in the changes we make has greatly improved.
00:03:29.230 Previously, we had a staging environment that was slow because you had to wait for it to deploy and then test it there. If it didn't work, you'd return and deploy again.
00:03:41.060 But with this new setup, you can see what fits a lot better. We want our demos to act like local staging systems, as close to production as possible.
00:03:49.270 To do this, one thing we want to have is a domain name. You shouldn't run your services online by simply exposing ports and giving people access.
00:03:55.480 You need domain names, so the first thing to do is get a domain. This can be done quite easily by using test domains reserved specifically for this use case.
00:04:05.630 Choose a domain with the TLD test. If you want to run this on your local network, you need a nameserver that responds to requests for this domain.
00:04:13.130 You can start a DNS server in your environment to handle requests for the domain. For this example, I'll choose the domain 'devbox.test'.
00:04:23.850 With this setup, you need to make sure your system knows where to ask for the domain name.
00:04:32.330 Unfortunately, many operating systems, like Linux, have to be configured to ask the specific DNS server for those requests.
00:04:42.610 You can bind your web server to this IP address and ensure that your DNS server returns the right IP when queried.
00:04:49.070 With Docker set up on this port, it will always return the same IP address, which is the IP address used for your containers.
00:05:01.230 The first part is done: I can go to my browser and type 'devbox.test', and it should point to the container environment. However, if nothing is running, I won't see any response.
00:05:11.580 This can also be easily solved. There is a nice project by Jay Wilder called NGINX Proxy that acts as a proxy for your containers.
00:05:20.480 It comes with many features, the main one being that you can plug it into your Docker Compose file, and it will parse the configuration for different services.
00:05:29.690 For example, it will look for virtual host and port environment variables and use those to determine under which subdomain a service should be hosted.
00:05:40.230 You bind this proxy to the HTTP and HTTPS port and configure it with the virtual host settings for your services.
00:05:49.670 So when you launch your service, it will automatically become accessible under its appropriate subdomain.
00:05:57.570 Now we are capable of adding services to our Docker Compose setup, and they will all be available under subdomain names.
00:06:07.790 Next, you probably want to enable HTTPS for your services, which is a default requirement in most cases.
00:06:15.270 There are tools available, such as Let's Encrypt, which allow you to generate valid certificates.
00:06:22.200 Alternatively, you can use OpenSSL to generate your own CA and sign a self-signed certificate for your test domain.
00:06:31.200 Once you have this certificate, you can proceed to configure NGINX as a proxy to use these certificates for HTTPS.
00:06:39.420 Unfortunately, for self-signed certificates, you will need to import the public key of your CA into your browser or operating system.
00:06:49.175 Be careful with the privacy of the CA key; if it gets leaked, someone could generate certificates that might compromise your applications.
00:06:57.770 Always keep this key safe, or you can throw it away after generating certificates and create a new CA if needed.
00:07:07.410 Now, we have set up Docker with a DNS that responds to our domain, a proxy that forwards requests to our services, and certificates for HTTPS.
00:07:13.820 The next step is to run all your services inside Docker Compose.
00:07:22.560 This can get messy depending on how your current setup is.
00:07:30.660 Typically, we have the luxury of already running all services on Docker in production.
00:07:39.100 For every service, you need a Docker image as an artifact, and when setting up the dev box, we could theoretically use those images.
00:07:49.740 However, there can be issues with existing Docker images. Sometimes configuration files are baked into the images, or you may have to deal with multiple configurations.
00:08:07.640 For instance, if you are using Webpack, you might have different configurations for different environments.
00:08:16.600 This situation can lead to a lot of work, and there are many strategies for how to handle it, depending on the specifics of your setup.
00:08:26.300 If you have any questions about how a specific service could be imported or the strategies there, please feel free to come talk to me afterwards.
00:08:32.880 Assuming you successfully import all the services you need, you may find that it sounds a bit daunting at first.
00:08:40.170 However, if you start tackling it, you might find it's not so bad.
00:08:47.990 This is especially true for databases, caches, and infrastructure, because there are already images available.
00:08:54.850 Even if you don't find an image for your Ruby service, you can find one based on existing technology.
00:09:01.520 Generally, for most projects, you can just take the code you have and plug it into one of the existing containers.
00:09:05.940 You may need to do some adjustments, but you shouldn't be afraid—it can actually be quite nice.
00:09:14.260 When you have the service running properly in Docker, you won't have to keep reconfiguring your local system repeatedly.
00:09:23.080 Instead, you can just take the Dockerfile and give it to your colleague, who can replicate the same service with the same configuration.
00:09:30.750 Everyone's setup would be identical, eliminating discrepancies.
00:09:38.610 Now that we have a running system with all the necessary services, let's talk about data.
00:09:46.750 If you have pictures or other necessary data for your system, you need to ensure that your database migrations are also run.
00:09:54.300 There are also additional gems or packages that you may want to install.
00:10:00.350 To approach this, you can call the same functions you would on your local machine using commands within Docker.
00:10:06.080 Use the command 'docker-compose run' to execute commands inside a container.
00:10:14.920 I have had successful experiences with executing such commands within the containers.
00:10:22.700 Sometimes, you might need additional configuration, but with some adjustments, it tends to work quite well.
00:10:30.700 However, you may encounter issues—especially concerning SSL communication among services.
00:10:44.150 Sometimes, services are configured to exclusively use SSL for communication, which is a good practice.
00:10:53.470 In such cases, you'll need to provide the necessary development CA certificate to your services.
00:11:03.700 This is straightforward; import the certificate into the CA of your operating system.
00:11:09.830 Simply place the certificate in the certificate store, and it will generate the necessary files.
00:11:18.750 Unfortunately, this process doesn't work seamlessly for all languages.
00:11:26.750 For instance, the security models differ across programming languages, and you'll have to ensure that libraries are configured to use system certificates.
00:11:33.060 Make sure your service can access your local certificate store, so SSL communication can occur securely.
00:11:41.000 Once everything is running, including SSL communication, we should consider mounting our source code into the container.
00:11:48.800 This works well with scripting languages like Ruby, PHP, and Python, and even JavaScript.
00:11:56.520 For compiled languages like Go or Java, you might want to compile artifacts and run them.
00:12:02.450 However, you may be pleased to know that there are tools available that can detect changes and rebuild containers.
00:12:11.800 On a development machine, you can mount your source code into the Docker container.
00:12:20.000 You might also want to bind your project's directory to reduce the need for repeated installations.
00:12:27.050 If you adjust configurations like environment variables or system settings, you can do so through your Docker Compose file.
00:12:34.850 Furthermore, Docker Compose supports inheritance, allowing for multiple Compose files to be combined.
00:12:44.670 In the first file, you define the standard setup, and then in a second file, you add specific development configurations.
00:12:51.970 By using this feature, you can create a setup tailored for the specific service you're developing.
00:12:59.990 You can also ensure that any changes trigger relevant file system events forwarded to the container.
00:13:05.480 Be aware that if you're using Docker Machine, you may experience challenges with NFS mounting.
00:13:13.150 On the other hand, if you're running directly on Linux, those issues will likely be resolved.
00:13:20.870 Another feature of Docker Compose that can enhance your configuration is the use of placeholders, allowing for greater flexibility in environment variables.
00:13:32.100 For instance, you can define a DNS server configuration and reuse it across your setup. It makes managing configurations easier.
00:13:43.430 However, one thing I find missing is the ability to define included files in your configuration, which would simplify management.
00:13:51.150 A big Docker Compose file with all your services can quickly become unwieldy.
00:14:00.140 It’s preferable to have each service defined in different configuration files for better organization.
00:14:08.529 Additionally, it's important to keep up with the latest updates on how project names are being handled in Docker Compose.
00:14:16.170 This includes making sure that your project names are correctly stripped of dashes, which may not happen in new versions.
00:14:25.530 To ensure consistency, it's a good practice to define the project name in an environment variable.
00:14:30.980 As a side note, make sure that the docker version and network settings are aligned throughout your environment.
00:14:44.430 It is also advisable to create some toolset to help with setup, execution, and maintaining your local environment.
00:14:53.080 This includes placing DNS files in appropriate locations and managing migrations.
00:14:59.700 It's crucial to ensure that everyone is using the same tooling and that they are familiar with it.
00:15:06.800 That’s it from my side, and here are some links for the libraries I used and my contact information.
00:15:14.140 If you have questions or if you're interested in Docker, server management, or Raspberry Pi, feel free to reach out.
00:15:18.280 Thank you very much!
00:15:26.960 One thing I like to do during retrospectives is to ask for feedback.
00:15:30.200 I would like you to rate your time investment on a scale from one to five.
00:15:34.380 One finger means it was a very poor time investment, while five fingers indicate the best time investment.
Explore all talks recorded at Ruby Unconf 2018
+9