Keep Ops Happy, Designing for Production

by Jason Roelofs

In this talk at MountainWest RubyConf 2013, Jason Roelofs discusses best practices in application development to ensure operational efficiency and resilience during production deployment. His presentation revolves around the practices developers can adopt to minimize production errors and maintain smooth application performance.

Key points discussed include:

- The Importance of Documentation: Roelofs emphasizes that proper documentation is crucial for understanding what the application does, its resource needs, and how to run it on different machines. This should come in various forms, like README files and executable documentation.

Frequent Deployments: He advocates for deploying code as frequently as possible, ideally several times a day, maintaining code in a deployable state through tools like Capistrano and Heroku.
Configurable Applications: The speaker stresses that applications should avoid hard-coding configurations and instead utilize environment variables or configuration files, allowing seamless transitions between development and production environments.
Thread Safety: While thread safety has not been a major concern traditionally in Ruby, as the language evolves, developers are encouraged to avoid sharing mutable states between requests to maintain stability.
Security Best Practices: Roelofs underlines the necessity of robust security measures, including protection against attribute injection, using secure password hashing (e.g., bcrypt), and cautious serialization practices to avoid remote code execution risks. He mentions middleware tools like Rack Protection and rate-limiting mechanisms like Rack Attack as essential for fortifying applications against automated attacks.
Storage Solutions: Utilizing reliable cloud storage solutions like Amazon S3 is recommended over local file storage to enhance security and scalability.
Monitoring and Alerts: Roelofs concludes by advocating for the use of monitoring tools like New Relic and alert systems like Dead Man's Snitch to efficiently track application performance and catch discrepancies early.

The overarching message from Roelofs is that by implementing thoughtful documentation, frequent deployments, securing applications, and utilizing monitoring tools, developers can vastly improve operational readiness and efficiency, ultimately keeping both the operations team and customers satisfied with application performance.

00:00:20.480 Hello everyone, as Mike said, I'm Jason Roelofs. I work for Collective Idea, a small contracting company based in West Michigan, and I'm one of their two DevOps people.

00:00:27.039 I've been professionally developing for seven years and have spent my entire career in various operational roles. A lot of what I’ll discuss today comes from experiences I've encountered and lessons I've learned along the way.

00:00:40.480 This talk focuses on development practices, patterns, and techniques that you can incorporate into your code to make it easier for operations to manage and run your application on the internet. As has been mentioned in previous talks, operations teams already have to deal with their share of difficulties because the internet can be a hazardous environment.

00:01:10.000 There are constant threats, with scripts running to exploit common vulnerabilities. As an operations person, I appreciate when I can trust that the applications I manage run smoothly on their own without constant oversight. However, managing firewalls and ensuring threats are mitigated can become overwhelming.

00:01:24.320 The first and most important aspect of any project is documentation. Documentation often gets a bad rap; many people either do too much of it, do it poorly, or dislike it altogether. However, it is crucial for several reasons.

00:01:36.960 Firstly, you need to make sure that people understand what your application does and how it operates. Any documentation should address three key questions: What does the application do? What resources does it need? And how does one get it running on their machine? Each type of application has different requirements for hosting and operation, and understanding these needs is vital for operations personnel.

00:02:10.040 For instance, knowing if your application uses a database, needs a caching layer, or requires a load balancer is essential for provisioning that application on production servers. If you can't even get the application running on your own development machine, how can you expect it to run in production? Documentation should ideally come in forms like a README or well-maintained documentation files that provide clear instructions.

00:02:50.560 Another helpful approach is to ensure that documentation is executable, such as including information in a Procfile when using Foreman, or creating scripts that help automate setup and configuration. Applications without documentation can be incredibly frustrating, leading to unnecessary delays as developers spend time figuring out what is needed.

00:03:01.440 Next, a crucial part of DevOps philosophy is to deploy as frequently as possible. The code should always be in a deployable state. You should aim to deploy every day, or ideally multiple times a day. At the end of a sprint or iteration, you should have a working version of the application deployed so everyone can interact with it.

00:03:39.440 Deployments should be streamlined and standardized to reduce the risk of errors. Tools like Capistrano can help automate the process, and ensuring that deployments are codified minimizes the chance of mistakes. When you start new applications, leveraging services like Heroku is beneficial since their first dyno is free, allowing developers to understand deployment needs without financial pressure.

00:04:01.680 Now, let's dive into some specific things you can do in your code to enhance its robustness. First, we will discuss the configurability of your application. As someone who works in operations, I value applications that are easily configurable. It is not about excessive configuration files; rather, it is about separating what your application does from who it does it with.

00:04:22.240 For instance, your application may require a database, but it shouldn't be hard-coded with specific database credentials. Instead, it should be able to recognize configuration settings like host, username, password, and port from a dedicated configuration file or environment variables.

00:05:01.440 This approach allows you to easily switch your application's database connection based on the environment, such as using different databases for staging and production, maintaining separation of concerns regarding passwords and sensitive information.

00:05:22.320 One pattern to avoid is embedding configurations directly in your application code. Hard-code connections to services, such as Redis, can lead to potential issues. For example, if a Redis connection works in development but requires different settings for production, embedding that information into the code can lead to confusion and expose sensitive data.

00:06:02.880 Using Ruby, this can be made easy through the use of YAML files in your config directory, allowing you to store your configuration separate from your application logic. Additionally, tools like Figaro can help manage environment variables for productions like Heroku, easing configuration management.

00:06:23.520 The use of different environment variables can allow developers to seamlessly switch between production and development environments without exposing sensitive credentials in the code. It’s crucial to maintain cleanliness in your configuration, especially regarding the environment you are working within.

00:06:47.440 Moving further, I want to address the topic of thread safety in code, which has been less of a concern within the Ruby and Rails community, where we generally rely on multiple processes rather than threads. However, as Ruby evolves, the capacity to run multiple threads across CPUs using tools like JRuby and Rubinius becomes more relevant.

00:07:29.280 Rails is designed to be thread-safe, and while most of its components handle this effectively, developers still need to ensure their own code does not violate thread safety principles. The key rule for achieving thread safety is to avoid sharing mutable state between requests. Instead, store data in a session or database.

00:08:16.880 If you have to share certain values across requests, you can use a structure like `Thread.current`. However, be vigilant because if exceptions occur without proper clearing of your data, you might end up sharing unwanted state between requests, leading to unintended behavior.

00:09:02.360 Next, I want to stress the significance of security in application development. Every developer needs to have an understanding of security implications, especially in the context of web applications. Just throwing together functionality without considering potential security holes is a recipe for disaster.

00:09:37.680 The first area to focus on is preventing attribute injection. Rails provides mechanisms to protect against this by designating which attributes are accessible. This isn’t the default in many applications, and lack of this protection opens the door for vulnerabilities.

00:10:20.480 With the introduction of strong parameters, you will have a defined mechanism for which attributes are permitted to come from user input. It helps you maintain documentation and clarity regarding what input your application may accept.

00:10:54.800 When it comes to password handling, there’s a need to be vigilant. Passwords should never be stored in plain text, and methods like MD5 or SHA hashing are no longer secure. Today, bcrypt is recommended for hashing passwords.

00:11:22.720 Bcrypt introduces a complexity factor, making it slower to generate hashes and much harder for attackers to brute force them. For instance, as GPUs have become more rapid and efficient, it is essential to use resistant hashing algorithms like bcrypt that can be scaled up as hardware improves.

00:12:00.320 Additionally, serialization is a hot topic regarding security in the Rails community. Recently, issues have arisen around YAML serialization, which exposes applications to remote code execution risks. A safe practice is to avoid serializing custom types. Instead, stick to using simple data structures such as strings, hashes, and arrays.

00:12:36.560 When dealing with jobs, remember to use ids instead of full user objects during serialization to minimize overhead and avoid complications.

00:13:18.640 On the topic of security, elastic opens up challenges including cross-site scripting or clickjacking. Luckily, many of these vulnerabilities can be managed using middleware like Rack Protection, which automatically implements protections against common security threats.

00:13:56.960 We recently found ourselves bombarded with automated scripts attempting to identify security flaws in our applications. Implementing throttling mechanisms such as Rack Attack helped limit requests and mitigate the risk from these attacks. Also, using techniques like honeypots allows us to filter out automated submissions effectively.

00:14:33.280 In general, be sure to avoid storing user uploads or any important data on a local file system. Services like Amazon S3 offer scalable and secure storage solutions that take away the burden of managing local storage.

00:15:09.400 Additionally, don’t run scheduled jobs during hours where they may be affected by daylight savings—such as between 1 and 3 AM—as this can lead to jobs not executing or executing multiple times.

00:15:47.520 If your project requires managing payments, I highly recommend using established services for billing rather than attempting to handle this yourself, as issues could significantly disrupt your workflow.

00:16:21.040 To wrap things up, I’m a big fan of monitoring services to keep track of application performance. Tools like New Relic and Instrumental are fantastic for statistics and analytics. Log aggregation services like Loggly and Papertrail can simplify log management significantly.

00:17:03.120 One of my favorite services is Dead Man's Snitch. It alerts you if specific tasks don’t execute as planned, ensuring you stay on top of your monitoring capabilities. This service has saved my team during crucial moments.

00:17:45.680 That concludes my talk. I’m Jason Roelofs from Collective Idea. Feel free to check out my website, GitHub, or Twitter for more information. Thank you!

MountainWest RubyConf 2013