Talks
Building and Managing a Ruby Infrastructure
Summarized using AI

Building and Managing a Ruby Infrastructure

by Jeremy Hinegardner

In this keynote presentation titled "Building and Managing a Ruby Infrastructure" at the LoneStarRuby Conf 2008, Jeremy Hinegardner discusses the process of developing and managing Ruby applications within a company called Collective Intellect. The focus revolves around the pitfalls and best practices associated with Ruby infrastructure, particularly in the context of rapid growth and development.

Key Points Discussed:

  • Initial Challenges:

    When Hinegardner and his co-presenter, Fernand, began at Collective Intellect, they experienced issues with messy, unfactored code due to a startup mindset focused on rapid iterations and customer service over code quality. Problems arose from doing direct deployments from developer workstations causing inconsistencies and errors.

  • Transition to Plugins:

    As the team grew, they attempted to organize their code into Rails plugins. However, this made the system brittle due to the increase in external dependencies, complicating maintainability and encapsulation of the core business logic.

  • Embracing Gems:

    They ultimately shifted their strategy to using Ruby gems, leading to a more organized and maintainable codebase. This modular approach allowed for isolating different aspects of their application and developing around 20 specific gems.

  • Operational Improvements:

    The presentation emphasizes the introduction of continuous integration through tools like Cruise Control that checks builds continuously to avoid deploying bad code. They also established dedicated test databases for developers to streamline testing processes.

  • Documentation and Communication:

    Effective communication and documentation practices within the development team were deemed essential, especially as the team size expanded. The importance of keeping dependency specifications tight and managing gem versions effectively during deployments was highlighted.

  • Deployment Practices:

    Hinegardner shared insights on using Capistrano for deployment across applications and creating a gem server to manage both internal and external dependencies, ultimately leading to more stable and efficient deployments.

Conclusions and Takeaways:

  • The presentation concluded with the understanding that successful management of a Ruby infrastructure requires effective organization through gems, a solid continuous integration process, and robust communication and documentation practices. Hinegardner encouraged the audience to implement these best practices in their own Ruby environments to foster stability and efficiency in development processes.
00:00:06.359 Video equipment rental cost paid for by Peep Code.
00:00:18.720 Screencasts. Hi everybody, let's get going!
00:00:24.960 Okay, well first off, this is a keynote presentation at a Ruby conference. Coco is in the other room.
00:00:31.880 Sorry about that; we need to keep this focused on Ruby.
00:00:39.039 Alright, let’s get into it. What Fernand is doing here is something we do all the time: installing gems in our work environment.
00:00:44.640 We start up different things regularly, so how many people here use gems internally—not just third-party gems—but within your own applications? Raise your hands!
00:00:50.280 Looks like quite a few. Good job!
00:00:56.600 Okay, well, this is basically our slides as a Ruby application. Hopefully, it doesn’t look too bad. We never tried it on the projector yet, and so far, it looks pretty alright.
00:01:04.239 So first off, Fernand and I work for a company called Collective Intellect. We essentially collect data from blogs, message boards, news feeds, and other social media feeds.
00:01:13.720 We combine all of this data and provide a topic analysis and sentiment analysis, presented to our customers in various formats.
00:01:19.360 Let’s get going! First bug...
00:01:25.920 I'm Jeremy Hinegardner, and part of my day-to-day job involves handling a lot of data. I frequently shovel data muck around, moving it from point A to point B, converting other people's muck into ours.
00:01:41.600 I try to do a good job with this, and I also have a few open-source gems available, including 'Copious' and 'Rabble Keybox.' How many people have heard of any of these?
00:01:48.079 Alright, we see a few hands up. Great! The other thing I do is photography, and I'm curious about how many people we have here.
00:01:58.560 Okay, good to see! A quick question: I’ve spent a lot of my life trying to figure out who I am. So, let me tell you what I know so far.
00:02:13.720 You may remember me from one of my best-selling books, 'Get Confident, You Stupid!' Well, maybe that's not the case.
00:02:22.079 I was born and raised here in Texas, though I usually try to say I’m from France. This is a French TV anchor, who, in my opinion, is easier on the eyes than Ted.
00:02:34.920 Let’s see...
00:02:39.560 Oh, we're missing something. Well, I also host the Denver Rails User Group.
00:02:45.959 My company, Liquid Rail, produces a couple of things, one of which is called 'The Mall.' How many of you know what your web or Ruby applications are currently doing?
00:03:03.360 Is it still running? Is it performing? Do you know if anyone is using it? If you answered no to some of these questions, check out 'The Mall.' It allows you to inject information within your application code.
00:03:18.879 You don’t have to use Rails; it can be any plain old Ruby application. It helps you extract valuable information such as performance metrics and whether exceptions are thrown.
00:03:35.000 It also allows you to identify what features users are actually using, which is beneficial for your marketing team.
00:03:45.760 The second thing I wrote, along with Delin Berry, started as a Rails plugin and was converted into a gem. It's a framework that allows you to do charting in your Ruby or Rails applications.
00:03:58.520 This is a simplified version of our production environment.
00:04:06.879 Now, a couple of questions for everyone here: How many of you are working with more than one programming language in your environment?
00:04:16.720 Almost everybody! Great. How many of you have more than one application framework?
00:04:25.999 More than one application framework per language?
00:04:31.680 Alright, we see we've got different Ruby and Java applications.
00:04:39.600 How many of you are secretly Java developers disguised as Rubyists?
00:04:47.120 Come on, there’s more than that. Don’t be shy!
00:04:52.760 How about .NET developers? Any of those around?
00:05:00.639 Okay, a few. Any C programmers disguised as Rubyists?
00:05:07.480 Another question: How many of you are using more than one database in your environment?
00:05:14.919 Do those databases know about each other? No?
00:05:21.960 Well, in our setup, we have a transactional database with multiple slaves and a data warehouse.
00:05:27.959 This feeds data from our transactional database into the data warehouse where we conduct a lot of reporting.
00:05:36.680 Let’s start with the initial scenario.
00:05:44.600 When Jeremy and I first started at Collective Intellect, we were in startup mode, rapidly throwing code around and refactoring later.
00:06:01.639 You hear a lot about refactoring and patterns, but when you have a business to build and customers to serve, your customers aren't typically your code base.
00:06:11.320 You tend to become a bit sloppy about it.
00:06:19.560 The first version of our code was quite messy. It was unfactored code with a mix of framework components.
00:06:28.200 There was a lot of bad practices, and we can’t criticize this too much because, at the time, it felt like a good approach.
00:06:38.280 We had a single directory filled with a mixed bag of different components: models, utilities, and varying ways to connect to our databases.
00:06:46.800 As we increased our team size, the codebase grew rapidly and became unmaintainable.
00:06:54.560 With millions of records in our database, it became hard to create tests.
00:07:03.440 There was no testing at all, which turned out to be a really big problem.
00:07:11.440 We had models, logging, database plugins, and shared concerns scattered all over the place.
00:07:20.240 We also had SVN externals that pointed to additional locations, which added to deployment nightmares.
00:07:29.640 Worse yet, we were deploying code directly from developer workstations to production.
00:07:36.400 On multiple occasions, code that hadn't even been checked into Subversion ended up in production, leading to issues.
00:07:50.000 This made developers hesitant to deploy, as they were unsure of what was already there. As a result, they resorted to hand-patching production systems.
00:07:59.760 Over time, we realized that this wasn't the way to manage things.
00:08:06.800 We moved on to our second version, where we started breaking things down into Rails plugins.
00:08:15.360 We adopted a plugin directory structure and philosophy to address various concerns more effectively.
00:08:27.440 However, this approach quickly led to challenges as it introduced brittleness.
00:08:34.800 We found ourselves in a situation where we dealt with so many external dependencies that our application code became hard to maintain.
00:08:45.680 We thought dividing things into plugins would simplify our setup, but we learned otherwise.
00:08:54.000 By moving too far into modular approaches, we neglected the encapsulation of our core business logic.
00:09:05.760 In our refactoring process, we realized how essential it is to keep models cohesive, as they represent our business objects.
00:09:13.920 Over time, we learned that encapsulating our domain effectively was critical.
00:09:23.760 Here’s an example of our directory structure for plugins.
00:09:31.960 At this stage, we were only about 20% into our transition.
00:09:37.480 However, we noticed that certain plugins were being used across multiple applications.
00:09:43.960 This led us to think about the one true path in Ruby, which is gems.
00:09:49.920 We started fully embracing gems, which changed how we developed.
00:09:55.760 Moving into our third version, we realized that separating concerns into gems was far more beneficial than plugins.
00:10:04.240 We ended up refactoring our entire codebase into around 20 different gems.
00:10:14.839 Each gem serves its purpose, making our application easier to maintain.
00:10:21.040 The primordial gem became the basis of all our systems.
00:10:29.280 We also created a database interface through the sentiment gem.
00:10:35.840 We managed to isolate different aspects of the application's architecture.
00:10:44.520 One key aspect of this restructuring was the introduction of command line applications as gems.
00:10:51.679 We also made extensive use of Quartz, especially for scheduling jobs.
00:10:58.759 By integrating Quartz, we created a system to monitor various processes.
00:11:05.679 This allows for efficient job scheduling and management.
00:11:12.559 One of the critical components we developed was the primordial gem.
00:11:19.519 This gem handles project generation, managing dependencies, and ensures a consistent project layout for development.
00:11:27.840 This structure was extremely beneficial for new developers.
00:11:36.080 With uniformity, new and existing developers can navigate the code base much more easily.
00:11:43.760 Consistent project layouts also facilitated easier onboarding for new team members, enhancing productivity.
00:11:55.200 The primordial gem provides common tasks to every project ensuring all developers have the same tools at their disposal.
00:12:07.920 Sometimes, familiarity makes it easier to transition between projects.
00:12:15.520 This improved confidence in navigating the code base.
00:12:20.440 It promotes rapid development without fearing the unknowns of a new codebase.
00:12:29.760 This familiarity reduces onboarding time, increasing new developer productivity.
00:12:34.960 Testing became a fundamental part of our process.
00:12:40.760 In our new environments, every developer has their dedicated test database.
00:12:45.760 We're leveraging real objects for our tests, allowing for genuine interactions with the database.
00:12:53.680 This setup simplifies generating tests and quality assurance.
00:13:02.680 The challenge of dependencies led us to a more cohesive strategy.
00:13:15.000 We also implemented a parameterized approach for managing changes.
00:13:21.160 Next, we ensured database configurations remained consistent between development and production environments.
00:13:30.560 This check eliminates surprises during deployment.
00:13:37.600 Now we have tests that validate the configurations are the same.
00:13:43.280 All of our environments align, minimizing potential discrepancies.
00:13:51.840 Moving on to continuous integration, we’ve implemented a system with Cruise Control.
00:14:00.560 This CI system checks our builds continuously, preventing bad code from being deployed.
00:14:06.799 It encourages accountability among team members.
00:14:13.120 The environment works seamlessly with external dependencies.
00:14:21.840 If a gem changes, the system rebuilds all dependent applications.
00:14:28.760 This process has saved countless hours of debugging later.
00:14:37.760 As our team's size increased, managing code became a challenge.
00:14:45.480 Having a team of developers meant more integrations and potential issues.
00:14:53.880 Effective communication and solid structure fostered collaboration.
00:15:01.760 Our team eventually grew to 14 developers.
00:15:09.840 Fluctuations in team dynamics required us to develop an adaptable system.
00:15:17.920 We strive to foster an environment built on communication and teamwork.
00:15:24.320 We focus on continuous integration through feedback loops.
00:15:30.560 Part of our deployment involved building documentation as a phase.
00:15:38.560 Everything is made available through CI, ensuring everyone is on the same page.
00:15:44.640 We encourage effective documentation practices throughout our projects.
00:15:51.760 There's a method known as twiddle-waka, which helps manage gem dependencies.
00:15:59.760 It's important to tighten dependency specifications during deployment.
00:16:03.919 The importance of versioning cannot be overstated.
00:16:12.360 Deployment mechanisms need to be consistent and predictable.
00:16:18.599 We use Capistrano for managing deployments across varied applications.
00:16:25.080 This standardization has increased efficiency and reduced errors.
00:16:32.960 Cruise Control automates our workflow and manages build dependencies.
00:16:40.200 We create a gem server that handles internal and external dependencies.
00:16:48.320 As a result, our deployment process allows us to quickly respond to potential issues.
00:16:55.360 We’ve learned the importance of reliable package management at scale.
00:17:01.760 These measures have led to more consistent, stable deployments.
00:17:09.360 Overall, these experiences have shaped our approach to development.
00:17:16.320 Now, we have effective systems in place for managing our infrastructure.
00:17:22.080 Thank you all for joining us. We're happy to ask any questions!
00:17:30.560 We appreciate you being here and hope to connect with many of you afterward.
00:17:45.600 Thank you once again for your time!
00:17:51.200 (audience applause)
Explore all talks recorded at LoneStarRuby Conf 2008
+18