00:00:06.359
Video equipment rental cost paid for by Peep Code.
00:00:18.720
Screencasts. Hi everybody, let's get going!
00:00:24.960
Okay, well first off, this is a keynote presentation at a Ruby conference. Coco is in the other room.
00:00:31.880
Sorry about that; we need to keep this focused on Ruby.
00:00:39.039
Alright, let’s get into it. What Fernand is doing here is something we do all the time: installing gems in our work environment.
00:00:44.640
We start up different things regularly, so how many people here use gems internally—not just third-party gems—but within your own applications? Raise your hands!
00:00:50.280
Looks like quite a few. Good job!
00:00:56.600
Okay, well, this is basically our slides as a Ruby application. Hopefully, it doesn’t look too bad. We never tried it on the projector yet, and so far, it looks pretty alright.
00:01:04.239
So first off, Fernand and I work for a company called Collective Intellect. We essentially collect data from blogs, message boards, news feeds, and other social media feeds.
00:01:13.720
We combine all of this data and provide a topic analysis and sentiment analysis, presented to our customers in various formats.
00:01:19.360
Let’s get going! First bug...
00:01:25.920
I'm Jeremy Hinegardner, and part of my day-to-day job involves handling a lot of data. I frequently shovel data muck around, moving it from point A to point B, converting other people's muck into ours.
00:01:41.600
I try to do a good job with this, and I also have a few open-source gems available, including 'Copious' and 'Rabble Keybox.' How many people have heard of any of these?
00:01:48.079
Alright, we see a few hands up. Great! The other thing I do is photography, and I'm curious about how many people we have here.
00:01:58.560
Okay, good to see! A quick question: I’ve spent a lot of my life trying to figure out who I am. So, let me tell you what I know so far.
00:02:13.720
You may remember me from one of my best-selling books, 'Get Confident, You Stupid!' Well, maybe that's not the case.
00:02:22.079
I was born and raised here in Texas, though I usually try to say I’m from France. This is a French TV anchor, who, in my opinion, is easier on the eyes than Ted.
00:02:34.920
Let’s see...
00:02:39.560
Oh, we're missing something. Well, I also host the Denver Rails User Group.
00:02:45.959
My company, Liquid Rail, produces a couple of things, one of which is called 'The Mall.' How many of you know what your web or Ruby applications are currently doing?
00:03:03.360
Is it still running? Is it performing? Do you know if anyone is using it? If you answered no to some of these questions, check out 'The Mall.' It allows you to inject information within your application code.
00:03:18.879
You don’t have to use Rails; it can be any plain old Ruby application. It helps you extract valuable information such as performance metrics and whether exceptions are thrown.
00:03:35.000
It also allows you to identify what features users are actually using, which is beneficial for your marketing team.
00:03:45.760
The second thing I wrote, along with Delin Berry, started as a Rails plugin and was converted into a gem. It's a framework that allows you to do charting in your Ruby or Rails applications.
00:03:58.520
This is a simplified version of our production environment.
00:04:06.879
Now, a couple of questions for everyone here: How many of you are working with more than one programming language in your environment?
00:04:16.720
Almost everybody! Great. How many of you have more than one application framework?
00:04:25.999
More than one application framework per language?
00:04:31.680
Alright, we see we've got different Ruby and Java applications.
00:04:39.600
How many of you are secretly Java developers disguised as Rubyists?
00:04:47.120
Come on, there’s more than that. Don’t be shy!
00:04:52.760
How about .NET developers? Any of those around?
00:05:00.639
Okay, a few. Any C programmers disguised as Rubyists?
00:05:07.480
Another question: How many of you are using more than one database in your environment?
00:05:14.919
Do those databases know about each other? No?
00:05:21.960
Well, in our setup, we have a transactional database with multiple slaves and a data warehouse.
00:05:27.959
This feeds data from our transactional database into the data warehouse where we conduct a lot of reporting.
00:05:36.680
Let’s start with the initial scenario.
00:05:44.600
When Jeremy and I first started at Collective Intellect, we were in startup mode, rapidly throwing code around and refactoring later.
00:06:01.639
You hear a lot about refactoring and patterns, but when you have a business to build and customers to serve, your customers aren't typically your code base.
00:06:11.320
You tend to become a bit sloppy about it.
00:06:19.560
The first version of our code was quite messy. It was unfactored code with a mix of framework components.
00:06:28.200
There was a lot of bad practices, and we can’t criticize this too much because, at the time, it felt like a good approach.
00:06:38.280
We had a single directory filled with a mixed bag of different components: models, utilities, and varying ways to connect to our databases.
00:06:46.800
As we increased our team size, the codebase grew rapidly and became unmaintainable.
00:06:54.560
With millions of records in our database, it became hard to create tests.
00:07:03.440
There was no testing at all, which turned out to be a really big problem.
00:07:11.440
We had models, logging, database plugins, and shared concerns scattered all over the place.
00:07:20.240
We also had SVN externals that pointed to additional locations, which added to deployment nightmares.
00:07:29.640
Worse yet, we were deploying code directly from developer workstations to production.
00:07:36.400
On multiple occasions, code that hadn't even been checked into Subversion ended up in production, leading to issues.
00:07:50.000
This made developers hesitant to deploy, as they were unsure of what was already there. As a result, they resorted to hand-patching production systems.
00:07:59.760
Over time, we realized that this wasn't the way to manage things.
00:08:06.800
We moved on to our second version, where we started breaking things down into Rails plugins.
00:08:15.360
We adopted a plugin directory structure and philosophy to address various concerns more effectively.
00:08:27.440
However, this approach quickly led to challenges as it introduced brittleness.
00:08:34.800
We found ourselves in a situation where we dealt with so many external dependencies that our application code became hard to maintain.
00:08:45.680
We thought dividing things into plugins would simplify our setup, but we learned otherwise.
00:08:54.000
By moving too far into modular approaches, we neglected the encapsulation of our core business logic.
00:09:05.760
In our refactoring process, we realized how essential it is to keep models cohesive, as they represent our business objects.
00:09:13.920
Over time, we learned that encapsulating our domain effectively was critical.
00:09:23.760
Here’s an example of our directory structure for plugins.
00:09:31.960
At this stage, we were only about 20% into our transition.
00:09:37.480
However, we noticed that certain plugins were being used across multiple applications.
00:09:43.960
This led us to think about the one true path in Ruby, which is gems.
00:09:49.920
We started fully embracing gems, which changed how we developed.
00:09:55.760
Moving into our third version, we realized that separating concerns into gems was far more beneficial than plugins.
00:10:04.240
We ended up refactoring our entire codebase into around 20 different gems.
00:10:14.839
Each gem serves its purpose, making our application easier to maintain.
00:10:21.040
The primordial gem became the basis of all our systems.
00:10:29.280
We also created a database interface through the sentiment gem.
00:10:35.840
We managed to isolate different aspects of the application's architecture.
00:10:44.520
One key aspect of this restructuring was the introduction of command line applications as gems.
00:10:51.679
We also made extensive use of Quartz, especially for scheduling jobs.
00:10:58.759
By integrating Quartz, we created a system to monitor various processes.
00:11:05.679
This allows for efficient job scheduling and management.
00:11:12.559
One of the critical components we developed was the primordial gem.
00:11:19.519
This gem handles project generation, managing dependencies, and ensures a consistent project layout for development.
00:11:27.840
This structure was extremely beneficial for new developers.
00:11:36.080
With uniformity, new and existing developers can navigate the code base much more easily.
00:11:43.760
Consistent project layouts also facilitated easier onboarding for new team members, enhancing productivity.
00:11:55.200
The primordial gem provides common tasks to every project ensuring all developers have the same tools at their disposal.
00:12:07.920
Sometimes, familiarity makes it easier to transition between projects.
00:12:15.520
This improved confidence in navigating the code base.
00:12:20.440
It promotes rapid development without fearing the unknowns of a new codebase.
00:12:29.760
This familiarity reduces onboarding time, increasing new developer productivity.
00:12:34.960
Testing became a fundamental part of our process.
00:12:40.760
In our new environments, every developer has their dedicated test database.
00:12:45.760
We're leveraging real objects for our tests, allowing for genuine interactions with the database.
00:12:53.680
This setup simplifies generating tests and quality assurance.
00:13:02.680
The challenge of dependencies led us to a more cohesive strategy.
00:13:15.000
We also implemented a parameterized approach for managing changes.
00:13:21.160
Next, we ensured database configurations remained consistent between development and production environments.
00:13:30.560
This check eliminates surprises during deployment.
00:13:37.600
Now we have tests that validate the configurations are the same.
00:13:43.280
All of our environments align, minimizing potential discrepancies.
00:13:51.840
Moving on to continuous integration, we’ve implemented a system with Cruise Control.
00:14:00.560
This CI system checks our builds continuously, preventing bad code from being deployed.
00:14:06.799
It encourages accountability among team members.
00:14:13.120
The environment works seamlessly with external dependencies.
00:14:21.840
If a gem changes, the system rebuilds all dependent applications.
00:14:28.760
This process has saved countless hours of debugging later.
00:14:37.760
As our team's size increased, managing code became a challenge.
00:14:45.480
Having a team of developers meant more integrations and potential issues.
00:14:53.880
Effective communication and solid structure fostered collaboration.
00:15:01.760
Our team eventually grew to 14 developers.
00:15:09.840
Fluctuations in team dynamics required us to develop an adaptable system.
00:15:17.920
We strive to foster an environment built on communication and teamwork.
00:15:24.320
We focus on continuous integration through feedback loops.
00:15:30.560
Part of our deployment involved building documentation as a phase.
00:15:38.560
Everything is made available through CI, ensuring everyone is on the same page.
00:15:44.640
We encourage effective documentation practices throughout our projects.
00:15:51.760
There's a method known as twiddle-waka, which helps manage gem dependencies.
00:15:59.760
It's important to tighten dependency specifications during deployment.
00:16:03.919
The importance of versioning cannot be overstated.
00:16:12.360
Deployment mechanisms need to be consistent and predictable.
00:16:18.599
We use Capistrano for managing deployments across varied applications.
00:16:25.080
This standardization has increased efficiency and reduced errors.
00:16:32.960
Cruise Control automates our workflow and manages build dependencies.
00:16:40.200
We create a gem server that handles internal and external dependencies.
00:16:48.320
As a result, our deployment process allows us to quickly respond to potential issues.
00:16:55.360
We’ve learned the importance of reliable package management at scale.
00:17:01.760
These measures have led to more consistent, stable deployments.
00:17:09.360
Overall, these experiences have shaped our approach to development.
00:17:16.320
Now, we have effective systems in place for managing our infrastructure.
00:17:22.080
Thank you all for joining us. We're happy to ask any questions!
00:17:30.560
We appreciate you being here and hope to connect with many of you afterward.
00:17:45.600
Thank you once again for your time!
00:17:51.200
(audience applause)