A 4-pack of Big Lightning Talks

Big Ruby 2014

Chris Morris

@morriswchris

#developer-tooling

#automation

#internationalization-i18n

#ruby-on-rails

#software-development

#testing

#payment-processing

A 4-pack of Big Lightning Talks

by Chris Morris

In his lightning talk at the Big Ruby 2014 event, Chris Morris, a senior engineer at LivingSocial, shares insights into the importance of developing tools for oneself as a developer. He emphasizes building personal automation solutions rather than relying solely on existing tools for end users.

Key Points Discussed:
- Introduction and Background: Morris highlights his experience with LivingSocial's payment systems, which handle millions of API calls daily. He mentions the challenges faced while managing production support under strict PCI regulations.

- Building Developer Tools: He shares his journey of creating various console tools to simplify repetitive tasks. Examples include:
- Using Pry: He enhanced the Ruby development experience by integrating Pry, improving data readability through features like pretty-printing and history retention.
- Batch Classes: Morris created batch classes that simplify handling arrays returned from Rails, using method missing magic to enhance code readability and save time.

- Text-Table Gem: He implemented a gem for formatting outputs into clean, tabular formats.

- Email Automation: Morris developed a method to send quick emails via Mailgun directly, bypassing the complexities of ActionMailer.

- Troubleshooting Tools: He built functionality for Braintree that assists with troubleshooting payment systems, allowing efficient comparisons and resolution of issues.

- Log Analysis Scripts: Utilizing shell scripting to grep through logs proved effective in organized data management without access to traditional log aggregators.
- Fixtures and Mocks: He reflects on challenges with standard libraries like FactoryGirl and Mocha, advocating for clarity and simplicity in managing test data and mocks.
- Internationalization Tool: Morris discusses the development of Humperdink, a tool for tracking translation keys in real-time, which is crucial for managing unused translations in a large application.
- Coverband: He mentions the open-source code coverage tool that helps in tracking active parts of code in production, guiding developer focus during enhancements.
- Final Thoughts: Chris wraps up by drawing an analogy between software development and continual refinement, encouraging developers to create personal tools to enhance their productivity.

Main Takeaways:

- Building custom tools can significantly simplify developer workflows and enhance efficiency.
- Continuous iteration and improvement can lead to valuable solutions.

- Morris encourages developers to explore creating their own automation rather than relying on existing off-the-shelf solutions.

00:00:12.000 Hello everyone! I want to give a quick shout-out before I begin.

00:00:20.320 Is this too loud or too hot? Thank you, Glenn, for the feedback.

00:00:27.840 Last year, Joe Kuttner came to Big Ruby and did a presentation on his book, Healthy Programmer. I wanted to say thank you to him because, despite only following a few of the suggestions in the book, I've managed to lose about 20 pounds and I no longer feel sleepy in the afternoons. If you sit in a chair all day long, I definitely recommend checking this book out.

00:00:45.920 My name is Chris Morris, and I'm a senior engineer at LivingSocial. Over the past year, we've faced a bit of bad press, so I thought it would be a good idea to start with some positive news. This is a quote from our CEO, Tim O'Shaughnessy, from an article in CNN Money highlighting all the great things happening at LivingSocial. Best of all, we are hiring!

00:00:52.879 Brian Owens, our VP of Engineering, is here, and Glenn and Adam will be speaking tomorrow. Greg Vaughn is also part of our team, so if you're interested in a new position or if you want a change from your current job, please come talk to us!

00:01:14.400 As I mentioned last year, I did two lightning talks. I really like that format, and I didn't expect they would invite me back, but here I am, ready to share! My first topic today is to encourage you, as developers, to build tools for yourself, not just for your end-users. I'm often surprised to see not only myself but also other developers performing repetitive manual tasks instead of creating automation.

00:01:41.360 For some background: last summer, I took over as tech lead for our payment systems, which primarily handle credit card transactions, PayPal, and a few other payment methods. Our core payment system processes approximately a million and a half API calls daily, with credit card transactions reaching up to 150k. The system is designed to handle significant sales days we've experienced in the past. It's a decent-sized system—fairly mature in terms of codebase.

00:02:11.040 When I began overseeing this area, my main focus was on production support, particularly due to PCI regulations which restrict access to production consoles for most engineers. That meant a lot of lookup tasks and assisting with unusual cases. Today, I want to share the evolution of some console tools I've developed. They may not be particularly impressive, but I hope they inspire you to consider creating solutions for your own development challenges.

00:02:41.360 Let's start with an example. Here's the standard Rails output. Don't worry, Ryan, this isn't production data. It can be somewhat difficult to interpret, so I iterated on a few different approaches and managed to simplify the output. One of the first changes I made was to incorporate Pry. Pry is an interactive shell for Ruby that can significantly enhance your development experience.

00:03:19.440 With Pry, you can have features like history retention and pretty-printing for arrays of hashes, making the data much easier to read. Additionally, it comes with a rich toolset. Once you start using Pry as your console, you can type `help` to explore its functionalities.

00:03:58.560 Another improvement I made was creating a batch class that simplifies the way I handle arrays returned from Rails. Instead of typing out the mapping blocks repeatedly—which can become tedious—I implemented method missing magic that allows me to access attributes more elegantly. This not only saves time but also enhances the readability of my code.

00:04:37.040 I also added named constructors to my batch class. This was useful since finder methods often lack auto-completion, making it harder for developers to work efficiently. With this, I can easily call attributes and have them passed through the entire batch without confusion.

00:05:09.120 A more obvious enhancement I made was integrating a text-table gem. This feature allows us to format the output cleanly without fussing over column justification. We can select the columns we want and generate a nicely formatted output. I leveraged Active Record's serialization methods to obtain the necessary column names, ultimately resulting in tabular output that was far cleaner and more user-friendly.

00:05:58.560 As I continued refining this, I realized there were still extraneous details visible in the output. The solution was to override `pretty_inspect` when using Pry, enabling me to focus solely on the tabulated data.

00:06:39.440 Another frequent scenario in my work is that I often need to send out emails when I am researching something for someone. Rather than going through ActionMailer and its complexities, I created a way to send quick email messages using Mailgun directly.

00:07:08.480 I've also developed similar batch functionality for Braintree, our credit card gateway. This has been invaluable for troubleshooting when we need to compare our payment systems against Braintree’s systems or even repair transactions following brief outages.

00:07:24.960 Over time, by investing in and iterating on these tools, I've been able to return to them and either use them in production to solve problems or create new endpoints for our API, allowing clients of our payment systems to benefit as well.

00:07:53.760 Now, I've also been working on a Thor command line script, especially since we didn't initially have access to Splunk or any log aggregators. Instead, I utilized shell scripting to grep through our environment, enabling me to collect and analyze diagnostic information effectively.

00:08:23.760 This process allows me to track down logs across our machines and process them more effectively because, at times, our outputs can be overwhelming. I built a system that organizes these data into manageable chunks that can then be analyzed or repurposed as needed.

00:08:51.840 The point I want to make is really about encouraging you to do the same and building custom solutions for your challenges. I know that while building automation, sometimes it feels like I’m working on niche tools. However, by iterating and releasing often, you can create substantial tools from small steps that benefit your production environment.

00:09:14.080 That wraps up part one of my talk. Please hold your applause until all the lightning talks have concluded!

00:09:29.840 Moving into my next topic, I want to share insights from a presentation of the enhancements we made, which have gained some accolades.

00:09:40.960 I gave this talk during a job interview for a role at Relevance. To be fair to JB, what he commented on was indeed correct.

00:09:58.880 A bit of background: LivingSocial acquired a company in the Middle East where, if you want to conduct business, you need to acknowledge that a substantial percentage of the population prefers to use cash. This has been a challenge for us as we primarily shipped products but, in order to collect cash, we had to create a system.

00:10:40.000 Glenn Vandenberg and I worked together on a project to address this need, which he named Codmonger—a humorous title indeed. In establishing this system, we incorporated FactoryGirl to facilitate the creation of test data.

00:11:10.640 Here's a compressed view of our FactoryGirl fixtures. We developed a structure where customers have addresses and purchases, with each purchase associated with a state machine to manage its various stages.

00:11:40.000 We quickly got this into production, as only a few internal staff were utilizing it at that stage. I started writing RSpec tests to ensure everything was functioning properly. However, I encountered issues where the collections did not have the necessary dependencies to work correctly.

00:12:16.000 After some frustration with FactoryGirl, I decided to switch to Machinist. However, I ran into similar issues, which led me to reconsider the components of the creation process for these elements.

00:12:51.680 The key takeaway for me was that, although FactoryGirl and Machinist use DSL to instantiate ActiveRecord objects, they often handled this process in ways that created additional complications. Simplifying this with direct code, while perhaps less elegant, provided clearer pathways to satisfactory function.

00:13:26.880 While bothFactoryGirl and Machinist abstracted some complexity, the ActiveRecord API is challenging enough on its own without additional layers of abstraction complicating it further.

00:14:05.600 Let’s move on to discussing mocks and fixtures—this section is humorously subtitled 'Not All Expectations Were Satisfied.' I want to highlight an instance where using Mocha didn't provide the clarity I needed after encountering non-useful error messages.

00:14:31.920 I learned that when our Purchase ActiveRecord has too many fields, even if Mocha is functioning correctly, it can still become cumbersome to manage and lead to poor readability in error messages.

00:15:32.080 Another example I can share is utilizing struct or open struct for certain data objects in less complex cases can create a clearer codebase. In many circumstances, it becomes simpler to define a method on a class instance to maintain behavior, especially with small test cases.

00:16:11.040 Overall, when weighing whether to build a custom solution or buy tooling, it's essential to understand that when selecting off-the-shelf products, you give up some familiarity and transparency.

00:16:55.840 Now, regarding tracking and managing internationalization, I have undertaken this as well while working on our main app that handles complex tasks around product offerings.

00:17:32.000 This particular main Rails app is robust, with approximate numbers indicating we had around 2500 translation keys. However, through static analysis, we've confirmed that only about half were actually in use at any time.

00:18:17.040 To address this, we built a tool called Humperdink that allows real-time tracking of keys that are actively being utilized in the application. We were particularly cautious of performance impacts when implementing this solution.

00:18:47.680 Working on these real-time tools poses challenges but provides immense value, especially given the overhead that can come from large, unused translation keys cluttering our application.

00:19:42.560 The tool not only allows us to manage translations effectively but also streamlines our internationalization processes, which is pivotal when exploring new markets.

00:20:26.240 Now, let’s talk about Coverband, an open-source tool developed by Dan Mayer that allows us to track code coverage in production. Providing insights into which parts of the code are running can guide our efforts during development.

00:21:06.840 To conclude, I’d like to share that software development is comparable to creating analogies. We continually design and model our realities, refining and optimizing as we go.

00:21:51.760 Moving forward, I want to emphasize that all these enhancements and tools point towards one crucial takeaway: if you are interested in joining our team at LivingSocial, please do talk to us!

00:22:11.920 Thank you for your attention, and please remember, we're hiring!

See Slides on www.slideshare.net

Big Ruby 2014