00:00:18.240
Hi everyone! We're here to talk about Living Social. We build a consumer marketing platform that you can visit at livingsocial.com. At first glance, it may appear to be just a deal site and seem small and boring, but underneath, there is a phenomenal amount of work being done. Most of our applications are written in Ruby on Rails, but we also have teams working in Clojure and Scala. I almost mentioned Smalltalk, but that would be inaccurate. We have native iOS and Android applications, as well as big data operations. There's a wealth of interesting and complex problems we face, and today, we're going to present a few lightning talks on these technical challenges and our approach to operating as a distributed development team.
00:01:02.399
The first speaker will be Ed Weng, a graduate of our Hungry Academy training program. We essentially took about 30 random people from the streets. Just to clarify, they applied voluntarily; we didn't press gang them! They went through an intensive six-month training program, and they have all become amazing developers. Ed contributes to our mobile consumer applications. If you see him later, don't hesitate to ask him about his pen pal.
00:02:06.479
Hey guys! My name is Ed Weng, and today I will talk about sharing templates in a service-oriented architecture. If you've never been to our site before, if you visit livingsocial.com, this is the page you'll land on. To give you some context about our architecture, about two and a half to three years ago, we were still operating under a monolithic Rails app. Everything was housed within this gigantic Rails app. However, since then, we have been working diligently to break things down into smaller microservices.
00:02:31.840
For example, this page is part of an app that we call 'Browse,' which is responsible for providing the browsing experience to our customers. As you can see, you can view various deals on our site. If you click on, let's say, the Audible deal, you will be directed to a page that appears differently. Most of our users might not realize it, but they've actually transitioned from the Browse app to another app we call 'Sponsors.' This app is responsible for servicing sponsored deals, such as the one from Audible.com, which is why it’s free.
00:03:17.760
If the user then decides to click on the 'Shop' tab, they are taken to yet another different app called 'Products.' This app is responsible for servicing physical goods that we sell to our customers. If we look back at the last three slides that I just showed you and I ask you what is the one common element that appears on all three slides, you might say it’s the navbar, and you’d be correct.
00:03:54.720
One of the insights we've gained from splitting things into smaller services is that while certain tasks may seem straightforward in a monolithic Rails app, they can become quite complex in a service-oriented architecture. The big question we need to address today is: when it comes to sharing views, how do we keep ourselves DRY (Don't Repeat Yourself)? In a monolithic Rails app, using partials and layouts is standard practice. For example, to display the navbar on every page, we would typically include it in a partial called navbar and render it in the application layout, ensuring it appears everywhere.
00:04:44.959
This method works well in a monolithic Rails app, but once we transitioned to a service-oriented architecture, this approach began to break down. The first issue is the increase in copy-and-paste coding. Whenever there's a need for a change, the navbar must be copied and pasted into multiple locations. Another drawback is that making changes requires updates in various places, which slows down development. If you need to tweak the navbar, you might end up making changes in five, ten, or even fifteen different services, and as a result, the navbar can quickly become inconsistent, leading to a poor user experience.
00:05:15.440
One might consider placing everything inside an engine or a gem. This can be a much better solution because it eliminates the repetitive copy-and-paste coding discussed earlier. Once everything is stored in a centralized repository, any necessary adjustments can be made by simply updating the gem. Additionally, using an engine or a gem allows for sharing not only the markup but also translation files, images, JavaScript, stylesheets, and so on.
00:06:23.920
However, the problem arises again when, after an update, you still have to go to each individual app and issue a 'bundle update.' This approach might work fine if you have just a couple of apps, but as the number of apps grows to five, ten, or more, it becomes cumbersome.
00:06:31.400
At LivingSocial, we have a saying: if services aren't solving all of your problems, you’re probably not using enough services. So, to tackle our issue, we built another service known as 'Stepford,' designed by Eric Brody. Stepford is responsible for sharing our views across our service-oriented architecture.
00:07:14.720
During the initial development of Stepford, we adopted a Domain-Driven Design (DDD) approach, considering what we really wanted the interaction with the service to look like. Essentially, we wanted the client app to ask Stepford, 'Send me the footer.' Stepford would then process this request and return a package.
00:07:47.280
Let’s break down this process into three steps. The first step is the request: 'Send me the footer.' We settled on a simple GET request to a JSON endpoint. This request is made to the packages controller in the Stepford app, where we specify the elements we want within the query string. For instance, we're requesting the footer, and we can also pass additional parameters such as the city ID if we need the footer to be customized or personalized.
00:09:11.360
So, what exactly is inside a package? A package consists of three components: markup, styles, and scripts. Everything revolves around the markup being requested. For instance, if you're asking for the footer, an associated set of styles and scripts comes along with it. If the styles aren’t delivered, the footer may look unattractive, and if the scripts aren't included, it may not function as intended.
00:09:49.920
Thus, we've consolidated all three components and refer to this concept as a package. Next, after Stepford receives a request, it goes through a more complex process behind the scenes, which I’ll explain in a simplified manner. When the GET request reaches the package's endpoint, Stepford attempts to create the package by collecting the markup, styles, and scripts. It even tries to pre-compile styles, pre-compile scripts, and then store everything in the database.
00:10:40.480
An interesting feature is how Stepford renders the markup on the server side. It takes the ERB template stored within Stepford and renders it into raw HTML, which is then sent back via JSON. Let’s take a look at what a package response looks like. This response includes the three components we discussed: styles, scripts, and markup. Typically, when an app requests just the footer, it may also request a navbar or button styles, etc. The advantage of using Stepford is that it aggregates and packages all necessary styles and scripts for every markup element you've requested, resulting in one pre-compiled stylesheet and one pre-compiled JavaScript bundle.
00:11:37.439
Now, let’s briefly consider the pros and cons of implementing a service like Stepford. The primary advantage is that it keeps your views DRY. With all your views centralized within a single repository, this results in substantial benefits in terms of maintenance and ease of use.
00:12:01.760
Updating elements, like the navbar or footer, becomes a simple process. You won’t have to notify multiple applications about changes. Stepford returns the latest version of each element directly from the database with every request.
00:12:46.640
However, there are some drawbacks. The first con is the introduction of another service. At LivingSocial, we’ve learned that with the addition of each service, we trade application complexity for network complexity. This means that our tech operations team must manage one more application, ensuring it remains available; otherwise, styles will fail to render.
00:13:39.200
Additionally, because Stepford is another service, it introduces a new network request which can increase latency within our application. To mitigate this, we implemented extensive caching, such as Varnish caching around Stepford, as well as deploying memcached. The rationale behind this approach is that many of our main UI elements do not change frequently, allowing us to safely place caching layers in front of the service.
00:14:32.320
Moreover, it’s important to note that changes can have a much larger impact in this architecture. While it’s easy to release changes, it can be too easy, leading to unforeseen consequences, especially when modifying shared components like JavaScript. As a result, close monitoring of JavaScript is essential to detect errors effectively.
00:15:10.080
That concludes my talk on Stepford. If you have any questions, please feel free to come and talk to me afterwards.
00:15:49.840
Thanks, Ed.
00:15:50.240
Next up, Tyler, someone whom you get to watch as he uses a computer over my shoulder, is also working on our consumer apps team. He hails from Southern California— which I mistakenly said earlier, but he's still handsome— and he's taken. According to his website, he wants to learn everything. If you see him in the hallway, just tell him anything; it will make his day.
00:16:20.960
Howdy! I'm Tyler, and I live in Northern California while working remotely. Today, I'm here to discuss the internal gem infrastructure. Essentially, I’ll cover how we streamline the process of writing and releasing gems. Sharing code effectively across applications enables a conducive work environment, allowing for seamless extraction of functions into our services.
00:16:57.440
Working on shared code through gems significantly reduces the rampant copy-paste scenarios we’ve encountered before. When we set up little services, people frequently copy client code into multiple locations, making it hard to track who’s hitting what endpoints. Establishing a culture around building and sharing code in gems has really helped us streamline our operations.
00:17:43.680
Every gem gets its own repository and README file, making it easy for anyone to add contributions while simplifying code management. You get all the advantages of pull requests, which enhances our workflow by helping to distinguish varying concerns. However, one of the bigger challenges has been figuring out where to house all these tiny objects of code.
00:18:05.840
The solution of utilizing gems helps establish boundaries around your code. This practice lets me isolate the interaction between my code and Rails, how I access the database, and even logging when I work in isolated environments. This isolation is beneficial for code deletion later as well; tracking becomes much simpler, and you won’t find yourself digging through old commits.
00:18:37.040
A practical example: a few weeks back, I was tasked with automating a daily CSV report from our database to upload it to a third-party FTP site. I noticed that some code would be dying under the weight of being buried in a model; the code would accomplish the job, but I knew I would soon forget its purpose as time passed.
00:19:16.080
Thus, I created a gem. Moving forward, I will quickly run through the process of building and releasing gems while also covering how to host them internally.
00:19:59.440
Building a gem is straightforward. If you have Bundler installed, you can simply execute `bundle gem your_gem_name` to set it up. The command creates a basic gem template including a gemspec, a license (typically MIT), a README, and a lib directory containing your program code. The most crucial file is the gemspec itself, as it contains all metadata and dependencies. Bundler does a good job of filling in defaults.
00:20:32.760
For namespacing, it's beneficial to create a short representation of your company name within a module structure. This approach has been useful as it clarifies where your code originates. It's crucial, especially to distinguish between open-source implementations and internal projects. Here’s a quick example of setting up and structuring a gem appropriately.
00:21:11.200
Versioning following semantic versioning principles is essential. You can refer to semver.org for more details, but generally, the last number signifies bug fixes, the middle number indicates new features, and the first number denotes breaking changes. A stable gem should have a major version of 1.0, while unstable gems should regard a version below that as indicative of its reliability.
00:21:55.920
Once your gem is built and you aim to release it internally, this can often be a challenging step. There are several important tasks such as remembering to tag version numbers, building the gem with rake, and deploying it without accidentally open-sourcing it.
00:22:31.360
We’ve rewritten a gem that inherits bundler's gem helper to ensure that when we release, it doesn't push to rubygems but rather to our own gem server, Geminabox. We configure the tasks in our rake file to reflect this.
00:23:01.920
As for gem servers, there are several options available. If you host your own, it should remain behind a firewall for internal code. Geminabox offers a user-friendly web interface for code management, requiring some authentication setup. Moreover, it has mirroring capabilities for rubygems, ensuring uninterrupted access to dependencies.
00:23:45.600
Another option for an open-source gem server is Stickler. It features customized mirroring capabilities as well as comprehensive command-line support. For hosted servers not behind firewalls, Gemfury.com is an option, but it doesn’t provide mirroring.
00:24:27.200
To summarize, the easy process of ensuring your gems are hosted internally involves creating a solid setup around naming and versioning while maintaining a repository for your gems within your organization.
00:25:07.680
Next up is Dan Mayer, also a member of our consumer app team. Dan actively engages in teaching Ruby in the DC area and has a unique skill for fixing bugs over the phone while atop ski lifts.
00:25:48.560
Hello! I’m going to discuss the analysis of production code. As services and applications grow, debugging becomes considerably more complex. A smaller application is easier to comprehend mentally, but as the system scales, it becomes challenging to keep track of everything.
00:26:36.880
To improve things in a meaningful way, one cannot enhance what cannot be measured. While we often focus on performance and exception monitoring, there are significant insights to be gleaned from understanding what your code does while it’s running in production. Ruby tools are still improving in this area, so there’s plenty of work ongoing to enhance these tools.
00:27:14.639
I want to acknowledge the good work from Etsy, who have some informative posts regarding measuring production systems. By graphing data and analyzing release statistics, we can tremendously improve our understanding of what’s happening.
00:27:54.560
As we separate our codebases, we work diligently to eliminate unnecessary code. Dead code often occupies production environments for various reasons, prompted by teams growing or neglecting old code over time. While we might not delve deep into every possible reason, if you've worked on a team, you’ve undoubtedly stumbled upon code written long ago that’s no longer in use.
00:28:38.800
There are numerous approaches to identify dead code. Simple tools like New Relic can help by showing transactions and requests over a timeframe, enabling us to identify potentially useless code.
00:29:22.639
Through StatsD and performance instrumentation, we’ve successfully deleted over 20,000 lines of application code. If you include all assets, test files, and scripts, that number may climb into the hundreds of thousands. Regaining control over what you’re running improves the whole system.
00:30:06.560
With StatsD, you can trace background jobs, measuring their execution time and success rate. For example, we build a wrapper around our background jobs using StatsD, allowing us to measure performance efficiently.
00:30:55.360
In terms of email templates, transaction emails can become one-off instances that may no longer be triggered. Reviewing email codes ensures we maintain relevance across our applications, especially during significant updates.
00:31:39.600
Your view layer can become needlessly complex with various partials. Active Support Notifications can help you track which partials remain in use and which ones have been refactored entirely. We’ve incorporated a gem called FlatFoot to simplify this tracking process.
00:32:26.960
If you uncover code that you need more clarity on, StatsD can also serve as your one-off tracker. This allows you to assess which routes or formats are still active, leading to potential code cleanup.
00:33:10.160
The final discussion revolves around tracking translation usage. As your application expands with various internationalization efforts, translation files can inflate with unnecessary keys that consume memory. You can leverage performance metrics to monitor which translations are actively used and which can be removed.
00:34:06.240
An experimental project we run involves monitoring production. Unfortunately, I have to navigate around bugs in Ruby affecting coverage at production. We deploy the following setup with sampling, and we use it to check our app paths while efficiently tracing it through Redis and logging features.
00:34:41.919
In terms of logs, consolidating all log data from various applications in one place, such as Elasticsearch or Splunk, is paramount for understanding request flows. I've worked on a gem called 'Imprint,' and there's a potential use in Zipkin as well to enable tracking across systems.
00:35:10.080
These tools help maintain cleaner code while providing insights into what's actually happening during production. That concludes my segment.
00:35:59.200
Thank you, Dan.
00:36:00.400
Next up, Rodrigo, a developer on our Merchant team, is both a board game and video game enthusiast from Brazil! If you catch him at the mall, don't forget to ask him about his goats.
00:36:45.360
Hello everyone, my name is Rodrigo. I’m here to share one trick to boost your productivity while working from home.
00:37:01.440
You can find me online, and I wrote an article for our tech blog detailing how I manage my focus at home. One major point of interest mentioned by readers was my pet goats featured in the article.
00:38:06.320
To illustrate my workflow, I’ve prepared some visuals. I've been working remotely for nearly a decade, and it was essential for me to identify methods to maintain focus while also resisting the temptation to simply engage in video games all day.
00:38:34.560
One way I accomplish this is having a structured startup ritual. Each workday begins when I get presentable, preparing myself for video calls, which includes making coffee. It’s critical to eliminate distractions; hence, I personally rely on a good pair of headphones. They signal to those around me, like my wife, that I’m in work mode.
00:39:14.159
Now that I’ve established my workspace, I strive to minimize multitasking. I prevent myself from opening new browser tabs for social media or non-work-related activities while working. For example, using an Arduino light system allows me to establish what messages require immediate responses.
00:39:48.640
Finally, maintaining a balanced approach is essential—working without breaks can lead to burnout. Spending time with others, pursuing new knowledge, or even engaging in activities like cooking can enhance your work-life equilibrium.
00:40:37.520
That's all I have to share with you today! Thank you very much!
00:41:15.120
Thank you!