Delivering Autonomous Rails Apps behind Corporate Firewalls

by Nick Merwin

In this session titled "Delivering Autonomous Rails Apps behind Corporate Firewalls," Nick Merwin, founder of Coveralls, discusses strategies for deploying Software as a Service (SaaS) applications within corporate environments, particularly for enterprise clients. The primary motivation for such deployments relates to security, reliability, and cost considerations. The key points covered in the presentation include:

By the end of the session, the attendees are encouraged to consider how these insights can help them deploy their applications securely behind corporate firewalls, unlocking a previously inaccessible market segment.

00:00:12.480 Thanks for coming out late Thursday, everyone. It's been a long couple of days. I'm Nick Merwin, a founder of Coveralls. I hope you guys have seen some of our badges around town—GitHub, ReadMe, etc. We're kind of all over the place, hopefully right now.

00:00:28.400 Today, we're going to discuss how I approached getting Coveralls, our Software as a Service (SaaS) app, into the hands of customers who wanted to run it within their corporate networks. This was pretty uncharted territory for me, but I hope this general overview, along with a basic prototype that we'll see later, will help you gain some confidence to also take your app to the next level by offering a private version to a customer base that may have seemed impossible before. This discussion will partly revolve around business models and general sysadmin topics, all viewed through the lens of Rails, since Coveralls is a Rails app.

00:01:10.600 So, why would you want to set your app free into the outside world, beyond your secure deployment environment? Let's say your app is currently hosted on Heroku or a VPS, like Coveralls is on Digital Ocean. If you have a subscription-based service where users are paying monthly, and maybe there are a few tiers and usage-based add-ons, and it’s humming along, accruing users, it might seem like your potential customer base is completely covered.

00:01:40.159 However, you might be neglecting an even more lucrative and preferable customer base—enterprise clients. These could be large companies where development teams face difficulties in getting upper management to embrace cloud-based tools, or perhaps the issue is that setting up a $5 monthly service fee necessitates an extensive amount of vendor approval and paperwork.

00:02:05.119 So, let's chat about how offering a hosted version of your app, in addition to your cloud service, can make sense. I came across three key reasons for wanting to pursue this: first is security. It's clear that we all worry about using cloud apps since we essentially give up control over our data. We hope that the developers on the other side use bcrypt for our passwords, but we never really know for sure and just hope there will never be a real data breach.

00:02:36.120 Beyond passwords, any sort of data that a company’s security department has forbidden from leaving their networks effectively precludes the possibility of them ever using your app. For instance, a medical company might want to use your data mining tool but cannot upload anything to the cloud due to HIPAA compliance issues. Or perhaps it's a development team that wants to use your source code analysis tools but cannot risk sharing or leaking proprietary code.

00:03:00.360 Additionally, there are general business apps like CRMs, internal task management, chatting, and scheduling that contain plenty of information those companies wouldn't want to share with competitors. By running a hosted version of your app within their networks, they can be more confident that they won’t ever have their accounts hacked or their data compromised on the open internet.

00:03:42.799 The second reason is reliability. This is a worst-case scenario consideration. If the app you're offering is a DevOps tool that's part of the build or deployment pipeline, teams won’t want any downtime or unplanned maintenance interruptions. For example, with services like GitHub or Travis CI, if something goes down, you do not have control over it, which could disrupt your deployment process. By managing it internally, your development teams can coordinate to ensure everything runs smoothly during crunch time.

00:04:30.360 Lastly, we have cost. The traditional subscription model of simply paying monthly doesn't work once you transition your app to a customer. Often, these potential customers are less price-sensitive and might be willing to pay a premium for your hosted service, along with some extra white-glove support. However, once you deliver the app to them, you must assume it won't be able to 'phone home' to check the subscription status.

00:05:12.520 Instead of a time-based subscription like a monthly fee, you’ll likely need to implement some sort of seat-based usage or another model beyond just monthly or yearly billing. When considering seating, you can create seat packs as tiers based on the number of active users accessing the app within their network. Additionally, you might consider building a self-destruct or shutdown mechanism that forces the customer to purchase or renew the license after a set period, such as six months or a year, to ensure they keep a valid license.

00:05:43.519 With that said, there is considerable opportunity to address business model challenges beyond the traditional SaaS model. Before diving into how we figured this out, let me briefly discuss what Coveralls does and why we aimed for the enterprise market. Coveralls is a tool for tracking code coverage and providing notifications. It annotates pull requests with success or failure statuses based on changes in coverage percentage and sends messages to chat or email with updates.

00:06:19.060 This functionality helps development teams ensure they do not deliver untested code, especially in production. In the on-site version, you can see line-by-line coverage of the codebase. The cloud version for open-source projects is free, but we charge for private repositories.

00:06:49.600 The challenge arises because while we don’t store source code on our servers, we do store private scoped OAuth tokens for GitHub, and for some companies, that's a complete dealbreaker. They cannot allow us to have access to their private repositories. Additionally, some companies are utilizing GitHub Enterprise and are not using cloud GitHub to begin with, making interoperability a challenge.

00:07:19.760 To explore this market, we gathered interest from our potential customer base over several months. It became apparent that there was sufficient interest to pursue this initiative, which ultimately brought us to today.

00:07:48.200 Now, let's discuss the significant hurdles I encountered in converting a general cloud Rails app to one that operates within corporate firewalls, often referred to as 'the fog'. The first hurdle we need to tackle is delivery and installation.

00:08:04.680 The best user experience for your customers would be to minimize the amount of work required to get your app running in their environment. I believe that the most consolidated method to achieve this involves three downloads: the virtual machine that the app will run on, the packaged application files themselves, and a license file specific to each customer.

00:09:18.680 The virtual machine could just be a Linux box, typically between 800 MB and 1 GB in size. The application package can vary considerably, sometimes between 50 MB to 100 MB or more, depending on how many gems need to be bundled and vended into the package.

00:09:46.399 Lastly, the license file would be generated uniquely for each customer. From those three files, customers will probably only need to download the virtual machine once in a while, such as when you implement significant upgrades or new dependencies. The application package is something they would download whenever updates occur, and license files would be relatively quick to download during trials or when converting to active status.

00:10:17.960 Furthermore, let’s talk about those incremental updates. Normally, when your code is hosted on Heroku, deploying features and fixes is simple. However, when your app is running in the wild with no access to it, the app packages come into play.

00:10:51.160 These packages contain only the most up-to-date bug fixes, features, etc. This smaller download is much quicker than having to redownload and reconfigure an entire virtual machine. Regarding the license files, it makes sense to build a secondary app in addition to your main cloud app to service those customers and manage subscription fees, trial accounts, and so forth.

00:11:24.360 Installation-wise, the first thing customers will interact with is the networking configuration. Since you wouldn't want them to log into the VM themselves, the optimal solution is to create a tiny, menu-driven Ruby app presented on boot.

00:11:42.360 This way, customers won’t need to know the specifics about Linux networking. Also, assume no external access will be possible. If they're purchasing the hosted version, it is for the confidence of security behind it; they wouldn’t want any external access to it, ensuring it remains entirely sandboxed within their network.

00:12:12.040 Next up is process management. You need a strategy to manage background jobs and ensure that the server remains alive and starts up on boot.

00:12:43.480 Another hurdle relates to product support. Typically, if a user encounters a 500 error, you immediately know about it via email or monitoring services. However, in this scenario, if something goes wrong, you have no visibility. You will only receive an email from the customer reporting an issue.

00:13:12.760 So, you must provide them a way to send you details about the exception, storing those exceptions in a way that can be sent over, either in an email attachment or another method to keep track of them.

00:13:45.760 Log management is another crucial aspect. If users experience issues, you want to be informed about any unusual parameters passing through the routes, especially if they aren’t resulting in exceptions.

00:14:14.080 Lastly, you must consider resource management. If it seems like performance is lacking, you will need a way to address that. This is more of a system admin topic, so I won’t delve any further here, but it's definitely something to keep in mind.

00:14:56.240 Now, let's address the intellectual property concerns. This is a big one because you're providing them a VM, which they can unpack, mount the disk, and review everything included. Although it comes packaged into one file, it is extractable, and users can investigate it as if they were a root user.

00:15:21.399 Understanding that database access is difficult to secure, it’s wise to assume that companies will have access to the database. Furthermore, there’s no thorough method to protect your source code once it is running in somebody else’s environment. You can write a code obfuscator as a deterrent—but ultimately, there’s no foolproof way.

00:15:45.720 Instead, consider covering yourself with your licensing agreement. State in the license that the software is for use only, and modifications and redistribution are not permitted. It all comes down to legal concerns being your last line of defense. If worries about this keep you awake at night, remember that your customers should primarily be interested in receiving updates, bug fixes, new features, and support rather than attempting to breach your source code.

00:16:39.600 Now, let’s delve into the nitty-gritty of the architecture we use. This diagram, while complex, shows the general structure of how the app runs within our VM, which is Ubuntu. We chose Ubuntu due to its widespread usage and ease of configuration. Key components within the VM include a network config app that customers see when they boot it up.

00:17:27.720 This app allows users to set static networking or DHCP, manage name servers, and perform actions like rebooting or shutting down the VM. Behind this web server, we'll explain our choice of Passenger, housing a pre-installed app designed to handle package file extraction containing the actual application and its updates.

00:18:13.080 Within this architecture, the Rails app will also be running. The license file reader will load and cache the license upon the Rails app booting and check it on every page load. Actions like enabling and disabling features or locking down the app can be achieved based on this check.

00:18:41.360 Data import and export is another significant feature. When customers need to retrieve data from the old virtual machine to the new, the app requires a way to facilitate this transfer. This includes dumping the database and potentially aggregating user-uploaded files. Lastly, we need a support package generation and download functionality.

00:19:24.000 We can use Rails' rescue feature to collect exceptions occurring in controllers, archiving them into a temporary directory. Admins can generate an archived file when they need support that can be encrypted and emailed or Dropboxed to us.

00:20:05.520 Setting up the environment during development is crucial. I recommend using VirtualBox, as it is free and straightforward. Exporting an appliance from your machine as an OVA file makes it easy to link to your CDN.

00:20:51.960 Given Ubuntu's reliable built-in security settings and standards, we can trust it grants our customers that confidence from the outset. When provisioning for your Rails app, it’s essential to use the minimum dependencies necessary to keep the download size small.

00:21:19.160 While many installations quickly balloon to 2 GB or more, you can keep it down as small as 1 GB by avoiding unnecessary installations. In the development phase, it also helps to have two separate versions of the virtual machine: one for packaging the app and the other for local development that can assist with platform-specific issues.

00:22:10.120 Next, let's discuss the networking configuration. A customer interacts with a simple menu-driven Ruby app upon their first load of the VM, assisting them in gathering crucial information from system calls. This module permits user interaction without requiring complete blocking.

00:22:50.520 I chose Passenger for managing the server because it allows your app just by placing it in a predefined directory and starting to serve it. The setup page assists users in uploading the packaged and license files, which will then be extracted for future integration.

00:23:30.440 This installation package handles placing both the apps in their designated folders and preparing them for future access by the user. The encryption library utilized to decode data is effectively an added layer of protection over the package.

00:24:33.440 The user experience is meant to be simple, assisting customers in the transition from the old version to the updated package without excessive complexity. If the package and license files are present, they will be decrypted, cached as the app operates, and applied instantly.

00:25:28.160 We also discussed how to handle the license file within our framework. The license file contains unique data for each customer and checks against every page load. Any new information related to Rails secrets should likewise be included within the license file for every boot, preventing cross-customer interference.

00:26:19.760 A straightforward licensing module demonstrates how to read and write the license file, using the same encryption key as earlier indicated. This module is loaded at the initialization stage, ensuring seamless access throughout the app, a crucial aspect of compliance and security management.

00:27:09.480 Support package management allows us to track user interactions of exceptions via an encrypted file for easier access when customer support is needed. It’s vital to securely track backtrace and error reports that facilitate prompt customer service responses.

00:27:51.920 Data import and export management comprises critical functions of PG restore and PG dump commands. These functions enable full database backups and integration with the client’s updated packages, representing important features of maintaining operational transparency and support.

00:28:26.440 When it comes to background job management, I won’t delve too deeply, but we use 'foreman' and 'export' to maintain running processes tied to Ubuntu, increasing the efficiency of server operations. The next logical step is creating a dashboard within the app to provide users with real-time updates improving their overall operational awareness.

00:29:31.600 When all these pieces are in place, finding the best way to make them distributable is key. Streamlining to a single script for deployment could simplify user experiences significantly. This strategy entails linking updates to a CDN, indicating a new version for customers to download—essentially making updates user-friendly.

00:30:25.760 The process of creating this streamlined version is straightforward: execute a rake task that examines the version, runs 'bundle vendor' to prepare vendor directory packages, and compresses relevant files while excluding unneeded components. The task then seamlessly encrypts this structured update with the shared key before uploading it to the CDN and notifying users.

00:31:31.920 By managing each download's version effectively, your users will have a seamless way of identifying their updates and making the transition a stress-free process.

00:32:01.680 Upon successfully landing in customers' hands, if they are already using services like AWS, they might want to see it as an Amazon image. This conversion can typically be done quite easily through available command-line tools, making the process accessible for all clients.

00:32:42.920 As your package evolves, consider including further enhancements like resource management for admins to gauge VM performance. Introducing clustering may improve performance for clients using multiple VMs simultaneously. The need for a specified mail server will also arise, as traditional cloud sources won't be applicable under the VM setup.

00:33:15.640 Discuss incremental VM updates that require bash scripts for package evolution within this framework. All changes should consider the lack of internet access in the VM.

00:34:00.440 While I mentioned the enterprise side as somewhat of a joke, it's essential to recognize the scope for uncharted territory in that market.

00:34:40.080 As we approach the conclusion of this session, let's use the remaining time for any questions you may have! The question was raised about our approach toward AISC, which remains a work in progress as we navigate using a Ruby C extension similar to GitHub’s compiled Ruby.

00:35:19.360 In replying to how we aim to minimize the updates required to ship, while it is complex, it boils down to customer relations management. Bug fixes will certainly need to be bundled and delivered swiftly where necessary.

00:35:59.960 If bugs are non-critical, they can be saved for a major point release. If it critically affects the business, a new version needs to get released immediately so that everyone impacted is updated accordingly.

00:36:41.720 When discussing scalability, our support team hasn’t yet needed to scale significantly due to a manageable customer load. We anticipate that a review system will also need adjustment when supporting various package versions and handling incoming inquiries, particularly those requiring detailed responses.

00:37:53.920 Ultimately, we’re here to support our clients effectively and efficiently. There are two prototype apps to demonstrate our features—the first, an elementary blog app, utilizes license reading—while the second is our app loader.

00:38:40.760 Thank you all for attending today, and I hope this session has provided you with valuable insights and ideas for deploying your own applications securely behind corporate firewalls.