Running a Government Department on Ruby for over 13 Years

by Jeremy Evans

In the presentation 'Running a Government Department on Ruby for over 13 Years' by Jeremy Evans at RubyConf 2018, the speaker shares insights from his extensive experience managing the software development unit at the California State Auditor's office, where Ruby has been the primary programming language. Key points discussed include:

Software Development Approach: Jeremy contrasts traditional government software development practices, characterized by lengthy approval processes and dependency on contractors, with his department's agile approach, focusing on smaller projects developed internally.
Gall's Law: Emphasizing the importance of starting with simple, working systems over complex, unwieldy projects, he cites Gall’s Law as a guiding principle to avoid pitfalls in software development.
Direct Stakeholder Engagement: Developers engage directly with stakeholders to identify essential requirements, which enables the creation of user-friendly systems designed for maintainability.
Project Completion and Testing: The department aims to complete projects in under a month with robust automated testing, ensuring high code coverage and reliability.
Use of Ruby: All software development is conducted in Ruby, which is deemed suitable for government applications due to its ease of learning, quick development capabilities, and maintenance simplicity.
Security Measures: Jeremy discusses the security strategies employed, including handling authentication through a custom Ruby proxy and implementing defense-in-depth approaches to mitigate vulnerabilities.
Web Applications Developed: Several key applications are built, such as an intranet site, a recruiting system, and a recommendation system for audit findings. These applications feature unique workflow capabilities.

In conclusion, Evans advocates for exploring innovative approaches to government IT projects, emphasizing the importance of adopting adaptable frameworks that prioritize stakeholder needs and apply lessons from past experiences. He asserts that Ruby remains an excellent choice for government software development due to its accessibility and fun atmosphere, which helps retain talent amidst lower compensation in the public sector.

00:00:15.529 Good morning, everyone! Thank you for coming out. I'm excited and honored to be presenting here today.

00:00:21.510 Today, I'll be sharing my experiences running the programming unit for a government department for over 15 years, using Ruby almost exclusively for the last 13.

00:00:27.720 My name is Jeremy Evans, and I've been using Ruby since 2004. I'm the maintainer of numerous Ruby libraries, the most popular of which is Sequel, the database toolkit for Ruby.

00:00:35.309 I also maintain a web toolkit for Ruby named Rhoda, as well as an authentication framework built on top of Sequel. I didn't start out wanting to maintain these libraries.

00:00:48.660 The reason I maintain these libraries is that they form the core of many personal projects as well as the applications I'm responsible for at work.

00:00:53.879 I work for the California State Auditor, and our office is located in the capital of California, Sacramento. Our auditors work all over the state, and our mission is to promote efficient and effective government by performing independent evaluations of other government departments.

00:01:09.420 We publish reports of our audit findings and make recommendations to other government departments and the legislature. Our national equivalent in the United States would be the Government Accountability Office.

00:01:18.820 All 50 states have equivalent offices, but the level of independence and exact responsibilities vary from state to state.

00:01:25.679 Before we go any further, please be advised that all opinions expressed in this presentation are my personal opinions and not those of my department.

00:01:31.439 Now, let’s talk about software development as it is typically done at other departments in this state. I will simplify it substantially for the interest of time and exaggerate slightly for comedic effect.

00:01:37.740 I'm not sure how similar this is in other states or countries; I'm guessing there is some overlap. Government software development often revolves around large projects.

00:01:44.520 In many cases, there is an existing system in place, but it has various issues such as running on a mainframe.

00:01:51.540 Executives get the idea of building a modern system and throwing away the old one. After the government decides to build this modern system, before they can even start the project, they will spend a year going through a four-step project approval process with the Department of Technology.

00:01:56.969 And this is one of those times I'm not exaggerating— the median time for this project approval process is over a year.

00:02:02.790 After approval, the government will develop a long request for proposal detailing what they think the requirements are for this system.

00:02:09.720 This request for proposal is prepared by analysts and program managers without the involvement of the internal developers working on the current system. The government will solicit proposals from companies to build the system, and after reviewing proposals and removing those not considered acceptable, the government follows standard purchasing regulations to award the contract to the company with the lowest bid.

00:02:25.599 Payment on the contract is based on a checklist of deliverables, with minimal consideration for how easy the software is to use, and absolutely no consideration for how easy the software will be to maintain.

00:02:39.580 The contractor builds the system using their own developers who are incentivized to check off the deliverables as quickly as possible, usually using C# if they are on a Microsoft stack, or Java otherwise.

00:02:48.930 Near the end of the contract, the software developer will train government staff on how to maintain the system. Minimal, if any, unit or model testing is done during development. Good integration or acceptance testing is usually done, but it is performed manually using checklists.

00:03:02.590 The system contractor government staff or external independent validation and verification vendor does the testing. Sometimes the contractor will base their solution on an existing, expensive enterprise resource planning system such as SAP or PeopleSoft.

00:03:18.369 The contractor will heavily customize the installation to meet system requirements, allowing them to claim that they are just using common off-the-shelf software in their request for proposal, when in reality so much custom work is done that the benefits of using common off-the-shelf software do not apply.

00:03:34.680 There will be a large amount of oversight on this project. Project managers write quarterly progress reports and send them to the Department of Technology, executive management, and an external independent project oversight company.

00:03:48.740 When the project runs into problems during development, these stakeholders will be aware. However, the stakeholders are not experts in software development, and other than approving additional funding or extending the project schedule, there may not be much they can do to fix the problems.

00:04:02.280 Most of us know how well adding developers to a late project actually works. As you might guess, I'm not a fan of this approach to software development.

00:04:16.169 Let me share the alternative approach to software development that we use in our department. This comparison may not be fair since our department is much smaller, but it is what we do.

00:04:35.370 First, we try as much as possible to avoid building large systems. When stakeholders request a large system, we discuss the situation with them and try to convince them to build a smaller system initially that only does what is most important.

00:04:52.750 In some cases, we are successful and in others, we aren’t, but we always try to reduce the scope. If a large system is requested, we can always add more features later after the system is functioning. This approach emphasizes starting with a working, simple system to avoid complexities.

00:05:06.640 The idea that you should start with a simple working system before expanding complexity instead of attempting to build a complex system from the start has been around for decades. John Gall wrote a book called "Systemantics" in 1975 discussing how systems work and fail. A couple of sentences in that book eventually became known as Gall’s Law.

00:05:31.547 Gall’s Law states that a complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and can't be patched up to make it work; you must start over with a working simple system.

00:05:44.460 I know that Gall was not specifically referring to software development; he discussed systems in general, be they computer-based, mechanical, or manual. We keep Gall’s Law in mind when building systems in our department.

00:05:55.750 In most cases, we try to add new functionalities as subsystems or new features to our existing working systems. We aim to complete the initial development of almost all projects in under a month, with most projects finished in about a week.

00:06:10.200 When I say completed, I mean that all features are implemented with complete automated tests. While systems may take longer to go into production due to delays in getting final approval, stakeholders are usually quick to approve projects.

00:06:25.270 However, they can be less quick to put the developed system into production. A typical project involves taking a paper-based process with one or two levels of review and integrating it into our existing intranet site.

00:06:40.460 We develop a forum for employees to complete forms for reviewers to approve or deny requests, including emails to notify the next reviewer and reports for the employee, reviewers, and management to track the status of requests.

00:06:56.050 We build all these systems internally using government staff, without contractors. The developer responsible for building the system meets directly with all stakeholders to determine the system requirements.

00:07:11.270 Often, stakeholders will request features that are not strictly necessary or will question what they think will meet their needs rather than simply stating their actual needs.

00:07:26.800 It is the developer's job to discuss the system requirements with the stakeholders, figure out which features are important, and understand the underlying needs, then try to build the simplest system that meets those needs.

00:07:42.210 The developer knows they will be responsible for maintaining the system in production, motivating them to design the system for easier maintenance. Furthermore, they understand that if users have challenges, they will be the ones responding to their complaints.

00:07:57.200 This encourages developers to design the system to be user-friendly so they don't have to spend excess time making modifications later.

00:08:13.350 As the title of this presentation indicates, all our internal custom development is done in Ruby. Ruby has been our primary development language for new projects since mid-2005, and all our existing internal custom systems were converted to Ruby by 2009.

00:08:27.840 I will discuss the stack we use and how it has evolved over time. We do not have a large amount of oversight during our development process.

00:08:41.360 The only external oversight we receive comes from external security assessments and penetration tests, which are conducted approximately every three years. In terms of internal oversight, developers work on their systems until they believe it is ready.

00:08:58.540 If they encounter problems during development, they typically talk to me, and we often pair program to resolve the issue. After the developer thinks the system is ready, they'll request a code review from me.

00:09:09.030 I'll review all the code and provide a list of requested changes. This process continues until all issues are addressed.

00:09:23.980 Following a clean code review, the developer will notify the stakeholders who will use a development version of the system to verify if it meets their needs.

00:09:38.780 In some cases, developers or stakeholders may request changes at this point, where the developer will further discuss the system requirements, agree on any necessary changes, and this process may repeat until all issues are settled.

00:09:54.460 For any of our systems going to production, we require automated testing at both the model and web levels for all parts of the system. We perform coverage testing on a regular basis, ensuring our line coverage ranges between 93% and 100%, depending on the system.

00:10:10.120 Since we prefer building smaller systems before larger ones, we deal with prioritizing a multitude of incoming requests. One factor we consider is the size of the request.

00:10:25.170 Our aim is to build smaller systems before larger systems, as this encourages stakeholders to be more open to the idea of developing smaller systems.

00:10:39.390 Another consideration is how often the system will be used. If a system automates a manual, paper-based process, we inquire about the volume of forms submitted per month.

00:10:56.000 This may result in prioritizing the system or even informing the requester that due to low volume, automating this process may not make sense.

00:11:07.760 We consider the system's importance to the organization. If the current process is paper-based, we ask what the consequences would be of losing a form.

00:11:19.490 If there are legal implications, we may prioritize automating that system even if it has low volume, just to guarantee we can closely track its progress and comply with the law.

00:11:31.890 Lastly, we consider who is requesting the system. As much as we prefer an egalitarian approach, if an executive requests the system and wants priority, it will receive priority.

00:11:45.160 Now, earlier I mentioned that Ruby is the primary language used for software development, essentially boiling down to the question of what software development we engage in. As you may expect, the main software development revolves around web applications.

00:11:59.290 Our largest application is our intranet site, known as 'The Hub,' where the majority of our development takes place. The Hub contains standard intranet features, providing extensive information about our internal processes.

00:12:16.700 The site includes a comprehensive manual, an employee directory with profile pages that exhibit employee details, such as division, position, and is inclusive of audits they have worked on or are currently working on, along with any awards they have received.

00:12:33.080 The employee's profile page also features a link to a map of their respective work floor, highlighting their desk location, aiding new staff in navigating the office and allowing existing staff to easily find new colleagues.

00:12:50.260 Most new development is centered around automating existing processes. When I started in 2000, processes were just manual and paper-based. Now, most common processes have been automated through online forms.

00:13:06.830 For example, the processes for submitting requests to take time off, attending training conferences, getting reimbursed for overtime, and purchasing supplies/equipment are now automated. Records are kept so employees can monitor the status of their previous requests.

00:13:19.200 Most of these processes have custom review and approval workflows based on different requirements. Some processes, the simpler ones, might only require supervisory review, but others can have between two and five levels of review depending on the specifics of the requests and the employee's position in the department.

00:13:36.040 Another notable web application we develop is our recruiting system, which is divided into two parts: an externally accessible part that integrates with our public website, and an internal system dedicated to Human Resources staff.

00:13:52.040 Most of the employees we hire are entry-level auditors right out of college or graduate school, and the recruiting system is designed to facilitate the recruiting process for these applicants.

00:14:08.320 The recruiting system allows prospective auditors to apply to take our online exam, and after their application is reviewed and approved by our Human Resources staff, they are notified to take the exam.

00:14:24.660 The online exam is timed and consists of 75 multiple-choice questions. Applicants must score around 80% to rank highly enough and advance.

00:14:38.400 The exam is relatively difficult, with approximately only 30% of applicants scoring high enough to advance. Assuming they do score high enough, applicants are then notified they can take our online writing assessment.

00:14:53.579 In the writing assessment, applicants receive a prompt via an upload form, providing them a couple of hours to write and submit their sample.

00:15:09.570 This writing assessment is graded by our Gatehouse Editor, with only one-third of applicants scoring high enough to advance to the next stage, which includes a phone interview followed by an in-house interview.

00:15:23.170 The system manages all information about these stages and handles internal workloads related to processing applications. It also features extensive reporting capabilities.

00:15:39.490 There are smaller subsystems within the recruiting system that manage recruiting for advanced auditing positions. Another significant web application we develop is our recommendation system.

00:15:58.160 In addition to reporting our audit findings, one of our primary functions is to make recommendations to improve government operations. Departments we audit are required to respond regularly to our recommendations about their progress in implementing them.

00:16:14.200 The recommendation system is divided into three parts: the first part is externally accessible and allows other government departments to submit their responses to our recommendations through our website.

00:16:29.000 There is an internal part that allows our staff to add recommendations and to review those submitted by the departments. Each response goes through four levels of review.

00:16:44.750 Once fully reviewed, the department's response, along with our assessment, is posted on our website, ensuring accountability and making sure that the departments’ implementation of recommendations is taken into account during legislative budget reviews.

00:16:58.970 We continue to follow up on these recommendations for up to six years post the release of our audit report.

00:17:14.680 The third part of the recommendation system also has external accessibility, allowing the legislature, the press, and the public to subscribe for notifications about new report releases and responses to recommendations.

00:17:29.500 This system allows subscriptions to be filtered by specific policy areas, providing targeted updates.

00:17:45.200 The last major web application we developed is our public website, which, like our recommendation system, is also split into three distinct parts. Over 99% of the content on our public website is generated by an internal system that caches static pages.

00:18:03.040 To update the content of our public website, we run a web crawler over our internal system that caches all pages, which are then copied to our public web servers via our sync system whenever we want to update content.

00:18:18.340 We have approximately 20,000 pages on our public website, and this approach is feasible for our operations. Having most of our website consist of static pages is a significant advantage from a reliability perspective.

00:18:35.980 In rare instances when we have encountered issues with the dynamic sections of our public site, our users typically don’t notice if they are accessing only the static content.

00:18:51.320 A small dynamic system runs on our public site, addressing various actions mainly pertaining to search features, along with a diminutive internal administrative application used for adding reports to the site.

00:19:04.490 As I mentioned previously, we use Ruby for all development. Besides web applications, I'd like to discuss some non-web applications we develop.

00:19:19.600 To log in to any of our internal web applications, employees are required to use the same Windows username and password they use to log into their computers.

00:19:35.190 We prefer this to avoid employees having to remember separate usernames and passwords. From a security perspective, we do not want our web servers communicating directly with our Windows domain controllers.

00:19:50.460 Thus, all our web applications authenticate by using SSL to connect to a custom authentication proxy written in Ruby. The web applications submit the username and password provided by the user.

00:20:05.920 Our proxy first checks if the user belongs to an allowed usernames list and then connects to Windows using LDAP over SSL to authenticate the password.

00:20:22.780 We also employ Ruby to implement two-factor authentication for our VPN. We use OpenVPN, which does not support native two-factor authentication.

00:20:36.920 After OpenVPN authenticates their certificate, we have OpenVPN call a statically compiled C program that uses a UNIX socket to connect to a server written in Ruby. This Ruby server then connects to Windows using LDAP over SSL to authenticate the password.

00:20:56.080 Our approach is slightly more secure than the standard two-factor authentication method. Typically, the authentication is performed using the username provided by the client.

00:21:14.840 If an attacker can compromise one employee's VPN certificate and another's password, they could potentially gain remote access.

00:21:29.140 In our case, an attacker would need to compromise both the VPN certificate and the password for the same employee, which is hopefully significantly more difficult.

00:21:47.260 We also use Ruby to download financial information from the state’s mainframe daily for our financial auditing team. Previously, this required a gem capable of FTP over SSL.

00:22:02.840 However, after the release of Ruby 2.4, we were able to transition to using Net::FTP from the standard library.

00:22:17.690 We develop custom programs to assist auditors with their audit work. For instance, we recently used Ruby with Capybara, Selenium, and headless Chrome to download nearly 3,000 PDFs from a separate government website written in ASP.NET that requires JavaScript to function.

00:22:33.240 We then used Ruby, combining PDF extraction libraries to obtain a specific page from each of these PDFs and compile all the extracted pages into a single PDF.

00:22:49.300 This allows auditors to review the results easily and incorporate the data into their audit work papers.

00:23:05.300 We also utilize Ruby alongside a Microsoft program called AccessEnum to generate reports on file access permissions, as well as changes in file access permissions over time.

00:23:18.400 These reports are examined annually with management to determine if the permissions are appropriate. In contrast, permissions changes are reviewed monthly to detect any obvious security issues that have arisen.

00:23:32.640 We routinely use Ruby to check the remaining free space on our web servers each month to ascertain if additional space needs to be allocated.

00:23:46.140 For critical file servers prone to exhausting available space, we have a similar Ruby program that automatically alerts the appropriate manager when free space falls below a certain threshold.

00:24:02.540 We employ Ruby to ensure our auditors comply with our data retention policies by scanning for materials related to released audits that staff should no longer retain.

00:24:15.800 We also have many reporting programs using Sequel to connect to internal PostgreSQL and Microsoft SQL Server databases to generate reports.

00:24:30.680 In essence, Ruby serves as our programming toolbox, and we can easily build virtually everything we need with it.

00:24:45.660 How did we begin using Ruby? In 2003, when tasked with maintaining our websites, I inherited them as static pages developed using a product called NetObjects Fusion.

00:25:01.500 Even though I had no prior professional programming experience, I had some exposure to PHP, so I initially opted to use it. In late 2004, I discovered Rails and tried it out.

00:25:18.220 I recognized it as a significant improvement over the spaghetti PHP I was writing previously. After several months of utilizing Rails in personal projects and within work, I transitioned our intranet site to Rails in mid-2005.

00:25:32.350 In 2008, I came across Sinatra and was drawn to its much simpler approach to web development, subsequently using it for all new developments. The initial versions of our recruiting and recommendation systems were both built in Sinatra.

00:25:47.420 In 2014, I was exposed to the routing tree approach taken by Cuba, recognizing how it could address complexity issues we faced in our Sinatra applications, while remaining much simpler than Rails.

00:26:03.460 I ended up creating a fork called Rhoda and transitioned all our applications to it before the release of Rhoda 1.0.

00:26:19.890 On the database side, while using PHP, I relied on raw SQL for all my queries on a database driver. When I first began using Rails, I utilized Active Record, which I found a huge time-saver.

00:26:35.220 After being introduced to Sequel in 2008, I recognized its method chaining approach for building queries as beneficial, converting all our Active Record usage to Sequel that year.

00:26:51.080 We have consistently used PostgreSQL as our database starting from version 7.1. Our operating system has always been OpenBSD, prioritizing security and simplicity over performance.

00:27:07.200 We originally utilized an Apache web server, transitioning through various servers including Lighttpd, CGI, and then to Nginx and Unicorn, which we've been successfully using for several years.

00:27:24.140 For testing, we use MiniTest, MiniTest hooks, Rack test, and Capybara.

00:27:38.680 All of our web applications are designed around a simple Web 1.0 experience. While we utilize some HTML5 features like required inputs, we avoid JavaScript whenever possible—only using it when absolutely necessary.

00:27:52.040 Most frequently, JavaScript is used for input forms that can accept an arbitrary number of line items. We use so little JavaScript that we manually test it whenever we alter related code.

00:28:06.840 All our applications run on the same stack and use the same library versions, simplifying the switching process between applications during development.

00:28:23.510 When we need to upgrade a library, we execute a single command which runs tests for all applications using the new library version, allowing us to check for any issues.

00:28:37.400 This command can also be employed to test across multiple Ruby versions, ensuring all our applications pass our test suites on Ruby 2.3, 2.4, and 2.5.

00:28:53.960 After library upgrades, we may decide to implement new features added to those libraries. Typically, this process starts with our simplest application, then applies to remaining applications in increasing order of complexity.

00:29:09.750 Examples include switching to using frozen Sequel datasets, databases, and models, as well as adopting Refrigerator to freeze all Ruby core classes simultaneously.

00:29:26.640 When using Ruby without Rails, it may limit the number of developers available with experience on your stack. Finding developers with Rails experience may be easier for some companies.

00:29:43.590 However, when we attempted to recruit developers at various experience levels, we explicitly highlighted our Ruby usage but received no applications from candidates with Ruby or Rails experience.

00:29:58.820 Thus, Rails' popularity did not significantly impact our search for candidates. Fortunately, based on our limited experience with Ruby and the Sequel-based web stack, new programmers can learn easily.

00:30:14.640 While extrapolating from a sample size of one may be unwise, our current developer became productive quickly with no prior programming experience and no exposure to Ruby before being hired.

00:30:30.400 Having adopted Ruby, I believe it would have taken her significantly longer to become productive if we had been using Rails.

00:30:45.420 As the information security officer for the department, one specific focus area for me is security. We strive to protect against common vulnerabilities associated with web applications.

00:31:00.660 We mitigate cross-site scripting using Rhoda and Ruby to automatically escape output in our templates, while any cross-site request forgery is addressed by using Rhoda's CSRF plugin.

00:31:15.800 We enforce the use of path and action-specific CSRF tokens for any requests modifying the application state. Many security vulnerabilities within Ruby web applications stem from the introduction of unexpected parameter types via attackers.

00:31:32.210 To prevent these types, we utilize Rhoda’s typecast params plugin alongside Sequel’s type conversion to ensure all accessed parameter inputs are of the correct type.

00:31:49.120 We defend against SQL injection vulnerabilities by ensuring all our queries are constructed using Sequel, avoiding any raw SQL usage during runtime.

00:32:02.350 We apply a restrictive content security policy across all applications to help mitigate potential browser problems should an attacker manage to exploit a cross-site scripting vulnerability.

00:32:15.940 In addition to this, we adopt a defense-in-depth strategy towards our web application security. A majority of our planning starts with the premise that there may be SQL injection or remote code execution vulnerabilities present.

00:32:29.660 We implement features that act to make those vulnerabilities harder to exploit and minimize the potential damage.

00:32:45.790 All applications run under separate operating system users with limited privileges, along with individual database users per application.

00:33:01.920 For public-facing applications like the public components of our recommendation and recruiting systems, these run as distinct operating system users with even fewer privileges.

00:33:18.420 This limited access means that if a vulnerability exists in our public recruiting system, an attacker could exploit it but would not be able to change exam or essay scores since the database user lacks necessary privileges.

00:33:34.040 In like manner, a security vulnerability in our public recommendation system would only enable additions to responses and not modifications.

00:33:48.830 We use security definer database functions in our systems to grant specified types of database access. For instance, password authentication for users in our recruiting system is handled through a database function that checks the password and returns a match for the stored hash.

00:34:05.480 This database user cannot access the password hash directly, preventing any exported attempts to perform offline attacks on them.

00:34:19.840 We also utilize security definer database functions in tests when setting up database state, enabling fully transactional testing even when different database users are employed.

00:34:34.900 All our applications run on our own hardware within isolated subnets, featuring strict ingress and egress firewall rules that limit allowed types of connections to and from the servers.

00:34:49.650 Per operating system firewall rules permit our internal applications to connect to our custom authentication proxy, while external applications cannot access it.

00:35:04.900 To prevent arbitrary file access and remote code execution vulnerabilities, all applications are configured with CHROOT.

00:35:20.760 Our applications start as root and afterward, just prior to accepting connections, they drop privileges to their respective application-specific operating system user.

00:35:34.570 In conjunction with restricting access to application folders, we utilize file access permissions to ensure attackers cannot read sensitive configuration files containing secrets or other critical information.

00:35:48.240 CHROOT is an effective security feature, but it presents challenges when used with Ruby. The main problems involve its shortcomings with runtime requires and Ruby's autoload feature.

00:36:02.010 The use of autoload has been largely discouraged by Matz for several years, but popular Ruby libraries like Rack and Mail still utilize it, complicating their use in chroot environments.

00:36:15.000 To further hamper blind return-oriented programming attacks predicated on consistent memory layouts, each Unicorn worker process is executed after forking. This ensures that every Unicorn worker process has a unique memory layout.

00:36:31.520 While this will incur a substantial memory cost, we are currently operating all our applications on a server with 256 gigabytes of RAM, using around 4 gigabytes for all applications.

00:36:46.460 Thus, the additional security measures are worth the extra memory costs. Finally, we aim to complicate exploitation in general and reduce the attack surface for privilege escalation by limiting the set of kernel system calls.

00:37:01.979 Applications' web processes are not permitted to fork, exec, or send signals to any other processes. If a process does not need to accept uploaded files, it is also restricted from modifying or creating any files.

00:37:15.920 Collectively, these security controls raise the difficulty level for successful exploitation and make it more challenging to use a successful exploit to attack other applications and systems.

00:37:32.680 Certainly, some trade-offs exist. Certain restrictions complicate development and introduce challenges with specific libraries, necessitating workarounds, and a few Ruby libraries needed adjustments to facilitate their operation in chroot and fork plus exec environments.

00:37:50.140 Therefore, after utilizing these defense-in-depth security features in production for the past 18 months, here are my recommendations on whether to consider implementing them.

00:38:06.880 Assess the value of the data that requires protection. Consider how sensitive or confidential the information is you deal with. If only working with anonymized or public datasets, these data may not require implementation of the security controls.

00:38:23.080 Examine who can access your application. Should it be exclusively for internal organizational use and not accessible from the internet, the risks diminish; in this case, it may be more beneficial to enhance user experience.

00:38:35.570 Reflect on how much control you have over your system's environment. If operating on your own hardware or virtual machines, you are likely to utilize most or all of these security features.

00:38:49.020 If relying on a platform-as-a-service provider, your options will be limited to what the provider supports. If your application is internet-accessible and contains sensitive or confidential data, my recommendation is to first consider utilizing multiple database users with restricted permissions.

00:39:05.160 Utilize database functions to limit potential exposure from SQL injection vulnerabilities. If feasible, configuring firewall rules is advisable.

00:39:19.130 The initial implementation is generally straightforward, and ongoing maintenance costs are low. If you have spare memory and the augmented usage won't pose problems, consider adopting fork plus exec.

00:39:32.860 When your application or server has special access to other applications or systems not publicly accessible, where security is prioritized over maintenance ease, consider implementing chroot and/or system call filtering.

00:39:45.920 In conclusion, I believe that successful government IT projects necessitate executive management willing to explore new approaches. As Grace Hopper, one of COBOL's creators noted, the most dangerous phrase in our language is 'We've always done it this way.'

00:40:01.920 Given the history of government IT projects often being delivered late, over-budget, and riddled with bugs, if they work at all, new development approaches are worth exploring.

00:40:14.860 Next, project managers need an in-depth understanding of the technology employed, so they can identify system challenges promptly and resolve them.

00:40:27.800 Developers should be directly discussing system requirements with stakeholders while aiming to minimize project scope.

00:40:39.280 Developers should also take responsibility for maintaining the system in production, encouraging designs that facilitate ease of maintenance and usability.

00:40:57.720 Finally, all systems should commence with the development of the simplest possible system that addresses the most pressing needs, ensuring that it functions effectively before incrementally adding features.

00:41:11.440 This approach can prevent running afoul of Gall’s Law, which points to the risks in attempting to create a complex system from the outset.

00:41:32.480 From personal experience, I can attest that Ruby is an excellent fit for building government systems, at least those where runtime performance is not critical.

00:41:47.640 Ruby facilitates quick development towards functional systems. If you have robust tests that encompass most of your system's functionalities, maintaining Ruby web applications becomes significantly simpler.

00:42:04.540 However, a lot of this depends on your selection of Ruby libraries. My experience indicates that external requirements often necessitate more code adjustments than internal refactoring.

00:42:20.300 Thus, the limited ability to automate refactoring within Ruby is not a critical issue. Ruby is easy to learn and to teach to new programmers.

00:42:33.080 This fact is especially relevant for government work, as I have observed that applicants for government programming positions typically lack significant programming experience.

00:42:47.920 Importantly, Ruby retains the fun aspect of programming, which is often overlooked. Given government work's lower compensation, using a fun programming language can help retain talented staff.

00:43:00.640 After being responsible for all programming within a government department for the last 15 years, I can confidently state Ruby is a superb choice for application development.

00:43:13.600 I highly recommend it.

00:43:24.620 Thank you for listening, and if you have any questions, I have 45 seconds.