Running a Government Department on a Roda Sequel Stack

Talks

Jeremy Evans

#software-development

Running a Government Department on a Roda Sequel Stack

by Jeremy Evans

In the video titled "Running a Government Department on a Roda Sequel Stack," Jeremy Evans discusses his experience with using Ruby, Sequel, and Roda to develop and manage software for a small government department. The California State Auditor's office, where Evans works, transitioned to Ruby for all custom software development starting in 2005. He emphasizes the choice to build software internally with government staff rather than outsourcing to contractors.

Key Points Discussed:
- Introduction to Ruby, Sequel, and Roda: Jeremy Evans has been using Ruby since 2004. By 2009, all internal systems were transitioned to Ruby, and Sequel became their primary method for SQL database access.
- Applications Developed: The department’s notable web applications include:
- The Hub: An intranet site featuring employee profiles, information on internal processes, and automated features replacing manual tasks.
- Recruiting System: A system managing recruitment processes, including applications, tests, and interviews for hiring entry-level auditors.
- Recommendation System: This application holds departments accountable for implementing recommendations made in audit reports, with public visibility into responses.
- Transition from Rails to Roda: Evans narrates the evolution of their software stack from Rails to Roda, explaining how they started with Rails in 2005, later adopting Sinatra, and eventually creating Roda to address complexities in Sinatra. The transition resulted in enhanced speed and maintainability of applications.
- Benefits of Sequel and Roda: Sequel and Roda have facilitated simpler and faster development, with less code to manage database interactions compared to Rails. The systems also support higher productivity due to their ease of learning.
- Key Achievements: With over 100,000 lines of code managed effectively by one and a half full-time positions, the efficiency of development and maintenance has been significantly improved.

Conclusions and Takeaways: Jeremy encourages other developers to try Sequel and Roda, highlighting that even programmers with no prior experience could become productive quickly with these tools. The transition has led to faster application performance and has streamlined development workflows, confirming their suitability for the department's needs.

00:00:14.460 Bonjour! Today, I'll be speaking about my experiences using Sequel and Roda to run a small government department, as well as what led us to adopt Ruby and these libraries.

00:00:20.740 My name is Jeremy Evans, and I have been using Ruby since 2004. I work for the California State Auditor. Our office is in Sacramento, California, in the United States. We perform independent evaluations of other government departments and produce reports with recommendations to improve government operations.

00:00:26.200 In France, our department's software development approach differs from many other government departments. We build all of our custom software internally using government staff, without relying on contractors. All of our internal custom software development is done in Ruby.

00:00:39.580 Ruby has been our primary development language for new projects since 2005, and by 2009, all of our existing internal custom systems were converted to Ruby.

00:00:52.410 Consequently, all of our SQL database access uses Sequel. We use PostgreSQL for our custom software development and occasionally interface with Microsoft SQL Server. We've also had to access other database libraries or databases, such as Oracle and Microsoft Access, and thankfully, Sequel ships with support for all the databases that we have needed to use.

00:01:09.820 Most of our software development is delivered as web applications, and for those, we have used Roda since 2014. Let me discuss some of the web applications that we develop.

00:01:30.700 Our largest application is our intranet site called The Hub, and the majority of our development focuses on adding new features to The Hub. The Hub contains many standard intranet features, providing a lot of information on our internal processes, a comprehensive manual, and an employee directory.

00:01:47.660 Each employee has a profile page that displays information about their division, position, audits they've worked on, audits they're currently involved in, and the awards they've received. The employee profile page also includes a link to a map of the floor they work on, with their desk location highlighted, making it easy for new staff to navigate the office and for existing staff to find new colleagues.

00:02:20.390 Most of the new development revolves around automating existing processes. When I started working for the department back in 2000, almost all processes were manual, paper-based tasks. Now, most common processes are automated via online forms.

00:02:46.570 For example, we automate the submission of requests to take time off, attend training, get reimbursed for overtime, purchase supplies, and equipment, among many others. Employees can see the status of all their previous requests through automated record-keeping.

00:03:06.360 Most of the processes now have custom review and approval workflows based on different requirements. Some processes only require a supervisory review, while others may need between two and five levels of review, depending on the request and the employee's position.

00:03:34.150 Another web application that we develop is our recruiting system, which is split into two parts: a publicly accessible part for applicants and an internal system used by our Human Resources staff. Most of the employees we hire are entry-level auditors directly out of university, and the recruiting system is designed to handle the recruitment process for these applicants.

00:04:54.560 The recruiting system allows prospective employees to apply and take our online exam. After applying, our Human Resources staff review the application and, if approved, the applicant is notified that they can proceed to take our online exam. The online exam is timed and consists of 75 multiple-choice questions. To rank high enough to advance, applicants must score about 80 percent.

00:06:09.391 Assuming an applicant scores high enough, they are notified they can take our online writing assessment. During the writing assessment, the applicant is given a prompt along with an upload form, and they have two hours to submit a writing sample similar in style to our audit reports. Our staff grades this writing assessment.

00:07:05.539 If the applicant passes the writing assessment, they undergo a phone interview followed by an in-house interview. The system handles all the related real information and workflows required for processing these applications, and it offers extensive reporting capabilities. There are smaller subsystems within the recruiting system that handle the recruitment process for more advanced auditing positions.

00:08:54.280 Another major web application we develop is our recommendation system. One of our department's primary functions is to make recommendations to improve government operations. The departments we audit are required to respond regularly regarding their progress in implementing our recommendations until they are fully implemented.

00:09:05.190 The recommendation system is divided into three parts: the first part is externally accessible, allowing other government departments to respond to our recommendations. The second part is internal, enabling our audit staff to add recommendations and review responses. Each response goes through four levels of review.

00:09:43.169 After thorough review, the department's response and our assessment of it are posted on our website. This process holds departments accountable, and their implementation of our recommendations is often considered during legislative budget reviews.

00:10:12.689 We continue to follow up on the recommendations made for up to six years after the release of our audit report. The third part of the recommendation system is externally accessible, allowing the legislature, the public, and the press to subscribe and be notified about new report releases and responses to our recommendations.

00:11:03.299 Additionally, the system enables users to subscribe to receive notifications filtered to specific policy areas of interest. Now, how did we start using Ruby and end up using Sequel and Roda? I was first tasked with maintaining our websites back in 2003, which were initially developed as static pages using NetObjects Fusion.

00:12:47.250 While I had no professional programming experience, I had some exposure to PHP and decided to use it. I cannot recall the exact PHP code we used, but it included a library for shared code that established the database connection and stored it in a variable.

00:13:14.689 All SQL statements were written manually, requiring values to be escaped and interpolated into the SQL statements. The data returned from the database was accessed by indexing into an array instead of by column name.

00:14:08.170 In 2004, I learned about Rails, and after trying it out, I discovered that it was an improvement over what I had been doing with PHP. It made SQL creation much easier, as it handled most of it for me.

00:14:40.500 After several months of using Rails for personal projects and work, I transitioned our intranet site from PHP to Rails in the summer of 2005, making my work notably easier. After being exposed to Sequel in 2008, I recognized the benefits of its method chaining approach to building queries.

00:15:56.780 I converted all of our Active Record usage to Sequel that year, and we have been using Sequel exclusively since then. Here's an example of the Active Record code we used before our upgrade.

00:16:22.370 Back then, Active Record did not support case-insensitive searching through its public API, so we had to write database-specific SQL manually.

00:16:39.870 Sequel, on the other hand, had built-in support for case-insensitive searching, making it significantly easier to handle queries. With Sequel, we no longer needed to modify a conditions array for filtering.

00:17:20.840 After our switch to Sequel in 2008, we needed significantly less code to interact with our database, and as Sequel became more powerful, we found we no longer needed to write raw SQL or SQL fragments at runtime.

00:18:02.590 In 2009, I was exposed to Sinatra, which offered a much simpler approach to web development. We began using Sinatra for all new development, and the initial versions of our recruiting and recommendation systems were written in Sinatra.

00:19:00.040 In Sinatra, routes are specified directly with a block to handle the route with the return value of the block being the response body. I found Sinatra less complex than Rails while still handling the needs of our recruiting and recommendation systems.

00:20:27.690 However, another issue with Sinatra was its lack of support for handling emails, as it has no equivalent to Action Mailer in Rails. The email handling within our Sinatra applications was often ad hoc and messy.

00:20:59.890 In 2014, I became acquainted with the routing approach used by Cuba. This approach addressed some complexity issues we faced in our Sinatra applications while remaining simpler than Rails.

00:21:61.919 With Cuba, routes are handled in a nested fashion, assisting in reducing code duplication while maintaining localized route handling. Although I appreciated the nested routing of Cuba, I found some of Sinatra's behaviors, such as blocks returning the response body, to be friendlier.

00:22:34.570 This ultimately led me to create a fork of Cuba called Roda, integrating ideas from both Sinatra and Sequel. In the summer of 2014, I converted all our applications to Roda, and once Roda's mailer plugin was released, we found email handling significantly simpler.

00:23:36.210 After upgrading, we noticed that our web applications performed noticeably faster, especially during tests with the same set of Capybara tests, which ran twice as fast using Roda compared to Rails.

00:24:21.020 Now, with Roda, all our applications use the same set of libraries and versions, making it easier to switch between applications during development. The plug-in architecture of Sequel and Roda facilitates what I call complexity scaling, allowing you to start simple and add complexity as necessary.

00:25:30.190 Our department has now been using Sequel for over 11 years and Roda for over five years, and the combination has made web development productive and enjoyable.

00:25:59.799 Here are some main advantages we've experienced using Sequel and Roda: they tend to be easier to understand and therefore easier to learn. I'm currently the manager of the department's IT unit, and I no longer handle the day-to-day aspects of programming on our web applications.

00:26:25.870 The day-to-day updates and addition of features are handled by another programmer who had no professional programming experience before joining our team. She quickly became productive and was able to implement new features using Sequel and Roda without a hitch.

00:27:38.830 Had we been using Rails, it would have likely taken her much longer to become productive. Applications converted from Rails to Sequel and Roda have proven to be easier to maintain.

00:28:10.460 With few exceptions, major version updates of Sequel and Roda have required minimal backwards compatibility changes and have been less cumbersome compared to similar updates in Rails.

00:29:20.920 This is partly because Sequel and Roda offer plugins that allow new behaviors to be implemented before they become defaults, enabling more gradual updates.

00:29:52.690 Our applications utilizing Sequel and Roda contain over 100,000 lines of Ruby code, and we effectively maintain and add new features using the equivalent of one and a half full-time positions.

00:31:10.910 I find that remarkably efficient.

00:31:28.530 We have found that Sequel and Roda are significantly faster than Rails in terms of both request handling speed and development time.

00:31:51.300 If you haven't tried Sequel and Roda yet, I encourage you to experiment with them and see if you experience the same benefits.

00:32:09.700 This concludes my presentation. I'd like to thank all of you for listening.

00:32:42.040 I have about five minutes for questions, so please feel free to ask.

00:32:54.550 Merci beaucoup!

00:33:02.040 Thank you for that.

00:33:05.000 How long did it take you to convert the Rails apps to Roda?

00:33:09.300 Let's see. I believe the conversion of the Rails app that we maintained happened between the releases of Roda 0.9 and 1.0. That is, our applications were converted before then.

00:34:11.040 I would say it probably took a week, maybe two. The application was the largest one, probably having around 40,000 lines of code at the time. A lot of it wasn't too difficult due to the way we structured our Rails applications.

00:34:41.850 It was fairly easy to take each method from Rails and convert it into a block in Roda to handle requests. The actual transition was more mechanical than difficult. It could have been done by hand, but most of the changes were straightforward.

00:35:44.020 First, I want to thank you for those two amazing libraries. I always encourage people to try them, and I'm curious: how do you use Sequel? Is it as a low-level database access library or with the Active Record pattern?

00:36:09.060 In most cases, we use two different approaches. When modifying a single row, Active Record works well. You retrieve the database value and update it with the new value, which makes sense for many reporting systems.

00:36:55.880 However, for larger datasets, we often utilize Sequel in its low-level form where we deal with hashes. Each row in the database is represented as a hash. Much of the design of Sequel focuses on these primitives like hashes and arrays which makes it easy for Ruby users to understand.

00:37:45.330 Are you using ActiveSupport in your libraries?

00:38:36.800 No, I don't use ActiveSupport in any of my libraries. Even when my libraries don't utilize it, there can be bugs reported because of changes in ActiveSupport that affect the API.

00:39:34.780 Thank you for the presentation and the libraries. The problem I face sometimes regarding dependencies relates to Active Record, making transitions of applications nearly impossible. It seems that the trade-off of the libraries comes with a learning curve that might not be beneficial for newcomers.

00:40:36.390 That's true! In general, when using Sequel and Roda, you're creating a trade-off due to fewer available resources compared to Active Record. However, once you understand the libraries like Roda, it can make things a lot more manageable.

00:41:43.500 Any other questions? Thank you very much.

Paris.rb Conf 2020