Reconstructing Taiwan’s Internet-Based Party Politics

DevOps

Mu-Fan Teng

1 talk

#content-management-system-cms

Reconstructing Taiwan’s Internet-Based Party Politics

by Mu-Fan Teng

In this presentation titled "Reconstructing Taiwan’s Internet-Based Party Politics," speaker Mu-Fan Teng discusses the significant overhaul of the digital infrastructure for the New Power Party (MPP) in Taiwan. Established in 2015, MPP utilizes an internet-based system for various internal operations, such as party membership, donations, and online elections. After years of development, the original system encountered various inefficiencies, leading to a decision to completely rebuild the platform using Ruby 3 and Rails 7. Teng outlines the challenges faced with the legacy system and the methodologies employed to develop a new, efficient version.

Key points of the presentation include:
- Introduction to New Power Party (MPP): Founded in 2015, MPP leverages online tools for membership applications, donation processing, internal elections, and a content management system.
- Legacy System Issues: The original website suffered from accumulated technical debt, lack of testing, poorly documented features, frequent data errors, and over-engineered functionalities that hampered usability.
- Rationale for Redesign: Collaboration between MPP staff and Teng’s team led to the decision to overhaul the system due to growing complications with managing the bug-ridden legacy code.
- Project Goals and Methodologies: The redevelopment focused on normalizing the database schema, enhancing security, and ensuring over 80% unit test coverage. The adoption of DevOps practices through continuous integration and deployment was emphasized to streamline development.
- Election System Development: A significant aspect covered was the creation of a new online election system that improved verification processes for voters, transitioned to encrypted data handling, and automated various aspects of elections, reducing reliance on manual tasks.
- Data Migration Strategy: The presentation discussed the implementation of a dedicated migration tool to effectively transition data from the old schema to the new system while ensuring a smooth launch without data loss.

Teng also highlighted the methods adopted for DevOps integration, including the establishment of CI/CD systems that automated coding checks and deploy workflows. The integration of Docker within the development process further improved the project’s deployment efficiency.

In conclusion, the session emphasized the importance of modernizing political party operations through technology, showcasing how strategic planning and execution in efficiently managing legacy systems lead to improved functionality and reliability in party politics. This case study illustrates the utility of software development practices in an innovative political landscape, advocating for a future where political engagement can flourish through robust online platforms.

00:00:04.279 Hello, everyone! Thank you very much to the organizing team for giving me this opportunity to speak here. I'm very glad to meet you all. It takes nine hours to travel from Taiwan to Australia, and this is also my first time in Australia, as well as speaking outside of Japan and Taiwan.

00:30:679 My name is Mu-Fan Teng, and I have been a Ruby enthusiast for 17 years. I started participating in Ruby communities in 2008 and have hosted Ruby community events in Taiwan for more than a decade. You can find me on Twitter and GitHub. My company, Five Time Lby, was established a decade ago, and we are dedicated to providing professional technical advice and services to businesses and startups worldwide, utilizing Ruby and various technologies.

00:01:12.439 Before starting my presentation, I would like to introduce the main activities of the Ruby Taiwan community. RubyKaigi Taiwan is the conference for the Ruby programming language in Taiwan. It has been held ten times since 2010, attracting speakers and attendees from all over the world, similar to this event. It's also one of Matt's most recommended Ruby language conferences outside of Japan. Since 2012, Matt has been invited as a keynote speaker for each edition. Last December, we held the first physical RubyKaigi Taiwan after the pandemic, and we were honored to invite Matt and Tim as keynote speakers. Unfortunately, Matt was injured a week before the conference and could not attend in person. However, we still attracted nearly 200 attendees and had 28 wonderful speakers from over ten different countries.

00:02:11.800 We have a monthly sharing and networking event called Ruby Jam at P office in Taiwan. If you happen to be in Taiwan, you are welcome to join us. Although RubyKaigi Taiwan will not be held this year, we will continue the tradition with the community agenda track at Coast Cup. Coast Cup, like FOSDEM in Europe, is an extensive open-source conference where over 40 communities gather each time, allowing each one to host its own agenda. This year's event is set for August 3rd and 4th at the National Taiwan University of Science and Technology in Taipei. The call for proposals is currently open, and we would be delighted to see all your proposals for the Ruby track.

00:03:00.440 To start the agenda, I will briefly introduce how the political party, New Power Party, has been using the internet for its internal political operations since its establishment in 2015. Next, I will discuss the state of the system after seven years of development, including the accumulation of technical debt and the reasons why I chose to redevelop and refactor it. Finally, I will discuss the improvements and challenges in terms of complexity, security, and agility before and after the upgrade.

00:03:36.919 The New Power Party, abbreviated as MPP, is a political party established in 2015. As of this year, it has been eight years since the general election, and the party holds a seat in the Parliament, making it one of the major political parties in Taiwan. Apart from its political agenda, MPP has been an internet-based political party since its establishment. The MPP website system mainly provides five major online functions: membership application, due payment and validity period calculation, donation, internal elections, and a content management system for party members and non-member accounts. Most party members join the party and pay the membership fee online or in person at any branch office. The party joining requests require an identity review by the staff of the central party headquarters before officially becoming a member. If the review does not pass, the fee will be refunded.

00:05:01.000 Members who pay annual fees, like most online subscription services, need to renew their fees once a year. It’s important to note that it's not an automatic process. Annual fee members must pay the membership fee for the year before the end of June or December, depending on the joining date. If the payment is not made after the deadline, the membership qualification will become expired and can be recovered within two years by the MPP. Otherwise, it will be considered leaving the party, and the membership cannot be restored through a payback. Regarding donations, there are three types accepted on the MPP website. The one-time donation allows supporters to make a single contribution of any amount, while the premium rewards contributors with exclusive party-branded items. Lastly, the monthly recurring donation establishes a steady flow of support.

00:06:03.919 Technically speaking, one-time and premium donations are processed the same way in terms of payment flow, while recurring donations are a unique feature of MPP compared to other parties in Taiwan. One of the most significant aspects is its implementation of a completely online election system, enabling the selection of its most important decision-making and legislative entities. The Decision-Making Committee functions similarly to a parliamentary cabinet, with its chairperson elected internally. The party representative committee is responsible for legislative tasks, and both bodies have a term of two years.

00:07:03.000 Let's dive into the internal election process. The official announcement marks the start of the election. From this point forward, being a valid party member who has completed a certain period of membership becomes a requirement for voting eligibility. Members with expired status, even if they pay the party dues after the election starts, will not be eligible to vote. During the candidate registration period, individuals can register as candidates through the system. Once the period closes, the election committee will conduct a qualification review and announce the results. In the question period, members can use the election questioning system to pose public questions to the candidates. One day before the voting starts, ballots are sent to the email address of all eligible party members, followed by several days designated as the voting period.

00:08:33.640 After voting concludes, the system calculates the results and publishes them to all party members. The entire election process was originally completed by volunteers using Ruby on Rails on the first day of the party's establishment. After that, the party hired an employee with a background in IT to continue development. Eventually, in 2021, my company took over. The party suffered from a lack of effective software project management, leading to numerous features being developed with little or no subsequent use or only being utilized once. Certain functionalities were so obscure that even the most senior party staff were unclear about their original purpose. Some features were over-engineered, incorporating too many unnecessary possibilities.

00:11:39.760 One of the issues with the legacy version was the exceptions caused by data errors happening very frequently. Due to the lack of error tracking, these exceptions were often only reported by users, making it time-consuming for staff to troubleshoot. Additionally, the database migration of this project could not proceed smoothly. I took over the maintenance of this project in 2021, initially focusing on front-end adjustments and infrastructure maintenance. However, by the end of 2021, amendments to the party charter regarding the method of calculating overdue membership fees led to numerous issues, resulting in time-consuming remediation and data adjustments.

00:13:08.680 Consequently, in the third quarter of 2022, the party staff decided to initiate a project rebuild, which formally began in October. The main goals of the new version's improvement included normalizing the database schema to ensure data consistency and efficiency. It was crucial to retain only the essential functions related to election functionality. As the project progressed, it was necessary to isolate this feature in new system development. This approach allows compatibility between the old and new versions while simultaneously enhancing security and reliability.

00:14:47.759 We aimed for over 80% unit test coverage, ensuring all significant functionalities were backed by automated tests. This step was vital for maintaining code quality. Our DevOps practices also saw improvements with the implementation of continuous integration and deployment automation, streamlining the development process and enhancing productivity. Finally, integrating data from the legacy version into the new system was a critical goal that allowed us to preserve historical data while moving forward with a more robust and efficient system.

00:15:57.560 The schema improvement involved separating general user data from party member data, including membership information and fee records. This separation allows member-related functions to exist strictly within the membership-related models, avoiding interference with non-member users. In the user and membership models, I used AASM to reduce code coupling and complexity. In the original user model, most state changes related to party members, such as confirming payments and handling approvals, were written in a single confusing method, making it very difficult to maintain. In the new version, I added the AASM state machine, allowing for clearer and more maintainable state change operations.

00:17:39.400 For example, when a membership fee is confirmed as successfully paid, the state changes from pending to paid through the payment event, which in turn triggers an operation to extend the validity period for the party member. Reviewing the state-related code, it is evident that the number of lines has been greatly reduced, allowing for better compliance with code quality checks. One of the major issues with the legacy version of the election functions was that many tasks had to be completed manually, and not all operations could be executed just by following the documentation. Additionally, the voting mechanism required no sign-in; voting could be completed merely by using a token in the URL, meaning the voter's identity could not be verified. Finally, the voting data was not encrypted, so if the database was compromised, the voting information would also be leaked.

00:19:23.840 The election function was originally part of the website. However, during the development of the new version, internal elections were scheduled to take place. Due to political issues, an early decision was needed for the supplementary election of the decision-making committee. Consequently, the party requested us to expedite the work, and we developed an independent election website within one month. This site needed to be compatible with both the legacy and new versions, requiring database synchronization.

00:20:35.160 To establish seamless sign-in, we installed and configured the Doorkeeper gem on both codebases, facilitating single sign-on functionality for the election system. At the start of the election, a voter roll is generated from the member database to send notifications and ballots to eligible voters, making it essential to have complete member synchronization from the official website. Due to our tight development timeline, we initially used database synchronization to transfer member data from the legacy system to the voter model on the election system. Later, when we developed the new official site for launch, we switched to using a RESTful API method.

00:22:24.640 Considering dispute handling, the election system needed to keep a record of voters’ identities. Although the legacy version had a record, it was unencrypted. In the new version, we use ActiveRecord's encryption feature to secure this record, ensuring actual voting data remains protected even if the database is compromised. The process for generating election announcements, ballots, and emailing election results had previously depended on manual tasks. This entire process has been redesigned to utilize Active Job, automating workflows and reducing the chance of errors caused by human operation under tight time constraints.

00:24:07.320 Despite the challenges, we successfully completed the development and launched the new election system with just a two-person team within one month. We conducted extensive testing, resulting in test coverage exceeding 80%. Now, I would like to share my experiences in improving our DevOps practices. Good DevOps design significantly accelerates the product's development process. Our primary goal was to automate as much of the code checks as possible.

00:25:56.960 Further, we aimed to create an independent review environment for each pull request. When a pull request is merged, it can automatically deploy to the production site utilizing continuous deployment services, thereby minimizing human errors. To achieve this, we outlined several objectives: we needed to orient the project towards automation by writing CI/CD pipelines and selecting a Ruby stack suitable for automation. I have summarized our objectives and actions to achieve these goals.

00:28:39.800 To further enhance automation, we implemented an automatic detection system for the Ruby project environment to generate a Dockerfile. Supporting user options with a DSL, our generator can be extended with additional requirements. The Docker stack allows running different commands under the same entry point, as well as customizing checks for database connection and migrations. The generated Docker container only by default includes Puma and Sidekiq commands, but custom commands for other services can also be integrated.

00:30:06.560 The live stream feature offers middleware options for health checks and can easily be mounted in routes, automatically adding health check settings when generating the Dockerfile. This streamlined process was a significant improvement. As we leverage GCP's continuous integration for development, many variable settings are predefined in the CI environment included in our GitLab project.

00:31:38.080 There's also a mention of review apps, which create an independent real-time environment for each pull request. The ability to create these apps with pre-defined variables corresponding to the code branches allows for improved workflow during development. However, deployment design and the CI setup are complex topics, and I'm unable to delve deeply into them today.

00:32:55.520 To aid in simplifying the set-up process, I recommend a specific open source GitLab CI script template designed to facilitate rapid establishment of CI workflows, from linting and testing to automatic deployment for typical RESTful projects. It serves as a valuable resource in my projects.

00:34:23.920 This is what the process looks like for a pull request in GitLab where we perform linting, testing, build the Docker image, and deploy review apps accordingly. When a pull request is merged, it automatically deploys the staging environment and allows for further manual deployment to the beta and production environments.

00:35:21.320 Lastly, I want to share my experience regarding data migration. A dedicated migration tool was crafted for migrating data from the old schema to the new schema, ensuring only necessary data was transferred. It also facilitates the transfer of attachment files from the legacy carrierwave to active storage.

00:36:58.960 Given that this tool is intended for one-time use, I created a separate Rails engine, hosting it in a distinct repository. It is referenced in the project Gemfile and can be removed once the migration completes. For implementation, I first defined a migration method in a base class, allowing subclasses to only need to implement the corresponding methods for their specific migrations.

00:39:01.480 We also used valid data and callbacks when saving, configured through constants, to define what scope needed to be migrated, effectively managing associations to avoid N+1 queries during migrations. When migrating attachment data, we utilized a method to check the existence of files before transferring.

00:39:52.560 As our project generates Docker images during the CI pipeline build, it was essential to ensure that our engine is added to the Docker image at build time. To avoid security risks, we coded sensitive tokens in the GitLab CI variables and passed them as arguments during the build process to secure them as environment variables.

00:40:43.600 Finally, on launch day, the migration process took around 30 minutes, followed by two hours spent on manual verification of accuracy. The party's staff subsequently inputted the new CMS content following the new format and, fortunately, we experienced no data loss or any incidents, making the migration process remarkably smooth. Thank you for your attention.

RubyConf AU 2024