Ruby Video
Ruby
Plugin-based software design with Ruby and RubyGems
Sadayuki Furuhashi
@frsyuki
#ruby
#rubygems
#open-source
#data-processing
#software-design-patterns
Summarized using AI

Plugin-based software design with Ruby and RubyGems

by Sadayuki Furuhashi

In the presentation titled "Plugin-based software design with Ruby and RubyGems," Sadayuki Furuhashi discusses the advantages and challenges of utilizing plugin architectures in software applications, particularly within the Ruby ecosystem. The speaker, who is also the founder of Tri Data and has designed software such as Fluentd and Embulk, emphasizes the importance of extensibility in programming through the use of plugins.

Key points discussed include:

- Definition and Importance of Plugin Architecture: Plugin architecture allows applications to extend functionalities without altering the core software. Examples include browsers with add-ons and WordPress with open-source plugins.

- Robust Plugin Design: Two approaches are highlighted: traditional extensible architecture, which increases host complexity; and purely plugin-based architecture, which relies on a thin core and modular design.

- Key Techniques:

- Dependency Injection: This programming technique separates interfaces from implementations, facilitating testing and plugin replacements.

- Dynamic Plugin Loading: This allows the core application to manage and execute plugins seamlessly, promoting modularity without altering the core functionality.

- Real-world Implementations: Furuhashi uses Fluentd and Embulk as examples.

- Fluentd: Serves as an event collector, utilizing over 300 open-source plugins to enhance its data collection abilities without the need for extensive scripting.

- Embulk: A bulk data loader that supports multiple data sources through flexible input and output plugins, ideal for handling large datasets efficiently.
- Challenges: Specific hurdles include managing library conflicts from multiple plugins depending on the same libraries and keeping plugins updated to avoid performance issues.

In conclusion, Furuhashi encourages the audience to explore plugin architectures for their extensibility benefits and proper data handling, inviting collaboration and proposed solutions to common challenges in plugin implementation.

00:00:01.439 Thank you for coming. I'll present on the topic of plugin-based software design with Ruby and RubyGems. My name is Sadayuki Furuhashi. I was born in Japan and moved to the US four years ago. I currently live in Montville, California, and I am the founder of a company called Tri Data.
00:00:14.080 We provide a cloud-based data management service where you can collect data, store it, and query it. Even if the data is large, such as hundreds of gigabytes per day, you can perform queries smoothly. For data collection, we have open-sourced two projects: Fluentd and Embulk. I designed both of these projects and also created another project called MessagePack, which is a serialization library similar to JSON. However, because it is binary-based, it is faster and more compact. I encourage you to check it out since it is open source.
00:00:47.239 Today's topic is plugin architecture. You're likely familiar with plugins from various applications. For example, browsers have add-ons to extend their functionalities. Eclipse has its own marketplace for plugins, and WordPress has thousands of open-source plugins available for download, allowing you to enhance its capabilities. A plugin architecture allows for similar enhancements in applications. With something like Fluentd, you can add extensions that enhance its features, such as integrating with PostgreSQL. By doing so, you can run queries seamlessly across different databases.
00:01:37.680 Fluentd has over 300 open-source plugins available. The benefits of a plugin architecture are evident, as it allows a wide range of features provided by a community of developers. You can continuously add new features while keeping the core application simple. Since the plugins are isolated, it's easier to test them individually. This leads to a more active developer community, which is crucial for an open-source project. However, the success of this architecture largely depends on its design. If it is not designed well, we may encounter significant downsides. Therefore, today I will touch on how to design a robust plugin architecture.
00:02:23.920 To begin, I'll discuss some design patterns crucial for plugin architecture. There are two main approaches: traditional extensible software architecture and pure plugin-based software architecture. Traditional extensible software typically features a host application that provides extension points where plugins can connect. While this is straightforward, adding more flexibility increases the complexity of the host application. On the other hand, a purely plugin-based architecture consists of a thin core with most functionalities implemented using plugins. This structure simplifies the core, as it does not become more complex with additional features, but it requires a well-thought-out modular design.
00:03:38.920 We can implement plugin-based architectures with two key techniques: dependency injection and dynamic plugin loading. Dependency injection, a well-known programming technique, involves creating software using interfaces and classes. This method is popular in Java due to the clear separation it provides between these elements. By configuring the dependency injection container, you can easily replace interfaces with plugin implementations for testing and clarity. The concept allows for effective unit testing, as different plugins can be replaced with dummy implementations to isolate functionality.
00:04:54.800 Another approach is to utilize dynamic plugin loading, which allows the core application to query the plugin loader for executing plugins. This creates a network of plugins that compose the functionality of the application. Adding a new plugin can enhance existing capabilities without modifying the core. Additionally, the combination of both dependency injection and dynamic plugin loading can yield a robust and flexible framework.
00:06:01.039 However, as we implement these designs, we also need to consider potential trade-offs. Changing an existing monolithic application to a more flexible architecture can be challenging. While adapting to a new system might push for dependency injection and dynamic plugin loading, those strategies are best suited for new developments. Existing applications may still benefit from traditional architectures, which is often sufficient. Moving on, I'll discuss actual implementations of plugin architectures, specifically with Fluentd and Embulk.
00:07:18.200 Fluentd is an event collector designed to aggregate data from various sources and transport it to multiple outputs. It resembles a traditional logging system but is more structured and programmable, allowing the addition of plugins to modify its behavior. The core of Fluentd is implemented in C for performance, while the plugins are designed in Ruby. This approach facilitates extensive usage and encourages production readiness.
00:08:07.040 One challenge faced by Fluentd was the need to write separate scripts for each input-output combination, resulting in a labyrinth of management. By implementing a plugin system, adding a new input allows for immediate copying of data across all outputs without individual scripts. This configuration flexibility is essential for managing log data efficiently and adapting to user requirements.
00:09:50.080 I will now show an example of a configuration file showcasing how plugins interact within Fluentd. Each type in the configuration file defines the plugin name, such as 'tail' for reading files. The core architecture remains minimal, focusing only on routing events, while other functionalities are executed via plugins. For instance, you can use a meta-plugin called 'copy' to duplicate data and send it to multiple outputs, like Elasticsearch for real-time querying. This modular framework emphasizes flexibility, allowing users to choose appropriate plugins depending on their requirements.
00:12:17.680 Fluentd has effectively positioned itself within the RubyGems ecosystem, allowing users to easily install and utilize plugins. The core implementation leverages RubyGems to load necessary plugins at runtime, primarily focusing on a simple and efficient method. By utilizing existing Ruby tools, developers can harness the functionality they need, rapidly enhancing their data processing capabilities. Notably, over 300 plugins are available for Fluentd on RubyGems, marking its wide acceptance and utility in the community.
00:13:43.640 Next, I will discuss another project called Embulk, an open-source bulk data loader that supports both Java and Ruby. Embulk is designed for efficient data transfers, particularly for batch operations. Much like Fluentd, it utilizes a plugin architecture, allowing developers to create input and output plugins to handle data seamlessly. Vendors can create their custom plugins, enabling them to adapt to specific use cases without the need for extensive core modifications.
00:16:02.760 When using Embulk, you define a configuration file specifying data input and output types, while the core handles the data processing efficiently. Embulk's flexible plugin system ensures that operations can be performed across multiple data sources, optimizing data loading times. This adaptability is critical for handling large datasets often encountered in real-world applications.
00:17:40.520 Both Fluentd and Embulk illustrate the power of plugin architectures in simplifying complex data movement tasks. With various community-contributed plugins available for both tools, users have the freedom to leverage existing features or create new functionalities tailored to their specific needs. The ongoing evolution of these projects highlights the importance of modular design and encourages active community participation. Now, I will focus on some challenges faced in the plugin architecture to conclude this talk.
00:20:29.760 One challenge is managing library conflicts when multiple plugins depend on different versions of the same library. In such cases, isolating dependencies through unique class loaders can help, but it doesn't completely eliminate conflicts at runtime. The polymer the community continues to share solutions and approaches to navigate these complexities as they arise. Keeping plugins and their dependencies updated is another aspect that requires diligence from both the developers and the users.
00:22:32.080 In closing, I encourage everyone to explore plugin architectures in their applications. This design approach can significantly enhance extensibility while allowing for efficient data handling. Please feel free to reach out if you have insights or questions regarding plugin implementations, as I am continuously looking to learn from the community. Thank you very much for your attention.
Edit
Explore all talks recorded at RubyKaigi 2015
+43