00:00:18.490
Hi everyone, welcome! Today I'll talk about integrating Elasticsearch into your Rails applications.
00:00:22.670
In particular, I want to share details about a gem called 'search_flip' that I wrote. Although it isn't completely new, as I started implementing it back in 2013, I hadn't marketed it much before. I used it internally for a long time, but now I've decided to release it publicly.
00:00:36.680
Search_flip is a gem that features a chainable DSL, which I will demonstrate in the next slides. Many gems out there lack this chainable DSL, making them quite cumbersome to work with. Search_flip essentially removes this bloat from your codebase.
00:00:59.960
A little about myself: I currently work at a company called Really Fat Vas, which as of Friday is now under the visible umbrella. You can find me on GitHub, and I have a strong passion for search technologies. I’ve been working with search for nearly ten years, starting with Elasticsearch when it was version 0.9. Before that, I worked with various tools like Solr.
00:01:14.240
You might wonder, why create another Elasticsearch client when there are already good ones available? Let's take a step back and consider a simple query in sample code. For instance, we have some comments in our database that we want to search through, and they should contain the text 'hello world'.
00:01:32.960
We also want these comments to have an approved state since we have a review process. Publicly, we’ll only show the approved comments. We may need to aggregate information about users to display counts of how many comments they had. Let’s examine how this query could look with different tools.
00:02:02.149
Additionally, we want to sort the results by ID. First, we look at querying with Elasticsearch Ruby. This approach often leads you to writing raw Elasticsearch queries, which can be tedious and error-prone. You don't need to focus on the code listed there; it's merely illustrative.
00:02:22.640
Writing raw Elasticsearch queries can feel cumbersome, requiring complex hashes built up for the query, which can easily go wrong due to syntax issues. I wanted to simplify this process, making it look more intuitive. For instance, there is another popular gem called 'search_cake' that aims to alleviate some of that bloat.
00:02:49.460
Search_cake has improved readability, but the underlying structure still resembles old Active Record methods from several years back, making it necessary to bundle all the clauses into one method.
00:03:04.710
The challenge remains in collecting all the clauses we want to send to Elasticsearch for the desired output. This often leads to a messy code structure, especially when user inputs dictate certain query clauses. My goal was to enable the chaining of these clauses seamlessly.
00:03:35.240
With Search_flip, the structure allows you to write queries that feel distinct from an Active Record query. You can chain clauses together, providing flexibility for optional additions based on parameters, much to the delight of users.
00:04:05.210
Who is search_flip for? Typically, when you're integrating search into an application, it is reasonable to approach it within your existing database. If that fulfills your needs, it's best to stick with it, as introducing a new service or database can be quite challenging. However, when your data needs outgrow your database capabilities, that’s when tools like Elasticsearch come into play.
00:04:35.300
Feature-rich options in Elasticsearch, such as scoring, aggregating, and suggestions, might compel you to explore it further, especially if you're struggling with complexity or performance with your current setup. Let’s delve deeper into how Search_flip integrates within a Rails application.
00:05:05.460
Essentially, you begin by creating an index class that includes the search index module. You can specify a model and then inform Search_flip how to serialize it. The serialized data will be sent to Elasticsearch for indexing.
00:05:34.140
These fields will be searchable, aggregable, and usable for highlighting. In your index classes, you can also specify mappings, index settings, aliases, and more—all designed to flexibly integrate with Elasticsearch optimally.
00:06:01.173
After building your index, you can perform several operations such as creating the index, checking if it exists, deleting it, or updating its mappings. You can import records efficiently, whether all records, a single record, or even arbitrary active record relations.
00:06:43.777
Similarly, you can delete records using the same commands for creating indexes. This overview is intended to familiarize you with the concept of Search_flip, providing a basic understanding of its purpose.
00:07:05.961
As mentioned, the Elasticsearch queries created with this gem are chainable and lazily evaluated. That means they only get executed when you request results, like with the 'total count' or 'records' method, mirroring how you might work with Active Record.
00:07:34.610
Let’s explore a practical example. Imagine a hotel booking platform where we maintain a hotel index. We can execute a search only for hotels with a minimum ranking, in a specific location, aggregating ratings to display on a sidebar and paginate results.
00:08:02.600
Looking into aggregations, these are about counting items, a feature easily found on popular sites that show brand counts as you filter results. With Search_flip, this can be as simple as calling 'aggregate' on a brand.
00:08:31.970
Additionally, you can perform date histogram aggregations effortlessly. One noteworthy thing is that Search_flip, although abstracted, still allows you full interaction with Elasticsearch.
00:09:07.350
Many users prefer sticking to the raw Elasticsearch features for flexibility. The motto of search_flip is to keep the simple things simple while permitting complex functionalities. You can incorporate various Elasticsearch features alongside the chainable queries.
00:09:27.710
Exploring more complex aggregations, for instance, you can create a sales histogram restricted to specific price ranges, showing aggregations only for documents priced above certain thresholds.
00:09:55.880
With Search_flip, building such deeply nested aggregations is manageable while preserving code clarity. This approach results in readable and maintainable code without losing the powerful capabilities of Elasticsearch.
00:10:19.060
Let’s switch gears to a practical demo. I’ve prepared a movie index using IMDB data. When you create an index, you can define its structure within your Rails app, including mappings that may seem cumbersome but are essential.
00:10:49.990
It’s important to serialize your index properly. The movie index I’ve set up has a familiar structure, highlighting fields like movie names, release years, and ratings. Now let’s import specific records into the index.
00:11:17.340
For example, we might choose to import only movies that currently have a rating. Once indexed, querying allows us to find and sort these records effectively.
00:11:49.310
When we query the index, we can also look at the database records, ensuring that results are sorted correctly—something that doesn’t always happen with traditional database queries.
00:12:15.330
Returning to our querying capabilities, we have the ability to create range queries and utilize aggregation features to extract valuable insights from the dataset.
00:12:39.000
The elegance of Search_flip lies in these flexible interactions that make filtering and retrieving data intuitive, even as your queries grow in complexity.
00:13:06.710
As we enhance our query requests, they can start to become intricate quickly, transitioning from straightforward searches to more elaborate aggregations. The software’s design aims to accommodate this growth.
00:13:35.310
Finally, we can execute bulk actions such as deleting records using the queries established earlier, showcasing how seamless operations remain even in varying scenarios.
00:14:02.490
Integrating with Elasticsearch poses challenges, such as maintaining synchronization with your databases, which many gems struggle with. While Search_flip does not natively address syncing, it can leverage hooks to update your index using a streaming system.
00:14:28.540
Using tools like Kafka or Redis, ensures smooth operation across systems while keeping data accurate and up-to-date. In practical applications, especially in large datasets, it is vital to manage how data flows between services.
00:14:55.370
So, are there any questions? Throughout this journey, we have only touched upon the surface of what can be accomplished with Search_flip. Both your use case and curiosity will guide how you explore and implement these features.