Suggest modification to this talk

Title

Description

Date

Summarized using AI?

If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.

Show "Summarized using AI" badge on summary page

Summary

Markdown supported

In his talk at RubyConfTH 2023, Benji Lewis, a software engineer at Zappy, presents an innovative project focused on a custom data indexing system created using Ruby, graphs, and bitmaps. Benji begins by discussing his background and the goals of Zappy's research and development team, Zappy X, which aims to explore concepts that may initially seem impractical.

**Key Points Covered:**
- **Overview of Data Management**: Zappy collects survey data and uses it to model insights that help clients in advertising. The existing system showed a need for improvement in data context, storage, and connections between data points.
- **Challenges Identified**: 
  - Context Management: Ensuring all data points reflect the same concept to avoid misalignment in data analysis.
  - Storage Problems: Efficiently managing a large number of survey measures to maintain a comprehensive dataset while avoiding redundancy.
  - Data Harmonization: Implementing a system to equate different measures and stimuli effectively for comparative analysis. 
- **The Measure Store Solution**: Benji introduces the measure store, which simplifies querying data by focusing on three components: context, measure, and dimensional filters, enhancing real-time data retrieval. 
- **Live Demonstrations**:  The session included GIFs showing various examples of the measure store in action, including performance analyses across countries and studying the relationship between ad watchfulness and persuasion sentiments. 
- **Architecture of the Measure Store**: The design utilizes bitmaps for dimensional data storage, allowing rapid access while employing a compression algorithm to optimize performance. A graph structure is implemented to model the relationships among different data points effectively. 
- **Technology Used**: The measure store relies on Redis, integrating a graph database and bitmap functionality for efficient data management, significantly reducing query times and storage requirements. In benchmarking tests, the efficiency of retrieving complex datasets showed improvements from 90 seconds to just 495 milliseconds.
- **Future Directions**: The project has moved past its research phase into production, aiming to develop a new reporting platform that maintains data integrity while leveraging Redis's in-memory capabilities.

Benji concludes by expressing enthusiasm about the opportunities ahead for their data index system, emphasizing the advantages of using graphs and bitmaps for indexing, ultimately enhancing their data analysis processes at Zappy.

Cancel