Ruby Video
Talks
Product metrics for developers
Dmitry Salahutdinov
@dsalahutdinov
#backend-development
#data-analysis
#continuous-improvement
#user-experience-ux
#minimum-viable-product-mvp
Summarized using AI

Product metrics for developers

by Dmitry Salahutdinov

Introduction

In the video "Product metrics for developers," speaker Dmitry Salahutdinov, a backend engineer at Amplifier, explores the significance of product metrics in software development. He emphasizes how developers can benefit from integrating product analytics into their workflow to enhance decision-making and feature performance evaluation.

Key Points

  • Understanding the Product Approach: Dmitry introduces the concept of a 'product approach,' focusing on solving client pain points efficiently. This involves starting with an idea, validating it through Minimal Viable Products (MVPs), and iteratively improving based on user feedback and metrics.
  • Importance of Measurement: He stresses the necessity of making data-driven decisions rather than relying on emotions. Establishing product metrics enables developers to gauge the effectiveness of features and adjustments accurately.
  • Practical Examples:
    • Success Rate Metric: This metric quantifies the reliability of integrations, such as with third-party services. By tracking successful and failed API requests, developers can measure integration performance using analytics tools like Amplitude.
    • Error Handling: Dmitry discusses the importance of managing API errors effectively. By documenting error types and enhancing error handling strategies, developers can improve user experiences and maintain service reliability.
    • Using Visualization Tools: Tools like Redash and Grafana help developers visualize metrics seamlessly, allowing for better insights into product performance and monitoring operational aspects.
  • Continuous Improvement: The video emphasizes that an ongoing cycle of measurement, modifications, and analysis leads to continual enhancements in product development. By understanding metrics and experimenting with changes, teams can systematically improve offerings and address user needs.

Conclusions

Dmitry urges developers to adopt the product approach, highlighting the relevance of establishing metrics, making informed changes, and analyzing results. This proactive engagement with product development and metrics fosters a culture of continuous improvement and enhances the overall value delivered to users.

00:00:14.920 oh
00:00:48.559 Our next speaker is Dmitry Salahutdinov. He is a mathematician and a passionate Ruby backend engineer at Amplifier. Today, he is here to talk to us about product metrics. I told Dmitry backstage that I'm eager to hear his talk because, as a product engineer and lead of a product engineering team, one thing I've noticed is that sometimes we have a hard time understanding and communicating the value of what we do for our users and our businesses. It’s a really important conversation to have.
00:01:11.600 There are tools that can serve us in our careers for years to come. My colleagues, Luca and Alberto, and I, while screening the talk submissions, all agreed that this is an important topic. Dmitry, I admire your ability to contribute to open source projects, especially considering you have three kids. I often feel overwhelmed with my daily job and a cat. Your dedication is commendable. Thank you for being here today. I’ll leave the stage to you.
00:02:01.119 Thank you! My name is Dmitry. I’m the backend engineer of a product team. Throughout my work, my team has fallen in love with modern product development. I use the term 'product approach' to describe my perspective on product development. This term may seem unusual, but once you understand it, you’ll find yourself doing the same things.
00:02:19.440 My talk today is a reassembled experience of using the product approach. It will be a mix of concepts where we’ll explore the idea of the product approach in backend development. We’ll define product metrics to help us work systematically and meaningfully. I hope to share several insights that you can incorporate into your daily activities to achieve more meaningful and measurable results. I invite you to dive deep into this topic with me.
00:02:47.799 The product approach is a working method that allows you to solve problems optimally and without unnecessary actions. It all starts with an idea or a key action, whether you're working on a website, an online school, or assembling a car. You begin by selecting one key task, focusing on the pain points you aim to alleviate for your clients.
00:03:06.000 Next, this idea goes through critical examination. You check if people indeed have the same problem you plan to solve. If yes, the next step is to create a Minimal Viable Product (MVP). Instead of developing a huge system, you build a simple and cost-effective prototype to test your idea. Then the fun begins; the process becomes continuous. Any project generally undergoes multiple stages, leading to gradual product improvements until it reaches your desired standard. Most importantly, regardless of how long this process takes, you already have a functional solution in the initial stage.
00:03:38.000 Measurement is essential for determining and quantifying results. You should make data-driven decisions based on facts rather than emotions. Forming a set of parameters helps you evaluate the effectiveness of your model, referred to as product metrics. For example, suppose you have a product that addresses user needs, but visitors don’t grasp its value. You could redesign your landing page to communicate the product's value better.
00:04:18.000 You aim to quantify the conversion of visitors into trial users. This method provides an objective way to evaluate the results. A new design, along with a product change and subsequent analysis, constitutes an experiment. You may run multiple redesign experiments until you achieve the desired improvement. Keep in mind that everyone on the team might have different backgrounds, perspectives, and understanding of the product. However, the numbers provide a universal truth to evaluate success.
00:04:55.600 The commonality of the product approach is so intrinsic that many professionals take it for granted. Think of a doctor examining a patient. First, they measure the temperature and conduct tests before diagnosing and administering treatment. They continue to monitor metrics to assess whether the treatment is effective. This process exemplifies the product approach: measure first, then implement a change, and finally analyze the results.
00:05:39.600 Let’s transfer the principles of the product approach to everyday backend development. For instance, when assessing the performance of a particular feature, you should generate a hypothesis for improvement and measure positive effects. Consider a product that automates social media content workflow, allowing clients to distribute content across multiple platforms simultaneously. This automation enables clients to focus on higher-level tasks while the product manages content delivery.
00:06:10.600 As a backend engineer, you might face challenges with third-party service integration. Such external integrations are often unpredictable, which can hinder system reliability. Your customers pay for your product, believing everything works perfectly. But how do you truly verify whether your Facebook integration is functioning properly? You could display a success rate—a quantitative indicator showing the percentage of successful API requests.
00:06:35.500 To gauge success, consider using analytics services like Amplitude, which can track success and failure events. Amplitude is easy to set up, and you can get started without any upfront payment. The Ruby client for Amplitude has a straightforward API; you just need to send events along with any metadata you want. Metadata becomes particularly helpful when generating analytical charts based on additional parameters.
00:07:05.440 In this case, it would also be beneficial to track the total number of API requests made to external services, distinguishing between social networks like Facebook and LinkedIn.
00:07:34.640 Once you've prepared all the necessary metadata, you send it to Amplitude via HTTP. You integrate the Ruby gem and start logging successful and failed events, as demonstrated on the slides. This process yields a metric called the 'success rate,' which serves as a quantitative representation of feature reliability. With this data, you can numerically estimate how well the integration performs. Additionally, you should continue monitoring the metrics to analyze the results after implementing any product changes.
00:08:46.880 The success rate metric provides clarity regarding your product's basic features at any moment in time. This number is easily understood by product managers, developers, and all team members. Its beauty lies in its impartiality; it remains unaffected by emotions or subjective opinions. Furthermore, what’s great about Amplitude or any similar service is its user-friendly API and easy setup, allowing you to embed powerful visualization features.
00:09:20.560 Once you comprehend how to evaluate product characteristics using comparable numbers, you gain insights into the product framework. The product framework fosters continuous project improvement through iterative changes, reinforced by ongoing outcome analysis. This method allows you to make improvements systematically and continuously. Once you understand what happens under the hood, you can experiment and implement enhancements, like a new error handling algorithm, and observe their impact on the metrics.
00:10:00.560 Regardless of your role, whether you’re a backend developer or work in another area, you're creating a product. Your system may not just be about excellent source code; it's about creating an environment where you can evaluate the effectiveness of your product, make decisions, and analyze results. This approach allows you to appreciate the value of your contributions to the project.
00:10:42.360 The product framework is rooted in the rationality of data-driven decision-making. You may face challenges like imperfect source code or significant technical debt. However, if your product addresses user needs and metrics support that, then it’s a success. Furthermore, this system enables you to improve your product through continual experimentation, fostering meaningful, conscious development informed by data.
00:11:30.480 Let's explore another scenario in your project. If you've improved the primary operational performance, you might next consider enhancing error handling. Why is this important? The reasons for failures in external API calls can vary. Some issues can be resolved, while others cannot. A more thoughtful approach to error handling enhances positive user experiences.
00:12:12.199 For instance, while you may not be able to rectify unavailability in external services, you can efficiently handle 400 Bad Request errors by adjusting parameters. Therefore, when an error occurs, classify the exception in your Ruby code according to its type and persist the error code in your database. Unclassified errors should be logged as 'unknown error code.'
00:12:40.079 You can visualize this data using a tool like Redash. Redash simplifies the process of connecting to various data sources, allowing you to write queries and render visualizations of the results in a user-friendly format. With just a few clicks, you can create charts showing the percentage distribution of error types over days or months. By improving error handling, you'll want to measure the percentage of unknown or unprocessed errors as a vital metric.
00:13:48.440 This metric indicates that while errors are occurring, your handling logic may not be able to identify the underlying reason. Therefore, the more users encounter these 'unknown' faults, the worse their experience becomes. By pinpointing the problem and updating your source code, you can roll out changes and analyze the results.
00:14:15.960 Redash proves valuable as a standalone service that’s easy to deploy without the need for complicated infrastructure. You can visualize database data and share insights with team members effortlessly. As you explore metrics systematically, the overarching approach remains consistent: continue to improve your product.
00:14:57.679 An additional benefit of managing metrics effectively is the ongoing quality oversight of your product. The number of handled errors will help you monitor this aspect over time, particularly during updates to APIs or integrations. By tracking unknown errors, you can ensure that everything remains fine. Additionally, automation eases monitoring by utilizing Redash's alerts feature. You can set up special queries to monitor metrics and receive notifications when specific limits are exceeded.
00:15:45.760 Regardless of how you implement metrics or the various processes involved, the principle remains steadfast: measure first, then make changes, and finally analyze the results. You might discover essential product features, where everything is clear, and you know how well the service operates. Now let's consider another task where you will need to gather and analyze social network metrics.
00:16:32.200 Suppose clients report the absence of follower data on social networks. Where do you begin? You’re right—metric quantification. In this case, you may leverage existing monitoring infrastructure with established Ruby modules for storage, Grafana for visualization, and a service like Ops Genie for alerts to notify the duty engineer.
00:17:17.280 Next, implement Prometheus to log metrics to capture data affordably. You’ll tap into the Yabeda framework, which provides a common DSL for metric declaration and management. Although several metrics exist, we will focus on 'counters,' which monitor the occurrence of events like web requests or background job executions.
00:18:01.760 Additionally, Yabeda supports exporting these metrics into the Prometheus exporter format. To proceed, you need to expose a specific endpoint that presents the metric values in textual form. The exporter will periodically scrape this endpoint to gather and store the values you define, similar to how events are logged in Amplitude.
00:19:01.439 Instead of just logging events, like using Amplitude, you will increment the success and error counts. You can then visualize this data in Grafana. By employing specific functions to calculate the increase of values in the time series while accounting for metric resets, you will gain deeper insights into your operations and ultimately refine your metrics.
00:20:00.000 Afterward, consider setting alerts based on these metrics. If something goes wrong, the engineer on duty should receive notifications via Slack or SMS, depending on your setup.
00:20:46.080 Ultimately, regardless of the tools you utilize, the concept of the product approach remains the same. It’s a consistent methodology applicable across various disciplines. Depending on your role in product development—whether defining product metrics or optimizing SQL query response times—you’ll want a standard strategy to ensure improvements are made effectively.
00:21:28.560 A fundamental aim of any conference talk is to spur action among listeners. I hope you have recognized relevant experiences in your own daily activities and that you'll apply the concepts of measurability and the product approach in your practice moving forward. Next time, establish metrics, implement changes, and analyze outcomes to understand the ease and efficacy of these methods.
00:22:09.840 By the way, the project we’ve discussed today is not theoretical; it reflects my real work on Amplifier. I’m here to answer any questions, whether now or later, using the contact details provided in the slides. Thank you for your attention.
00:22:56.800 And welcome back! Thank you! This is a great setup. It’s fantastic to be a speaker while having another person give a talk, which certainly eases any pressure. I appreciate the great tools we utilize; we use Amplitude for user data analysis and behavioral metrics, looking to see if released features meet our expectations.
00:23:48.600 Often, Amplitude is associated with product web development, but backend developers can log anything meaningful for their products. You can assess the performance of both product and technical aspects. It’s important to go beyond basic observations of backend operations and evaluate them in an overall context.
00:24:29.600 Absolutely! You can certainly scale from small metrics to higher levels of analysis to understand overall product performance. Do you have any recommended resources for further learning related to these product metrics?
00:25:06.640 One of the best resources for product metrics is Amplitude's communication. I find it straightforward and user-friendly, although many alternatives exist with a quick search. While I’m fond of Amplitude for its simplicity, I suggest you explore all options to find what suits your needs.
00:25:54.080 I advise connecting with product personnel who deal directly with product metrics. Gaining insights from their expertise will aid your understanding.
00:26:51.360 Thank you! That makes perfect sense. Engaging with the product manager provides invaluable perspectives on the metrics and their significance. Thank you for your time. As I haven’t seen any questions in the chat, we can now transition to the lunch break.
00:27:07.360 If anyone else has questions for our speakers, please don't hesitate to connect with them in the chat or directly message them. Thank you, Dmitry, for joining us today. I greatly appreciate your insights!
00:27:29.440 Thank you for having me! If anyone has questions, feel free to reach out to me.
Edit
Explore all talks recorded at rubyday 2020
+2