Building Fast, Lightweight Data-driven Apps Using The Infochimps API

LoneStarRuby Conf 2010

Jesse Crouch

1 talk

#performance-optimization

Building Fast, Lightweight Data-driven Apps Using The Infochimps API

by Jesse Crouch

In this video titled "Building Fast, Lightweight Data-driven Apps Using The Infochimps API," Jesse Crouch, a product manager at Infochimps, discusses the creation of a data-driven application utilizing the Infochimps API within a Rails framework. Crouch introduces the application that will showcase user interactions on Twitter by utilizing the Infochimps API to visualize these interactions, helping users discover new connections and expand their social networks.

Key points discussed include:

- Introduction to the Infochimps API: Crouch explains Infochimps as a data marketplace similar to Amazon but focused on data, providing vast datasets, including historical Twitter data.

- Technical Environment: The app will leverage Ruby on Rails and MySQL, utilizing tools like Chef and Cassandra for big data handling, with AWS's EC2 for scalable API services.

- User Interaction Visualization: The central feature called 'Strong Links' is introduced, which will identify the top 100 users a person interacts with on Twitter and highlight their strongest connections based on replies, retweets, and mentions.

- Implementation Steps: Crouch shares his coding process, starting from setting up a Rails application, including API key configuration, and building out the application's controller and view.

- API Functionality: He discusses how to gather strong links data and analyze it, emphasizing streamlined API calls and effective data handling techniques.

- User Experience Focus: The overall goal is to enhance user engagement via a clear visualization of interactions, facilitating easier navigation and understanding of the social dynamics in users' Twitter networks.

- Conclusion and Takeaways: Crouch recaps the importance of APIs in creating insightful applications, urging viewers to leverage the concepts and functionalities presented in the session to further their development skills.

In conclusion, the session emphasizes the potential of the Infochimps API in building interactive, data-driven applications while providing viewers with practical coding knowledge and insights into effective data visualization techniques. Crouch encourages ongoing engagement and experimentation with the API to maximize its benefits.

00:00:00.000 Hello everyone, I'm a little half off stage here. I'm going to take you through a data-driven app that shows how to use the Infochimps API.

00:00:12.360 We'll explore how to use APIs to pull things together in a Rails application. By the end, you'll have a clear idea of what our app will look like.

00:00:24.090 Currently, this app might not look like much, but it represents a pool of users and how they can be sorted based on different metrics.

00:00:37.200 This shows a network of users, a feature that is currently difficult to monitor.

00:00:54.629 A little bit about me: I'm Jesse Crouch, a product manager at Infochimps. I started working with Rails in 2009, but before that, I was involved in advertising for some big companies. Now, I’m back in the Ruby world and working with Infochimps, where we provide a data marketplace.

00:01:18.970 As a product manager at Infochimps, I also serve as a developer evangelist, actively coding while not claiming to be the best programmer out there. My background includes providing enterprise content management solutions with tools like SharePoint and Drupal.

00:01:32.050 Infochimps operates like an Amazon for data, offering a marketplace where anyone can buy or sell data online. We have both large-scale databases and more niche datasets available.

00:01:51.399 Additionally, we handle massive datasets, like Twitter data collected since 2006, at terabyte scale. This isn’t something you’d manage on a personal laptop; instead, it requires cluster computing resources.

00:02:07.629 To give you a sense of our technical environment, we predominantly utilize Ruby technologies, including Rails and MySQL. Our infrastructure utilizes various open-source tools designed for big data handling.

00:02:37.900 These tools include Chef, Cassandra, and others for data retrieval and processing. We use Amazon EC2 for scalable API needs.

00:03:02.230 Our platform is predominantly Ruby-based; even our big data solution, Pink, is Ruby-based despite being complex. Let's dive into the app we've built.

00:03:14.390 The real magic happens because our API allows you to visualize Twitter interactions beyond just viewing a user's page. With our API, you can see the interactions between users and how strong those connections are.

00:03:40.490 This is useful because it expands the circles of influence, letting you see not just who you know but also who your friends know, creating a broader network perspective.

00:03:58.150 We'll develop a feature called 'Strong Links' where querying will return the top 100 users you're most interactive with on Twitter. This interaction is based on metrics such as replies, retweets, and mentions.

00:04:15.799 When you query for a username, the API returns a list of user IDs and the weight of each link, representing the strength of those interactions.

00:04:34.250 These strong links provide a way to analyze your connections and discover new users to interact with, similar to expanding your social circle based on existing relationships.

00:04:54.670 It's valuable to see who your strongest connections are and whom they associate with, potentially leading you to interesting new accounts to follow.

00:05:06.440 So, we are going to implement functionality that loops through your top interactions and finds their strongest associates, building a comprehensive view of your Twitter network.

00:05:31.600 This means using a method of querying that can handle vast amounts of data, allowing us to discover users beyond our immediate circle, enhancing our social networking experience.

00:05:59.600 The process will involve analyzing connections and ranking them to maximize the value of interactions and to understand better the social dynamics at play.

00:06:17.640 In the end, we can create a rich visualization that helps users understand the relationship dynamics within their network, adding value to the social media experience.

00:06:43.829 So, I'm going to go ahead and build it. If you want to follow along, here's the GitHub URL, where you can also find the API documentation and more resources.

00:07:01.970 Remember that as we build this app, you'll see me coding. Feel free to check out my tags along the way, as they'll help guide you through.

00:07:32.150 I've set up our server and installed the required gems including the Infochimps library. It’s essential you already have Rails installed to proceed further.

00:08:05.560 We'll create a Rails app called 'Strong Links' which will consist of a single controller and one view called show.

00:08:31.930 Now I’m configuring the environment for our Rails app, specifically for handling Infochimps API interactions.

00:08:38.409 Setting up configuration files correctly is crucial as this is where we will specify our API keys and other settings for our environment.

00:09:02.290 With the configuration set, I'm going to move on to how we can build out our controllers and views...

00:09:23.610 We're going to go through how to create the strong links feature in our app, examining users who interact with you the most.

00:09:45.880 This represents an essential part of the app where we will analyze the strength of these interactions.

00:10:01.800 While coding this, I'll show you how we handle API calls and collect relevant user interaction data.

00:10:13.370 We will write methods to fetch and analyze these strong links to provide deeper insights into your social graph.

00:10:37.080 Each step of building will uncover ways to enhance data handling through Ruby and Rails, optimizing our interactions with the API.

00:11:09.290 As we proceed, remember that the main goal is to understand the connections within your network so we can visualize them effectively.

00:11:34.440 By the end of this coding session, you'll be equipped with the knowledge to efficiently leverage the Infochimps API alongside Rails and view the data relationally.

00:12:00.480 Let's move on to the next part where we'll dive deeper into the API calls we're going to make for gathering strong links data.

00:12:14.010 We'll construct a method that will loop through your top strong links to uncover significant connections in your network.

00:12:38.390 You should be getting familiar with utilizing the API as we make these calls, which will help us gather strong connection data effectively.

00:12:58.030 This whole process hinges on efficiency, and we want to reduce any potential delay while interacting with the API during user queries.

00:13:23.970 As we write our methods, I'll emphasize how to handle API responses properly and how to ensure your application remains fluid and responsive.

00:13:43.060 The end goal is to enhance user interactions dramatically by taking advantage of strong connections and providing better relationship insights.

00:14:10.640 Now that we've laid the groundwork, we’ll transition into some code that highlights the methods used to fetch this data and how to handle it properly.

00:14:38.930 Throughout the implementation, I’ll showcase best practices for error handling and optimizing data retrieval from the API.

00:15:03.210 The key takeaway is to format your API requests optimally to minimize issues and manage the data efficiently once received.

00:15:30.980 Let’s work through the complete logic of how this interaction unfolds, validating each part of our process as we connect to the API.

00:16:02.590 Now, I’ll run through another example of how to make these strong link calls more effectively to gather the right data promptly.

00:16:32.410 This includes creating a robust querying system that works seamlessly with the API to return meaningful connections.

00:16:58.970 We're developing a unique angle toward visualizing user interactions, especially for how you might network on Twitter.

00:17:29.410 In this coding adventure, you should gain insights not just into your network but understand potential avenues for growth and discovery.

00:17:46.640 So, to reiterate, the main goal here is to build a seamless interface that expands your understanding of connections across Twitter.

00:18:05.460 As we develop the functionality, keep an eye on how to maintain clarity in presenting data effectively within our application.

00:18:25.200 The approach we'll take through coding these API interactions aims for simplicity and clarity, making it easy for users to navigate.

00:18:47.980 Building this out requires thoughtful consideration of user engagement and interaction, shaping an experience that feels intuitive.

00:19:10.930 Once everything is situated, we’ll be positioned to validate the functionality of the strong links feature and optimize as necessary.

00:19:33.320 Throughout this process, we will continuously encourage feedback and adjustments, ensuring the strongest form of usability for our target audience.

00:19:51.610 I anticipate you’ll find the overall process enriching and rewarding as we tie together API usage, Rails capabilities, and user engagement features.

00:20:11.970 Stay engaged as we move through executing this plan, paying close attention to the finer details that make an application outstanding.

00:20:37.560 As we finalize building the strong links feature, note how functionality improves the overall user experience and drives interaction fluidly.

00:21:10.170 When all is said and done, we should walk away with valuable tools and knowledge for building applications that effectively use the Infochimps API.

00:21:45.510 We are nearing the end, and I’d like to recap what we’ve accomplished together regarding the potential of the Infochimps API.

00:22:12.040 Exploring the details of API calls, the significance of robust routing, and how to maintain interactions will be critical moving forward.

00:22:34.340 I encourage you all to dive deeper into the tools and practices we discussed to elevate your understanding of data-driven applications.

00:23:02.280 As we round up this session, let’s ensure we continue empowering ourselves with the knowledge shared during this presentation.

00:23:24.980 Takeaways from this session encapsulate the essence of applying Ruby and Rails in a unique context like the Infochimps API.

00:23:53.670 Your engagement here is a vital part of appreciating the potential of data-driven applications, particularly as we look at APIs like Infochimps.

00:24:24.000 I wish you all the best as you proceed with integrating these concepts into your projects, looking forward to seeing what you can create!

00:24:47.370 Don't hesitate to reach out if you have questions or need further clarification on any of the points we've discussed today.

00:25:14.890 You can find me on Twitter or through my email. I'm excited about the possibilities to leverage the Infochimps API for your applications!

00:25:42.780 Before we wrap up entirely, I welcome you to take a moment to reflect on the coding journey we undertook together and the insights gathered.

00:26:06.490 Your contributions are the key to fostering a vibrant community, where we can learn and grow together through knowledge sharing.

00:26:32.370 In conclusion, remember that the utilization of APIs opens the door to new opportunities, especially in crafting data-driven applications.

00:26:54.670 Thank you for joining this session. I hope you gained valuable insights from our exploration of building with the Infochimps API.

00:27:18.330 It’s been a pleasure sharing this information with you, and I look forward to seeing how you put these ideas into practice.

00:27:42.530 Please feel free to reach out anytime for any follow-up questions or discussions. Thank you again, everyone!

LoneStarRuby Conf 2010