Ruby Video | JSON and the Argonauts - Building Mashups with Ruby

Talks

JSON and the Argonauts - Building Mashups with Ruby

Wynn Netherland

5 talks

JSON and the Argonauts - Building Mashups with Ruby

by Wynn Netherland

In the video titled "JSON and the Argonauts - Building Mashups with Ruby," Wynn Netherland delivers a comprehensive talk at the LoneStarRuby Conf 2010 focused on the creation and utility of API wrappers in Ruby development. The presentation emphasizes how these wrappers can greatly enhance the interaction with various APIs, making the integration more idiomatic and user-friendly for Ruby developers.

Key Points Discussed:

API Wrappers Definition: Netherland introduces API wrappers as tools that help simplify interaction between applications and APIs. He shares his personal experiences, highlighting projects like Tweet Congress and the Twitter gem, which sparked his interest and expertise in API integrations.
Importance of Idiomatic Access: A major theme in the talk is that Ruby developers prefer a more idiomatic approach in accessing API data. Netherland explains that while APIs often return data in camel case, Ruby's conventions prefer underscore notation. Wrappers can facilitate this transformation, enhancing readability and usability.
Method Naming and Syntactic Sugar: He discusses how customizing method names in API wrappers (e.g., renaming methods for clarity) and incorporating method chaining can streamline interactions and make development smoother.
Handling Various API Structures: The presentation showcases the diversity in API architectures and stresses the need for wrappers to accommodate these variations. Examples include how different APIs return JSON structures and handle data retrieval.
Tools and Libraries: Various Ruby libraries are discussed, such as HTTParty, Patron, and Faraday, which aid in building efficient API wrappers. He highlights the use of libraries like Oj for JSON parsing and VCR for testing API responses.
Authentication Methods: Netherland extensively covers the complexities involved in API authentication, pointing out the differences between basic authentication and OAuth, and the importance of adapting to evolving security standards in APIs, particularly with Twitter’s recent transitions.
Testing API Wrappers: He stresses the significance of testing wrappers using tools that can simulate API responses, ensuring robust development without unnecessary server calls.

Conclusion and Takeaways:

Wynn Netherland concludes his talk by inviting questions, emphasizing that creating API wrappers not only shields developers from frequent changes in APIs but also enhances the functionality and reliability of their applications. The key takeaway is that thoughtful design of API wrappers can significantly improve integration processes, making interactions more intuitive and aligned with Ruby's conventions.

00:00:14.960 This talk is more informational than cute, but it totally goes against the grain of most of my job. Let me introduce myself: I recently joined Shield Pack's cloud computing group, where I'm building some cool tools that you probably already use. So keep an eye out for that. By the way, we're hiring and ramping up our team over the next couple of months. If you're interested in cloud computing—whatever that means these days—and you can sling Groovy, we’d love to hear from you.

00:00:33.440 I also have a podcast called The Change Log, which features open source projects on GitHub. Hopefully, everybody has at least one open source project there. Cool! Does anyone have anything they want to be featured on The Change Log?

00:01:02.960 Great! All right, the purpose of this talk is to cover API wrappers, which is kind of what I do on Sunday afternoons when I’m not busy editing my Avatar football team. I’ve created a lot of API wrappers, starting with an application we called Tweet Congress. This app functioned as a directory and aggregator for all congressional U.S. Senate and House tweeters.

00:01:14.720 The project compiled all their tweets so that users could easily find their congressman on Twitter. We were submitting a lot of patch requests to John Maker, who created the Twitter gem, and I think my frequent updates became a bit of a nuisance for him. So, he suggested that I take over the maintenance of the Twitter gem since I seemed to be more interested in it than he was. This eventually led me to write the LinkedIn gem.

00:01:41.440 I've also worked on a project called Octopussy, which dealt with wrapping various other APIs. However, I didn't like how they handled the GitHub API, which is where my focus shifted. The truth is, we need API wrappers because they simplify the process of interacting with APIs.

00:01:58.560 Before discussing why we need them, let's remember that as Ruby developers, we prefer not to deal with low-level libraries. Curl is a great way to interact with APIs, where you can just pass the URL and get some JSON or XML back. However, in Ruby, we want something that feels more idiomatic.

00:02:28.960 This desire for idiomatic access has been highlighted in a post by Chad Fuller, written back in 2007. It sparked a big debate between various APIs at the time. What I mean by idiomatic access is that when we receive a JSON response, we want it to transform cleanly into a Ruby hash. Many JSON responses use camel casing for keys, which while usable, can feel uncomfortable in Ruby. Therefore, we want our Ruby hashes to use underscore notation.

00:03:40.720 As a result, I’ll show you some methods later that help achieve this conversion. One of the benefits of creating an API wrapper is that you can present the data in a way that feels more natural to the language consuming it. So we take those camel case keys and transform them into underscores, which seems to be the convention in the Ruby world. This shift enhances readability, especially since services in PHP or Java often expose keys in camel case.

00:04:08.160 Additionally, we can modify method names when exposing API features. For example, if we have an API method called 'getInfoByEmail', we could instead expose it as 'infoByEmail', which feels more idiomatic. We also incorporate syntactic sugar into our Ruby code, which allows for method chaining. Let’s look at an example from the Twitter gem: we can build a search query by chaining method calls together.

00:05:10.080 This lets us compose complex HTTP requests more simply. Furthermore, the Remixer gem serves as a wrapper for the Best Buy API, facilitating method chaining. In RESTful APIs, we often think of data being returned as JSON, but that term can be quite broad. Each API can have its unique structure, such as Best Buy’s complex URL composition to return particular product listings.

00:06:23.840 In fact, they allow you to reverse query results and approach product searches from multiple angles. This complexity can be simplified using Ruby, allowing you to construct those requests more naturally and programmatically. Let’s touch on another method I recently stumbled upon with Hayes Davis, a local Rubyist who created a Twitter client called Grapple.

00:07:43.200 His client implements method chaining to construct URLs that align with the Twitter API. He maintains the typical API endpoint structure but enhances it with Ruby syntax to post updates easily. Different architectures and methodologies are essential when building an API wrapper. For instance, Michael Bleight from Intridia created a Twitter API authentication wrapper that communicates directly on behalf of the authenticated user.

00:09:09.680 He utilizes a method to wrap the API calls, passing in the URL endpoint and the tweet's content seamlessly. On the flip side, the Grackle gem takes a different approach by masking the underlying HTTP calls to streamline user interactions with the Twitter API. By abstracting the API complexities, users can focus on functionality without concerning themselves with the intricacies of HTTP requests.

00:10:01.920 Now, why would we wrap APIs like this? The primary goal is to insulate against API changes. Instead of your client code breaking due to changes on the server side, a wrapper can provide stability and allow for seamless integration. You can leverage existing API documentation, ensuring that the client-side mirrors the server-side setup, thus improving usability.

00:11:10.720 Now let's talk about the tools used to create these wrappers. The transport layer primarily involves HTTP in Ruby. However, working with regular HTTP can be tedious due to its synchronous nature. You might find yourself making numerous requests, leading to lag in your application's responsiveness. This can be countered with libraries like Patron or HTTPClient, which allow for asynchronous requests.

00:12:25.560 Patron, in particular, allows you to perform multiple requests simultaneously, thus improving performance. In the Ruby community, we also have EventMachine, which offers an alternative threading model for handling requests asynchronously. It’s essential to choose a parsing layer carefully as well. If an API returns XML, you might need to convert it into JSON for better integration with Ruby.

00:13:27.520 Tools like Crack and ActiveSupport are excellent for parsing data, while libraries like Oj can help optimize your parsing further and give you significant performance benefits. At one point, we switched to Oj for our Twitter gem due to its speed. However, some users have reported compatibility issues, particularly on Windows.

00:14:25.840 So we came up with multi-json, which provides a unified interface to handle multiple JSON parsers and improve compatibility across different Ruby environments. When writing an API wrapper, high-level libraries are usually preferable, especially if you intend to interact with APIs frequently. A great example of this is HTTParty, which simplifies making API calls considerably.

00:15:34.960 HTTParty provides a straightforward way for you to include methods for common HTTP requests like GET and POST by turning your classes into HTTP clients. This can also be useful for handling default parameters, which is especially vital for APIs requiring authentication tokens with each call.

00:16:30.800 Following the structure of HTTParty can also help to avoid the serialization problem that arises from standard HTTP libraries. The sample code for using HTTParty illustrates how you can create a nice wrapper for your API interactions.

00:17:31.200 Moving on, Faraday is another excellent option that provides a DSL for composing wrappers, supports nested resources, and allows for middleware options, which can be really handy for specific requirements within your APIs.

00:18:49.720 Additionally, it integrates easily with testing frameworks and provides support for asynchronous calls—this is particularly useful for modern web applications that scale better with non-blocking operations. The parallelization of requests is crucial for speeding up interactions with APIs, especially for integrating multiple services.

00:19:59.520 Testing your API wrappers is as crucial as writing them. I recommend using tools like FakeWeb or VCR to help simulate API responses without hitting the actual API. This means you can run your tests without worrying about the API's state, and you can control and validate the responses that your code is working with.

00:21:09.760 When it comes to more complex testing scenarios, consider integrating rack applications that can mirror your API setup for more thorough testing. Utilizing libraries such as ShamRack can help avoid unnecessary calls to production APIs during development.

00:22:25.440 Finally, we have to discuss authentication. This can be one of the most challenging aspects of creating an API wrapper, mainly because different APIs have different authentication schemes. Basic authentication is straightforward but not secure, while OAuth provides a higher level of security but can be quite complicated.

00:23:43.920 Many APIs have moved to OAuth 2.0, which offers better security through various flows tailored for different applications and devices, although implementation still requires careful handling.

00:25:00.000 Twitter's API, for instance, has embraced OAuth, but don't expect a seamless upgrade path if you're already integrated with the previous version. It often requires significant refactoring to adapt to their new structures, which presents both challenges and opportunities for development.

00:27:00.880 Now, as we wrap up this talk, do you have any questions about any part of creating an API wrapper or about specific tools? Feel free to ask!

LoneStarRuby Conf 2010