What's Love Got To Do With It? Ruby and Sentiment Analysis

by Ben Greenberg

In the engaging presentation titled "What's Love Got To Do With It? Ruby and Sentiment Analysis," Ben Greenberg explores the intersection of linguistics, Ruby programming, and natural language understanding. The session highlights the importance of analyzing human language through sentiment analysis and demonstrates its application within a Ruby application.

Key points discussed include:
- Introduction to Linguistics and Its Importance: Greenberg begins by emphasizing that language shapes our thoughts and perceptions, illustrated through an example comparing English and Hebrew.
- Understanding Natural Language Processing (NLP): He defines NLP and sentiment analysis as tools to analyze language similarly to human understanding.
- Challenges of Sentiment Analysis: Various linguistic challenges are discussed, such as polarity, context, and contradictory emotions that complicate machine interpretation of human language.
- Practical Application: Greenberg introduces a project where attendees will build a Ruby app that sends WhatsApp messages to gauge the mood of news headlines related to specific topics.
- API Integration: He explains how to incorporate multiple APIs: the News API for gathering headlines, IBM Watson for sentiment analysis, and Nexmo's Messages API to relay results.
- Building the App: The session outlines technical details from setting up the gem file to defining routes and methods for processing incoming messages and conducting sentiment analysis through the app.
- Final Results: An example is presented by analyzing the sentiment of news about Ruby, illustrating a positive sentiment of 70% joy.

Greenberg's session effectively combines theoretical learning with practical programming, demonstrating how sentiment analysis can provide insights into the emotional landscape of news media. The main takeaway is the potential of Ruby and NLP in creating meaningful applications that bridge technology and human language understanding.

00:00:12.080 Good afternoon, everyone! How are you doing? That was very subdued. That's better! Everyone had coffee or tea? They're feeling ready for the middle of the day! It's awesome! This is going to be the best session of the hour.

00:00:32.009 We're going to spend some time talking about the tension between these two statements that are in front of you. We're not going to address these statements in particular, but they represent, in many ways, a dialectic between how we understand human language and what it's meant to be. This will be our objective today as we discuss Ruby and sentiment analysis.

00:00:58.170 In this talk, we're going to cover a few things. First, I will introduce some high-level ideas around linguistics and explain why studying language is important. Then, we'll delve into natural language processing and sentiment analysis. Finally, we're going to build a small but really cool Ruby app that ties together this technology in a way that's relevant to our world.

00:01:16.350 Before we get started, let me introduce myself. Hi everyone, I'm Ben! I really like Ruby; Ruby's awesome! I am also a co-editor of a weekly newsletter called 'Torah and Tech,' which combines Jewish insights and tech insights. It's very Jewish! This comes from my background as a second-career developer. I spent ten years as a rabbi and community organizer, working on issues like immigration reform and gun violence.

00:01:34.800 I have lived in various places where language is expressed differently and communication styles vary—from San Diego to New York, Boston, Denver, and now I live outside Tel Aviv in Israel. Currently, I am a Developer Advocate at Nexmo, the Vonage API platform, which is a cloud communications API provider. We are sponsoring RubyConf, and we have some awesome stuff. Be sure to check it out! We're even giving away a Nintendo Switch, which is really cool.

00:02:15.930 So now that that’s out of the way, let’s talk about why this matters. We’ll spend some time exploring sentiment analysis. What can we build with it? After that, we will actually build it. Sound good? Yes? Great!

00:02:44.970 So why does this matter? Let me give you an example. Despite what many may think, language is not simply a passive conduit or vehicle for information. In fact, language does a lot more than that. I want to present a small example from linguistics before we dive into the actual topic, as this concept is deeply intertwined with our discussion today.

00:03:08.690 Imagine a scenario around the water cooler at work, where we're all standing around in our suits talking about our evening. We might say something like, 'Last night, I had dinner with someone. It was delicious.' What are we conveying in that sentence? It seems simple, right? I did something last night—I had dinner. Who did I have dinner with? Someone. Do I know anything about that someone? No, presumably not. It was just someone, and dinner was delicious—what was delicious? Dinner! That’s English for you!

00:03:51.390 Now let’s take that same sentence in another language. In Hebrew, you might say: 'אכלתי ערב לאכול עם מישהי שהיתה טעים.' (I had dinner with a female someone, and she was delicious.) In English, we didn’t assess the gender, but in Hebrew, gendered speech presents a challenge because 'dinner' itself is a feminine word. So when you say, 'It was delicious,' you're grammatically referring to the feminine form, indicating that dinner is not 'it,' but 'she.' Hence, this reveals that language actually makes you think about certain things differently—in this case, gender! Language constructs the paths of our thoughts around what's important.

00:04:55.240 It's just one small snippet from the field of linguistics that underscores the importance of studying and understanding language. However, we must take it to the next level. How can we apply this understanding when working with machines? How can we start grasping language’s nuances with computers? This is where sentiment analysis comes into play.

00:05:23.680 Sentiment analysis is a subset of natural language processing (NLP). Has anyone worked with NLP? A few hands? It's pretty cool, right? You get a lot of interesting data from it. The essence of natural language processing is to reach a point where we can analyze human language with the same effectiveness as humans do.

00:05:54.790 Alan Turing optimistically declared back in the 1900s that we would eventually reach a point where it wouldn’t even be questioned whether computers can think. However, we are not there yet. Current NLP processors can analyze human language at a similar success rate to humans analyzing it. We know this through comparative studies with humans, as it's crucial to measure these constructs based on human behavior.

00:06:40.340 To illustrate sentiment analysis, let’s take the example sentence: 'Last night, I had dinner with someone. It was delicious.' The data returned from analyzing this might suggest that it is 97% positive. The subject is 'I,' and the object is 'dinner.' The algorithm segments and processes the sentence, identifying tenses, key words, and categories like 'food and drink.' Overall, this gives you a high-level perspective of how text analysis works through NLP.

00:07:28.050 Then, there are linguistic challenges such as parsing sentences with different meanings based on structure. For instance, the sentence 'Police helps dog bite victim' has a formal structure that requires analysis. Different languages utilize different structures and grammar rules, which means understanding these variances is critical to process emotions correctly. Parsing sentiment also requires us to consider emotionality, which remains a challenge for sentiment analysis.

00:08:12.110 One example of a challenge is polarity. Polarity refers to the extremes of sentiment; a statement could express extreme happiness or significant displeasure. For example: 'I dislike broken cars' indicates anger; 'I truly love going out in this weather' could either convey excitement or sarcasm, which requires context—something that's difficult for current machines to interpret.

00:08:48.240 The last example, 'I love my cell phone,' is particularly challenging because it reflects contradictory human emotion. Such contradictions can confuse sentiment analyzers as there's no clear categorization. We often state opposing feelings, which is a fundamental aspect of human speech yet remains perplexing for AI sentiment analysis.

00:09:15.940 With this brief overview, you are now experts in natural language processing and sentiment analysis! Now let’s discuss what we can actually create with this technology. Something that bothers me is how overwhelming my news alerts are. I tend to look at my phone a lot, especially when waking up, which often becomes overwhelming with all the topics I see.

00:09:44.080 There are times when I just want to gauge the mood of the news for the day. Am I waking up to a great day, or should I pull the covers over my head and go back to sleep? How can we achieve this? Well, my friends, we can do that with natural language processing and emotional analysis!

00:10:09.600 We're going to build an app that allows users to send a message via WhatsApp. The message will include a topic, such as asking for the mood of the news on a specific subject, like 'What’s the mood of the news on Nashville weather today?' I will scour the entire internet for news headlines related to Nashville weather, send that data to an API for analysis, and receive feedback on the news’s overall mood.

00:10:43.140 I won’t have to read through headlines—I can skip that part! I’m busy and have things to do, like catch up on the latest series of The Crown. So how does it work? We’ll use three APIs: the News API to gather headlines, IBM Watson for analysis, and Nexmo's Messages API to send the consolidated results back via WhatsApp.

00:11:00.820 All the APIs are free to access initially, which is essential, and I love good documentation. If I can’t set it up quickly, I'm out of there. Additionally, since I enjoy Ruby, we’ll be using Ruby gems in our app. Let’s define our gem file.

00:11:31.570 In our gem file, we’ll need to add the Nexmo Rails gem, which serves as a Rails initializer for the Nexmo Ruby gem. We’ll also include the News API gem and the IBM Watson gem. Plus, for good practice, we will manage environment variables via 'dotenv' in Rails. Lastly, since who doesn’t love emojis, we’ll add parsing functions for them as well!

00:12:09.850 We only need to establish two routes for our application: one for inbound webhooks, and the other for status updates. For those that work with webhooks, this allows an API to send you data without needing to ping for updates repeatedly. You simply provide an externally accessible URL for the API to hit, allowing them to reach your application.

00:12:58.960 In terms of environment variables, we'll include API keys, our WhatsApp number, and any other necessary secrets. Most of the processing will occur within our controller, where we'll create a News Controller with methods for handling inbound messages and retrieving status.

00:13:19.180 When a message is received from WhatsApp, we need to assign the incoming data to variables so we can later manipulate it. We'll extract the topic from the message as well as the recipient’s number, which is necessary to send a response back.

00:13:51.830 With the topic established, we will define a method called 'analyze_headlines' that will fetch and analyze the news based on the user’s query. This method will interact with the News API to gather relevant headlines, even though we haven’t created the actual method yet—we’re opting for optimism here.

00:14:30.760 Once we have the headlines, we will need to create an instance of the IBM Watson Natural Language Processor, providing it with the required API key and version details. It’s crucial to the process since this allows us to analyze the gathered headlines for sentiment, keywords, and sentiments—all vital information for determining how our topic resonates with the current news landscape.

00:15:39.720 We can invoke the analyze method, specifying the features we want: entities, keywords, sentiment, categories, and emotions. These details will help us to gain a complete understanding of the sentiment surrounding the topic. Additionally, we will output the results into the Rails console for further insights.

00:16:09.250 Next, we’ll gather the headlines required for our analysis. Since it won’t be human-readable right away, it’s just a large text string. We want to sort these headlines by popularity as they will shape the mood of the news based on what resonates most with readers.

00:16:41.440 Once we collect the relevant headlines, we will send a message that provides the user with an overall sentiment regarding the news. We will assess how much joy or sadness is associated with the news topic. Creating this message involves compiling all the emotions captured during the analysis and returning it as a concise summary for the user.

00:17:07.600 Once the message is formed, we will generate a JWT token to authenticate our WhatsApp requests. After all this setup, we will issue a request to send the message back to the user. Now, I want to see what the mood of the news is regarding Ruby!

00:17:46.740 By sending this request through all the APIs, we should receive a response within a few seconds. The mood of the news about Ruby today is overwhelmingly positive, with a score of 70% joy! We, as a Ruby community, are joyful folks.

00:18:24.300 Did you know that the number one emoji used on GitHub for Ruby projects is a heart? We love each other and we’re good people! Overall, it seems we have a very positive sentiment towards Ruby. Thank you very much!

What's Love Got To Do With It? Ruby and Sentiment Analysis

What's Love Got To Do With It? Ruby and Sentiment Analysis

RubyConf 2019