00:00:12.080
Good afternoon, everyone! How are you doing? That was very subdued. That's better! Everyone had coffee or tea? They're feeling ready for the middle of the day! It's awesome! This is going to be the best session of the hour.
00:00:32.009
We're going to spend some time talking about the tension between these two statements that are in front of you. We're not going to address these statements in particular, but they represent, in many ways, a dialectic between how we understand human language and what it's meant to be. This will be our objective today as we discuss Ruby and sentiment analysis.
00:00:58.170
In this talk, we're going to cover a few things. First, I will introduce some high-level ideas around linguistics and explain why studying language is important. Then, we'll delve into natural language processing and sentiment analysis. Finally, we're going to build a small but really cool Ruby app that ties together this technology in a way that's relevant to our world.
00:01:16.350
Before we get started, let me introduce myself. Hi everyone, I'm Ben! I really like Ruby; Ruby's awesome! I am also a co-editor of a weekly newsletter called 'Torah and Tech,' which combines Jewish insights and tech insights. It's very Jewish! This comes from my background as a second-career developer. I spent ten years as a rabbi and community organizer, working on issues like immigration reform and gun violence.
00:01:34.800
I have lived in various places where language is expressed differently and communication styles vary—from San Diego to New York, Boston, Denver, and now I live outside Tel Aviv in Israel. Currently, I am a Developer Advocate at Nexmo, the Vonage API platform, which is a cloud communications API provider. We are sponsoring RubyConf, and we have some awesome stuff. Be sure to check it out! We're even giving away a Nintendo Switch, which is really cool.
00:02:15.930
So now that that’s out of the way, let’s talk about why this matters. We’ll spend some time exploring sentiment analysis. What can we build with it? After that, we will actually build it. Sound good? Yes? Great!
00:02:44.970
So why does this matter? Let me give you an example. Despite what many may think, language is not simply a passive conduit or vehicle for information. In fact, language does a lot more than that. I want to present a small example from linguistics before we dive into the actual topic, as this concept is deeply intertwined with our discussion today.
00:03:08.690
Imagine a scenario around the water cooler at work, where we're all standing around in our suits talking about our evening. We might say something like, 'Last night, I had dinner with someone. It was delicious.' What are we conveying in that sentence? It seems simple, right? I did something last night—I had dinner. Who did I have dinner with? Someone. Do I know anything about that someone? No, presumably not. It was just someone, and dinner was delicious—what was delicious? Dinner! That’s English for you!
00:03:51.390
Now let’s take that same sentence in another language. In Hebrew, you might say: 'אכלתי ערב לאכול עם מישהי שהיתה טעים.' (I had dinner with a female someone, and she was delicious.) In English, we didn’t assess the gender, but in Hebrew, gendered speech presents a challenge because 'dinner' itself is a feminine word. So when you say, 'It was delicious,' you're grammatically referring to the feminine form, indicating that dinner is not 'it,' but 'she.' Hence, this reveals that language actually makes you think about certain things differently—in this case, gender! Language constructs the paths of our thoughts around what's important.
00:04:55.240
It's just one small snippet from the field of linguistics that underscores the importance of studying and understanding language. However, we must take it to the next level. How can we apply this understanding when working with machines? How can we start grasping language’s nuances with computers? This is where sentiment analysis comes into play.
00:05:23.680
Sentiment analysis is a subset of natural language processing (NLP). Has anyone worked with NLP? A few hands? It's pretty cool, right? You get a lot of interesting data from it. The essence of natural language processing is to reach a point where we can analyze human language with the same effectiveness as humans do.
00:05:54.790
Alan Turing optimistically declared back in the 1900s that we would eventually reach a point where it wouldn’t even be questioned whether computers can think. However, we are not there yet. Current NLP processors can analyze human language at a similar success rate to humans analyzing it. We know this through comparative studies with humans, as it's crucial to measure these constructs based on human behavior.
00:06:40.340
To illustrate sentiment analysis, let’s take the example sentence: 'Last night, I had dinner with someone. It was delicious.' The data returned from analyzing this might suggest that it is 97% positive. The subject is 'I,' and the object is 'dinner.' The algorithm segments and processes the sentence, identifying tenses, key words, and categories like 'food and drink.' Overall, this gives you a high-level perspective of how text analysis works through NLP.
00:07:28.050
Then, there are linguistic challenges such as parsing sentences with different meanings based on structure. For instance, the sentence 'Police helps dog bite victim' has a formal structure that requires analysis. Different languages utilize different structures and grammar rules, which means understanding these variances is critical to process emotions correctly. Parsing sentiment also requires us to consider emotionality, which remains a challenge for sentiment analysis.
00:08:12.110
One example of a challenge is polarity. Polarity refers to the extremes of sentiment; a statement could express extreme happiness or significant displeasure. For example: 'I dislike broken cars' indicates anger; 'I truly love going out in this weather' could either convey excitement or sarcasm, which requires context—something that's difficult for current machines to interpret.
00:08:48.240
The last example, 'I love my cell phone,' is particularly challenging because it reflects contradictory human emotion. Such contradictions can confuse sentiment analyzers as there's no clear categorization. We often state opposing feelings, which is a fundamental aspect of human speech yet remains perplexing for AI sentiment analysis.
00:09:15.940
With this brief overview, you are now experts in natural language processing and sentiment analysis! Now let’s discuss what we can actually create with this technology. Something that bothers me is how overwhelming my news alerts are. I tend to look at my phone a lot, especially when waking up, which often becomes overwhelming with all the topics I see.
00:09:44.080
There are times when I just want to gauge the mood of the news for the day. Am I waking up to a great day, or should I pull the covers over my head and go back to sleep? How can we achieve this? Well, my friends, we can do that with natural language processing and emotional analysis!
00:10:09.600
We're going to build an app that allows users to send a message via WhatsApp. The message will include a topic, such as asking for the mood of the news on a specific subject, like 'What’s the mood of the news on Nashville weather today?' I will scour the entire internet for news headlines related to Nashville weather, send that data to an API for analysis, and receive feedback on the news’s overall mood.
00:10:43.140
I won’t have to read through headlines—I can skip that part! I’m busy and have things to do, like catch up on the latest series of The Crown. So how does it work? We’ll use three APIs: the News API to gather headlines, IBM Watson for analysis, and Nexmo's Messages API to send the consolidated results back via WhatsApp.
00:11:00.820
All the APIs are free to access initially, which is essential, and I love good documentation. If I can’t set it up quickly, I'm out of there. Additionally, since I enjoy Ruby, we’ll be using Ruby gems in our app. Let’s define our gem file.
00:11:31.570
In our gem file, we’ll need to add the Nexmo Rails gem, which serves as a Rails initializer for the Nexmo Ruby gem. We’ll also include the News API gem and the IBM Watson gem. Plus, for good practice, we will manage environment variables via 'dotenv' in Rails. Lastly, since who doesn’t love emojis, we’ll add parsing functions for them as well!
00:12:09.850
We only need to establish two routes for our application: one for inbound webhooks, and the other for status updates. For those that work with webhooks, this allows an API to send you data without needing to ping for updates repeatedly. You simply provide an externally accessible URL for the API to hit, allowing them to reach your application.
00:12:58.960
In terms of environment variables, we'll include API keys, our WhatsApp number, and any other necessary secrets. Most of the processing will occur within our controller, where we'll create a News Controller with methods for handling inbound messages and retrieving status.
00:13:19.180
When a message is received from WhatsApp, we need to assign the incoming data to variables so we can later manipulate it. We'll extract the topic from the message as well as the recipient’s number, which is necessary to send a response back.
00:13:51.830
With the topic established, we will define a method called 'analyze_headlines' that will fetch and analyze the news based on the user’s query. This method will interact with the News API to gather relevant headlines, even though we haven’t created the actual method yet—we’re opting for optimism here.
00:14:30.760
Once we have the headlines, we will need to create an instance of the IBM Watson Natural Language Processor, providing it with the required API key and version details. It’s crucial to the process since this allows us to analyze the gathered headlines for sentiment, keywords, and sentiments—all vital information for determining how our topic resonates with the current news landscape.
00:15:39.720
We can invoke the analyze method, specifying the features we want: entities, keywords, sentiment, categories, and emotions. These details will help us to gain a complete understanding of the sentiment surrounding the topic. Additionally, we will output the results into the Rails console for further insights.
00:16:09.250
Next, we’ll gather the headlines required for our analysis. Since it won’t be human-readable right away, it’s just a large text string. We want to sort these headlines by popularity as they will shape the mood of the news based on what resonates most with readers.
00:16:41.440
Once we collect the relevant headlines, we will send a message that provides the user with an overall sentiment regarding the news. We will assess how much joy or sadness is associated with the news topic. Creating this message involves compiling all the emotions captured during the analysis and returning it as a concise summary for the user.
00:17:07.600
Once the message is formed, we will generate a JWT token to authenticate our WhatsApp requests. After all this setup, we will issue a request to send the message back to the user. Now, I want to see what the mood of the news is regarding Ruby!
00:17:46.740
By sending this request through all the APIs, we should receive a response within a few seconds. The mood of the news about Ruby today is overwhelmingly positive, with a score of 70% joy! We, as a Ruby community, are joyful folks.
00:18:24.300
Did you know that the number one emoji used on GitHub for Ruby projects is a heart? We love each other and we’re good people! Overall, it seems we have a very positive sentiment towards Ruby. Thank you very much!
