Lightning Talk: Synthetic Focus Groups - Predicting Human Behavior

by Lee McAlilly

In this lightning talk presented by Lee McAlilly at the Blue Ridge Ruby 2024 event, the focus is on using AI for predicting human behavior, specifically through the use of synthetic focus groups. McAlilly works at an agency specializing in political campaigns and consumer marketing, where he aims to enable non-technical personnel to utilize AI effectively. He discusses the challenges marketers face with traditional data analytics tools and introduces a multi-agent approach that utilizes multiple instances of GPT to simulate human respondents in surveys.

Key points discussed include:
- Multi-Agent Approach: The idea of employing numerous instances of GPT to act as individual participants in surveys has garnered attention, thanks to researchers like John Horton from MIT, who has been validating aspects of behavioral economics within this model.
- Expected Parrot: Horton has created a Python-based domain-specific language for conducting synthetic surveys, which McAlilly believes could also be effectively implemented in Ruby.
- Ruby's Potential: Ruby is lauded for its ability to consolidate data from various sources and send it to AI models, which could make it a strong player in this new domain, particularly compared to Python.
- Synthetic Focus Groups: The concept of synthetic focus groups allows marketers to test ad strategies quickly without the financial burden and time lag of traditional A/B testing. McAlilly mentions that this method could lead to significant cost savings and faster results, highlighting potential increases in ad conversions.
- Emerging Technologies: The evolving landscape of AI tools indicates that technology firms like Facebook may develop similar analytical tools, but McAlilly stresses the need for open-source alternatives to avoid dependency on major corporations.
- Future of AI in Marketing: The advent of GPT-4 has made implementing such strategies feasible, pointing towards a broader recognition of its applications outside of coding.

In conclusion, McAlilly encourages exploration of this research area, emphasizing that synthetic surveys could revolutionize how marketers approach consumer feedback and decision-making processes. The insights from this talk reveal possibilities for cheaper and more efficient marketing strategies using advanced AI techniques.

00:00:12.240 Today, I want to discuss an aspect of AI that has me quite excited. I work at an agency that focuses on political campaigns and consumer marketing, and I'm one of the few tech specialists there. My goal is to find ways for the average person in our company to use AI effectively.

00:00:21.279 As developers, we clearly recognize the power of AI, but for marketers, engaging with Google Analytics or similar data can be somewhat unappealing. About six months ago, I had an idea: what if we employed a multi-agent approach, spinning up numerous instances of GPT that act as individual respondents, and tasked them with taking surveys?

00:00:34.719 It turns out, I wasn't alone in that thought. A researcher at MIT, economist John Horton, has been examining this concept for the past couple of years. He's started to demonstrate that some classic laws of behavioral economics—more than just theories—hold true when applied to a multi-agent GPT model. This approach effectively applies to classical auction theory, for instance.

00:01:07.560 Horton has designed several experiments that validate this concept. He has created a domain-specific language (DSL) for synthetic surveys called Expected Parrot, which is built in Python. If you're interested, it's worth taking a look at. However, I believe Ruby is an excellent programming language for this kind of work because it excels at funneling data from various sources—whether it be from Facebook custom audiences or Google Analytics data—and then sending that information to the OpenAI API or whichever model you prefer.

00:01:44.159 You might be familiar with LangChain. It’s crucial for the Ruby community to engage more with this type of work instead of leaving it largely to Python developers. Earlier, Daniel demonstrated the Assistant API, which illustrates how to implement this with the OpenAI API.

00:02:13.840 It's important to note that our goal is not to perfectly mimic human behavior. For example, if one were to analyze ad conversions on Facebook, the platform will likely roll out enhancements to their advertising tools. However, if you could pre-test various headlines for your ads—like through synthetic focus groups—you could quickly assess different metrics.

00:02:30.720 Even a 20% to 30% increase in conversions using this method would be significantly cheaper and faster compared to traditional A/B testing on Facebook ads. A/B tests require running actual campaigns where you'd need to wait a week to obtain a statistically significant sample and spend money accordingly. This alternative is a fraction of the cost and time, and those incremental improvements can compound into substantial gains for a business.

00:03:06.480 I anticipate that Facebook will eventually develop a similar tool, but we also need open-source alternatives. We don’t want to be dependent on a single company's tools. The implications of this approach extend across various fields.

00:03:39.040 I encourage you to explore the research around this area. That's my company, and I'll leave you with a quote from the CEO of Box: I believe this idea is among the first applications of GPT outside coding. It truly wasn't feasible until the advent of GPT-4. Moreover, its potential is being recognized now, as emerging papers begin to validate its effectiveness.

00:04:08.040 Thank you.