00:00:06.549
Hello, everyone! We're glad to be here. This is actually my first talk ever, and I tend to get excited, which means I often speak quickly. So, if I get too fast, just shout out for me to slow down. I reviewed our title and thought it was quite long, so I consulted with my team, and we decided to go with a more concise title today. Our topic is managing complexity through Domain-Specific Languages (DSLs) and a bit of metaprogramming.
00:00:40.640
A little bit about myself - I'm Melvrick Goh from Singapore, which is just a short hop away. If you ever visit, you can stop by for some great food, or you might even have heard of the show 'Crazy Rich Asians' which is set in Singapore. Just a disclaimer: most of us are not rich, but we might be a little crazy. I work at a loyalty technology company called a Jeanette, and later, I'll share more about that.
00:01:06.300
For some context, I work alongside a talented group of engineers, some of whom are here today. Today’s discussion centers around the collective experiences our team has had while tackling complexity. I also enjoy reading and running, mostly to stay fit and enjoy delicious Pad Thai.
00:01:39.260
Speaking of movies, has anyone seen 'John Wick'? I was watching the latest installment while flying here and noticed something interesting. In 'John Wick 3', he’s back for a third time, still trying to stay alive. I've watched the first movie, the second, and now the third, and it's always the same plot: he's got to survive against overwhelming odds.
00:02:15.659
This brings us to today's topic: have you ever worked on projects that come in different forms, much like how John Wick faces a variety of enemies? Even though the settings change, the overall objective remains the same. Similarly, our projects have unique contexts, yet the challenges often mirror each other. Today, we’ll explore how DSLs can aid us in addressing the problems we face.
00:02:46.780
To give you the necessary context, let’s dive into an example. Imagine I want to redeem my points to fly on Thai Airways and need to exchange my Amex points for Thai Airways points. Our product essentially facilitates such loyalty transfers between various banks and airlines. Picture this multiplied by 20x – that’s the scale we are dealing with.
00:03:30.670
However, this scale introduces challenges in development and maintenance, especially when working within a sizeable development team. Each integration feels unique, much like how John Wick tackles each fight differently. Every integration has its peculiarities, and this diversity is both a blessing and a challenge.
00:03:38.910
As we progress at a project level, we often encounter new connections, migrations, and enhancements. Businesses always want changes delivered yesterday, which puts pressure on our teams. Sometimes, we struggle to keep up with the demands at scale, and it becomes crucial to standardize our approach for the sake of efficiency.
00:04:40.150
Here’s a glimpse of how our content might appear, like a simplified CSV file containing essential pieces of information. When exchanging points, you need to include your membership ID for a loyalty program. The integrations vary widely, and we handle everything—CSV files, spreadsheets, JSON formats, and API interactions. Everybody loves RESTful interfaces, right?
00:06:37.000
I want to show you an example of a JSON request body which highlights some of the challenges we face. For instance, some partner systems cannot handle HTTP status codes or expected headers, so they require everything to be included in the body. There are other inconsistencies, like receiving different response formats—sometimes JSON, other times plain text.
00:07:48.300
An essential point is how we manage customization and configuration. Each project must be attentive to unique requirements while minimizing duplication of effort. Our aim was to standardize the approach to integrations so that we focus on what’s truly custom or unique. In many cases, what’s consistent across integrations can often be distilled into a configuration format.
00:09:25.480
As an analogy, think of the character John Wick again. When faced with different integrations, we want our team to respond as quickly and flexibly as he does, regardless of the unique challenges presented. Consistency allows us to effectively simplify the domain complexity encountered in our work.
00:10:00.230
To distill the essence of what we face, I’ve outlined three main areas that need addressing to achieve consistency. These include the data formats we handle, the business logic involved, and the mechanism for delivery itself. These elements combine to contribute to an effective integration strategy.
00:10:35.250
Another major aspect of our approach is the challenge of customizations. Our configuration can address specific needs, from encryption standards to timezone specifications. The goal is to standardize what's common while ensuring each integration remains adaptable to unique specifications.
00:12:38.860
Now, let’s explore how our DSL looks. The DSL was designed to streamline our high-level configuration, simplifying the complexities we encounter. By consolidating much of the boilerplate code into a few lines, we enable developers to focus on customizations instead of reinventing the wheel.
00:13:50.080
Taking parts of our system to simplify further, our builder facilitates the dynamic aspect of the configuration. If a specific partner requires a custom solution, the builder helps define that in a concise way, reducing code redundancy while allowing for individualized configurations.
00:14:34.040
As we wrap up, it’s important to note that despite the flexibility and efficiency our DSL brings, challenges remain. The underlying code can be harder to modify, especially for newer developers. However, good test coverage ensures that even with changes, we can maintain functionality across our diverse integrations.
00:17:29.119
In summary, using a DSL has significantly sped up our development, simplifying testing as we’ve extracted common components. Newcomers find it easy to join and contribute. However, modifications can still be tricky for newer engineers, especially as this constructs the foundation of our entire integration architecture. Thank you for your attention and participation today!