A Plan towards Ruby 3 Types

Talks

Yusuke Endoh

A Plan towards Ruby 3 Types

by Yusuke Endoh

In this talk presented at Euruko 2019, Yusuke Endoh, a full-time Ruby committer at Cookpad, explores the plans for implementing static types in Ruby 3, primarily focusing on the development of a tool called Type Profiler. The motivation behind Ruby 3's static type feature is to enhance the programming experience while also identifying potential type bugs without imposing the need for type annotations on developers. He emphasizes that maintaining the excellent Ruby programming experience is a priority for the Ruby development community.

Key points discussed include:
- Ruby 3 Goals: The main goals for Ruby 3 are performance improvement, enhanced concurrency, and the introduction of static analysis through type checking.
- Static Analysis Overview: Endoh introduces a three-pronged approach to static type analysis, which includes:
- A standardized type signature format that resembles Ruby but is used to indicate method types.
- A Level 1 type checker that infers types without requiring explicit signatures and identifies potential bugs.
- A Level 2 type checker that verifies code against defined signatures, facilitating deeper error checking but requiring more complexity.
- Type Profiler: The Type Profiler is an experimental tool that will allow developers to analyze Ruby code statically. It uses abstract interpretation techniques to evaluate method types and identify potential errors.
- Implementation Challenges: Endoh discusses challenges faced in the development of the Type Profiler, particularly concerning analysis time for complex applications and ensuring comprehensive test coverage.
- Future Direction: The Type Profiler is still in development, focusing on supporting more Ruby features and enhancing its type checking capabilities while ideally requiring minimal changes from developers.

Endoh concludes by inviting feedback and collaboration from the community as the project progresses, underscoring the open-source nature of the Type Profiler and the Ruby Signature proposals, ultimately aiming to blend effective static analysis within Ruby's flexible design framework.

00:00:06 What we have now is a talk by a full-time Ruby committer, Yusuke Endoh, who works at Cookpad. In that capacity, he enjoys working alongside Yukihiro Matsumoto on at least a monthly basis. I've heard that he is a very good programmer, and to be precise, a brilliant programmer. It's wonderful to have him on stage with us today. He is also the winner of the International Obfuscated C Code Contest, so you could say he never gets seasick. Let's give a fresh round of applause for Yusuke!

00:00:51 Thank you. My name is Yusuke Endo, and I come from Japan. Today, I will be discussing plans toward Ruby 3 types. Unfortunately, I will not be talking about the International Obfuscated C Code Contest, but I will delve into Ruby 3. I work at Cookpad as a full-time Ruby committer, where my job involves developing MRI, the mature interpreter.

00:01:23 Cookpad employs two full-time Ruby committers: myself and Koichi Sasada, who you may know as the person making Ruby faster. I want to thank Cookpad for sending me to this event, and briefly introduce our company. We provide services with a mission to make everyday cooking fun. Our biggest service is Cookpad.com, which is a sharing platform where users can submit their original recipes and also find reviews from others. We have around 90 million monthly average users across the world and currently support 29 languages in 72 countries.

00:02:06 This service is still growing with the aim to become the number one cooking site in 100 countries. As I mentioned, Cookpad is built using Ruby on Rails, and to develop and maintain this large-scale service, we need many great Ruby engineers. We're hiring! Although I'm working in Japan, Cookpad's headquarters is in Bristol, United Kingdom, so feel free to reach out to me if you're interested.

00:02:51 In Cookpad, I am developing Ruby, and my main contributions include the implementation of keyword arguments in Ruby 2.0, testing coverage measurement features, and a de-facto standard benchmarking program called Ruby 3C. Recently, I proposed, implemented, and committed 'endless' and 'beginningless' lens features in Ruby 2.6, which are set to be expanded in the 2.7 release this December. We are actively developing Ruby 3, which is planned for release in 2020.

00:04:07 As Matsumoto mentioned in his keynote, Ruby 3 has three primary goals: first, to improve performance such that Ruby 3 will be three times faster than Ruby 2.0. The second goal is to enhance concurrency, providing a better concurrency model than the traditional thread model. The third goal is the implementation of static analysis, which is my primary focus. Ruby 3 will provide a way to analyze Ruby programs statically. This goal has been the most uncertain of the three, but over the years, I have engaged in monthly discussions about it with Matsumoto and other contributors.

00:05:06 Recently, we have been able to articulate the goal more concretely. In this talk, I would like to introduce the current plans for Ruby's static types and how Ruby will handle static analysis. I will explain one of the components of this program, called Type Profiler, which I am currently developing.

00:06:02 Let's start with the plan. The objective of Ruby 3 types is to identify potential type bugs. This initiative aims to provide useful information for developers to write and debug Ruby code effectively. While type analysis can sometimes be utilized for performance improvement, that is not the main focus of our scope. One critical requirement is that we must maintain the excellent programming experience that Ruby offers. For example, we should not impose on users the need to write type annotations to facilitate type checking.

00:07:03 If users want to write annotations, we will not stop them—feel free to do so—but we want the default experience to be without Ruby being type-annotated. Based on our objectives and requirements, Ruby will provide three items for static analysis. The first is a standardized type signature format. The second item is a level one type checker, which can infer types without requiring signatures and report possible bugs. It can also suggest a prototype type signature for untyped Ruby code.

00:08:08 The third item is a level two type checker, which does require type signatures and verifies that the code complies with its signature. I will explain these three items one by one. The first is the type signature format, which resembles Ruby but is not Ruby code per se. This format complements Ruby code and indicates the types each method accepts and returns.

00:09:02 For example, a method could accept a block and return itself. This signature format supports various features essential for representing Ruby types, such as generics, mixing, optional types, and interfaces. You can find the current proposal for this format under the name Ruby signature.

00:10:03 The second component is the type checker that does not require signatures. This checker attempts to identify possible errors, including unresolved messages and incorrect argument types. It works even if the application does not have any signatures, reporting potential issues such as type errors.

00:11:02 There are two purposes for this item: the type profiler that I am developing, which I will explain later, and an experimental compiler developed by Emerald called MLRuby, which performs similarly to reduce long-term type-checking overhead. The level one type checker is expected to suggest prototypes of type signatures for untyped application code.

00:12:05 The level one type checking and type inference can be performed through an abstract interpretation technique. This approach allows Ruby to analyze code at the type level by identifying the types each method accepts and returns in a type signature format. The second component is the level two type checker, which represents more traditional static type checking.

00:12:51 This type checker requires signatures for each record and verifies that code complies with the defined signatures. It employs established techniques for gradual type checking. Currently, there are three proposals for level two checking, which involve signatures and method annotations. Despite the increased complexity, they offer deeper error checking compared to level one type checking.

00:13:38 Let me review the three items: First, we have a type signature format for Ruby code, which almost all libraries will need to bundle. Second, there's a level one type checker that produces possible type errors for both library and application code while suggesting type signatures for any untyped application code. Lastly, the level two type checker provides consistency between application code and its respective signature, reporting any inconsistencies as warnings.

00:14:59 There are various use cases for these tools: First, if you're not interested in static checking at all, you don't have to use anything. If you want to find possible errors but aren't focused on types, you can write code as usual and apply the level one type checking. If you prefer to verify your code more deeply, you can write typed code with prototype signatures and employ the level two checker.

00:15:48 All tools in the pipeline are under development, and all projects are open source, including Ruby Signature. Discussions are mainly led by Rust developers, but any comments regarding these proposals are welcome. The type profiler project is still experimental and has its own challenges. However, I believe it provides a rich way to analyze Ruby code effectively.

00:16:48 Type results can vary widely between implementations, so we need to address this and enhance functionality. For instance, in a scenario where multiple branches can lead to a significant increase in the states being analyzed, a technique called statementing can be implemented to simplify how we handle and represent these variables. For instance, we can summarize multiple variable states into broad categories rather than analyzing them comprehensively and exhaustively.

00:17:51 My earlier implementation faced issues with analysis time, taking too long for complex applications. Therefore, I spent significant time researching effective techniques from the field of abstract interpretation and symbolic execution. By adopting a more efficient algorithm and statementing, I improved the performance of type profiler significantly. Now, it only takes about 2.5 seconds to analyze the profiler's own code and 6 seconds for benchmarks with more complex Ruby applications.

00:19:00 However, my work with type profiler is not without challenges. For example, it requires comprehensive test coverage with method definitions to check for errors effectively. Without test cases, the inability to check methods means some issues might be overlooked. Despite meticulous typing sequences provided by type profile itself, we still need to enforce rigorous testing to catch edge cases.

00:20:06 In summary, while the advantages of type profiler abound, it also can produce excessive and long error suggestions for developers to manage, due to ambiguity in variable types. Furthermore, the system might sometimes struggle with selection of methods dynamically based on runtime information. Nonetheless, with diligent efforts going forward, we aim to enhance the type checking capabilities, ensuring it is aligned with Ruby's flexible and dynamic nature.

00:20:55 Finally, I would like to discuss the development status of type profiler. While the basic algorithms for analyzing Ruby code using abstract interpretation are laid out, the functionality is still in rapid development due to the complexity and variability of Ruby's features and syntax. Our current focus is to support more Ruby features, including built-in classes and complex argument patterns like optional and keyword arguments.

00:21:56 This project is still very much a work in progress, and although there are many tasks ahead, I am dedicated to continuing development on type profiler. In conclusion, I hope I have communicated the importance and utility of Ruby 3 types, the foundational elements towards achieving effective static analysis. I have presented the type profiler, which is integral for this purpose.

00:23:06 It is the only approach applicable to non-typed Ruby code based on abstract interpretation techniques. I genuinely hope to analyze Ruby code with minimal changes to the developer experience, retaining Ruby's powerful programming sensibilities. The type profiler project is open source, so I welcome any comments, suggestions, or collaborative ideas on enhancing its capabilities.

00:25:06 Thank you for your attention! Thank you very much! Here’s some treasure!

EuRuKo 2019