Search
Regex
Ten Unicode Characters You Should Know About as a Programmer
Edit
#unicode
#ruby
#internationalization-i18n
#regex
#character-encoding
#programming-best-practices
Summarized using AI

Ten Unicode Characters You Should Know About as a Programmer

Jan Lelis • December 01, 2017 • New Orleans, LA

In the talk titled "Ten Unicode Characters You Should Know About as a Programmer," Jan Lelis provides an insightful examination of Unicode, emphasizing its significance and complexities for programmers. The discussion is anchored in Ruby, showcasing how its robust support for Unicode facilitates the management of different characters and their corresponding code points.

Key Points Discussed:

- Unicode Overview: Unicode assigns a unique code point to every character, making it essential to understand that multiple code points can represent a single visual character. The Thai character is introduced as an example that illustrates this concept.

- Character Normalization: The demonstration of the character 'Ä' highlights issues with keyboard input and how Ruby's normalization feature allows for transforming characters composed of multiple code points into a single cohesive representation.

- Confusable Characters: Lelis discusses the confusion that arises between visually similar characters, like different versions of the letter 'I', especially in relation to Turkish capitalizing rules. Programmers face security risks due to these similarities, necessitating tools like the Unicode Confusable gem to identify and manage them effectively.

- Locale-aware Methods: Enhancements in Ruby (post-2.4) allow the use of locale-specific arguments in methods like 'upcase' and 'downcase', improving internationalization support for varying languages.

- Control Characters: The importance of understanding control characters such as newlines is emphasized, including how they behave differently in various operating systems and affect string manipulation in Ruby.

- Legal but Unassigned Code Points: Lelis explains the concept of non-assignable characters within Unicode, where certain code points, like 10FFFF, are legal but not assigned any character. Knowing this can help prevent potential application errors.

- Regular Expressions and Unicode Properties: The use of Unicode properties in Ruby's regular expressions enables developers to match unique and sometimes visually ambiguous characters effectively, enhancing string manipulation capabilities.

In conclusion, Jan Lelis highlights that understanding Unicode intricacies is crucial for programmers, especially when building robust applications. By recognizing confusable characters and learning how to handle them correctly in Ruby, developers can avoid potential pitfalls associated with Unicode data. This knowledge not only enriches a developer's toolkit but also enhances application performance across diverse linguistic settings.

Ten Unicode Characters You Should Know About as a Programmer
Jan Lelis • December 01, 2017 • New Orleans, LA

RubyConf 2017: Ten Unicode Characters You Should Know About as a Programmer by Jan Lelis

There are a lot of things that can go wrong when working with Unicode data. Some examples of unmeant behavior:

You try to downcase "I" to "i", but your Turkish friends want it to be a dotless "ı"
Your UI is broken, because people use empty usernames, despite the String#blank? check
You think "C" is the same letter as "С", but your system does not think so and crashes
Using ten characters as representatives, I will highlight some Unicode characteristics which require a programmer's attention and demonstrate how Ruby's solid Unicode support can be of useful assistance!

RubyConf 2017

00:00:11.050 I'm Jan Lelis, and I'm here to talk about some interesting Unicode characters that you should know about as a programmer. Additionally, I'm working on a Ruby project where I highlight amusing aspects of Ruby that you may either know about or want to explore.
00:00:27.200 The first character I'm sharing is a Thai character. Although it might appear to be a single character, it consists of multiple strokes. To understand this, we need to grasp what Unicode is. According to the Unicode website, Unicode provides a unique number for every character. This means it's not about how the actual bytes appear; rather, it's about identifying each character with a unique code point.
00:01:12.049 Often when we talk about characters, we're actually referring to code points, which are unique numbers assigned to every character in Unicode. A single character can be composed of multiple code points, which is quite fascinating. This is crucial for understanding how Ruby processes these characters.
00:01:45.430 In Ruby, we are fortunate to work with these characters efficiently. With the backslash-X notation, we can easily represent various Unicode characters. For instance, you can use regular expressions in Ruby to match any character, and the dot operator will match across code points.
00:02:01.400 The second example is the character 'Ä', which is very common in the German language. This character is also made from two code points, yet when I try to input it using my keyboard, it doesn't produce a single code point that represents the character correctly.
00:02:27.200 In Ruby, we have the ability to normalize characters. Normalization can transform the two code point version into a single code point representation. This normalization feature has been included in Ruby since version 2.3 and is required due to its importance in handling Unicode.
00:03:07.920 The next character is a small letter 'o', which may seem ordinary, but it holds a special distinction. It's not just the regular letter 'o'; it's a different character altogether. There are many characters that could be confused with each other, especially when they look similar.
00:03:10.379 The Unicode Consortium has created a list of characters that often get mistaken for others. One example is the question mark character, which can easily be confused with its counterparts. Such confusability is a significant issue for programmers and can lead to serious security vulnerabilities.
00:03:44.720 We can mitigate confusion by using tools such as the Unicode Confusable gem, which helps identify similar-looking characters programmatically. Furthermore, even though a character may look like an 'I', it could be a different version of 'I' with a dot above, especially in Turkic languages like Turkish where the capitalization rules differ.
00:04:22.080 In Ruby, methods like 'upcase' and 'downcase' now support locale-aware options. For instance, with Ruby 2.4 and onward, you can pass in locale-specific arguments to correctly handle casing. With these enhancements, Ruby has become more equipped to handle internationalization.
00:05:04.509 Next, let’s discuss control characters, particularly the newline character. Each operating system has its method for handling new lines. Unicode introduced a character called 'next line', but its usage has been limited due to confusion among different systems.
00:05:52.500 It's essential to be aware of how these control characters interact within Ruby. For instance, how you match newline characters can vary based on the specific code and functions used. Consequently, when working with strings in Ruby, approaching them with care is imperative.
00:06:32.480 As developers, knowing about these representations is vital. Some characters are not supported or may not display correctly, particularly in fixed-width environments. Characters that are ambiguous or undefined can lead to mishaps in your applications or outputs.
00:07:29.210 Unicode also has a range of code points that the Unicode Consortium has deemed non-assignable. It's crucial to know which characters are legal but unassigned, like the highest code point 10FFFF, which is never given a defined meaning. This knowledge can help prevent confusion in applications.
00:08:58.580 Finally, while utilizing regular expressions in Ruby, the Unicode property syntax can help match all these unique characters, including those that may not always render or have recognizable visual representation. The Unicode landscape is vast, and being aware of these intricacies ensures smoother handling of character data.
00:10:07.510 In conclusion, understanding Unicode can significantly enhance your capabilities as a programmer. From recognizing confusable characters to knowing how to process them correctly in Ruby, each of these concepts plays a critical role in crafting robust applications. Thank you for your attention!
Jan Lelis
@janlelis
Explore all talks recorded at RubyConf 2017
+83