Hello, computer. Writing Ruby with voice recognition

Ruby

Nat Budin

Hello, computer. Writing Ruby with voice recognition

by Nat Budin

In the talk titled Hello, computer. Writing Ruby with voice recognition, Nat Budin shares a personal account of transitioning to voice coding after suffering a finger injury that hindered her ability to type. The presentation explores the importance of accessibility in tech workplaces, especially for developers who, like Nat, depend on their hands for coding. Key points include:

Personal Injury Experience: Nat details her struggles with pain in her right index finger after camping, which eventually led to a diagnosis of trigger finger, underscoring the need for proper medical care and telemedicine consultation during the pandemic.
Introduction of Voice Recognition: As part of her recovery, Nat sought alternative methods to code and discovered Talon, a customizable voice recognition software designed for programmers.
Talon Features: Nat explains how Talon operates, allowing for not just dictation but also specialized commands for coding. It has a community-driven repository of scripts that tailor the software to individual needs, enhancing productivity and convenience.
Voice Coding Challenges: While demonstrating how Talon functions in real coding scenarios, Nat emphasizes the initial learning curve involved in using voice recognition effectively, likening it to solving a puzzle.
Best Practices for Voice Coding: She offers practical advice for new users, such as integrating traditional input methods, ensuring audio quality, and customizing vocabulary for technical terms. Nat highlights the significance of comfort and adaptability, suggesting that users should listen to their bodies to avoid strain.
Broader Accessibility Perspective: Nat connects her experience to a wider conversation on accessibility, stressing that it's not only important for end-users of technology but also for developers who may encounter physical challenges as their careers progress.

The presentation concludes with resources for getting started with Talon and a reminder of the importance of awareness regarding body health and adaptive technologies in programming. The event took place at RubyConf 2021, emphasizing community support and shared learning in tackling accessibility challenges in tech.

00:00:11.040 Hi, welcome to "Hello, Computer: Writing Ruby with Voice Recognition." Before I begin, I'd like to give a small content warning; there will be some medical information in this presentation.

00:00:18.160 I'll indicate when I'm about to reach the part with a diagram. There will be one slide featuring a diagram of the anatomy of a hand. It's not super graphic, but I understand it might be uncomfortable for some people. I won't be offended if you prefer to step out and return later.

00:00:30.160 Most of the medical content is at the beginning of the presentation. I will get to the coding demo part of it around five to eight minutes in. So anyway, hi, my name is Nat Budin. I work at AKBlue Technical Services, and we are hiring. We are a remote-first organization, but our main office is in the Boston area. Feel free to chat with me about that if you’re interested. If you'd like to follow me on Twitter, my handle is @nat_budin.

00:01:10.560 Before I dive into the main presentation, I want to give some context. I don’t code by voice full-time anymore; I did so for a few months while recovering from an injury. As such, I'm not really an expert in this software. I haven't developed the level of expertise that someone who uses it regularly would have. There are people who are much better at this than I am.

00:01:37.680 I consider myself fortunate in terms of being a voice coder. I speak in what is often referred to as Middle American English, which definitely helps with the voice recognition software's understanding. I don’t have the kind of medical issues that would require me to use voice recognition full-time, so I feel very lucky in that regard.

00:01:59.280 Now, speaking of my hand injury, here’s how I got into voice coding. The story begins in August 2020. This is a vacation photo from when I was camping with my partner in Central Washington. In the background is Sun Lakes Dry Fall State Park, which used to house the largest waterfall in the world during the ice age. It's since dried up, but you can still see the remnants. You can camp right in the basin of the cliff, and I highly recommend it if you're ever in the area.

00:02:23.600 While we were there, I started to feel some pain in my right index finger. Initially, I couldn’t identify the exact movements causing the pain, as it didn’t occur in the same spot every time. I tried to ignore it, thinking, ‘I’m just getting older.’ I figured I’d be fine, but the pain did not improve.

00:02:45.360 Eventually, like anyone would do, I self-diagnosed on Google. I read about various medical conditions, one of which was De Quervain's tenosynovitis. Spoiler alert: that wasn't the issue, but I thought it was for quite a while. The next step after self-diagnosing was, of course, self-treating with Amazon.

00:03:02.720 I purchased a finger splint that I wore 24/7 for about ten days straight, attempting to work with it on. It was supposed to help by preventing my finger from bending to relieve any pressure causing the problem. However, at the end of those ten days, my condition had worsened.

00:03:25.760 I discovered that my work's health plan included a telemedicine option, which was critical during the peak pandemic period. I was reluctant to see a doctor in person, particularly before vaccines were available. Finally, I consulted with someone through telemedicine, and she advised me to stop wearing the splint, stating it was causing more harm than good. I was told to only wear it at night, if at all. She also recommended seeing a hand specialist.

00:03:51.360 Here’s a diagram illustrating what was going on with my fingers. If you're uncomfortable with it, feel free to step out now. The hand specialist confirmed that I had a trigger finger. There's a tendon running through each finger and down to the arm, known as a flexor tendon, which is depicted in this diagram. When you bend your finger, it acts like a rope moving through a pulley, which, in my case, had become inflamed.

00:04:35.200 The inflammation prevented my tendon from gliding smoothly, causing pain that could manifest anywhere in the finger. The hand specialist provided an injection of cortisone directly into the affected area, which worked almost immediately. I began to feel significantly better within a few days.

00:05:14.120 However, this was a gradual healing process that took months. During that time, I still wanted to code, so I tried various methods, including coding with one hand, which was incredibly frustrating, or trying to avoid using my index finger altogether, which only made things worse.

00:05:51.680 I remembered a talk from the Perl Conference in 2019 that my friend Amy Newell had told me about but I had never watched. The speaker, Emily Shea, uses voice recognition full-time due to severe arthritis, making keyboard use nearly impossible. In her talk, she demonstrated a piece of software called Talon.

00:06:17.920 Talon is a customizable system for desktop voice recognition. Unlike traditional voice recognition software, Talon not only transcribes speech but recognizes commands. It supports several voice recognition engines, including its own optimized model called Conformer, which is incredibly effective.

00:06:43.920 Talon was developed for programmers by a programmer, and its community repository contains various scripts and commands written in scripting languages. While the software isn’t open source, users can support its development via Patreon.

00:07:05.840 Now, I'd like to demonstrate how Talon works. I'll run a few commands for you. Let me open my editor.

00:07:45.440 Talon, wake. Say "now Talon is active." Enter. Go up. Go word right. Clear word right. Space. Say Talon. Go down.

00:08:03.760 As you can see, there are plenty of commands available for editing text. While it doesn't always recognize my commands correctly at first, it's actually quite efficient most of the time.

00:08:27.120 You might think that while text editing is useful, as a programmer, I need to type things differently, like writing in Ruby or JavaScript, which have specific naming conventions. For example, the commands include prefixes such as 'snake' for underscores or 'camel' for CamelCase, enabling different outputs as needed.

00:09:04.000 This feature can be especially useful for someone like me who often switches between Ruby and JavaScript in my work. I'm constantly adjusting to different variable naming conventions across different languages.

00:10:07.680 After getting the hang of it, I found that Talon significantly elevated my programming capabilities and allowed for precision that keyboard typing alone might not offer. This makes it an exciting prospect for anyone dealing with repetitive stress injuries or those simply looking for an innovative way to code.

00:10:56.560 Talon has additional prefixes available when I’m in Ruby mode. For example, saying 'instance loaded records' automatically generates the instance variable format, which is incredibly helpful when working in Rails.

00:11:42.240 Talon adapts to various programming languages as well, detecting which language you're using based on the active window title. This means if you open a Ruby file, it tailors the commands to fit Ruby programming, making it versatile.

00:12:36.400 Talon also includes a phonetic alphabet for those moments when it doesn't recognize certain words. It’s user-friendly by using one-syllable words to represent letters, making it easier to spell out commands.

00:13:20.480 You can use the Talon online help resource to get assistance with commands or debug issues. It features a command history window, allowing you to see what commands have been executed.

00:14:15.680 Now, let me copy some actual Ruby code to demonstrate how Talon works in practice. This is code I have from an open-source project, not proprietary code.

00:15:00.160 Talon, wake. Snake, rescue from space.

00:15:05.440 Hammer pundit, colon, twice. Hammer "not authorized error".

00:15:20.960 State do, arg pipe, say "error". Slap.

00:15:36.560 Hammer rails, dot, dotted logger, warn, paren.

00:16:05.040 Many of the commands involve going left and right to make precise adjustments. Talon can identify where my cursor is and make the necessary changes.

00:16:22.960 While I’m not entirely practiced with the software at the moment, I’m hoping this demonstration shows you how possible it is to program using Talon, honing in on voice commands and editing features.

00:16:59.520 Over time, people using this software full-time can become as efficient with it as they were with a keyboard within a few months.

00:17:40.480 However, for beginners, the first couple of weeks can be challenging, but it gradually becomes more intuitive, almost like solving a puzzle.

00:18:17.600 If you plan to start using Talon, remember some tips to ease that process. For one, don’t attempt to do everything at once, initially, use both your voice commands and manual input with a trackpad or mouse.

00:19:14.720 Listen to your body; if something hurts, it’s crucial to modify the way you attain your tasks. Take breaks, adjust your approach, and don't push yourself into discomfort.

00:19:53.360 Excessive gaming during my injury recovery period played a part in how I developed it in the first place.

00:20:42.640 Finally, if you’re keen to learn more about Talon or get started, the resources provided will be immensely useful.

00:21:24.200 Thank you for joining my talk, and I’ll be available on Discord for further questions.

RubyConf 2021