Talks
Drones Galore: controlling multiple drones using mruby/ruby
Summarized using AI
Drones Galore: controlling multiple drones using mruby/ruby
by Shashank DatéThe video 'Drones Galore: controlling multiple drones using mruby/ruby' features Shashank Daté presenting at RubyConf 2021. The main focus is on utilizing mruby, a lightweight implementation of Ruby, for programming Tello drones—a popular category of programmable drones. Daté begins by engaging the audience and demonstrating the operation of two Tello drones, configured via a local Wi-Fi setup. The session covers the evolution of Tello drones from SDK versions 1.0 to 2.0 and discusses how mruby can be optimized for drone control.
Key Points Discussed:
- Introduction to Tello Drones: Affordable and ideal for educational purposes, these drones help beginners learn programming concepts.
- Drone Control Modes: Explanation of the drones' operational modes—AP (Access Point) mode and station mode—highlighting how each mode impacts the ability to control multiple drones.
- Technical Requirements: Discussed the necessity of UDP communication for command processing and the importance of concurrency to handle command-response systems effectively.
- Programming in Ruby and mruby: Details about implementing UDP communication in Ruby with examples of socket programming and concurrency using fibers, threads, and actors.
- Challenges and Limitations: Daté discusses the current limitations of mruby, emphasizing the absence of video processing capabilities as of the event's date. He outlines the differences between Ruby and mruby, mentioning mruby's advantages in deployment.
- Demo and Code: The presentation culminates in a live demonstration of drone acrobatics using the developed code, showcasing practical applications of the discussed concepts. Daté invites attendees to engage with the codebase available on GitHub.
In conclusion, the talk not only delves into the technicalities of controlling multiple drones with mruby but also encourages the integration of community contributions to the project. It exemplifies how programming can be made engaging through hands-on experiences, exemplified by the interactive demo of the Tello drones performing acrobatic maneuvers. Daté's motivation stems from personal experience and fosters a community-driven approach to programming and robotics.
00:00:06.710
Good afternoon, everybody.
00:00:12.240
Thank you very much for being here. There are so many other interesting talks, but you chose to come here.
00:00:17.760
Oh, I'm Shashank Daté.
00:00:23.279
This is my hobby, and I do it for fun and, unfortunately, some losses.
00:00:29.119
I have lost a couple of drones and never got them back.
00:00:35.040
Now, to get the energy flowing in this room, I want you all to participate.
00:00:41.440
On the count of three, I want you all to read the next slide as loudly as you can so that the energy in this room goes up.
00:00:48.320
Okay? Alright. One, two, three.
00:01:00.079
Thank you! Now, if you have blessed me, I'll show you why that blessing is required.
00:01:07.040
And here, pretty soon, these two should take off using my hello_edu.rb file.
00:01:13.439
I have to do one little thing that I forgot.
00:01:19.040
Give me just a second; I need to find the IP address of these two drones.
00:01:25.759
When they booted, they received different IP addresses.
00:01:31.439
The way I communicate with them is through a Wi-Fi router that I have set up here.
00:01:37.680
My laptop is connected to them through that, so I'm going to scan for these devices.
00:01:43.920
It shouldn't take more than a few seconds to detect their IP addresses.
00:01:49.520
The last three digits are 102 and 103.
00:01:56.320
If our energy is right, they're going to take off and land— not exactly in the same spot, but somewhere nearby.
00:02:02.719
If they start drifting, I have drone catchers in the audience who will catch them.
00:02:08.560
Okay? Let's see if it turns green. There you go.
00:02:24.239
That’s like my teenage daughter who doesn’t listen to me. She has a knife.
00:02:31.519
It will land in a few seconds; it has a default 15-second wait time.
00:02:44.560
They behaved much better today than they have in the past.
00:02:51.440
Okay, so now, these are a special class of drones called Tello drones.
00:02:57.840
Tello is the type of drone that I've been programming.
00:03:04.879
These are probably the cheapest programmable drones you can find at under $150 each.
00:03:10.959
The first few that I lost were during tests to see how high they could fly, but then the wind came up and they were gone.
00:03:19.040
Nonetheless, they are perfect for educational purposes.
00:03:25.760
They introduce beginners to programming.
00:03:31.920
For those of you who attended the previous talk, it mentioned a language called Logo, which taught kids to draw diagrams using turtles on a 2D plane.
00:03:38.159
These drones represent a 3D version of that. You can move not only horizontally but also vertically.
00:03:43.840
The purpose of these drones is to teach programming.
00:03:50.319
While there are many simple concepts to grasp, there are also deeper constructs that more advanced programmers can benefit from.
00:03:57.439
The type of drones I’m using are called Tello EDU type drones.
00:04:02.799
They have progressed from SDK 1.0 to 2.0, which includes the EDU capabilities we just saw.
00:04:09.680
This enables us to program swarms of drones.
00:04:15.519
I didn't have the budget to buy the full EDU kit that comes with 20 drones and costs about $5,000.
00:04:20.560
But I bought a couple and started experimenting.
00:04:26.400
These kits come with official support for Python, Scratch, and Swift.
00:04:32.880
I thought, 'Why not Ruby?' So, I took it on myself.
00:04:38.160
I got it working in Ruby as well.
00:04:44.080
These drones require concepts of network programming and concurrent communication.
00:04:49.280
They also involve video streaming and related tasks.
00:04:55.600
In terms of specs, these drones have an Intel Core i4 processor.
00:05:01.680
They are very lightweight and come equipped with a camera.
00:05:08.639
Believe it or not, they also have collision detection sensors.
00:05:14.320
There are two flight modes available in the EDU that you just saw.
00:05:19.680
There’s AP mode and Station mode.
00:05:24.880
AP mode is the access point mode that comes by default when you pull it out of the box.
00:05:30.000
In this mode, you can only control one drone directly from a single computer.
00:05:36.560
A single computer cannot control multiple drones.
00:05:41.680
I have never tried controlling multiple computers to manage one drone.
00:05:47.520
In this mode, video streaming from the camera is possible.
00:05:52.960
Then there is the swarm mode, also known as station mode, which is available in the second version of the Tello EDU drones.
00:05:58.960
To communicate with the drones in swarm mode, some manual setup is required.
00:06:04.639
With this mode, you use a router to communicate from your laptop to the drones.
00:06:11.039
In the AP mode, the drone itself acts as the access point.
00:06:16.080
It publishes its own SSID and password, which you must capture in your program.
00:06:22.160
This is why switching between those two modes is challenging.
00:06:27.840
These are the two modes of communication.
00:06:33.280
Now, let's talk about the programming prerequisites to control these drones.
00:06:39.120
First, you need UDP/IP communication between the drones.
00:06:44.800
Second, you should have some form of concurrency because you'll be sending commands in one thread while receiving responses in another.
00:06:51.120
You need to keep track of which command corresponds to which response.
00:06:57.120
Sending one command and waiting for the response in the same thread doesn’t work.
00:07:02.479
UDP is fast enough, and we'll see why.
00:07:09.759
Lastly, video streaming is optional; I couldn't get it to work for Ruby.
00:07:16.319
If you want to see real-time video, you need a suitable library in your programming language.
00:07:21.759
In Ruby, there's a UDP socket class available in the standard library.
00:07:28.840
I use that for communication.
00:07:34.240
Ruby 2.x has fibers and threads, and there are also reactors.
00:07:40.720
There are many ways to implement concurrency in Ruby.
00:07:47.680
When it comes to video streaming, I was excited when I discovered Ruby OpenCV.
00:07:54.720
I'll discuss this library a bit later.
00:08:00.319
So what is UDP/IP?
00:08:05.599
You must have heard of TCP/IP. UDP/IP is a sibling of TCP/IP.
00:08:10.639
It’s based on IP sockets and uses datagrams and port numbers for communication.
00:08:18.240
It is connectionless, meaning there is no need for handshaking.
00:08:24.639
You send a command and forget about it.
00:08:30.479
No buffering of datagrams occurs at either end.
00:08:36.479
While it is fast and meant for real-time communication, there is no guarantee of message delivery.
00:08:42.399
Also, commands might return in different orders.
00:08:47.600
Programmers need to keep this in mind when using this protocol.
00:08:53.200
You witnessed an example of this today.
00:08:59.600
One drone took the command to land while the other ignored it due to a packet loss.
00:09:05.200
That's the drawback of UDP; however, speed takes precedence.
00:09:11.839
In Ruby, if you wanted to see an example, this is a basic UDP socket implementation.
00:09:19.360
You require the socket library of Ruby, set up a host and a port,
00:09:26.080
and then prepare a message to be sent to that object.
00:09:31.920
You create a new socket, binding it to the host and port, then send the message.
00:09:37.600
Afterward, you’ll wait for the data to arrive.
00:09:44.320
In this example, it's not threaded; it's waiting for data to return.
00:09:51.040
Since it's essentially sending data to itself, it’s highly reliable.
00:09:56.880
Printing the data reveals the output received.
00:10:01.920
In summary, the first prerequisite for using UDP/IP is satisfied by Ruby.
00:10:07.760
In terms of concurrency, Ruby offers fibers.
00:10:13.600
Fibers handle cooperative concurrency, meaning the programmer must control the flow.
00:10:20.959
Threads provide preemptive concurrency, meaning the programmer has no control over thread switching.
00:10:26.800
Reactors come next, which are still in the experimental phase.
00:10:32.560
Ruby might soon include them as a core part of the language.
00:10:38.079
I wanted to highlight a library called Ruby OpenCV for video streaming.
00:10:44.880
This library wraps the OpenCV C++ library for video processing.
00:10:50.560
Unfortunately, I faced difficulties compiling it since it has been stagnant for eight years.
00:10:56.400
Indeed, I couldn't get OpenCV running properly on my Mac.
00:11:03.680
Because of this, we won't see a video streaming demo today.
00:11:10.560
Now, concerning mruby, by default it has only fibers; there are no threads.
00:11:15.600
To enable socket communication, you need to compile mruby with the socket gem.
00:11:21.600
This core gem comes with the mruby installation.
00:11:27.760
If it's not present in Ruby, it's not likely to be in mruby either.
00:11:34.800
As of now, I haven't seen attempts to integrate video processing or streaming into mruby.
00:11:40.240
For those unfamiliar with mruby, it’s a lightweight implementation of Ruby.
00:11:47.200
The 'm' stands for Embedded Ruby, Modular Ruby, or Minimalistic Ruby.
00:11:53.360
It's a proper subset of Ruby; anything you can do in mruby can also be done in Ruby, but the reverse isn't true.
00:12:00.000
In terms of deployment architectures, mruby can be deployed in more spots and environments than Ruby.
00:12:06.800
A key advantage of mruby lies in its ability to compile your code into a single executable.
00:12:13.440
This reduces dependencies on the operating system library.
00:12:20.079
Typical use cases include bare metal programming.
00:12:26.639
There might not even be an operating system or file system available.
00:12:31.760
For memory-constrained devices, I previously attempted to program point-of-sale systems using mruby.
00:12:38.240
However, it's likely still bulkier than what those devices actually require.
00:12:45.040
With progress made since my last attempt, it may now be more feasible.
00:12:51.920
The latest stable version of mruby (3.0) can launch an app within 100 KB of RAM.
00:12:58.080
The devices I had back then required less than 64 KB of RAM, so I'm skeptical.
00:13:05.600
Nevertheless, progress is ongoing, and this is one of Mats's favorite projects.
00:13:12.480
You'll see regular commits coming into mruby.
00:13:20.079
Game engines and other applications could also see enhancements.
00:13:26.639
Now, moving towards the implementation, you notice one line that is different: there is no require.
00:13:32.560
Why no require? It's because mruby does not assume the existence of a file system.
00:13:40.000
The require statement requires files to exist from a specified path.
00:13:45.440
So, you'll have to recompile the interpreter with the core gem.
00:13:51.440
There's a build configuration script in mruby.
00:13:58.400
If you add an extra line in the build configuration file to compile with mruby, make sure to clean up previous builds.
00:14:05.600
Once you clean the previous version, you can compile it with the socket layer.
00:14:10.240
Once compiled, you’ll be able to run the example.
00:14:16.880
What if you want to convert this program into a single executable?
00:14:23.360
This process is slightly convoluted, so let's break it down.
00:14:30.080
You can compile your example.rb code into a bytecode using the mrbc command.
00:14:43.920
This bytecode can then be distributed to anyone who has mruby installed.
00:14:50.959
They won’t see the original source code, just the bytecode.
00:14:57.679
If you want to go further and ensure the person doesn't need mruby installed,
00:15:04.480
you’ll need some C programming.
00:15:11.200
It’s a two-step process.
00:15:16.000
First, use mrbc to generate a symbol table, which is a data structure for the bytecode.
00:15:23.360
You can assign a name to that table for easy reference.
00:15:30.280
This is important for structuring your code.
00:15:36.320
In the next step, you'll write boilerplate code and include the emitted file.
00:15:44.320
Then you open the VM (virtual machine), which is standard for compiling any mrbc code.
00:15:50.959
Use the generated symbol table in the C file to execute your code.
00:15:56.160
You can compile it using your regular C compiler, and this creates a usable binary file.
00:16:02.079
That binary file can then be distributed without needing mruby installed on the user's machine.
00:16:09.200
So, this is the process to create a native binary.
00:16:15.360
For this talk, my three focus areas were Tello drones, actors and fibers, and mruby.
00:16:23.040
I opted to go deeper into mruby since that's my favorite.
00:16:29.840
Now, how do we get these drones to fly? Are they still on?
00:16:36.240
I should have turned them off earlier.
00:16:41.440
One observation I've made regarding the battery life is that the reported flight time is around 15 minutes.
00:16:47.040
However, I've never experienced a full 15 minutes on a single charge.
00:16:53.440
I usually only get about five minutes of flight time.
00:16:59.919
It’s vital to manage your battery effectively.
00:17:05.440
So, let's see how much time we have left.
00:17:11.600
I have a project on GitHub for Tello, where I’ve written the same code in Ruby, mruby, and Crystal.
00:17:17.760
A friend contributed code in Elixir as well.
00:17:23.760
Remember, those two concurrency prerequisites are essential to control these drones.
00:17:30.480
If you have a favorite programming language not included in the project, please submit a pull request.
00:17:37.440
Today, we're going to focus on the GitHub project where I've curated Ruby files for this conference.
00:17:44.400
The code we saw running was hello_edu.rb, which we'll review.
00:17:50.560
This all started because of a gentleman named Tom Black.
00:17:56.160
In a previous RubyConf, he conducted a five-minute lightning talk showing off these drones.
00:18:02.400
I got excited by his demonstration. Though he used an earlier version without swarm capabilities.
00:18:08.880
I collaborated with him to create the second version of the gem.
00:18:15.760
My pull request is still pending, but I am confident he will merge it once he sees the quality of the code.
00:18:22.640
If you visit that GitHub repo, you'll find a gem that allows you to set up a local Tello server.
00:18:30.320
This way, you don’t even need an actual Tello to test your client code.
00:18:37.680
Tom has put a lot of effort into this gem, and I'm grateful to him.
00:18:42.720
Now, let’s focus on the code we will be modifying today.
00:18:49.640
Since I'm not on the public Wi-Fi, I'm on a Wi-Fi that isn’t connected to the internet.
00:18:54.960
So I won’t be able to drive the drones right now.
00:19:02.560
I'll show you the code, which is not production-quality code; it's more like a hello world example.
00:19:08.280
I used global variables, which isn't a good practice.
00:19:13.920
This code is specifically designed for two drones.
00:19:19.440
So, to communicate with the two drones, I’ll use methods for sending commands.
00:19:26.160
All my work was done on a Raspberry Pi equipped with a camera.
00:19:32.320
As soon as the drones take off, the camera starts recording.
00:19:40.000
If something goes wrong, I will at least have the video evidence.
00:19:44.640
Thankfully, everything has worked so far.
00:19:51.680
Here’s the command structure I used.
00:19:56.880
This if statement will handle the response from the drones.
00:20:03.520
I send commands to set the Tello into command mode.
00:20:09.520
Now I will check the battery level since certain maneuvers are only possible with higher battery values.
00:20:15.840
For take-off and landing, you don't need much power.
00:20:22.080
Now, I will attempt to take off.
00:20:29.440
We’re going to do some live testing with the drones.
00:20:35.360
Let’s hope there’s enough battery for the drones to perform the tasks.
00:20:42.560
While they boot up, I'll try to get them to flip.
00:20:50.080
Let’s see how that works out!
00:20:55.360
One of these drones will flip while the other will shift left.
00:21:00.960
Let’s give that a shot.
00:21:05.760
I’m hoping everything goes as planned.
00:21:11.200
They should be up and flying now!
00:21:24.080
Alright, I have about five more minutes. Should we keep going?
00:21:30.399
Let’s try to do two flips this time!
00:21:35.919
They should perform their flips now.
00:21:42.080
If there aren’t any errors in the console, they will return together.
00:21:48.000
Okay, here we go.
00:21:54.000
Now they should perform their flips!
00:22:00.480
And they're off!
00:22:12.720
I just want to do one more thing, and then we can switch to Q&A.
00:22:22.080
This time, one drone will rotate clockwise while the other will rotate counterclockwise.
00:22:28.560
Let’s see if that works.
00:22:35.680
Okay, let me clear up some space here.
00:22:42.080
Can you all read the size of the text?
00:22:49.360
Let’s get ready!
00:22:57.440
The drones should begin their rotations!
00:23:05.600
Here we go!
00:23:11.680
If everything goes well, they should execute flawlessly.
00:23:19.200
There you go! Thank you very much!
00:23:30.399
That’s it, everyone! They’ve been on their best behavior today.
00:23:35.679
I have about three minutes for questions.
00:23:39.680
As you can see, it’s really straightforward.
00:23:45.920
Please submit pull requests (PRs) for the code in the GitHub repository.
00:23:51.920
If you do submit a PR, please ensure it follows the model created by my friend.
00:23:58.400
I’m going to switch over to the public internet now.
00:24:05.920
In my codebase, I have contributions in other languages like Crystal, Elixir, and more.
00:24:11.920
Just last week, a friend submitted a PR that included detailed instructions.
00:24:18.000
He provided examples of how to install and use it, which is great!
00:24:24.600
I would love if PRs maintained a similar level of detail.
00:24:28.600
Thank you very much for your attention!
00:24:34.000
Any further questions? Let's give that a try!
00:24:39.679
I have only one minute left.
00:24:45.679
So let's proceed to try that!
00:24:52.560
It is possible, let’s explore this option.
00:24:59.679
I’m going to navigate to my Tello gem folder.
00:25:07.360
I believe rake builds it.
00:25:13.600
I might have the Tello gem and the EDU gem installed.
00:25:19.560
So let’s see if that works.
00:25:26.560
This is like an interactive mode, similar to IRB.
00:25:34.560
Unfortunately, I’m encountering a permission error.
00:25:39.520
The Tello gem allows for interactive commands, but I need to remember how to start.
00:25:45.920
You must first issue a command to initiate the Tello, followed by others like takeoff.
00:25:52.560
Alright, I’m almost out of time.
00:25:59.200
If you want, we can try this outside and capture live interactions.
00:26:06.560
You’re welcome to join and test things out!
00:26:13.279
Thank you once again!
Explore all talks recorded at RubyConf 2021
+92