Exploring Real-time Computer Vision Using ActionCable

Talks

Justin Bowen

#real-time-applications

Exploring Real-time Computer Vision Using ActionCable

by Justin Bowen

The video titled "Exploring Real-time Computer Vision Using ActionCable," presented by Justin Bowen at RailsConf 2021, delves into integrating Rails with Python for real-time computer vision applications. The primary focus is on analyzing images of cannabis plants, particularly in tracking changes in leaf and bud areas. Bowen articulates that leveraging computer vision (CV) tools can significantly enhance the agricultural process, especially in cannabis cultivation, while also discussing architectural methods for deploying Rails applications that incorporate CV techniques.

Key Points Covered:
- Introduction to Computer Vision: Bowen begins with an analogy between Teslas and dairy barns, illustrating how both utilize cameras for insights through CV, notably in Tesla's autopilot features and dairy farm monitoring systems for cow comfort and health.
- Experience with Drones and Agriculture: Describing his past roles in the drone and agricultural sectors, Bowen emphasizes how his experiences shaped the development of computer vision applications in cannabis cultivation, focusing on biomass tracking and disease detection via image analysis.
- Technical Challenges in Agriculture: Common issues such as powdery mildew and bud rot significantly impact crop yields. He proposes using CV to monitor plant health by analyzing leaf and bud areas to detect disease early and track growth rates.
- Implementation of Computer Vision Techniques: Bowen discusses employing OpenCV with ActionCable and Sidekiq to process imagery, enabling real-time data analysis and providing actionable insights to users. He details specific methodologies for measuring healthy versus unhealthy areas in crops through pixel analysis.
- Architecture and Scalability: The talk explores a scalable architecture for deploying computer vision applications using Rails, alongside Python. He highlights the use of services such as Puma, Redis, and Google Cloud for performance optimization and data management.
- Machine Learning Integration: Bowen touches on how to incorporate machine learning techniques into Rails applications, including the potential use of neural networks for enhanced detection capabilities. He emphasizes learning by doing and provides resources for further exploration, such as Adrian Rosebrock's work.
- Conclusion and Future Directions: The talk concludes with encouragement to engage in these technologies while sharing a whimsical note about his cat, underscoring a light-hearted approach to the complex world of computer vision in agriculture.

Overall, attendees gain insights into the foundational tools needed for integrating machine learning and computer vision with Rails projects, especially in the context of agricultural applications.

00:00:07.220 What do Teslas and dairy barns have in common? Welcome to "Exploring Real-time Computer Vision Using ActionCable" with me, Justin Bowen. First, I want to thank you for joining my talk at RailsConf 2021. In this talk, we'll go over some of the computer vision experiences I've had and how I use those experiences to develop cannabis computer vision.

00:00:18.240 Then, we'll review the typical architecture I like to use for deploying Rails apps and explore how we can utilize it for processing imagery with computer vision. This is a cat—how about that? She is a cute cat. We'll just put her down here and move along.

00:00:44.340 So, what do Teslas and dairy barns have in common? Well, they both utilize cameras to derive insights using computer vision. Probably the most advanced computer vision application available to consumers is Tesla's autopilot.

00:01:02.879 Tesla vehicles sport a suite of seven cameras—three front-facing with varying fields of view for short (60 meters), mid (150 meters), and long (250 meters) range. The forward-facing cameras use wide-angle, mid-angle, and narrow-angle lenses, while there are two side front-facing cameras with an 80-meter range and two side rear-facing cameras with a 100-meter range.

00:01:20.640 These cameras, combined with their specifications, can be utilized to measure and assess object size and distance in front of and around the vehicle. All of these cameras are processed in real time using GPU-powered computer vision onboard each Tesla to detect lane markers, vehicles, bikes, motorcycles, cones, pedestrians, trash cans, and intersections using neural networks combined with some rules.

00:01:40.979 These components are used to provide feature sets known as autopilot. Autopilot is a driver assist feature with the ambition of achieving full self-driving capabilities. At the time of this talk, Teslas are able to navigate freeways, change lanes, and take exits automatically—this feature is known as 'nav on autopilot.'

00:02:03.659 Teslas can also stop at intersections with red light and stop sign detection. With Tesla's full self-driving beta, they're capable of turning at intersections and driving users from point A to point B with little or no intervention. Teslas today are able to safely maintain their lanes with auto steer by monitoring the lane markers.

00:02:20.580 They're also able to park with auto park and unpark using the enhanced 'summon' feature, which can allow your car to come to you from across a parking lot. Okay, well that's impressive, but what was I talking about with dairy barns? Some modern dairy farmers have computer vision currently being used to monitor dairy barns for cow comfort because comfy cows produce more milk.

00:02:45.959 Cameras monitor, log, and report eating and drinking behavior to provide farmers with real-time insights into cow comfort and health metrics. I had the honor of leading the team building this dairy barn offering where we were tasked to deploy computer vision capable of detecting and identifying cows' eating and drinking behavior across multiple pens. After all, cows need water and food to make milk.

00:03:22.860 This provides farmers with the ability to determine cow health by alerting them to feeding, drinking, standing, and laying activity. The first thing to go when you're not feeling well is your appetite and energy—this is also true for cows. Similarly to Tesla's approach for computer vision, GPUs were deployed in the field to analyze cow data on the farm in locations with limited bandwidth, where streaming video from one camera would have been challenging.

00:03:45.540 These remote locations sometimes had tens or hundreds of cameras to cover a herd of 4,000 cows across multiple barns. Edge computing was the only option for computer vision since there was no way to upload all that data to the cloud. My experience actually started with drones and pivoted to dairy cows, and eventually, I ended up exploring cannabis computer vision.

00:04:04.319 Let's talk a little bit more about my drone experience. In 2016, I joined an Irish company called Kanthus with the challenge of building a data pipeline for processing drone flight imagery for field crop analysis. Interestingly, they switched to dairy cow monitoring in one of the most fascinating pivots I've experienced in a startup.

00:04:53.220 When I first joined, they were trying to load massive geotiffs (gigabytes in size) into memory and analyze all of the pixels at once, reporting back pixel similarity across a field based on user input. The only problem with this process was that it took seven minutes from when the user selected a target to receiving a heat map end result. While the industry standard for field crop analysis at this scale was a 24-hour turnaround, we were determined to devise a new process for breaking up the work to make it more scalable and respond in under a second for a better user experience.

00:05:41.220 The process involved unzipping archived JPEGs and stitching them into a giant mosaic geotiff—again, these were gigabytes in size, but this was only happening once when the flight data was imported into the system. These geotiffs were then tiled with GDAL to output tiles to be served in a tile map service (TMS) URLs. These TMS URLs could be used for mapping in Google map-style interfaces rendered by Leaflet.js, allowing farmers and agronomists to select an area of interest on the map to search the rest of the field for similarity by outputting a heat map of the similar pixels across the field or a subset of field tiles.

00:07:09.600 This could be used to determine harvest stability or crop failure rates. The business logic required tiles to be processed at render time as TMS image URLs in order to visualize a heat map of this similarity match, as well as return a percentage match across the field. We simply did this at the highest zoom level to reduce response time—the higher the zoom, the fewer the pixels. We also rendered and processed heat maps for tiles currently in the viewport at the zoom level the user was on.

00:08:37.500 This created the illusion that the whole field had been processed when really only the TMS URLs requested by Leaflet were processed on demand. These experiences gave me exposure to the architecture and implementation strategy I've used for computer vision. This inspired the work I've done on GreenThumb.io, my blog focused on cannabis cultivation and automation using computer vision as the driving data set.

00:09:57.060 While it seems overwhelming, there are common techniques employed in this process, which I've used for scaling all of the Rails apps I've worked on over the last ten years. First, let's consider some of the challenges that cultivation has to offer, which can be observed visually. Detritus is one of the most common diseases that impacts various crops, from grapes to cannabis, also known as bud rot.

00:11:02.880 Cannabis plants will present discolored bud sites. These spots change color from a healthy bud color to gray to brown, eventually covering most of the leaf and bud, causing them to wilt and die. White powdery mildew is a fungal disease that affects a wider variety of plants. Powdery mildew can slow down the growth of plants, and if affected areas become serious enough, it can reduce yield quality. Plants infected with white powdery mildew present an appearance of being dusted with flour.

00:12:26.700 Young foliage is most susceptible to damage, and the leaves can turn yellow and dry out. The fungus might cause leaves to twist or break and become discolored and disfigured. The white spots on the powdery mildew spread over to cover most of the leaves in the affected area. Another challenge is tracking biomass rate of growth, which is a key metric when growing any crop. Conventionally, NDVI is an agricultural approach to computer vision used to calculate plant health, usually at very low resolution.

00:13:39.660 More recently, with drone plates or stationary cameras indoors, conventional yield estimates have been calculated in the past by literally throwing a hula hoop into a field and counting the number of plants or bud sites within the known area of the hoop. The agronomists can then extrapolate yield potential from the hoop for a rough statistical estimate of the field's potential yield. So, how do we do this with computer vision? First, we perform a leaf area analysis.

00:14:49.680 This can be used to track our biomass rate of change to indicate healthy growth or decay. Increases in the number of pixels matching the healthy target color range indicate biomass increase, while changes in target color, something outside of the range, will trigger our matching pixel values to drop. We can also perform a bud area analysis, which can be used for tracking yield estimates by counting bud sites, tracking their size, and monitoring signs of stress due to color change at each bud site.

00:16:01.260 This can be tracked for pixel area growth and decay based on their similarity to a healthy bud target color range. To begin, we need to define a calibration value for our target of interest. We can use the range detector I found on piimagesearch.com's imutil library to select a low and high HSV color value for the target—in this case, the target is the bud sites. HSV is a color space value with values of hue, saturation, and value. By dialing up the low end and dialing down the high end, we can capture a clean mask of the bud sites without any additional cleanup.

00:17:13.740 We can use OpenCV for post-processing the image and pixel selection using the calibration HSV target range. We take advantage of Python's scikit-image library and its measuring and labeling tools to have a good understanding of the size of each of our contours. We take these components and use OpenCV to find and annotate images with contours of leaves and buds. Going back to the problem of white powdery mildew, we can use the leaf area mask to detect it by tracking the decay in healthy leaf area pixels.

00:18:41.400 You can see in this example that the leaf on the left is healthy, while the leaf on the right shows only a small percentage of its area as healthy leaf pixels. You can also track the rate of biomass increase from these test images to determine week-over-week growth. The output from week one's leaf mask to week two's leaf mask shows that the plants grew 27,918 pixels in five days. Using some basic trigonometry, we can calculate pixels to millimeters; cameras are really just like pyramids.

00:19:56.580 With the base of the pyramid being the frame of the image and the peak representing a horizontal and vertical angle of view based on the camera's specs and distance to the canopy, we can calculate that approximately 27 square centimeters of cannabis grew in five days. Bud site detection is similar to leaf stress in that we are looking for a rate of growth or decay in healthy bud area pixels and a change in the color of the contours we've detected as individual bud sites.

00:21:33.300 In this example, you see a large portion of the bud site has turned from a healthy target color to brown. We can also measure and count the size of individual bud sites we've detected, so if any bud site changes color or size, we can alert users of growth or decay. We can be very specific about which bud site is growing and which one may not be doing so well.

00:22:38.760 Here's an early stage visualization of the heat map, with the data set side by side with the source imagery. In this example, we see leaf area and bud count and size decaying over a matter of hours. This series shows minute-by-minute image analysis after drought-induced plant stress. The plants will recover with water, but the time spent in this stressed state will reduce yield potential and may cause infections to the bud sites, such as white powdery mildew or botrytis.

00:23:30.840 So, how do we deploy these visible indicators of plant health with Rails? Well, there are common component services that I prefer to run for all of my Rails apps. For background jobs, I like to use Sidekiq. For ActionCable, I usually start with Ruby's cable process, but I've found great success with AnyCable, Go, or web workers. I use Puma for essential queuing and Pub/Sub message service.

00:25:01.020 I use Redis for persistent data storage and querying of data sets, and I utilize PostgreSQL or MySQL depending on the use case. I really appreciate PostgreSQL's extension PostGIS for its ability to geo index and geo query. When I use OpenCV, I prefer Python as it's a developer-friendly language that I've come to appreciate, very similar to Ruby but with some differences.

00:26:19.500 The nice aspect of this architecture is that you can scale up individual services independently as work and jobs increase. For the background workers, we can scale up our Sidekiq processes or servers as WebSocket connections increase, and as our CPU load or memory increases, we can scale up our cable service. When HTTP requests rise and we need more workers, we can scale up our Puma processes. All this can be done independently without one overloaded service taking down the entire system.

00:27:45.240 It can be overwhelming to view the architecture all at once, so we'll take things step by step and visualize how each component of the system interacts to provide a user experience that exposes the inference derived from our computer vision analysis. The OpenCV code can run on devices like the Raspberry Pi, which can be accelerated with Intel's Movidius vision processing units or even the new OpenCV AI Kit, which has the Movidius Myriad X VPU integrated.

00:29:05.340 This can significantly increase your frame-by-frame processing time with an edge-based implementation. After image capture and analysis, the number of bud sites and their individual sizes can also be reported to the Rails app either through Puma and an HTTP request or ActionCable using a Python WebSocket connection to our ActionCable service. This way, we can offload some of the server-side processing such as leaf area and bud area to the camera.

00:30:12.600 In this case, the pre- and post-processed images can be uploaded to the Rails app or directly to S3 using direct file uploads. The metrics, such as leaf area and bud area in pixels or converted to millimeters based on their distance from the camera, can then be uploaded to our database so that we can store them in a geo-index in PostgreSQL. Another option is cloud computing—the captured images can be uploaded from the browser or devices.

00:31:28.560 I prefer using Thumbor, which is a Python image service that can be extended with our computer vision code using Thumbor's OpenCV engine. The computer vision code is deployed to Google Cloud Platform's Cloud Run service, where you can deploy highly scalable containerized applications on a fully managed serverless platform. I use Docker to do this; with a URL from a direct file upload, the Rails app can use the image service in the cloud and provide users with processed image URLs.

00:32:48.300 I've packaged the leaf area and bud area into Thumbor filters that can be passed into the URL as parameters with the HSV color space lower and upper bounds as arguments. The Thumbor filters have been put into Python pip packages and are installable through PyPI. This code is used in a Thumbor container deployed to Google Cloud Run. The Thumbor image service is extended using the two methods, and all it needs is an input image URL, which then outputs a processed image masking leaf area or bud area.

00:34:05.640 The method can also take a channel identifier for notifying back to ActionCable, again using a Python WebSocket client or Puma through an HTTP request. Either method can be utilized, and both can notify through ActionCable. Eventually, you can see here that the Thumbor image on the right and the number image service on the left, when combined, output the masked image—I'll call this the Green Thumbor.

00:35:14.220 I actually named the repo that I pushed to GitHub 'The Green Thumb Image API,' but while I was putting these slides together for this talk, I thought of 'Green Thumbor.' I thought it sounded better. Anyway, I've mentioned direct file uploads a couple of times—what are they? Well, they're a great way to relieve pressure on our Rails app. Puma workers can focus on responding to HTTP requests quickly without having to wait for blocking I/O from large objects like images and videos.

00:36:36.660 With AWS S3 or Google Cloud Store, our Rails app can provide a client browser or device with a pre-signed key to upload files to a location on our storage service directly. Once the files are transferred, the client can notify the Rails app that these files can now be processed or served. From there, we can insert the image URL data into our database and enqueue a background job with the image ID and a channel identifier to a Sidekiq worker.

00:37:57.540 The job will make a request to the Thumbor image service to process the image URL and send back a response to the channel's report ActionCable. Sidekiq's third-party API request to Thumbor's service allows for keeping the external requests in the background and off our Puma workers' plate. The report Action will then receive a message either from the Thumbor service directly using ActionCable or from Puma.

00:39:06.240 In this visualization, you see it coming from the Thumbor service directly to our ActionCable service, which can enqueue a job to update the inference data into our SQL database and broadcast to our ActionCable subscribers. The ActionCable consumers that are subscribed to this channel can then render the image and graph the values. Looking at this graph, we see that our rate of decay in healthy leaves and bud area can be utilized to generate an alert or trigger an automated watering.

00:40:00.120 This is a time series over a span of three hours, and you can see there's plenty of time to alert the user before it's too late. This video was provided by my friends at SuperGreenLab, who sell smart micro grow systems using LEDs and sensors controlled by a custom-designed ESP-based PCB. They also use Raspberry Pi's for capturing time-lapses.

00:41:54.660 Another great use for ActionCable is WebRTC signaling. WebRTC signaling allows two clients to establish peer-to-peer connections, enabling bi-directional communication of audio, video, and data without a trip to our WebSocket server. This involves sending a session descriptor offer from one peer to the other and sending a session descriptor answer back to the other peer, and then both peers can request a token to create what's called interactive connectivity establishment candidates or ICE candidates.

00:43:18.360 Once the peer connection is established, bi-directional communication from device to browser can be accomplished. This allows real-time video feed from the camera to your browser and the ability to trigger actions from your browser to the camera, which could control sensors, actuators, fans, or pumps. While I've seen implementations of WebRTC signaling done with HTTP polling, I've had great success using ActionCable to handle the initial exchange of ICE candidates.

00:44:43.740 Now, does it scale? The short answer is yes. I've been using Puma for a while for my web workers. These are just some graphs of the memory footprint and responsiveness at scale that I pulled from their website. Nate Berkopec, the author of the Complete Guide to Rails Performance and co-maintainer of Puma, recently said that in order to scale a service properly, you need to know the queue times for work and jobs, requests, the average processing time, and arrival rate for work and jobs.

00:45:57.300 You also need to monitor CPU load utilization, CPU count, and memory utilization of the underlying machine. This principle applies to any language and any framework. I had a recent consultation with a startup founder who was asking if I thought .NET was scalable. I used our time together to help them better understand the systems and concepts required for a scalable system and emphasized that it's best to use what the developer feels most comfortable and productive with.

00:47:18.060 I love Ruby and feel most productive and confident scaling a Rails app. Python is very friendly for me too, but I wouldn't say I love or feel comfortable with Django, though I have deployed some Django apps before. I certainly wouldn't personally feel comfortable scaling a .NET application, but in that case, I told the startup founder to listen to their developer and ensure they follow scaling practices outlined here in .NET with different tools similar to Puma and Sidekiq, and potentially a WebSocket service if they need it.

00:48:44.520 Anyway, I highly recommend checking out Nate's book and following him on Twitter for great write-ups on optimizing Rails apps. I've been using Sidekiq since around 2012, and I really appreciate the work that Mike Perham has done with contribsys.com, which is responsible for Sidekiq. Factory provides a feature-filled job processing system for all languages, employing a common queuing mechanism using Redis that supports workers in Node, Python, Ruby, Go, Elixir, and more.

00:49:57.540 While I haven't used Factory, I am curious and excited to see how our architecture could be enhanced with background jobs across our Rails and Python code bases. Sidekiq itself is 20 times faster than the competition. I think the old saying goes that one Sidekiq worker can do the work of 20 Resque jobs or something along those lines. It does scale.

00:51:13.680 Nate also mentioned that WebSockets are far more difficult to scale than HTTP in many ways. The lack of HTTP caching and the ability to offload to CDN are significant challenges. In this graph, we can see the CPU load comparison on the left with ActionCable Go versus the right with ActionCable Ruby under 10,000 concurrent connections each. The Ruby cable service is fully loaded, while the AnyCable service has headroom to spare.

00:52:38.880 You can also notice in this comparison chart that Ruby uses four times as much memory as AnyCable Go during these load tests that I found online. AnyCable Go has performance under load that is similar to Go and Erlang natively. I haven't used Erlang Cable, but it appears to have similar performance metrics. This is a report showing the decay from the video we saw earlier with a graph, where the plants were wilting from not being watered.

00:54:06.660 This is over a span of 10 hours, so there was ample time to notify the user that there was a critical issue needing their attention sooner rather than later when they may not have come in to see the issue until the next day. Going back to the initial step of manual calibration, how can we overcome this step? Well, masked images are great annotated data sets that can be utilized as training data for a neural network.

00:55:22.680 We could potentially leverage the detection of bud sites and leaves without calibration. This would present a great opportunity to integrate with something like TensorFlow to utilize what's known as transfer learning. Transfer learning takes an existing neural network trained to detect objects and teaches it to recognize new ones, such as buds and leaves.

00:56:48.990 There is a lot of theory and techniques out there; I have experience with some of these but not all. It's important to explore and experiment with common solutions in practice but not to get too overwhelmed with theory, especially when starting out. So where do you go from here? Adrian Rosebrock, PhD from Pi Image Search, suggests learning by doing and skipping the theory.

00:58:13.260 You don't need complex theory; you don't need pages of complicated math equations, and you certainly don't need a degree in computer science or mathematics. I personally don't have a degree, so I appreciate him saying that all you need is a basic understanding of programming and a willingness to learn.

00:59:40.740 I feel that this goes for a lot of things, not just machine learning, deep learning, or computer vision, but programming in general. Don't get me wrong; having a degree in computer science or mathematics won't hurt anything, and yes, there's a time and a place for theory, but it's certainly not when you're just getting started. There is a lot you can do off the shelf.

01:00:17.880 High Image Search is a great resource for computer vision and deep learning examples and tutorials. There are many off-the-shelf solutions that Adrian will walk you through, including face detection, recognition, and facial landmarks used for things like Snapchat filters. Body pose estimations, like those used in human interface devices such as the Xbox Kinect, object detection and segmentation.

01:01:35.520 This applies to techniques similar to what we just did with leaf area and bud area, but can be adapted for things like cats, dogs, and other common objects, as well as optical character recognition for reading text in images. Super-resolution, akin to what is shown in CSI when they zoom in and say 'enhance'—that image gets clearer—is also something implementable using a deep neural network these days.

01:02:52.560 There are also interesting computer vision medical applications, such as detecting if people are wearing masks or identifying COVID in lung X-ray images. So what else exists? How can we integrate machine learning and deep learning into our Rails apps without computer vision? Aside from computer vision, there are disciplines like NLP, where you can analyze the meaning of text, computer audio, where you can analyze sound, recommendation engines, and even inverse kinematics for training robots to control themselves.

01:04:07.560 That could be an interesting use for WebRTC connections for low-latency peer-to-peer communication to control or send commands. But most importantly, have fun with it! Thank you for watching my talk. If you're interested in learning more about these topics, check out greenthumb.io or follow me on Twitter. I'm always happy to collaborate. There's my cat, who has been with me during this entire talk, and if you follow me on Twitter, you know she helped me out—so I've got to include the cat tax at the end of this.

01:05:15.960 I'm also open to consulting or potentially taking on a full-time employment opportunity if it's the right role. Whether it's through my agency, CFO Media and Technology, or as myself—an experienced product and team builder—I would love to chat and look forward to getting to know the community more. Thank you!

01:06:00.000 Look at this cat... that's all!

RailsConf 2021