Talks
Keynote: How to lie, cheat and steal
Summarized using AI
Keynote: How to lie, cheat and steal
by Florian PlankIn Florian Plank's keynote titled "How to Lie, Cheat and Steal," presented at wroc_love.rb 2013, he explores the potential of modern web browsers and how developers can leverage them to enhance applications by shifting workloads from servers to clients.
Key points discussed in the talk include:
- Introduction to the Topic: Florian introduces his focus on maximizing browser capabilities to achieve better performance in web applications, as inspired by Aaron Patterson's provocative talk at RubyConf.
- Browser Capabilities: He emphasizes the rapid advancements in client-side performance, including improved hardware and software technologies like HTML5, which open new possibilities for web developers.
- Experimentation and Demos: Florian conducts a live demonstration of a heart rate measurement application using the browser and its camera. Although the demo fails due to technical limitations, it highlights the feasibility of complex tasks within the browser.
- Challenges and Solutions: He discusses the challenges faced when working with live demos, including noisy data from the camera and environmental factors that hinder performance. However, he emphasizes that these challenges can be overcome with time and experimentation.
- Shifting Computation to the Client: By analyzing tasks typically handled by servers, he suggests moving computations like media processing and image handling to the client side. This can reduce server load and improve responsiveness in applications.
- Case Studies and Practical Applications: Examples include implementing client-side image processing for responsive UI, using JavaScript APIs to enhance user experience, and managing video scrubbing and image resizing directly in the browser.
- Web Workers: Florian introduces the concept of web workers to manage background tasks without freezing the user interface, creating a library to simplify job management in web applications.
- Conclusion and Encouragement: He concludes with a call for developers to embrace modern browser features and refine their practices to leverage the potential of client-side computation efficiently.
The main takeaway is that by utilizing modern browser capabilities, developers can improve application performance, enhance user experience, and offload processing from servers to clients, ultimately leading to better, more responsive applications.
00:00:18.560
Please welcome.
00:00:29.359
Yeah, that’s a tough job. It’s the last talk before the lunch break, and I’m going to try to keep it short.
00:00:34.800
My name is Florian, and you can find me online. Unfortunately, you cannot find me on Facebook.
00:00:42.399
This is my first time in Poland, and it’s been really awesome. The people here are super friendly. But the best thing that I’ve found so far is this.
00:01:01.199
Pretty accurate! That’s all I can say about this.
00:01:07.520
Now, I wanted to warn you about a few things. This isn’t really a soft talk, but it’s also not purely technical. As the title suggests, there’s going to be a lot of half-truths, experimentation, and guessing.
00:01:13.840
Or maybe not guessing, but it’s not particularly the kind of thing that you would want to take home and use directly in your product. Also, there’s going to be a lot of CoffeeScript code, so I apologize if you’re not a fan. I prefer CoffeeScript over JavaScript.
00:01:26.479
So, how did I come to this topic? I was watching a video of a talk by Aaron Patterson at Aloha RubyConf.
00:01:34.800
He basically suggested that we should take better advantage of the browser, among other things. He worded it a bit provocatively, saying we should steal, lie, and cheat more.
00:01:46.160
This means making better use of the resources we have, and that’s where the title and inspiration for this talk come from.
00:01:52.319
We have just had two big releases, and this community is not very old, but Ruby is nearing 20 years and Rails is 10. We’ve been doing things in a very similar way for a long time.
00:02:00.399
I’m not suggesting that we should completely change everything, but I would like to suggest that there might be some new things we could try, particularly those that involve the browser.
00:02:10.560
I’d like to give you some ideas that you could try and think about to find ways to improve your applications.
00:02:23.680
There’s a demo here, despite the common saying that you really shouldn’t do demos or live coding because they tend to go horribly wrong. I don't have the problem with the demo I’m going to make now because I know it’s probably going to fail. I’m at least 80% sure it will fail.
00:02:39.440
But actually, it doesn’t matter; you’ll see in a second why.
00:02:52.080
So, what I’m going to do is show you an application I’ve built. This is actually going to be quite difficult.
00:03:05.680
I created a small application that is supposed to measure your heart rate in the browser. Let’s see how this goes.
00:03:34.320
So, I placed my thumb on the camera, but this doesn’t actually fit. Sorry, everyone.
00:03:41.600
This is seriously difficult.
00:03:48.000
Alright, yes, good. So it failed.
00:04:00.480
The reason I’m showing you this is that I can show you why it failed. The problem is that the data is too noisy, and I cannot detect the heart rate well enough.
00:04:11.840
This is not actually the key point here; the point is that I was able to do this all in the browser. So, why did I attempt this?
00:04:23.440
I came across an article from the book 'Exploring Everyday Things with R and Ruby' that spoke to me. I highly recommend reading it, as it beautifully describes a complex process in a simple manner.
00:04:38.560
I come from a front-end background, and this really resonated with me because it addresses a very human problem involving hardware.
00:04:54.960
I realized I actually have all the tools to accomplish this right in the browser, so why not? It turned out to be doable.
00:05:06.720
Just to quickly explain the tools used, they require a good camera that can take a series of images, which you then read into your machine.
00:05:23.120
The code described there would take over, but in the browser, I can accomplish everything in one go. I have direct access to the camera.
00:05:36.559
I read a few frames using the camera and push them into a canvas. This setup is necessary, and I have the preview screen that you saw a moment ago.
00:05:51.440
I have some methods to track whether recording is running, and the last line gives us access to the camera, which has been possible in some browsers for a while.
00:06:07.680
This key piece made me want to try to work with the browser because I can perform everything seamlessly. While the developers who used Ruby and R would have to use an external camera and move files around, we can do it all in the browser.
00:06:21.920
I have access to data in real-time, but I’ll describe the technical aspects of how it works, as it highlights an interesting point.
00:06:33.200
You snap a few pictures, read them into a canvas, and process the data. Unfortunately, it’s quite bright here, and I might struggle to show you.
00:06:47.840
So we read a couple of frames using the camera and push them into a canvas. This is the setup required.
00:06:58.679
I have methods that keep track of whether recording has started or is currently running.
00:07:05.839
This is significant because now I can access the camera in the browser.
00:07:15.760
The browser can process pixels using requestAnimationFrame repeatedly. I also capture data of pixels for a defined duration.
00:07:24.960
Each pixel is represented by RGBA values, and I mainly focus on retrieving the red value.
00:07:32.959
This process allows me to determine how red a frame is and correlate it with heart rate.
00:07:41.200
So why does this not work well? It’s mainly that the camera isn’t good enough, and lighting conditions can be problematic.
00:07:51.760
Additionally, the algorithm detecting the peaks might be suboptimal, but injecting fake results works well in the browser.
00:08:01.920
There’s so much more that can be accomplished in today’s browsers than we’re currently doing.
00:08:10.080
You need to implement some shims to ensure broader compatibility across browsers.
00:08:17.360
This piqued my interest. Although things didn’t work perfectly, I knew with enough time, I could make them work.
00:08:25.840
I started considering how we build web applications right now. The computation generally happens on the server, while we focus on the UI.
00:08:36.960
Could we shift some tasks to the client? It seems the browser can handle much more than we currently let it.
00:08:48.000
I wrote a simple script to guess MD5 hashes to experiment with this in the browser.
00:08:56.800
It’s a naive approach that provides insight when comparing performance with Ruby.
00:09:08.160
Unfortunately, I found that the browser froze after generating four characters, which was disappointing.
00:09:18.960
I was moments away from giving up, thinking that this talk might be futile, but re-evaluating helped.
00:09:29.440
Now, I wonder what types of problems consume significant server resources.
00:09:39.680
Typical web applications involve media processing, and solving these problems in the UI could work efficiently.
00:09:49.680
For instance, I wondered if we could implement video scrubbing in the browser.
00:09:57.760
I created a solution that effectively retrieves the desired thumbnails from a video.
00:10:04.720
This all happens in the browser, meaning the user can access content without server delays.
00:10:15.440
Thanks to new JavaScript APIs, we can access temporary object URLs for files, enhancing user experience.
00:10:23.920
However, issues might arise if not all video formats are supported or JavaScript is disabled.
00:10:31.200
I worked on an image annotation application using Dragonfly, which initially seemed practical.
00:10:39.360
Despite this, I learned that handling multiple image sizes became burdensome without knowing the required sizes in advance.
00:10:48.560
I addressed this by implementing client-side image processing techniques, leveraging the browser's capabilities.
00:10:56.160
This enables a more responsive user interface, even if the uploaded image isn’t available on the server.
00:11:03.680
However, the success of this approach depends on the user's experience being smooth with minimal delays.
00:11:09.600
The potential implementation of this could involve creating a jQuery plugin to simplify the image resizing process.
00:11:17.840
This would utilize the canvas object for cropping images directly within the browser.
00:11:25.360
If we shift workloads to the browser, we might enhance responsiveness and overall user satisfaction.
00:11:31.760
However, challenges with browser support are still a concern, especially around certain functionalities.
00:11:41.920
Despite the difficulties, you’ve seen examples that stretch the boundaries of what’s possible in a browser.
00:11:51.360
The heart rate measurement demo is an example; it required significant processing, which I handled using web workers.
00:12:02.480
I found out that processing too many frames simultaneously can make the browser crash due to worker limitations.
00:12:11.360
I wrote code to manage the number of active workers to avoid crashing the browser, creating a much-needed library.
00:12:21.360
So, how do you use this library? You require it, create a new job set, and track completion through callbacks.
00:12:31.520
This will take care of not exceeding a predetermined number of workers and simplify job management.
00:12:40.799
The intention is to provide a cleaner API for defining your workers in the future.
00:12:51.360
The current process for defining workers isn’t very user-friendly. The inline workers are a somewhat complex workaround.
00:13:02.240
Organizing workers to handle simultaneous tasks leads to efficiencies in handling frame capture and processing.
00:13:12.000
If we can achieve this, it opens the door to shifting workloads from the server, reducing resource strain.
00:13:23.040
I believe that providing a persistent queue for jobs completed could improve user experience and allow execution after user closure.
00:13:37.600
Although browser differences exist, we shouldn’t let that hold us back. There’s an opportunity to leverage browser capabilities.
00:13:47.920
We need to refine our practices, incorporating these new possibilities in our applications.
00:14:05.680
Embracing modern browser features can enhance user experience, offload work from servers, and meet evolving standards.
00:14:17.360
This is the encouragement I leave you with.
00:30:53.840
Thank you!
Explore all talks recorded at wroc_love.rb 2013
+30