Talks
Speakers
Events
Topics
Sign in
Home
Talks
Speakers
Events
Topics
Leaderboard
Use
Analytics
Sign in
Suggest modification to this talk
Title
Description
by Aja Hammerly It is a fact of life: When you are running a website stuff goes wrong. Someone puts a dictionary on the keyboard and reloads your site a million times. Your mobile app hits an error state and sends messages that cause 500s on your server. An external service takes 5 times as long as normal to respond to a request. When responding to problems logs are frequently our go to for investigating events but plain logs aren’t user friendly or efficient. Using BigQuery for log investigation lets you use familiar tools like SQL to dig into your logs, extract the interesting data, and even make charts of the data. Help us caption & translate this video! http://amara.org/v/GWI9/
Date
Summarized using AI?
If this talk's summary was generated by AI, please check this box. A "Summarized using AI" badge will be displayed in the summary tab to indicate that the summary was generated using AI.
Show "Summarized using AI" badge on summary page
Summary
Markdown supported
The video titled 'Forensic Log Analysis with BigQuery' presented by Aja Hammerly at MountainWest RubyConf 2015 centers on the importance of analyzing logs effectively in the event of website performance issues. Aja discusses how common unexpected traffic spikes can lead to problems such as server errors and performance degradation, emphasizing that plain logs are often not user-friendly or efficient for troubleshooting. Instead, she advocates for the use of Google BigQuery, a tool designed for querying large datasets using SQL, to facilitate forensic log analysis after incidents occur. Key points discussed include: - **Introduction to Log Analysis**: Aja introduces the context of analyzing logs when a website faces issues due to non-malicious traffic or unexpected behavior. She shares anecdotes about her experiences with sudden spikes in requests and the challenges faced due to the volume of data. - **Challenges in Traditional Approaches**: She outlines the difficulties in troubleshooting traditional log data, mentioning the time-consuming process of data importation, the lack of SQL querying knowledge among team members, and the ineffectiveness of some log analysis tools. - **Need for Effective Tools**: Aja highlights the need for an efficient tool that can scale with data surge, like BigQuery, for forensic analysis. BigQuery helps in quickly locating anomalies by enabling ad-hoc SQL queries on massive datasets. - **Demo of BigQuery Capabilities**: She provides a demo comparing rainfall data between Seattle and Salt Lake City to demonstrate BigQuery's speed and efficiency in handling queries over large datasets, illustrating the power of familiar SQL syntax in a new context. - **Practical Application of Log Analysis**: Aja discusses setting up a simple blog application to generate logs, emphasizing the importance of structuring data for effective analysis. She details the steps to clean and upload log data into BigQuery, and how to perform meaningful queries, culminating in identifying performance bottlenecks. - **Conclusions and Benefits**: The advantages of utilizing BigQuery for forensic log analysis include improved confidence in data analysis capabilities, cost-effective querying, and the ability to perform historical data analysis retroactively. Overall, Aja's presentation illustrates how leveraging tools like BigQuery can enhance the capability to perform log analysis, ensuring effective troubleshooting and continuous performance improvement for web applications.
Suggest modifications
Cancel
