Data-Driven Government and the Ruby Developer

by Eric Mill

The video titled "Data-Driven Government and the Ruby Developer" by Eric Mill discusses the evolving relationship between developers and governments, emphasizing the importance of open government and making data accessible to citizens. Mill introduces the concept of open government and highlights that it has gained traction in recent years, with developers increasingly leveraging government data to create valuable applications that serve the public.

Key points discussed include:
- Broad Definition of Open Government: Mill explains that open government encompasses various aspects, including transparency, participation, and accountability, extending beyond traditional notions of government corruption or lobbying.
- Government Data Accessibility: There has been a paradigm shift in how governments view their data, with many agencies now committed to making information available in machine-readable formats. This shift responds to growing public demand for more transparency in government activities.
- Innovative Applications: Mill provides numerous examples of successful applications developed using open government data, including:

- EveryBlock: Founded by Avery Colabatti, this tool aggregates local data, allowing citizens to access information specific to their neighborhoods.

- Flyontime.us: A project that analyzes flight delay data, allowing users to better understand and choose their travel times.

- USASpending.gov: This platform tracks government spending and provides insights into fiscal management.

- Collaborative Initiatives: Mill emphasizes the role collaboration plays in enhancing government services, with local governments in cities like San Francisco and Washington, D.C. encouraging developers to participate in contests that generate usable apps from public data.

- Citizen Engagement: The presentation underlines the significance of citizen engagement in promoting transparency. Mill encourages developers to leverage their skills to contribute to their communities by utilizing available data.

- Future Prospects: The talk concludes with a call to action for developers to become more involved in open government initiatives, highlighting the potential for future advancements in delivering services and improving transparency.

The overarching message of the video is that a strong collaboration between developers and government can lead to a more informed public and improved governance, as developers are increasingly seen as partners in fostering transparency and accountability.

00:00:07.279 Thanks for that warm welcome. So, my topic is a bit of a mouthful, but what I'm really here to talk about is open government.

00:00:12.480 This is a very broad term that means a lot of different things to many people. It's the kind of term that can inspire a movement, but it also encompasses a wide range of efforts by various individuals and organizations.

00:00:24.880 There are many people working on issues related to government influence and corruption, exploring how lobbyists affect the system. This is a very valid and important sector of open government.

00:00:37.360 Additionally, many journalists approach open government through the lens of the Freedom of Information Act, secret courts, and executive privilege. However, there is something relatively new that focuses on data.

00:00:55.440 This talk won’t be super technical like some of the others; I’ll be bouncing around a lot of different ideas. Much of it will focus on what has been happening with government data.

00:01:07.760 Historically, the government has not concerned itself with data. But now, we are demanding as much information as possible, even the seemingly boring details—about bats and birds, for example—which can be quite valuable. We want this data to be available in machine-readable formats.

00:01:31.280 Three or four years ago, the request for such data was often seen as a pipe dream by a small community advocating for open government. Now, it's becoming a reality, and I will show you many examples of this initiative.

00:01:54.479 Another way to view open government as a movement is to consider how it operates: we want a government that is participatory, responsive, and transparent.

00:02:05.920 I come from Washington, D.C., which is a hub of happy hours, meetups, and bar camps focused on new media. You've probably heard the term '2.0' used frequently. We’re used to hearing the term 'Web 2.0' for years, but it often means very little.

00:02:21.760 Similarly, 'Government 2.0' has been widely discussed and means various things. I personally have a problem with appending '2.0' to concepts, as real progress refers to minor version numbers and signifies incremental improvement.

00:02:44.959 Real progress also entails taking risks, experiencing some failures—even publicly—and ultimately, shipping something. What I want to discuss now is what people inside and outside the government have been successfully shipping.

00:03:14.159 There are many initiatives occurring at local, state, and national levels. For example, San Francisco has made significant strides in open government.

00:03:27.840 The official website of San Francisco's government offers a variety of public data, including information about the environment, housing, and crime.

00:03:39.200 They showcase apps created by individuals who are not part of the government, including tools that many here would likely find useful. They keep this list updated regularly, demonstrating their commitment to transparency.

00:04:00.959 Much of the development in San Francisco focuses on crime, mobile apps, park systems, and public transit, which are all vital and interesting areas for innovation.

00:04:22.079 In fact, back in 2005, Avery Colabatti, who was running the Django project at the time, matched data from the Chicago crime blog with Google Maps. This was before the Google Maps API existed, so he reverse-engineered it with JavaScript.

00:04:44.240 He embedded recent Chicago crime incidents onto a map, which was quite significant at the time. The Chicago Police Department was taken aback by this, as it empowered citizens to be more informed regarding local crime.

00:05:02.400 Colabatti later launched EveryBlock, which aggregates data from multiple sources, allowing people to access information specific to their neighborhoods and wards.

00:05:29.039 Moving on, initiatives are also flourishing in other cities. For instance, New York City has developed its own data catalog called the Data Mine over the past few years.

00:05:53.280 Local governments are now conducting contests to encourage developers to create applications that utilize the data they provide. This practice was uncommon in 2006, but it has gained traction in recent years.

00:06:19.679 Many useful apps have emerged from New York City’s data, such as locating the nearest subway stop or building a taxi social network for users to review their taxi experiences.

00:06:29.760 In Washington, D.C., where I am from, there’s a notable story related to the New York City Data Mine’s launch. The initial version included a spreadsheet containing the contact details of women's organizations, which was inadvertently published to the public.

00:06:53.760 Although the emails and passwords were scrubbed, the spreadsheet still contained secret questions and answers, leading to a significant public relations issue.

00:07:08.960 Despite this, the launch represented a public success, and since then the city has continued to publish data consistently. However, many governments are wary about making their data open due to the large amount of information they handle.

00:07:40.480 Washington, D.C., has been a pioneer in many respects. The current Chief Information Officer (CIO) of the country started their career as the Chief Technology Officer of D.C., and they initiated a program called Absolute Democracy.

00:08:12.960 This was one of the first contests conducted by a local government to engage developers and encourage contributions to improve the city. The team organized a contest with a reward of $50,000, successfully generating 47 apps in just one month, which they valued at approximately $2.3 million.

00:08:45.520 Cities are increasingly adopting this approach, transitioning from a perspective of viewing developers solely as critics to recognizing them as vital collaborators.

00:09:07.760 Like other cities, Washington, D.C., has a data catalog that contains a wide variety of crime statistics, economic data, and 311 data, which illustrates the trend towards making government data more accessible.

00:09:25.120 California has also begun publishing its data catalog and has introduced several different datasets, although I haven’t explored this extensively myself. The national effort began in earnest in 2009.

00:09:50.320 The focus of this presentation is largely what has been happening over the past few years, and how it relates to the broader evolution regarding government data. This is not about partisan politics or any particular administration.

00:10:09.280 There’s a centralized catalog of data being produced by various federal agencies, aiming to consolidate where this information can be found. It’s not about creating new data but rather organizing it for public access.

00:10:28.720 One example is a feed from the Food and Drug Administration concerning recalls, which allows the public easy access to relevant information. Much of this raw data comes in CSV or XML formats.

00:10:50.720 As developments progress, RDF has become a focus, with lots of datasets available in various formats. I'm now going to showcase some of the innovative applications that have arisen from these efforts.

00:11:14.159 One notable project is flyontime.us, which emerged from a contest hosted by the Sunlight Foundation. This contest was inspired by the aforementioned Absolute Democracy contest, and it invited people to use data from data.gov to build innovative applications.

00:11:38.400 One developer took data released by the Federal Aviation Administration (FAA) relating to flight delays and developed a tool that allows users to analyze these statistics for their flights. It’s interesting to note that you might see similar functionality on sites like Kayak.

00:11:58.560 The developer created a searchable interface for users to analyze past delays based on numerous factors, providing insight into the best times to fly, including data on current delays.

00:12:22.680 The site also includes a RESTful API that makes this data even more accessible. This application was built simply as a public service, demonstrating the potential of grassroots efforts using government data.

00:12:44.400 The next project is USASpending.gov, which tracks government spending across various sectors. It was born from a collaboration between then-Senator Obama and Senator Coburn, one of the most fiscally conservative members of the Senate.

00:13:04.480 They recognized the need for better transparency and worked together to pass a bill mandating USASpending.gov, which provides robust insights into various government expenditures in multiple formats.

00:13:35.280 While USASpending.gov has faced redesigns and improvements, it also highlights the fact that not all published data is necessarily of high quality. My organization, the Sunlight Foundation, recently launched ClearSpending.

00:14:00.880 This new tool analyzes the reliability of the data on USASpending.gov, revealing significant discrepancies and misreported funds that could total over a trillion dollars. This indicates systemic issues within large bureaucracies.

00:14:28.720 Quality control remains a challenge, as various agencies might report spending inaccurately due to internal priorities. It is vital for the government to maintain an accurate and reliable data reporting framework.

00:14:47.920 However, not all attempts at transparency are doomed to fail; many efforts to release spending data result in useful insights that help identify areas for improvement.

00:15:03.440 Next, let's discuss the legislative branch. The Library of Congress runs a service called THOMAS, which provides access to bill information and legislative processes.

00:15:16.720 Although THOMAS does not output data in machine-readable formats, it does provide useful information online. Users can access bill texts, co-sponsors, and related documents.

00:15:33.720 In 2004, the govtrack.us site was created, which serves as a valuable front-end for congressional bills and actions, publishing all legislative data for public use in XML format.

00:15:50.640 GovTrack allows users to track bills and engage with congressional activities. OpenCongress is another platform built based on GovTrack data, providing a space for public discussion around legislation.

00:16:09.599 These platforms have successfully motivated Congress to publish their voting data in XML format. Legislative information is increasingly being made available in real-time, which enables developers to create applications based on current legislative developments.

00:16:27.760 At the state level, the legislative landscape is often where citizens can engage most meaningfully, but we’ve seen a decrease in public and journalist engagement with state politics as traditional news organizations shrink.

00:16:47.440 However, some states have developed open legislative APIs to provide access to legislative data. New York, for example, has made significant progress in this area.

00:17:03.119 California is also working on a legislative API, but there is still a long way to go; not all states are offering their legislative information in open, standardized formats.

00:17:21.760 Our organization, the Sunlight Foundation, has initiated the Open State Project, which is designed to scrape legislative information from the 50 states’ websites and standardize access to this data.

00:17:41.440 Each state has a unique set of data and requirements, which adds complexity to providing consistent access. Thus, it’s essential to develop a project flexible enough to accommodate these variations.

00:18:03.600 Now, I want to share an inspiring story from a contest we hosted, where several Ruby developers became engaged in the concept of open government.

00:18:26.720 Bob Burbank, David Augustine, and Andrew Carpenter, who work at a non-profit school in the West, had never encountered open government before, but they were intrigued by our contest.

00:18:45.200 They explored data on pokemondana.gov and eventually discovered the Federal Register, which is the official journal of the U.S. government. Does anyone here know what the Federal Register is? It seems some of you are unfamiliar.

00:19:01.760 The Federal Register covers public notices about rule-making sessions and opportunities for public comment on significant legislative changes. Those in the know, such as lobbyists, leverage this information advantage.

00:19:15.760 The developers I mentioned created a project called Gulp Pulse, which won an award in our contest. They harnessed the power of the Federal Register, presenting real-time information about nearby events and notices.

00:19:38.480 Gulp Pulse automatically geolocated users, showing events that opened recently and those closing soon, including sparklines indicating activity changes in various federal agencies.

00:19:56.720 In March, the success of Gulp Pulse caught the attention of the federal government. The Office of the Federal Register recognized the project’s potential as a significant improvement over existing services.

00:20:18.160 They reached out to the developers to collaborate, allowing the project to evolve rather than go through a lengthy proposal process.

00:20:37.920 The developers were tasked with creating a more modern, visually appealing design for the Federal Register site, something distinct from typical government interfaces.

00:20:57.920 Their non-profit employer supported their dedication, allowing them to reduce their hours at work to dedicate time to this project. By working closely with the government, they leveraged their experiences to improve their ongoing work.

00:21:20.000 Now we have a new website for the Federal Register, hosted on Ruby on Rails and Amazon EC2. This platform is a public resource filled with information on government regulations and actively records changes.

00:21:41.920 In addition, it is open source, which is not common for government projects. The commitment behind this launch shows how collaborative efforts between developers and government agencies can yield successful outcomes.

00:22:00.560 This journey exemplifies the transformative power of citizen-driven innovation using government data. Just a few developers took this initiative, but it resulted in a highly functional government property that contributes positively to the public.

00:22:22.720 This movement indicates a significant shift in how the government views developers, recognizing their value as partners rather than adversaries. Increasingly, the government is beginning to see developers as customers.

00:22:44.080 Historically, government officials have justified not providing raw data by claiming it doesn’t serve citizens' needs. They believed their efforts would be better spent creating visualizations rather than making raw data accessible.

00:23:09.440 However, more citizens and developers are advocating for this change, demonstrating that a significant minority can effectively utilize and create valuable applications from the data.

00:23:37.440 As developers become more engaged with their local governments, positive transformation continues to emerge. Numerous hackathons and meetups are occurring across various cities, facilitating communication and collaboration.

00:24:04.320 While I’ve shared numerous examples of ongoing efforts, it is essential to recognize the vast potential remaining in this movement. We are currently at what I would call 'Government 1.3.' There have been significant advancements, but much more progress is required.

00:24:28.720 The need for increased engagement from citizens is clear, especially as the 2010 Census data will be published next year. The most valuable government data comes from a variety of areas.

00:24:42.640 Transparency and accountability in money and politics data are critical. Organizations like the Sunlight Foundation have worked on various transparency initiatives, including campaign finance data and lobbying information.

00:25:05.920 This data requires analysis and attention, both from developers and citizens alike. Engaging in these conversations can help illuminate issues and promote governance that is equitable and responsible.

00:25:29.120 It's vital to recognize that many civil servants within the government are aligned with these ideals and are eager to assist. They realize the benefits of collaborating with engaged citizens like you.

00:25:48.000 It’s important to distinguish that outside of citizens' voices, there can often be many unrealistic requests demanding immediate attention from government agencies. Your presence shows that you have reasoned ideas and constructive approaches to governance.

00:26:15.680 With ongoing issues surrounding public services in cities like San Francisco, there is a great opportunity for developers to contribute to positive change. Ask yourself what drives your passion in your local community.

00:26:34.640 Public parks, transportation, education—these could all be areas where your involvement makes a difference. Ultimately, you can harness the power of public data to create solutions and advocate for transparency.

00:26:52.160 Your commitment to improvement is vital; commit to the evolving landscape of government 1.4 or whatever future iteration becomes relevant. Your dedication can spark meaningful changes in government practices.

00:27:10.880 Thank you.