00:00:07.279
Thanks for that warm welcome. So, my topic is a bit of a mouthful, but what I'm really here to talk about is open government.
00:00:12.480
This is a very broad term that means a lot of different things to many people. It's the kind of term that can inspire a movement, but it also encompasses a wide range of efforts by various individuals and organizations.
00:00:24.880
There are many people working on issues related to government influence and corruption, exploring how lobbyists affect the system. This is a very valid and important sector of open government.
00:00:37.360
Additionally, many journalists approach open government through the lens of the Freedom of Information Act, secret courts, and executive privilege. However, there is something relatively new that focuses on data.
00:00:55.440
This talk won’t be super technical like some of the others; I’ll be bouncing around a lot of different ideas. Much of it will focus on what has been happening with government data.
00:01:07.760
Historically, the government has not concerned itself with data. But now, we are demanding as much information as possible, even the seemingly boring details—about bats and birds, for example—which can be quite valuable. We want this data to be available in machine-readable formats.
00:01:31.280
Three or four years ago, the request for such data was often seen as a pipe dream by a small community advocating for open government. Now, it's becoming a reality, and I will show you many examples of this initiative.
00:01:54.479
Another way to view open government as a movement is to consider how it operates: we want a government that is participatory, responsive, and transparent.
00:02:05.920
I come from Washington, D.C., which is a hub of happy hours, meetups, and bar camps focused on new media. You've probably heard the term '2.0' used frequently. We’re used to hearing the term 'Web 2.0' for years, but it often means very little.
00:02:21.760
Similarly, 'Government 2.0' has been widely discussed and means various things. I personally have a problem with appending '2.0' to concepts, as real progress refers to minor version numbers and signifies incremental improvement.
00:02:44.959
Real progress also entails taking risks, experiencing some failures—even publicly—and ultimately, shipping something. What I want to discuss now is what people inside and outside the government have been successfully shipping.
00:03:14.159
There are many initiatives occurring at local, state, and national levels. For example, San Francisco has made significant strides in open government.
00:03:27.840
The official website of San Francisco's government offers a variety of public data, including information about the environment, housing, and crime.
00:03:39.200
They showcase apps created by individuals who are not part of the government, including tools that many here would likely find useful. They keep this list updated regularly, demonstrating their commitment to transparency.
00:04:00.959
Much of the development in San Francisco focuses on crime, mobile apps, park systems, and public transit, which are all vital and interesting areas for innovation.
00:04:22.079
In fact, back in 2005, Avery Colabatti, who was running the Django project at the time, matched data from the Chicago crime blog with Google Maps. This was before the Google Maps API existed, so he reverse-engineered it with JavaScript.
00:04:44.240
He embedded recent Chicago crime incidents onto a map, which was quite significant at the time. The Chicago Police Department was taken aback by this, as it empowered citizens to be more informed regarding local crime.
00:05:02.400
Colabatti later launched EveryBlock, which aggregates data from multiple sources, allowing people to access information specific to their neighborhoods and wards.
00:05:29.039
Moving on, initiatives are also flourishing in other cities. For instance, New York City has developed its own data catalog called the Data Mine over the past few years.
00:05:53.280
Local governments are now conducting contests to encourage developers to create applications that utilize the data they provide. This practice was uncommon in 2006, but it has gained traction in recent years.
00:06:19.679
Many useful apps have emerged from New York City’s data, such as locating the nearest subway stop or building a taxi social network for users to review their taxi experiences.
00:06:29.760
In Washington, D.C., where I am from, there’s a notable story related to the New York City Data Mine’s launch. The initial version included a spreadsheet containing the contact details of women's organizations, which was inadvertently published to the public.
00:06:53.760
Although the emails and passwords were scrubbed, the spreadsheet still contained secret questions and answers, leading to a significant public relations issue.
00:07:08.960
Despite this, the launch represented a public success, and since then the city has continued to publish data consistently. However, many governments are wary about making their data open due to the large amount of information they handle.
00:07:40.480
Washington, D.C., has been a pioneer in many respects. The current Chief Information Officer (CIO) of the country started their career as the Chief Technology Officer of D.C., and they initiated a program called Absolute Democracy.
00:08:12.960
This was one of the first contests conducted by a local government to engage developers and encourage contributions to improve the city. The team organized a contest with a reward of $50,000, successfully generating 47 apps in just one month, which they valued at approximately $2.3 million.
00:08:45.520
Cities are increasingly adopting this approach, transitioning from a perspective of viewing developers solely as critics to recognizing them as vital collaborators.
00:09:07.760
Like other cities, Washington, D.C., has a data catalog that contains a wide variety of crime statistics, economic data, and 311 data, which illustrates the trend towards making government data more accessible.
00:09:25.120
California has also begun publishing its data catalog and has introduced several different datasets, although I haven’t explored this extensively myself. The national effort began in earnest in 2009.
00:09:50.320
The focus of this presentation is largely what has been happening over the past few years, and how it relates to the broader evolution regarding government data. This is not about partisan politics or any particular administration.
00:10:09.280
There’s a centralized catalog of data being produced by various federal agencies, aiming to consolidate where this information can be found. It’s not about creating new data but rather organizing it for public access.
00:10:28.720
One example is a feed from the Food and Drug Administration concerning recalls, which allows the public easy access to relevant information. Much of this raw data comes in CSV or XML formats.
00:10:50.720
As developments progress, RDF has become a focus, with lots of datasets available in various formats. I'm now going to showcase some of the innovative applications that have arisen from these efforts.
00:11:14.159
One notable project is flyontime.us, which emerged from a contest hosted by the Sunlight Foundation. This contest was inspired by the aforementioned Absolute Democracy contest, and it invited people to use data from data.gov to build innovative applications.
00:11:38.400
One developer took data released by the Federal Aviation Administration (FAA) relating to flight delays and developed a tool that allows users to analyze these statistics for their flights. It’s interesting to note that you might see similar functionality on sites like Kayak.
00:11:58.560
The developer created a searchable interface for users to analyze past delays based on numerous factors, providing insight into the best times to fly, including data on current delays.
00:12:22.680
The site also includes a RESTful API that makes this data even more accessible. This application was built simply as a public service, demonstrating the potential of grassroots efforts using government data.
00:12:44.400
The next project is USASpending.gov, which tracks government spending across various sectors. It was born from a collaboration between then-Senator Obama and Senator Coburn, one of the most fiscally conservative members of the Senate.
00:13:04.480
They recognized the need for better transparency and worked together to pass a bill mandating USASpending.gov, which provides robust insights into various government expenditures in multiple formats.
00:13:35.280
While USASpending.gov has faced redesigns and improvements, it also highlights the fact that not all published data is necessarily of high quality. My organization, the Sunlight Foundation, recently launched ClearSpending.
00:14:00.880
This new tool analyzes the reliability of the data on USASpending.gov, revealing significant discrepancies and misreported funds that could total over a trillion dollars. This indicates systemic issues within large bureaucracies.
00:14:28.720
Quality control remains a challenge, as various agencies might report spending inaccurately due to internal priorities. It is vital for the government to maintain an accurate and reliable data reporting framework.
00:14:47.920
However, not all attempts at transparency are doomed to fail; many efforts to release spending data result in useful insights that help identify areas for improvement.
00:15:03.440
Next, let's discuss the legislative branch. The Library of Congress runs a service called THOMAS, which provides access to bill information and legislative processes.
00:15:16.720
Although THOMAS does not output data in machine-readable formats, it does provide useful information online. Users can access bill texts, co-sponsors, and related documents.
00:15:33.720
In 2004, the govtrack.us site was created, which serves as a valuable front-end for congressional bills and actions, publishing all legislative data for public use in XML format.
00:15:50.640
GovTrack allows users to track bills and engage with congressional activities. OpenCongress is another platform built based on GovTrack data, providing a space for public discussion around legislation.
00:16:09.599
These platforms have successfully motivated Congress to publish their voting data in XML format. Legislative information is increasingly being made available in real-time, which enables developers to create applications based on current legislative developments.
00:16:27.760
At the state level, the legislative landscape is often where citizens can engage most meaningfully, but we’ve seen a decrease in public and journalist engagement with state politics as traditional news organizations shrink.
00:16:47.440
However, some states have developed open legislative APIs to provide access to legislative data. New York, for example, has made significant progress in this area.
00:17:03.119
California is also working on a legislative API, but there is still a long way to go; not all states are offering their legislative information in open, standardized formats.
00:17:21.760
Our organization, the Sunlight Foundation, has initiated the Open State Project, which is designed to scrape legislative information from the 50 states’ websites and standardize access to this data.
00:17:41.440
Each state has a unique set of data and requirements, which adds complexity to providing consistent access. Thus, it’s essential to develop a project flexible enough to accommodate these variations.
00:18:03.600
Now, I want to share an inspiring story from a contest we hosted, where several Ruby developers became engaged in the concept of open government.
00:18:26.720
Bob Burbank, David Augustine, and Andrew Carpenter, who work at a non-profit school in the West, had never encountered open government before, but they were intrigued by our contest.
00:18:45.200
They explored data on pokemondana.gov and eventually discovered the Federal Register, which is the official journal of the U.S. government. Does anyone here know what the Federal Register is? It seems some of you are unfamiliar.
00:19:01.760
The Federal Register covers public notices about rule-making sessions and opportunities for public comment on significant legislative changes. Those in the know, such as lobbyists, leverage this information advantage.
00:19:15.760
The developers I mentioned created a project called Gulp Pulse, which won an award in our contest. They harnessed the power of the Federal Register, presenting real-time information about nearby events and notices.
00:19:38.480
Gulp Pulse automatically geolocated users, showing events that opened recently and those closing soon, including sparklines indicating activity changes in various federal agencies.
00:19:56.720
In March, the success of Gulp Pulse caught the attention of the federal government. The Office of the Federal Register recognized the project’s potential as a significant improvement over existing services.
00:20:18.160
They reached out to the developers to collaborate, allowing the project to evolve rather than go through a lengthy proposal process.
00:20:37.920
The developers were tasked with creating a more modern, visually appealing design for the Federal Register site, something distinct from typical government interfaces.
00:20:57.920
Their non-profit employer supported their dedication, allowing them to reduce their hours at work to dedicate time to this project. By working closely with the government, they leveraged their experiences to improve their ongoing work.
00:21:20.000
Now we have a new website for the Federal Register, hosted on Ruby on Rails and Amazon EC2. This platform is a public resource filled with information on government regulations and actively records changes.
00:21:41.920
In addition, it is open source, which is not common for government projects. The commitment behind this launch shows how collaborative efforts between developers and government agencies can yield successful outcomes.
00:22:00.560
This journey exemplifies the transformative power of citizen-driven innovation using government data. Just a few developers took this initiative, but it resulted in a highly functional government property that contributes positively to the public.
00:22:22.720
This movement indicates a significant shift in how the government views developers, recognizing their value as partners rather than adversaries. Increasingly, the government is beginning to see developers as customers.
00:22:44.080
Historically, government officials have justified not providing raw data by claiming it doesn’t serve citizens' needs. They believed their efforts would be better spent creating visualizations rather than making raw data accessible.
00:23:09.440
However, more citizens and developers are advocating for this change, demonstrating that a significant minority can effectively utilize and create valuable applications from the data.
00:23:37.440
As developers become more engaged with their local governments, positive transformation continues to emerge. Numerous hackathons and meetups are occurring across various cities, facilitating communication and collaboration.
00:24:04.320
While I’ve shared numerous examples of ongoing efforts, it is essential to recognize the vast potential remaining in this movement. We are currently at what I would call 'Government 1.3.' There have been significant advancements, but much more progress is required.
00:24:28.720
The need for increased engagement from citizens is clear, especially as the 2010 Census data will be published next year. The most valuable government data comes from a variety of areas.
00:24:42.640
Transparency and accountability in money and politics data are critical. Organizations like the Sunlight Foundation have worked on various transparency initiatives, including campaign finance data and lobbying information.
00:25:05.920
This data requires analysis and attention, both from developers and citizens alike. Engaging in these conversations can help illuminate issues and promote governance that is equitable and responsible.
00:25:29.120
It's vital to recognize that many civil servants within the government are aligned with these ideals and are eager to assist. They realize the benefits of collaborating with engaged citizens like you.
00:25:48.000
It’s important to distinguish that outside of citizens' voices, there can often be many unrealistic requests demanding immediate attention from government agencies. Your presence shows that you have reasoned ideas and constructive approaches to governance.
00:26:15.680
With ongoing issues surrounding public services in cities like San Francisco, there is a great opportunity for developers to contribute to positive change. Ask yourself what drives your passion in your local community.
00:26:34.640
Public parks, transportation, education—these could all be areas where your involvement makes a difference. Ultimately, you can harness the power of public data to create solutions and advocate for transparency.
00:26:52.160
Your commitment to improvement is vital; commit to the evolving landscape of government 1.4 or whatever future iteration becomes relevant. Your dedication can spark meaningful changes in government practices.
00:27:10.880
Thank you.
