'Good Luck With That' : Tag Teaming Civic Data

by Liz Rush and Hsing-Hui Hsu

The video titled "Good Luck With That: Tag Teaming Civic Data" features speakers Liz Rush and Hsing-Hui Hsu, who share their experiences as recent graduates of the ADA Developers Academy and their journey into software development. They focus on their capstone project, a parking app aimed at helping users navigate Seattle's complex parking rules using civic data from the city’s open data initiative.

Key Points Discussed:
- Introduction to Speakers: Liz and Hsing-Hui introduce themselves and their backgrounds, touching on their diverse educational experiences leading them to programming.
- ADA Developers Academy: Overview of the ADA Academy, emphasizing its focus on training women in technology through a comprehensive curriculum that involves both instruction and internships.
- Project Selection: Joyfully discussing their project choice, the duo realized the necessity for a more user-friendly means to access parking information amid Seattle's confusing regulations, inspired by personal experiences.
- Initial Challenges: They encountered various obstacles related to accessing and using the ArcGIS server and KML files to display dynamic parking data. Their exploration led them to discover many forums with limited useful information.
- Adapting the Approach: Faced with time constraints, they adapted their approach from using KML files to implementing pin overlays as a Minimum Viable Product (MVP).
- Lessons Learned: During development, they learned the importance of mentorship, agile methodologies, and not getting caught up in perfectionism. They also highlighted the need for continuous learning and adapting tools to better meet project demands.
- Project Presentation & Community Response: They shared experiences from presenting their app, receiving both positive feedback and constructive criticism, which helped them grow and refine their ideas.
- Post-ADA Development: After their capstone, they continued to enhance the app with new features, focusing on refining user experience.
- Conclusion: Liz and Hsing-Hui emphasize that developing software is not merely about coding but involves navigating community support, mentorship, project management, and adaptability in discovering solutions. They conclude with gratitude towards their community, which helped them in finding success and shaped their identities as developers.

Overall, the talk is a reflection on the transformation from coding novices to competent developers, underscored with valuable insights about collaboration, project adaptability, and community utility in civic tech.

00:00:17.730 Okay, welcome to our talk, 'Good Luck With That: Tag Teaming Civic Data.' I'm Liz Rush.

00:00:23.740 And I'm Hsing-Hui Hsu. We are both recent graduates of the first cohort of ADA Developers Academy. We want to share the story of our year-long journey to becoming real developers and some of the lessons we learned along the way.

00:00:30.100 Presented through the lens of a case study of our ongoing project, we will show you how we learned to overcome common problems that new developers face. Throughout the talk, we will share seven takeaways that we discovered during this project.

00:00:41.380 For the senior developers in the room, we think you'll recognize the experiences and lessons we learned. For less experienced developers here today, we hope you will walk away with more tools to help bridge the gap from beginner to professional developer.

00:01:00.000 A bit about ourselves: I graduated college with a degree in clarinet performance and French—something that happened by accident. This allowed me to work in France, which was really just a way to fund my ultimate frisbee habit.

00:01:11.500 Once this strategy proved successful, I went on to learn a few other languages. At that point, I thought that talking to computers was probably even more interesting than talking to humans, so I decided to go into programming.

00:01:22.630 Before learning to code, I worked in digital marketing, transitioned into technical and legal translation, and eventually found my way into translating apps. I even got roped into manual regression testing before I started automating some of those tasks, seeing the parallels between translation and code.

00:01:35.290 Luckily for us, a new experimental program called ADA Developers Academy was just starting up. For those who haven't heard of ADA before, it's a nonprofit, tuition-free code school in Seattle focused on training women making career shifts into technology.

00:01:49.390 The program is industry-driven, with professionals, companies, and community members contributing to an open-source curriculum. It focuses on web development using Ruby on Rails, as well as general software engineering principles like software architecture, test-driven development, and agile methodologies.

00:02:01.790 The program is a year-long experience, separated into two six-month phases. The first phase consists of in-class instruction, while the second is spent in an internship with a local tech company in Seattle.

00:02:15.459 Our six-month classroom experience started with one month of learning pure Ruby, followed by Rails projects that lasted one to two weeks. These projects were completed in pairs or teams of four, leading up to a final four-week-long self-directed capstone project.

00:02:39.940 Hsing-Hui and I had previously worked together on a two-week Rails app, and that's when we discovered our individual interests and strengths. Liz likes front-end work, JavaScript frameworks, UX, and mobile, while I'm more inclined toward databases, APIs, and back-end work.

00:02:47.160 We both really enjoy the challenge of architecting a project together, so we decided to pair on our capstone project. Our app was designed to utilize the city of Seattle's open data initiative, providing users with a way to discover street parking restrictions in their area.

00:03:06.850 This project is very much an example of the 'scratch your own itch' principle. About three months before we began considering capstone projects, I borrowed my brother's car and needed to park it near my apartment.

00:03:24.970 It took me 20 minutes of searching to find a spot, and once I parked, I got out and noticed the nearest pole with parking signs. It was then that I realized there were multiple conflicting rules, leaving me more confused than before.

00:03:35.800 Both Hsing-Hui and I are car-free cyclists, so we knew intuitively that this was a problem that needed solving for more than just ourselves. Parking in Seattle is controversial, but everyone agrees that the parking signs are often incomprehensible.

00:03:55.000 Here's a particularly confusing example: it appears as if every sign contradicts the one before it. As a final cruel joke, the direction mentioned 'west of here,' when the photo was taken all facing east.

00:04:13.630 With just a week of free play in ADA before the capstone started, I spent my time doing research. I simply googled whether anyone had already solved this problem. While there were no mobile apps addressing this, it turned out that the City of Seattle had a parking restrictions map on their website.

00:04:26.229 I investigated the different datasets available through the city's open data portal, and I even managed to get in touch with someone working there—not in the parking or transportation department, but an engineer on their web team.

00:04:38.410 The City of Seattle implements a parking information map using pin overlays on map tiles to show different parking categories. Unfortunately, the map is hard to read, the categories lack clarity, and the data is static.

00:04:56.290 We thought we could do better. Our idea was to use the open data to display parking information in a simpler, more user-friendly way—such as representing available street parking with green lines and no parking zones with red lines.

00:05:14.530 Excited about the open data availability and our contact in the city, we felt optimistic. We learned that sometimes the key to solving a difficult problem is knowing that a solution is possible—or at least convincing ourselves it is.

00:05:32.110 However, knowing this would require help from the City of Seattle government, we reached out to our contact to explain our idea, hoping for feedback. He said, 'Well, if you think you can build it, you'll be the first. Good luck with that!'

00:05:38.860 Then, he hung up on us. Seriously, despite this ringing endorsement, the guy did provide us with a link to their ArcGIS server, which is what their web application used to generate the parking maps.

00:05:53.860 However, we had never used an ArcGIS server before. The website informed us about the different tools that ArcGIS provided, including a way to dynamically generate KML files for our project.

00:06:11.080 This resource seemed perfect, except it turned out not to be so perfect after all. Many of the implementation links were forums with no helpful information, and several links produced empty files, leaving us unable to extract any data.

00:06:28.950 Not knowing what to do next, we reached out to other developers on Twitter, and one developer pointed us to a fully implemented demo ArcGIS server. We found that an ArcGIS server could do much more than what the city had, including projection tools and coordinate versions.

00:06:41.470 When we contacted our city contact again, he informed us that the person who originally implemented the product was no longer there, and they had no idea what was going on with the server or how to help us.

00:07:00.740 As mentioned before, the city was using pin overlays, but to implement the features we wanted, we decided to use KML files—Keyhole Markup Language files—which are XML format files used for geo data in applications like Google Earth.

00:07:19.039 KML files contain information tied to latitude and longitude, allowing for the rendering of shapes, place markers, or lines with various colors and textures to show features like region topography and points of interest.

00:07:43.430 Fortunately, ArcGIS servers include an API endpoint that returns these KML files, but when we tried to access it on the Seattle city's map server, the KML file rendered incorrectly.

00:08:02.060 Two weeks into the four weeks allotted for our project, we began to worry that we wouldn't finish on time. Therefore, we compromised our original idea and decided to use pin overlays instead to create a Minimum Viable Product.

00:08:13.909 We thought this approach would be easier to implement, given that it was already the strategy used by the Seattle government. After some struggle with the query parameters, we successfully generated the overlays.

00:08:30.259 For the next week, we focused on building our backend and figuring out how to handle all of these overlays until we discovered that the graceful failure looked suspiciously like a success.

00:08:50.120 Faced with this issue, we decided to delve into the city government's code to understand how they dealt with this problem. The answer was that they simply commented in their 'else' block that they didn't receive any parking results, so they decided to do nothing.

00:09:03.290 Realizing this felt silly, and we weren't trying to tear down the city's code; at this stage of our project, it was comforting to know that even real developers face uncertainty.

00:09:32.809 Our JavaScript implementation also caused significant headaches. Eventually, we began receiving results corresponding to our bounding box queries and decided to use the Google Maps API as our base layer.

00:09:51.230 However, we encountered difficulty with the overlays not lining up with the map. It was perplexing because we assumed that whatever Google used, everyone else on the internet would also use.

00:10:07.309 It turned out there are many different map projections, and different regions often use various systems. We found ourselves on a twenty-part adventure, researching different kinds of projections and spheroids.

00:10:37.910 We learned that civic map servers often use flat map projections, which are accurate locally but become less accurate the further away you get from the reference point.

00:11:01.950 For example, the Seattle government sources their geoinformation using a flat map projection specific to the Washington North state plane, while the Google Maps API uses a spherical Mercator projection, better suited for global geo data.

00:11:28.680 This mismatch meant we were trying to lay a flat map on top of a round map, leading to significant inefficiencies in our progress. After much effort researching projections, we eventually found a website that cataloged all the various projections for every region.

00:11:51.210 Even when we got the correct spatial reference ID for the Washington state plane, we struggled to implement it effectively in our queries against the Seattle map server.

00:12:04.030 In a stroke of luck, while experimenting with spatial reference codes, I inadvertently input the wrong number into the right box, and it yielded exactly what we sought, despite being the opposite of what any guide suggested.

00:12:15.800 With the correct pin files now available, we needed a way to store them. Recognizing the potential for building both a front-end web client and an iOS app, we didn’t want to overload the city’s map server with requests.

00:12:40.130 Thus, we separated our project into two Rails apps: a backend API that would make requests and cache the city's data to serve a frontend web client. This forced us to consider performance optimization and request caching according to previous parameters.

00:12:55.300 We used the CarrierWave gem to cache the pin files grabbed from the city server, storing them in an Amazon S3 bucket. Given that CarrierWave took around 14 to 15 seconds to complete uploads, we set the uploads to run as background jobs.

00:13:22.100 In the meantime, we sent the client a temporary file so they could see the map overlay immediately. Any further requests made with the same bounding box parameters would return the cached file.

00:13:39.810 As April came to an end, we had wrapped up the classroom portion of ADA—six months down and only six months left. We reached a pivotal point in our journey toward becoming developers.

00:13:59.000 We created a minimum viable product to show off on presentation day. We utilized the same pin overlays employed by the city but made our client mobile responsive, ensuring that all important information was easily accessible through touch-responsive JavaScript tooltips.

00:14:43.220 Although these were small improvements, they significantly increased accessibility for average users, providing explanations of what the categories meant. We've even received feedback from users who said this version helped them navigate parking in Seattle.

00:15:27.880 However, due to the classroom constraints, we had to cut features and scale back once we encountered unexpected roadblocks. The hard deadline pushed us to focus on delivering a usable product, even if it did not fulfill every desired feature.

00:15:40.490 This brings us to a common struggle we observed in ourselves and peers: we get so caught up in wanting to build everything perfectly that we fail to adhere to MVP strategies. After the capstone project wrapped up, we decided to continue our project with the goal of getting the app into the App Store and adding more features.

00:15:59.480 As we transitioned into internships and moved away from ADA structure, we lost sight of our MVP strategy, beginning to think about how to completely overhaul the project and all the million features we wanted to include before we felt it was ready.

00:16:10.160 What actually happened was we floundered under the weight of this extensive work and made minimal progress. In fact, after we began presenting our project's progress, someone inadvertently submitted a copycat app to the App Store because they committed to the MVP principle.

00:16:25.960 Lesson learned: don't forget to incrementally build your project. Focusing on perfection rather than creating a whole, albeit imperfect, product can hinder progress.

00:16:43.090 For us, as we wrapped up our MVP and transitioned from students to real developers, mentorship proved immensely beneficial. We were lucky to have a fantastic iOS contract developer named John Rush, who we compensated solely in donuts and coffee.

00:17:05.960 John guided us while still allowing us to decide the direction of the product, the design, and the user experience of the iOS app. Additionally, I had two mentors from my internship company who continued mentoring me throughout the capstone and into my internship.

00:17:20.330 Having continuous mentorship throughout such a large project turned out to be incredibly valuable. We had experienced developers to ask questions like how to use MongoDB, and they offered constructive feedback instead of one-off discussions.

00:17:38.579 These mentors understood our skill level and possessed an overarching knowledge of our project history, allowing them to guide us without direct intervention in our code. Supportive mentorship helped keep our motivation up as we pursued the project post-ADA.

00:17:55.890 We established a long-term mentorship relationship with John, who witnessed our project evolve from a seed of an idea into our MVP and helped us build our first iOS app. While John offered expertise on iOS development, other mentors provided guidance on broader software and product development.

00:18:12.360 If you don't have a friend or sibling to help you with your project, consider reaching out to local meetup groups or Twitter to connect with supportive developers. There are many in the community willing to assist beginners and junior developers.

00:18:28.520 Returning to our MVP, we presented our app in front of our ADA sponsors, mentors, future internship bosses, as well as friends and family. We were also invited to showcase our project at a local meetup for civic technology enthusiasts.

00:18:44.740 While many attendees responded positively, we did encounter some discouragement from spectators who wanted to point out every single flaw in our work. At another local Ruby meetup, someone attempted to break our app during our presentation.

00:19:06.350 While this was rude, it felt like being considered equals with a professional developer. We felt we were navigating an awkward phase between beginner and junior developer, trying to find our footing.

00:19:30.380 One of my co-workers trying to learn how to be an effective mentor attended a presentation we gave at another meetup. His first response was, 'That was better than I expected,' which, while not intended maliciously, illustrated how people in the community sometimes struggle to provide support and encouragement.

00:19:57.340 We often found ourselves serving as ambassadors for alternative learning paradigms like ADA. As we sought our place within the tech community, we needed to cultivate an understanding of how to assist others in supporting us.

00:20:15.320 What we learned was that code is only a small part of what makes development challenging. We had to clarify our product vision, identify what success would look like, and integrate into the community while fostering a collaborative environment.

00:20:28.510 This involved scheduling work days to meet and collaborate face-to-face, using tools like Pivotal Tracker to track our progress. One vital lesson we learned the hard way was not to code when you're hangry!

00:20:44.480 As I said, with Liz established as my boss, we approached our phase two app by following the original idea of manipulating the lines on the actual overlay. We eventually found a KML file with citywide data on the open data website but encountered difficulties parsing it.

00:21:05.480 The first rule of parsing KML is similar to the first rule of XML parsing: don’t do it. Though we pushed through to some degree of success, we ultimately discovered that the file didn’t contain all the data we needed.

00:21:35.920 Next, we found that different files contained the needed information, specifically shapefiles—another data format storing geo data in a non-human-readable digital vector format that required a third-party library for interpretation.

00:21:54.570 Frustrated but willing to explore, we sought gems suitable for this task, but open-source options like RGeo and GeoRuby proved insufficient. We learned that a gem was in development by S3, but it wasn't ready for release.

00:22:14.220 This stage became quite frustrating as we threw spaghetti at the wall, trying various ideas until we reached dead ends. It took us a while to realize that this exhaustive effort is what seasoned developers term 'spiking.'

00:22:29.917 Eventually, we discovered that Python libraries are excellent at handling shapefiles and performing necessary conversions. Although we wanted to keep our back-end API as a Rails app, we decided to integrate a Python script to handle shapefile processing as a cron job.

00:22:59.800 Hsing-Hui was working in Python during her internship, so having that language available proved to be beneficial. We delved into questioning whether Ruby was even the right tool, as this is a hard lesson for beginners.

00:23:22.370 It's a challenge to know what you don’t know; translating that can be tricky if you’ve only just begun learning. As the saying goes, when you only have a hammer, every problem seems like a nail.

00:23:38.290 Ultimately, we realized Python was the better tool for parsing shapefiles and coordinate conversions and decided to replace part of our backend with that Python script. Unfortunately, the data was inaccurate.

00:23:54.640 While our city contact assured us that the data was maintained and updated daily, we discovered that the last update on the open data site was in 2011! So, we pivoted back to the ArcGIS map server.

00:24:18.200 To connect public data to the city’s map server, we used geometry object information provided in a CSV on the open data website, such as the unit ID number, allowing us to query the ArcGIS server directly.

00:24:32.490 Our ongoing implementation now leverages both the open data and the city’s map server, meaning we potentially had to query their server over 2,300 times to ensure we had the most current data for every part of the city.

00:24:52.060 Our MVP iOS app, in its current state, is simplistic yet functional. It uses MapKit to display parking data, where the lines are color-coded: red means no parking, green indicates available parking, and yellow implies possible restrictions.

00:25:00.210 The app also includes filtering options to customize queries. Users can adjust their desired parking duration or maximum payment, and the dynamically updated lines will reflect their preferences.

00:25:17.660 Moreover, there's a profile system allowing users to enter their default filter preferences, including residential parking permits. If someone lives in a restricted area but has a permit, those lines will turn green for them.

00:25:32.040 We've also implemented a favoriting feature to allow users to store data for quick access. For those who struggle to remember where they parked, they can easily save their location and return to it later.

00:25:47.440 The next steps for the app include cleaning up the existing codebase and submitting it to the App Store. We aim to do this as soon as possible to gather user feedback.

00:26:01.720 In the upcoming months, we plan to transition to Swift for our iOS development and move our web client to Ember to maintain consistency with dynamically updated line rendering for parking data.

00:26:12.370 Following our presentation at Cascadia Ruby last August, we decided to take some time off due to burnout; we grew tired of the parking topic and faced challenging internships.

00:26:30.700 We realized we needed to refocus on completing ADA and preparing for job interviews. This was a tough lesson but highlighted the importance of sometimes deprioritizing side projects.

00:26:43.810 During milestones, such as our first conference talk or deploying a new feature, celebrating victories—no matter how small—is essential. Taking time off to reconnect with goals and reassess plans can be more valuable than endless hours of half-hearted coding.

00:27:05.130 It’s crucial to remember that working with someone should ideally feel like a friendship. Despite only a few months of training and having no coding experience before ADA, the hardest part was often not writing code; rather, it was dealing with uncontrolled data that lacked documentation.

00:27:22.500 We became increasingly aware that challenges didn’t follow a straight path to resolution, even if a task seems simple at first glance. This ambiguity made us reevaluate our expectations of what we believed was possible.

00:27:44.890 As we faced ambiguity, we had to redefine our understanding of success. Occasionally, building something innovative isn't as straightforward as just clicking the first Stack Overflow result that appears.

00:28:05.090 This process was frustrating but empowering, demonstrating that even developers in industry don’t have all the answers. We, too, belong to that group of individuals who may not always know what they're doing.

00:28:23.510 Initially, our goal was to create an app that transformed the utilization of open data, resulting in something interactive, user-friendly, and mobile-compatible. However, we settled for an MVP that diverged from our original vision due to the deadline of our four-week capstone.

00:28:40.150 That said, our MVP was something to be proud of. It was both exciting and a little terrifying to realize that, after just five months of coding, we could build something functional and envision a path to a more refined product.

00:29:00.090 Moreover, we appreciated the supportive community in Seattle, which proved committed to aiding us in our endeavors. Despite the occasionally awkward or inconsiderate feedback, we gained valuable lessons from presenting our app.

00:29:16.300 These experiences helped us establish our identity within the tech scene, compelling us to confront challenging questions that ultimately contributed to our growth as developers. We learned which features excited our audience and heard citizens' stories detailing their parking frustrations.

00:29:33.480 Receiving validation about our direction in product development boosted our morale. As new developers, it's easy to feel lost when working on self-directed projects, but sharing progress with fellow developers enables quicker feedback.

00:29:51.970 Meeting other developers who have tackled similar issues reassured us that when a tool fails, it's often not our fault—the tool might simply be broken. In conclusion, remember that presenting your work to receive feedback is one of the best practices.

00:30:06.289 Realizing that investigative efforts in potential solutions were worthwhile helped us understand that outcomes weren't just about usable code. This was a crucial lesson for us; each developer’s journey, especially in ADA, is shaped by the pivot away from traditional career paths into tech.

00:30:26.690 It instilled in us the importance of persistent exploration. Learning how programming projects naturally evolve taught us resilience and patience, qualities critical in a developer’s toolkit.

00:30:42.210 As we wrap up, we want to shout out to those who have supported us along the way. We received community support through our crowdfunding campaign, which helped us get to this event.

00:31:01.400 A special thanks goes out to our fantastic friends, peers, and community members who contributed. Also, a huge shoutout to our corporate sponsors who significantly helped with our travel costs today.

00:31:16.100 Once again, I'm Hsing-Hui Hsu.

00:31:37.700 And I'm Liz Rush. We hope you enjoyed our talk. Thank you for coming!