One of the under-reported stories in the recent revelations about the NSA's surveillance program is that the U.S. Government — the world’s largest consumer of information technology — administers a vast reservoir of information that is not only publicly available, but encouraged to be shared freely. In an effort to spur innovation and improve the quality of services, this Digital Government Strategy grants all Americans access to high-quality datasets in categories ranging from Energy to Education, made available by agencies from the Geological Survey to NASA.

However, chances are you weren't aware that these mountains of data are available, or if you were, you wouldn't know what to do with it. "We have a lot of broad data," former Presidential Innovation Fellow Ian Kalin explains, "but we lock it behind websites where we put it in PDF files or just take forever to publish it." The dilemma, in Kalin's words, is that "while the government is very good at wholesale, it usually needs help with retail."

This begs the question: What if all the data the government has compiled — from alternate transportation fuel consumption to median rent estimates for fair market rent areas — was being used by innovators and business people, tech geeks and artists, civic activists and city leaders, to co-create real solutions for real people in real places? Would it be possible to bring different stakeholders together to turn the government's data warehouses into well-organized storefronts, accessible to all?

The answer, in short, is yes.  And a solution is Data Jams.

Data Jams

Formerly known as Joint Application Modeling Sessions, Data Jams originated when Todd Park, the current Chief Technology Officer (CTO) of the United States, was appointed CTO at the Department of Health and Human Services (HHS) in 2009 after a career as a health care entrepreneur. Park knew that all the information the government stored could help people, but that the data by itself was useless. It needed to be made available and transformed by innovators to become valuable.

Kalin says Park's original idea was to lock 50 entrepreneurs in a room, throw the data on the table, and say, "Okay, you're all here, clearly we're all trying to solve the same problems. We (the Government) can't do it ourselves, but we'd love to do everything in our power to help the private sector come up with solutions without getting in your way."

While somewhat similar to hackathons, data jams tried to be nothing like them. "We love hackathons, but we hate them, too," Kalin admits, adding that even though the White House was already hosting hackathons, it's usually hard to get a product of meaningful value out of them. "I don't know a lot of people that have used a hackathon product in their daily lives. The truth is, if there's no market interaction of these ideas, that product is usually going to fail."

Data jams were intended as a remedy for this shortcoming: neither the fun weekend events nor the multi-year process of large government organizations, but the missing middle. Usually about 90-100 days, they are long enough to come up with new products and services, yet short enough for creative people in companies' development departments to volunteer their time.

Saving Lives and Energy

On March 11, 2011, Park tasked 45 leaders in the health care and tech industries who had never met to spend the next 90 days building technology tools around massive amounts of health care data the HHS was about to release. Since then, data jams have spread through every major federal agency, from the EPA to Treasury, Transportation, Education, and Energy. One of the most famous health care data jams brought together a group of emergency room doctors from around the country who created an app to help people figure out where to go for treatment. "Very few apps include the phrase 'this app saves my life,'" Kalin laughs.

Not to be outdone, an energy conservation jam resulted in one of the most fruitful sessions to date, the Green Button Initiative. Starting with the simple question, "Why is it that you can have a check engine light for your car, but not for your home?" one of the participants offered his company's smart-grid technology to test whether homes were wasting energy via the line graphs in the homeowner's energy bill. The following weeks, business, non-profit, and government representatives — some of them not even specializing in the energy sector — collaborated on refining the Green Button idea.

100 days later, the group introduced the Green Button Data Standard at a White House Datapalooza, an event that brings together thousands of people to showcase their data jam inventions on a big stage. Kalin says that in order to keep Data Jams creative they have to stay intimate, because once you need a microphone, "this bizarre thing happens where people stop interacting." The Datapalooza exposure and subsequent capital investment turned the Green Button into a national energy efficiency program spanning 20% of the country's utilities and 36 million homes and businesses.

Housing Data Jam

While a national scope has proven to be effective, the intimate nature of Data Jams lends itself to tackling local challenges. For example, fresh on the heels of its new Open Data Ordinance, San Francisco's Mayor's Office of Civic Innovation began co-hosting a Housing Data Jam last month with the California Governor’s Office of Business and Economic Development and Code for America, a new non-profit that brings coding expertise to the government.

The SF Housing Data Jam was launched during a half-day event that brought together tech entrepreneurs, civic activists, and city, state, and federal representatives with different perspectives on housing issues to brainstorm new services that could be developed in areas related to housing, like building safety, homelessness, or energy efficiency.

The value of having people from diverse backgrounds collaborate became apparent during the breakout sessions, when, for example, nonprofit leaders pointed out to entrepreneurs that Section 8 Housing vouchers aren't of much use in a tight housing market. After spirited exchanges in these smaller groups, everyone reconvened to vote on the Top 8 ideas (Community Crowdfunded Real Estate, anyone?) around which teams were formed to deliver working prototypes within 100 days.

Already, one of the interim results from the Housing Data Jam has been the announcement of a Healthy Housing Data Standard that provides a uniform format for reporting government data on the health and safety of residential buildings. "Admittedly, standards aren't very sexy," Kalin, who was instrumental in organizing SF's Housing Data Jam, concedes, "but it's an important way in which cities can work together. For example, what if there were a standardized inspection for lead poisoning or mold in housing developments, so cities could communicate and learn from each other in trying to improve conditions. Or an app to alert you to a bad landlord or an unsafe building in SF that would also work when you move to LA?"

Pushing the ball forward

Only five weeks into the SF Housing Data Jam, the cities of Las Vegas, Kansas City, and Bloomington as well as companies like Accela, Socrata, and the National Center for Healthy Housing have signed on to this new specification. And yet, while some of the top ideas can get 20-30 people volunteering their private time to carry them to fruition, others might not be so lucky. Kalin says that it's not uncommon for data jams to sputter, fizzle, or, excuse the pun, jam. "People get stuck in between data jams and datapaloozas for any number of reasons — no time, no money, data is in wrong format, or they get distracted by their actual jobs."

While these are all good reasons to get interrupted, Kalin points to a few things that have kept projects moving forward. For one, it helps if participants think that in addition to making a positive impact on society they can also make money on it. Perhaps even more important has been the civic commitment level. For example, having a custodian from a government agency usually increases engagement and trust in the process.

Also, having organizations like Code for America offer coding expertise and volunteer help has proven to be invaluable. In the case of San Francisco, having a full-time Code for America fellow in mayor's office, along with support from Chief Innovation Officer Jay Nath and Mayor's Innovation Fellow Krista Canellakis, has better ensured that projects keep moving forward.

Chief Innovation Officer Jay Nath talks about connected empowerment in San Francisco.

As we're all glued to our various screens in suspense over Edward Snowden's fate and debating how much private data the government should have access to, it's a good opportunity to direct some of our attention to the mountain of open data the government is making freely available to all of us. For if we are truly en route to a kinder, wiser, and more sustainable world, we'd be well advised to not only remove the needles in the haystack that might hurt us, but to find the ones that are going to help us weave the fabric of a more collaborative society.