Author: Gunnar Hellekson

The future of the government forges

The GSA is currently planning forge.gov, which is widely assumed to be based on forge.mil, the much-discussed collaboration platform from the Defense Information Systems Agency, or DISA. forge.mil is a pretty incredible idea: a single destination for testing, certification, and software development in the Defense Department.

It sounds obvious, but the idea remains revolutionary. For the first time, there would be a single repository for source code that could be shared between the hundreds of agencies, commands, and programs in DOD. Developers would be able to share their work in a familiar, web-based environment. A previous version of forge.mil was pulled for unknown reasons, but the current iteration is based on the TeamForge product from CollabNet. If you’ve used SourceForge, you get the idea. The DOD is the largest consumer, and one of the largest developers of software in the world. Much of this software is redundant, locked up by vendors and integrators, can’t work with other software, and nobody remembers how to maintain it. There’s no doubt forge.mil was long overdue.

It’s dangerous, though,  to assume that forge.mil is a useful template for forge.gov. I think forge.mil could lead forge.gov down the same road as core.gov and other failed attempts to encourage source code reuse in government. To understand why forge.mil can be useful and simultaneously poisonous to forge.gov, you have to first understand how the DOD does software.

COTS vs. GOTS

Before, say, the mid-1990s, much of the DOD’s software was owned by the government. GOTS, or “Government Off-the-Shelf” as it’s now called, was built and maintained by the DOD and its contractors. This was appropriate for some military-specific systems, but the strategy outlived its usefulness when the government could no longer keep up with commercial enterprises. For many pieces of common software, like operating systems, spreadsheets and web browsers, the open market produced more innovative and higher-quality products. So down came the order: use commercial software. COTS (“Commercial Off-the-Shelf”) was ascendant.

Boeing, Lockheed Martin, General Dynamics, and the other integrators no longer write as much software from scratch. Instead, their business model has changed. They now glue commercial software together, and wrote code to fill in the gaps. Their business model, however, remains the same. Instead of seeking rent on the proprietary software they’ve written on behalf of the government, they seek rent on the integration work they’ve done.

This is a lucrative business: the latest UAV is composed of hundreds of software systems, some commercially available, some written by the integrator. If I’m the integrator, I’m the only one who knows how all the pieces go together and I can charge a handsome sum to anyone who’d like to field their technology on my platform. Think of Apple’s locked-up App Store, but flying through the air with a missile strapped to its belly.

There’s nothing nefarious about this, of course. Integrators are doing what the market commands, and controlling access to platforms is an perfectly legitimate business model. It discourages reuse, though, which means that it’s difficult for the DOD to effectively use the software it purchased or developed. You can read more about this strange market and its consequences in the excellent “Losing the Softwar(e)” by my friend and fellow Open Source for America member, John Scott.

Openness as a Desperate Act

So Rob Vietmeyer of DISA decided to borrow from the principles of the open source community and Internet governance. A more open and transparent development process at DISA could remove barriers to reuse, encourage collaboration, and discourage proprietary or closed systems. COTS software is still king, but where the government needs to control its own integration, set its own standards, and exercise stewardship over its own infrastructure, it can still develop its own GOTS solutions — this time, in an open, collaborative manner. Where GOTS was once insular, slow-moving, and highly proprietary, it can now be produced at lower cost and with lower barriers to entry for new innovations. Because this “Open GOTS” is built using familiar open source methods, the projects have a fighting chance of working together.

The Walled Garden and the Moral Hazard

So far, the DOD software problem sounds similar to the government software problem. Don’t misunderstand forge.mil, though. It’s not your typical open source development environment. If you talk with the forge.mil project team, like Guy Martin of CollabNet, he’s quick to correct you if you compare forge.mil to SourceForge. forge.mil is something very different. With all the challenges of this profoundly broken market, forge.mil had to make some serious sacrifices.

The first thing you’ll notice about forge.mil is that you can’t get to it. Access to forge.mil is severely restricted. To get access, you must have an official DOD Common Access Card (CAC) or have a DOD employee sponsor you for an “ECA Certificate”. Dave Wheeler of the Institute for Defense Analysis describes it as “gated development.” On the mil-oss mailing list, this is referred to as the “CAC Wall”.

The CAC Wall has some unintended consequences, and raises some very difficult questions. First, it prevents many well-meaning developers who don’t work for the Defense Industrial Base from helping the projects inside. There are literally tens of thousands of developers behind that wall, and forge.mil keeps that group partitioned from both the public and government employees without CAC or ECA credentials. The bargain is that by keeping these undesirables out, they carve out a sense of safety so skittish program managers are more likely to host their code. The cost of that sense of safety is a much smaller audience than they could muster hosting on more public platforms.

The CAC Wall also creates a dangerous an incentive to split communities. What happens if someone in the DOD wants to hack on a piece of open source code? They’ll host the hack at forge.mil, and the public could never see those hacks again. If I’ve licensed my project under an open source license, it’s because I want others to contribute. If that code disappears behind the CAC Wall, I’m cut off from tens of thousands of DOD developers. This “forge.mil fork” scenario is serious business. It’s a scenario where everyone loses.

Finally, the CAC Wall may create a moral hazard for the developers who live behind it. In the open source community, folks are very careful about what code they commit, and since they’re never really sure who’s a friend and who’s a foe. In a community where everyone’s “trusted,” developers can become complacent, making them more vulnerable to poorly written or hostile code.

Despite these real concerns, the CAC Wall still makes sense for forge.mil. DISA has some very legitimate security concerns about the code that’s being developed. They’ve weighed that risk against the advantages of public scrutiny, cooperation with external projects, and they’ve made the informed decision to keep the code behind the “CAC wall”,  and to their credit they encourage forge.mil developers to contribute patches upstream wherever possible instead of hosting on forge.mil.

forge.mil as Role Model

So we have a kind of Judgement of Solomon in the forge.mil platform. Because of its very unique market dynamics, the DOD needs to take advantage of open source projects, the open source development model, encourage collaboration, and reduce its reliance on proprietary platforms. But for that to happen, DISA had to put all the work behind the CAC wall.

forge.mil has attracted the attention of other agencies. How could it not? The DOD has the worst-case scenario: the gravest problem, the most complex market conditions, and the most dire consequences. So other CIOs take notice and the press on this forge.mil experiment has been relentless.

This brings brings us to the news that the GSA’s Dave McClure is planning forge.gov, a civilian counterpart to forge.mil. This is exciting. This is also terrifying, because they seem to be following the forge.mil model by restricting access to only US citizens. Guy Martin says:

I realize that putting up a barrier to entry in the form of positive identification of US citizenship and a vetting process will irk some who believe that everything should be free and open…

Consider me irked. I want to be perfectly clear about this: the compromises made in forge.mil are dangerous for forge.gov. If forge.gov were to follow the forge.mil “CAC Wall” approach, it will permanently damage the “Open GOTS” movement.

forge.gov cannot be forge.mil

forge.mil may be instructive and inspiring, but it’s a corner-case and fraught with compromises that have diminished its utility. In the case of forge.gov, it would be hosting unclassified code for civilian agencies. There’s no need to create a “trusted” environment. There’s no need to verify the citizenship or security clearance of its participants. The standard open source mechanisms are more than sufficient: only project leaders can commit code to the repository, a semiformal review procedure for patches, and so forth. In any case, I’m struggling to imagine why the repository would be better secured by allowed access to 300 million people. Let’s agree that making US citizenship a prerequisite is counterproductive, unworkable, unnecessary and most important: it’s un-American.

The entire country of Jordan has adopted the VA’s VistA software for their national healthcare system. Countless overseas researchers collaborate with their US counterparts through open source projects. When we share our source code with the world, it improves the quality of the software and is, in fact, a uniquely practical kind of diplomacy.

Since we don’t need to control access to the projects as we do in forge.mil, I have to wonder why we would need a prescribed set of tools for hosting each project. TeamForge is a fine piece of software, but there are literally dozens of viable alternatives. Developers are very picky about their tools, and extremely picky about the version control systems that are at the heart of these code repositories. If we presumptuously select tools on their behalf, we create an unnecessary barrier to entry.

The ideal forge.gov is two forge.govs.

With this in mind, we should break the forge.gov project in two.

First, forge.gov is useful as a catalog of open source projects that are used and created by the civilian government, many of which happily reside on agency websites or public repositories already. Forge.gov could be the way that projects and developers easily find each other. The Freshmeat or Ohloh of government, if you like. Let’s call this the forge.gov Catalog. The forge.gov Catalog would be as inclusive as possible, tracking the progress of every Open GOTS project we can find.

There are still projects that need an infrastructure, of course, and forge.gov could provide that. The forge.gov Repository would be provided by the GSA as a service to agencies, and provide a complete development environment. Here, I’m thinking of github or SourceForge.

I think by splitting these roles, and avoiding the dreaded CAC Wall, we can include the largest possible group of contributors and take best advantage of the excellent open source work that’s already underway.

Am I missing something here? Is there a national security concern that I’m overlooking? Is there an advantage to a homogeneous set of developer tools that I don’t understand? Let me know.

Spook developer speaks!

Matthew BurtonI had a chance to talk with Matthew Burton, the former intelligence analyst turned open source cause celebre who just launched a tool that helps frame and understand arguments with imperfect evidence. It’s based on method called Analysis of Competing Hypotheses (ACH), which has been around for quite some time. Matthew and his friend Josh Knowles, though, have a tool that allows the ACH method to be used by multiple participants simultaneously. It’s fascinating stuff, so I’m grateful that he took the time to talk with me.

On a personal note: I’m delighted to see that Matthew is a fellow emdash enthusiast, as you’ll see below.

First, tell me a little about ACH and how you first became interested in the method.

In the fall of 2005, Dick Heuer, the creator of ACH, contacted me after reading an article I’d written for Studies in Intelligence. The article was about how Intelink could benefit by being more like the Web. Dick had been wanting for some time to build a Web-based, multi-user tool for ACH, so he asked me to build it. I spent the following summer at DS&T, interviewing ACH practitioners and trainers.

Intellectually, the most fascinating aspect of this project has been its applicability to groupthink and dissenting viewpoints. When I started, the Intelligence Community was still feeling the effects of Iraq WMD blowback. Dick referred me to a book, “Groupthink: Psychological Studies of Policy Decisions and Fiascoes,” by Irving Janis (ISBN: 0395317045). It’s a fascinating book. Janis evaluates several US policy failures from the 20th century. He not only makes it clear that a groupthink tendency had a hand in misguiding groups of otherwise brilliant men (they were all men); he also pinpoints the moments where much-needed dissent and skepticism were quashed by the desire not to disrupt the camaraderie that comes with consensus.

In my summer at DS&T, I learned that this was a problem with ACH as well. The existing software couldn’t record multiple viewpoints, meaning dissenting opinions evaporated; and analysts had to state their opinions in front of everyone else instead of from the pressure-free environment of their desks. I saw a real opportunity to make a dent, however small, in the problems that lead to intelligence failures.

Are you dogfooding? Has the hyper-rationality of ACH slipped into your real life? Did you apply ACH when you asked your wife to marry you? :)

Ha! Fortunately, I can weasel my way out of that question just by explaining the purpose of ACH. It’s a tool for discovering facts–either has already happened, or what will happen in the future.  When it comes to questions that involve personal preferences, it’s not going to work so well, because its goal is to keep you from thinking subjectively about objective matters, not help you objectify the subjective.

Were you a developer first, or an analyst first?

I actually wouldn’t call myself a developer even now. I maintain the ACH code, but most of it was written by my friend Josh Knowles, a classmate from ITP who’s collaborated with me on several projects. But I’ve been a geek for a long time, having been neck-deep in the Web since I was 15. I took that knowledge–and my expectations for the national security structure’s technological prowess — into my job at DIA. That led to disappointment, so I channeled that negative energy into a desire to change how this place works.

Does your analyst interest complement your developer interests, or are they two separate things?

Most of the development projects I work on are meant to fix a problem that is consuming me, whether it be intelligence analysis, legislative transparency, or political campaigning. I mentioned I’ve been a geek for a long time. But I consciously avoided studying computer science in college because I didn’t want to code for it for its own sake, as an end; I didn’t want to end up building Web sites for online pet stores. Very soon after joining the government, I realized that my interest in technology could be used to further a bigger goal, and I’ve been doing that ever since.

If you developed this tool while a contractor, how did you end up with this code? What kind of hurdles did you have getting a copyright?

Josh and I developed the code ourselves, so that’s why I have the copyright. We did it in collaboration with both Dick and a DC consultancy called Pherson Associates. The Phersons—two retired CIA veterans—have been training analysts to use more structured analysis techniques like ACH for years, so they and Dick were there to make sure the software doesn’t betray the methodology. We built this for an intelligence community platform called A-Space, which was supposed to cater to outside developers, much like Facebook opened its platform to third-party app developers. But it’s been two years since A-Space launched, and neither the development specifications nor the purchasing mechanics have been defined. In that time, our own lives have changed: I’m an entrepreneur handling multiple projects in New York, and the Phersons are neck-deep in their growing training business, so it makes sense for us to take the ACH software in a direction that keeps us from having to maintain dozens of licenses while still allowing us to shepherd it. That means open source.

I know you’ve spent a lot of time thinking about licensing. Can you walk me through your thinking, and how it’s evolved?

It hasn’t evolved so much as led me in infinite loops. My goals for open sourcing the code are in some ways conflicting: I think ACH can help the world make better, more fact-based decisions, so I want it to be as widely adopted as possible. But I’ve been on a personal crusade to change the way the government buys software—namely, they should spend less on it and avoid vendor lock-in. So these two goals pull me simultaneously away from and toward copyleft. Beyond copyleft, the intricacies are so fine that I find myself unable to really comprehend the eventualities that my various options would lead to. When I reach that point in my thinking, I realize that I’m probably overthinking it, so I stop. Then I do it all over again.

It sounds like you feel as though open sourcing the code would lead to less use in the government? Can you unpack that for me a bit?

I feel as though certain FOSS licenses would lead to less government adoption than would other FOSS licenses. Specifically, the government relies on lots of proprietary legacy systems that would be contaminated by GPL code. And any intelligence agency that modifies the code would be loathe to share those modifications back to the world. So according to some people I talk to, the GPL is a non-starter. However, other people say that the copyleft provision would not be invoked simply by providing GPL code to government users, as such code is only being used internally. Different people have different opinions about the GPL’s acceptability in government, which makes me think that the fate of the software depends less on my licensing decision than it does on the worldview of the prospective user.  This realization takes me back to the end of that thought loop, where I feel like I’m overthinking it.

Who do you imagine using this software, beyond government? If I work in a group of analysts, it makes sense — but do you imagine other, commercial uses? Non-commercial?

On the commercial side, anyone whose profession requires them to use the facts at hand to figure out a puzzle: pathologists, investigative journalists, detectives, investors. I’m really excited about the non-commercial side. It’s been frustrating to watch political discourse in our country devolve into nonsense that is less about the facts and more about what “feels” right. This attitude is beginning to take hold of the policy process as well, and that’s bad news. I’m under no illusions that ACH matrices will start appearing on the Senate floor, but the idea of using ACH-backed arguments to explain policy proposals to lawmakers is intriguing.  I think this falls in line with the mission of Expert Labs, and I plan to reach out to them and see if we can do something cool together. I also think universities are a good fit for ACH, as college is all about challenging your preconceived notions and teaching you the meaning of rigorous research.

If I’m a developer who’s intrigued by the ACH idea, how can I help?

Go to http://competinghypotheses.org. There, you’ll find the repository, the code documentation, and information on how to join the community.

[The site will be live in about two weeks, Matthew and Josh are working furiously on it, I’m sure. For now, you can sign up to get notified when it’s up.]

So what’s next for you and Josh? What other projects are you working on?

We’re big fans of Stack Overflow, the Q&A community for developers. We think it could be a great platform for answering citizens’ questions about government: navigating bureaucracies and legal codes is very daunting, and finding answers to simple questions—how do I amend my tax return, how do I fight a parking ticket, how do I get a permit—can be extremely frustrating.  Government agencies are experts at making you wait in lines and on the phone. We know that they answer the same questions many times a day, and that private citizens can usually add valuable information (“Don’t go at 1 pm, it’s really busy”) that the government employees can’t or won’t provide.  We’d like to form a community of confused citizens and people who are willing to help them, so in between ongoing projects, we’ve been sketching out how we want to accomplish this — whether it should be done through Stack Exchange, whether we should build our own, etc.

[You can learn more about the project at http://govdecoder.com/.]

Finally, what’s your favorite government open source project?

I’ve never seen it in action, but DHS’s Virtual USA project sounds remarkable. On top of using open source software to build it, the objective of the project is to break another government taboo: sharing information with other agencies and levels of government. Having been an intelligence analyst who relied a lot on mapping tools and was constantly frustrated by the inability to share geographic data even within your own building, it’s apparent that if Virtual USA delivers, it’s going to dramatically change how first responders work.

# # #

I want to thank Matthew for taking the time to speak with me. I think the project is fascinating, and I can’t wait for it to launch.

Open source in government: Who was first?

Brian Purchia of Burson-Marsteller has a post here on GovFresh about the value of open source to unions. His argument pivots on cost-savings. I think you could make a more expansive argument that includes risk mitigation and innovation, but describing the advantage to unions is an interesting angle I hadn’t seen before.

I noticed that Brian repeated the misunderstanding that San Francisco had the nation’s first open source policy. I don’t want to diminish his larger argument, but it’s important that we give credit where credit’s due. So for the record:

  • July 1, 2004: OMB issues OMB-04-16, making clear that open source can be used in the Federal Government
  • October 16, 2009: The US Department of Defense CIO issues a memo reiterating that open source software is commercial software for procurement purposes, and encouraging DOD branches to include open source when they’re picking software.
  • September 30 2009: Portland, OR is the first city to issue an open source policy.
  • January 7, 2010: California‘s open source policy is published.
  • February 1, 2010: San Francisco, CA issues their open source policy.

These are just what I could find, of course. If you know of others, let me know! If you’d like to see a comprehensive history of open source battles in national and state governments around the world, CSIS maintains an annual survey intuitively titled “Government Open Source Policies“. Even just skimming it, you’ll be surprised at how little progress the United States has made in open source policymaking.

Lockheed goes open source. Blankenhorn hates it.

I was really pleased to read the announcement that Lockheed Martin's social networking platform, EurekaStreams, was released as an open source project today. Lockheed is a very conservative company, and while they're happy to use open source internally and on projects for their customers, this is their first experiment with actually running a project themselves. I think it's a big deal, not just for Lockheed Martin, but for large corporations who are considering a more open, more innovative approach to software development. And yet, Dana Blankenhorn hates it:

I don’t see anything in Eureka Streams I can’t do in Drupal, or a number of other high-quality open source projects that have existed for years. Lockheed has reinvented the wheel — why?

So here's the nice thing about the open source community: competition. If I think I've come up with a better way to solve a problem, it can easily compete with the incumbents. Low barrier to entry, we say. Let the best ideas win. Unless, apparently, the best ideas come from a company I don't like.

Then things start going sideways:

The author of Eureka Streams, who goes by the name Sterlecki at Github, has left no previous tracks there. Linkedin lists the same picture as belonging to Steve Terlecki, a Lockheed software developer.

The stuff’s legit, so we’re left again with the question of motive. Is the military-industrial complex reaching out to open source, is this just proof of press reports showing our spy efforts have more bloat in them than a Macy’s Thanksgiving float, are we being co-opted, or am I just too suspicious?

Wait, what? Open source advocates have, for years, been trying to encourage more code to come out from behind corporate skirts. Where companies can build business models around governing and supporting open source projects, we want them to take the plunge. If more code is open, that makes everyone smarter. And that, my friends, is exactly what Lockheed Martin did today. Someone who probably never contributed code in their lives just gave the community a project they've been working on for months, or even years. I think that's amazing. In return, this brave developer gets painted as a nefarious secret agent out to steal our thoughts and bug our laptops. Or whatever.

So here's the great thing about open source: we can prove Blankenhorn wrong. They use the Apache license, and it's on Github. We can go through the code and find backdoors, secret plans, and mind-control rays. This reminds me very much of the reaction to the release of SELinux. Conspiracy theories everywhere, but code is auditable and now it's in the mainstream Linux kernel. Do we really want to throw out these contributions, when code doesn't lie? When it's so easy to ensure there's nothing nefarious inside?

You can feel however you like about Lockheed Martin or the US Department of Defense. You can choose to contribute to the project, or not. You can choose to use the software, or not. But is it in the community's interest to summarily dismiss contributions based on those preferences? Lockheed's thousands of developers are sending up a trial balloon. If they fail, we lose access to those developers forever.

I think this kind of fearmongering is exactly what prevents large corporations and government agencies from releasing their code. These knee-jerk reactions harm the open source community at large. We pride ourselves on our meritocracy. A 14-year-old in his mom's basement is the same as a 30-year-old Lockheed developer is the same as a UNIX graybeard. You are just as good as your contributions. We need to welcome Lockheed's contributions, not throw them back in their face. Whether the project is useful or not, they've enriched the open source community. Let them succeed or fail on their own merits. If they do fail, we hope that they'll do better next time. Maybe this is a Drupal-killer. Who knows? Let's give it a try.

Open source headlines from the Open Government plans

The Obama Administration’s Open Government Directive ordered Federal agencies to produce open government plans by April 7th, and while some advocates are disappointed, we have before us a bewildering number of initiatives to improve transparency, collaboration, and participation across the Government. It will not surprise you to learn that I spent some time looking for places where open source is being used in these plans.

I’m not sure I can recommend reading all of the plans cover to cover, but if you’re an advocate or have a vested interest in the future of a Federal agency, these plans are fascinating peek into each agency’s interior life. It’s not just the content of the plans, which run from exciting to comical to mundane. You can also learn a great deal about how agencies view themselves from the way these plans are presented and marketed. It will come as no surprise that the Department of Justice’s rather unlovely document spends a lot of time thinking about reducing its FOIA backlog. The Department of Energy clearly understands itself to be a first a research organization, based on its flagship data sets. The Department of Defense plan is crisp, to the point, and focuses on getting the behemoth to better collaborate and interact with other agencies, rather than the public.

The organizational psychology betrayed by these plans is for another post. My interest is in where agencies found open source. I’ve long advocated for open source as a concrete, tangible way to encourage collaboration between agencies and between the government and its citizens. I was pleasantly surprised, frankly, to see how many agencies agree. Here’s what I found, in no particular order.

US Agency for International Development

The USAID plan was a total surprise. I had no idea how many open source initiatives were being conducted by USAID. Page 30 contains this gem on their Global Development Commons work:

With over four billion subscribers in the world, the mobile phone is often the key to connecting and exchanging information with people in developing countries. The 2008 USAID Development 2.0 Challenge, implemented by the Global Development Commons, invited innovators and entrepreneurs from around the world to participate in a global competition to seek access to information and build new connections to the global community. Crowdsourcing and Open Innovation have become increasingly important engines of innovation globally, leveraged by the commercial, non-profit, academic and government sectors to identify opportunities and solve problems. USAID’s Development 2.0 Challenge yielded 115 submissions using high impact, low-cost, open source solutions.

The winner among the 115 submissions was the RapidSMS Child Malnutrition Surveillance system, which “enables health practitioners to share and track children’s nutritional information with the touch of a cell phone.”

The agency also operates the Intra-Health OPEN Initiative, which is “a suite of free open source solutions to supply health sector leaders and managers with a collection of new tools.”

Social Security Administration

The Social Security Administration is another open source underdog. Imagine all the pent-up innovation they can unlock once this project is underway:

We are in the process of creating internal capacity to host websites and applications based on open-source software solutions and we look forward to a lively exchange of ideas and program code within the growing Federal open–source software development community;

As part of SSA’s fifth goal, “making government more sustainable”, they see open source software as an essential tool:

We are a Federal leader in the use of Health Information Technology. Our work with the private sector may yield transferable ideas and tools. We will share our results and products as appropriate. For example:

  • We look forward to sharing the products of our open–source platform efforts across the growing Federal open–source development community, as well as partnering with other agencies in future endeavors; and
  • We are in the process of designing and developing an Electronic Technology Repository for communities of innovation. We expect this repository to employ open–source social networking and other tools to permit users to better manage agency knowledge, avoid unproductive duplication of effort, and share experiences. The repository will support the storage of shared materials and project artifacts, discussion boards, wikis, blogs, subscription feeds, and other pertinent information. We envision sharing these resources with other Federal organizations as well.

Others have criticized open source as being irrelevant to the open government movement, but I think interagency collaboration doesn’t happen anywhere near as often as it should, it can be made easier with open source, and it’s outstanding that the SSA seems to agree.

Department of Defense

The DOD has been using open source software for years. Though I was a bit surprised that it wasn’t explicitly mentioned in the DOD’s open government plan, and even more surprised that the much-hyped forge.mil project isn’t referenced at all, the plan includes a number of initiatives that happen to take advantage of open source tools:

The Wikified Army Field Guide, based on Mediawiki, will allow warfighters to collaboratively edit the Army’s field manuals, allowing the documents to be more accurate and responsive:

…as the battlefield changes rapidly, field manuals must keep pace.  Under the traditional process – in which a select few were charged with drafting and updating field manuals –manuals often failed to reflect the latest knowledge of Soldiers on the ground.

Using the same free software behind Wikipedia, the Army’s “wikified” field manuals invite military personnel – from private to general – to collaboratively update the Army Tactics, Techniques and Procedures Manuals in real time.  In so doing, the Army provides a secure means for battle-tested Soldiers to share their experience and advice from the field.  Wikified Army Field Manuals ensure the men and women who serve our Nation have access to the best possible information when they need it.

This is a very exciting opportunity to capture all the innovation happening “at the edge” and quickly incorporate it into useful, official documentation. It makes so much sense, I’m surprised it hasn’t already been done.

The plan also highlights XMPP, which is a tremendously popular instant-messaging protocol that runs, among others, Google Talk. It may surprise you to learn just how much XMPP’s most popular implementation, Jabber, is already being used inside the DOD. The Defense Connect Online program uses Jabber to provide secure IMs inside the DOD, and they announced in November that this would be opened up to the outside world. Because they standardized on an open standard with robust open source implementations, literally dozens of different chat clients are now available to these non-DOD DCO users.

It’s interesting how both Jabber and the Wiki Field Manual projects aim to improve collaboration, and do so on highly collaborative open source platforms. I don’t think that’s an accident.

Department of Homeland Security

VirtualUSA is DHS’ flagship initiative, which couldn’t be more appropriate. From page 23 of the DHS Open Government plan:

On December 8, 2009, Secretary Janet Napolitano publicly launched Virtual USA (vUSA), an innovative information-sharing initiative that draws on practitioner input to help Federal, State, local and Tribal first responders collaborate to make fast, well-informed decisions. vUSA integrates existing frameworks and investments to provide real-time access to operational information—such as weather conditions; traffic; the location and operational status of critical infrastructure; fuel supplies; availability of emergency shelters and medical facilities; and other critical information—that allows users to improve situational awareness and to respond quickly in emergencies.

vUSA currently operates as two pilots – one in eight southeastern states: Alabama, Georgia, Florida, Louisiana, Mississippi, Texas, Virginia and Tennessee; and the other in five states in the northwest: Alaska, Montana, Idaho, Oregon, and Washington. In Virginia alone, vUSA reduced response times to incidents involving hazardous materials by 70 percent.

VirtualUSA is much more revolutionary than this lets on. It’s a very disruptive piece of software in its space. FCW has a succinct overview of the project’s history and what it means for first responders. DHS is funding this project, based on open source and open standards, in part because it wants to encourage collaborative toolbuilding and cooperation among the states, and also because this capability is too important to be in the hands of a single GIS provider, like Google or ESRI. Because it is an open source project, and uses open standards, VirtualUSA ensures that critical assets are not locked into a single vendor, and simultaneously lower the barrier to entry for new GIS vendors.

Department of Commerce

On page 18 of the Department of Commerce plan, under “Open Source Information Technology”, we find some familiar prose:

Also emerging from Commerce’s Open Government Ideascale community was a suggestion to “become more open through the increased use of open source software.” The Department has already begun using the open source tool, Drupal, for a number of its new websites and plans to increase this use in the future. Using open source technology will allow Commerce to develop new technologies and collaborate more readily with the public and other government agencies, and within the Department itself.

To make this happen, the Office of the Chief Information Officer and the Office of Acquisition Management will be consulted to ensure that open source offerings are fully considered during procurement processes. That consideration will include the value that the Department can receive through increased collaboration with the public and as a contributor to open source communities.

Nothing short of victory at Commerce for Open Source of America, whose suggestion this was. Congratulations!

Department of Labor

You’ll find open source in the strangest places. Until I read Labor’s plan, I didn’t appreciate how much data the Department of Labor is responsible for analyzing and disseminating. With that in mind, it makes perfect sense to find this initiative on page 29:

Create a “Developer’s Corner”

We plan to establish a “Developer Corner” on www.dol.gov/open that specifically targets and engages developers. Our goal is to make it as easy as possible for developers to re-purpose our data, provide feedback, get technical help, bring developers with similar interests together and, ultimately inspire the best possible uses of our data for the benefit of the public. Ideas under consideration include a bug tracking system, RSS feeds for dataset changes, dataset versioning, public code competitions, data authentication, and an ideation platform to prioritize developer needs.

I think every department and agency that distributes data to the public (which is to say, all of them) should follow Labor’s lead and establish their own Developer sites. There’s no better way to stay engaged with this very powerful community.

Health and Human Services

The outstanding NHIN CONNECT project, which has a thriving open source community, got a mention on page 56 of HHS’s plan:

Nationwide Health Information Network – Direct

A key component of the Nation’s emerging health information technology infrastructure is the Nationwide Health Information Network (NHIN) – a set of standards, policies, and services that enable the secure exchange of health information over the Internet. “NHIN Direct” is the latest development in the evolution of the NHIN. It’s an important effort to develop a “lightweight on-ramp” to the NHIN that will enable simple, direct exchanges of information between providers, labs, pharmacies, and consumers — and which will be easy to adopt and implement. In a process that launched on March 1, NHIN Direct is being designed in close collaboration with the community of potential users, with the entire process taking place in the open, in public, on a NHIN Direct wikispace. NHIN Direct will then be implemented in real-world tests and deployments by members of the community – with HHS’s Office of the National Coordinator for Health Information Technology (ONC) playing a coordinating and convening role. The gist of the NHIN Direct strategy is to utilize a community-driven approach to ramp up and power NHIN Direct-powered health information exchange.

I should mention that my employer, Red Hat, is involved in this project.

National Science Foundation

There was nothing explicitly about open source in the NSF’s open government plan, but their plan is worth mentioning anyway, as the NSF already does a tremendous amount of work in the open source community. Here’s a search for “open source” on their web site, which yielded over 5,000 hits when I last tried it:

http://search.nsf.gov/search?access=p&output=xml_no_dtd&sort=date:D:L:d1&ie=UTF-8&btnG=Google+Search&client=NSF&oe=UTF-8&proxystylesheet=NSF2&site=NSF&q=open+source

NSF regularly awards grants under the condition that software developed under those grants is given an open source license. Some very progressive thinking, and shrewd IP stewardship from the NSF folks, so we’ll forgive them for not mentioning open source directly in their plan.

Department of the Treasury

Yet another revelation. Treasury plans to cultivate open source projects to facilitate collaboration between agencies and between Treasury and the public. I’ve written about exactly this kind of collaboration before, back in December, so I’m enormously pleased to see that Treasury agrees. I’ve emphasized my favorite passages here:

In the areas of transparency, participation, collaboration, and flagship initiative, Treasury strives to share its efforts across Government to avoid duplication across agencies and to improve value/outcome of efforts. Treasury seeks to manifest cross-agency transferability in at least two of the following ways:

  • Make training available to other agencies by opening up classes/webcasts to other agencies; providing slides, video and/or audio after the training; and posting on an e-learning platform.
  • Name an advocate who gets the word out about what the agency has to share and invites other agencies to contact that person to learn from him or her.
  • Design procurements for enterprise (where government is the enterprise) or in such a way that what is created can be shared across government at no cost.
  • Develop and post code so it can be shared with other agencies (open source or the contract written such that the government owns the code.)
  • Share platforms utilized by the agency with other agencies at no cost.
  • Create participatory events across agencies with related missions.
  • Collaborate on projects and challenges with the public and with the private sector in partnership with other federal agencies that have similar missions.
  • Share all materials, results, tools, and training that could be transferable to other agencies with the Interagency Working Group as an efficient central dissemination mechanism.

Veteran’s Administration

The VA is an enormous consumer of information technology, and gained early recognition from the open source community for its public domain VISTA electronic health record platform. On page 22 of the VA plan, it becomes clear that the VA is expanding its use of open source to lower the barrier to entry for developers who want to help the agency:

AViVA

A Virtual Installation of VistA Architecture (AViVA) is a recent innovation that we are using to support collaboration. AViVA creates a universal user interface for the electronic health record and includes prototyping of data connectors in order to securely link the AViVA platform to patient data from any source. The AViVA project incorporates HealtheVet as an update of the VistA legacy database system.

VA’s current electronic hospital management system uses a graphical user interface known as the Clinical Patient Record System (CPRS). CPRS data is stored in the legacy data system known as VistA. CPRS requires installation on each machine that operates the program rendering the program difficult to scale and expensive to maintain and update. AViVA’s implementation improves this model in two ways. First, AViVA creates a modular, web-enabled electronic health record system that can be easily and remotely maintained. Second, Medical Data Web Services (MDWS), which can be accessed by the Department of Defense, will allow the creation of applications for any data source to be plugged into the system.

AViVA is a very exciting program for the collaboration portion of our Open Government Plan and because we are committed to creating systems that allow health care providers to collaborate to provide the best care for Veterans. AViVA’s web based presentation layer will allow our doctors and nurses around the country to search patient records as simply and succinctly as you can search for pizza on Google Maps and as securely as the best retail financial service businesses. Additionally, AViVA creates collaboration between VA and DoD, our partner in caring for our nation’s heroes. Finally AViVA creates an open source platform that allows software to be shared with entities outside of VA, creating opportunities for further innovation and development beyond the agency.

National Aeronautic and Space Administration

“NASA is working to make open source software development more collaborative at NASA to benefit both the Agency and the public,” it says right on the first page of the NASA open government plan. Here’s an agency which has always relied on a vibrant research community, software developers, and a culture of innovation. I’m not surprised by their focus on open source, but I am delighted. Among other things, NASA will be sponsoring an open source code competition, has an entire section of their plan devoted to open source development, and will be developing their Nebula cloud computing system on open source software.

It’s fair to say that NASA’s plan is the strongest I’ve seen for the open source community.

Your Turn

Who did I miss? What other opportunities for open source have you found in the open government plans? Leave a comment and let us all know!

Open source matters to open government. Really.

“Open source and open government are not the same,” I’ve been reading recently. When discussing the role of open standards in open government transparency projects, Bob Caudill at Adobe, is concerned that open source and open standards are being conflated. He likes open standards just fine, but:

“Open standards are driving for interoperability between systems or applications, while, the goal of open source is to make high-quality software available to the market free of charge.”

As an open source advocate, I’m surprised. What, I have to wonder, is so threatening about open source? Why the effort to take open source off the table? I’ve written on the topic before, and I didn’t think this was controversial — but apparently I was wrong. Andrea DiMaio at Gartner is more pointed:

“For those who have been following some of the vintage discussions about government and open source, this will probably sound like a déjà vu. I honestly thought that people had finally given up pushing the confusion between open source and open standards or open formats, but here we are again.”

While they both agree on the importance of open standards (although transparency also seems to annoy DiMaio), they also remind us that tools, proprietary or open source, are a means to an end. An open standard is an open standard, whether implemented by an open source project or a proprietary one. What’s important, they insist, is more transparency, collaboration, and participation. Open source is immaterial at best, and a distraction at worst.

They’re right, of course, that open standards are crucial to ensuring meaningful transparency in government. It does not follow, however, that this precludes a role for open source.  Open source software is an invaluable tool — one of many — to approach all three goals (transparency, collaboration, participation) of the Open Government Directive. It’s not about open source software specifically, although the software helps. It’s about the process that open source projects use to create good software. Because the open source development process requires real collaboration, tangible progress towards a goal, and the participation of a broad community of users and developers, it’s an excellent mechanism for getting citizens involved in the work of government.

DiMaio couldn’t disagree more. Referring to Nat Torkington’s idea of using the open source development model to improve transparency projects:

“…there is a fundamental flaw in this line of thought. Open source projects cluster a number of developers who collaborate on an equal footing to develop a product they are jointly responsible for, as a community.

“Government does not have the luxury of doing so. An agency publishing crime statistics or weather forecast or traffic information is ultimately accountable for what it publishes.”

I couldn’t disagree more. Again, DiMaio and Caudill misunderstand how the open source process works and what it can contribute. The trouble, I think, is with a too-narrow understanding of what participation and collaboration might mean, and a similarly narrow view of what the open source development process has to offer.

The goal of open source is much more than just making no-cost software, as Caudill suggests. It’s about producing better software through a process of inclusion and rough consensus. The source code is free of charge largely because that is the best way to create a large community around the project, it’s not the final goal. And while some open source projects function better than others, they are not, as a rule, unaccountable. In order for the projects to succeed, they must be highly accountable to their community.  Further, many open source projects have commercial ventures (like my company, Red Hat) that live or die by their success, which makes them extremely accountable. So to say that the government cannot rely on open source software or the open source process because it is unaccountable is just not true. We know this to be the case because you can find the government using open source software in the Army, the NSA, the Census, the White House, and just about everywhere else. So there’s no reason to think that open source process cannot inform and support an open data project, as DiMaio suggests.

Setting accountability to the side, the more interesting conversation is how open source can bring some unique benefits to open government, unavailable any other way.

If you look at the outstanding work of pro-transparency organizations like the Sunlight Foundation, govtrack.us, RECAP, and others, nearly all are using open source and the open source development model. It’s not, as DiMaio and Caudill suggest, because they’re naive ideologues who are confused as to the meaning of “open”. These are smart people doing serious work. They’re using open source because it’s the best way to collect a large number of contributors around a common problem. They’re using open source because the transparency of the process and software makes their work credible. They’re using open source because they believe that free access to government data means free access to the tools that make that data useful.

The alternative is closed, proprietary tools, which do little to further the transparency goals. RECAP, for example, had a difficult time understanding the US Courts’ closed PACER system, and had to do a lot of difficult reverse-engineering to work with it effectively. The job would have been significantly easier if they had access to the PACER software source code. Fortunately, because RECAP is an open source project, their hard work making PACER usable is now available to everyone. So to dismiss open source as irrelevant to the crucial work of making government data available and valuable to the private citizen, and the even more important work of encouraging a collaboration between government and its citizen, is deeply misguided.

Again, even though data transparency seems to annoy DiMaio, I think there’s good reason for the tremendous transparency effort the administration and the private sector have brought to bear. First, data transparency is a relatively simple problem to solve. It’s easy to publish data on the Internet, and there’s a tremendous amount of value to be extracted. So while it’s only a part of the challenge — indeed, is only one leg of the Open Government Directive — it’s an easy win for both government and its citizens.

But DiMaio is correct that open government is about much more than just data, so let’s generalize this further. We could understand open government as an opportunity to increase the quality of interaction between citizens and their government through collaboration. “The government is not a vending machine,” as Tim O’Reilly paraphased Frank DiGiammarino of the National Academy of Public Administration, “which we kick when it doesn’t work right.” Instead of treating government as a black box, we should treat our government as the place where we, in the public and private sector, come together, to solve problems as a group. This is why we refer to “government as a platform.” Yes, as DiMaio says, each agency is responsible for its own output. But that doesn’t mean the public has no stake. Precisely because we want to hold agencies to a higher standard, we must provide a means of collaboration and participation.

The trouble is, there’s a lot more of us than there is of them. How can one agency effectively collaborate with 300 million constituents? Likewise, how can an agency effectively communicate with that many people? One of the reasons the open government movement is so preoccupied with technology and the Internet is that they represent a solution to this problem. For the first time, the government and its citizens have the means to work effectively at this scale. There are all kinds of tools for this: social networking, blogging, data.gov, the Ideascale Open Government sites, and so on. One of those tools, the one that is most interesting to me, is the open source development process.

Note that I didn’t say open source software. Although I love the software, and could talk for days about why the government should be using more of it, it’s the process that creates this software that is most valuable to the goals of collaboration and participation.

In the last 40 years, open source software communities have learned how to effectively solve complex tasks with large, far-flung, geographically dispersed communities. Why wouldn’t we take these methods, and apply them to the task of creating a better government? As I mentioned earlier, Nat Torkington suggested using the open source process to improve data quality. The NASA CoLab project uses open source software and the open source development process alongside other collaborative tools to get researchers from the public and private sector to work together. The Defense Information Systems Agency is using the forge.mil project to encourage collaboration between the DOD and its contractors — not just for software, but for testing, certification, and project management. The Apps for Democracy, Apps for Army, and Apps for America contests are all attempts to harness the collective intelligence of citizens and government to solve common problems using the open source model — not just building tools, but building the means to collaborate on top of open tools, like Open 311 and DataMasher.

So when DiMaio bemoans the lack of government employee engagement and the lack of community data, it may be because he doesn’t realize that this work is happening, and it’s happening using open source and (more generally) collaborative innovation models.

Both DiMaio and Caudill make the mistake of believing that open source is about making cheap bits. Instead, it’s a blueprint for effective collaboration on a massive scale. Advocates for open source in government, like me and my friends at Open Source for America, aren’t just talking about open source tools, although those are also useful. We believe that the open source development model has a concrete contribution to make to the open government movement — and those who dismiss open source as irrelevant don’t realize just how open a government can be.

Developers for Glory

Although it may be simple to conflate the Apps for Democracy and Apps for America contests with the exciting new Apps for Army contest, they really couldn’t be more different. Together they represent an exciting experiment in what it takes to pull communities together around a problem. Though they all offer cash prizes to the winners, they each took a slightly different approach, with different results.

Cash incentives are somewhat controversial in open source circles. Most old-school advocates for open source development strongly prefer developers who are personally invested — famously, those that “scratch their own itch.” Developers who are paid a salary to work on software are also invested, but perhaps less zealously than those who are solving a problem they are afflicted with themselves. Developers who are working for glory and cash prizes, the model used by the “Apps for…”  competitions, is yet another class of developer, and despite the excellent submissions to the previous contests, there are valid concerns that the quality and sustainability of the code is not as good as it could be with a different set of incentives. Time will tell, of course.

If I’m a developer for glory, I may compete for the cash prize, or for altruistic reasons, but I’m also competing for the notoriety I’ll get if I win. If I don’t win, what will I do with the code I’ve developed? Even if I win, what are my incentives to continue working on the project? Put another way: how can we ensure that all of this good work and goodwill turns into viable, and active software projects once the contest is over?

Apps for Democracy is instructive.  The contest encouraged developers to provide services on top of the “platform” of Washington, D.C.’s IT infrastructure. This platform includes 270 public data feeds and the city’s newly unveiled 311 API.  47 submissions were collected in 30 days, and the winner was an iPhone and Facebook application that enabled users to take snapshots of potholes, broken windows, and so forth, have them tagged with GPS coordinates, and submitted to the city’s 311 service. Very handy. Unfortunately, the ongoing care and feeding for the application doesn’t seem to be there. The Washington City Paper found in a January 25th, 2010 followup on the contest:

The “Touch City’s Heart” Social DC 311 Web site seems to have been abandoned—it hasn’t been updated for months—saying the “IPhone” app is still waiting for approval from Apple (Apple approved it long ago). Some members of the D.C. 311 team had never laid eyes on the Web site until City Desk asked about it. “I’ve never even heard of it,” said one 311 operator. It has only 27 active monthly users on its Facebook Fan page and 40 followers on Twitter.

I’ll also note that after some cursory research, the source code doesn’t seem to be disclosed to the public yet, which I understand was one of the intents of the contest. Now, to be fair, there seem to be bigger plans afoot:

The dismal following is not a sign of failure, Sivak says. The District intends to take Social DC 311 and revamp the current model into an app that’s “enterprise-ready and robust for a large volume of users,” Sivak says. “Think of this first step as a pilot.”

Fair enough, but I would think that one of the desired outcomes was an ongoing community of developers that are producing and maintaining applications like this — whether it’s for love, money, or fame. It would be a shame to see hard work like this die on the vine because we’ve lost the carrot of a cash prize.

The first Apps for America contest winner was Filibusted, a tool for outing Senate obstructionists. It measures obstruction by the Senator’s votes on cloture motions. You can find the source on GitHub, but there doesn’t seem to be much activity since the initial checkin. One bug was opened 8 months ago, and doesn’t appear to have been addressed. The last blog post was in December. At the same time, there’s not much to work on — the site has a single purpose, which it seems to fulfil even without much of a community around it. It doesn’t really need a large community, I’d guess, because it’s “done.”

The second Apps for America yielded DataMasher. This tools allows you to compare Federal data sets with each other. Once you have the data and visualization you like, you can share it with others on the site. The source code was released, per the terms of the contest, but doesn’t seem to have much of a community around it. In fact, the DataMasher website doesn’t seem to link to the code from their own site. That hasn’t made the application less popular, though — the community isn’t working on the code, it’s working on the datasets. There’s a steady stream of new mashups that other users rate and comment on. In all, a healthy community that relies on user-generated content to ensure it remains a useful tool.

The second Apps for America contest also produced the strikingly elegant govpulse.us. It’s a vastly improved interfact to the Federal Register developed by the gifted team at GravyCones. The code for this application is available to the public, and seems actively developed to this day. This is, I think, exactly what the organizers had in mind when they started this contest: the tool is popular, the development community is active, and the project continues to improve.

Which brings us to Apps for Army, which is a serious departure from the other contests. First, it’s available only to Army soldiers and civilian employees, nobody from the public — not even reservists. In fact, you need a DoD ID card to go to the official contest website. Second, it seems that only the first 100 teams can participate. From a community standpoint, the project is wading into very unfamiliar territory. Rather than gathering the collective wisdom an initiative of thousands of interested developers, they’ll be picking 100 volunteers, seemingly at random.

The Apps for Army contest further diminishes its potential reach by dictating the tools developers will use: the DISA RACE environment to host the project, and the forge.mil repository for code. Since these resources are being paid for by the Army’s CIO, who is sponsoring the contest, what will happen to the competitors once the competition’s over? There are, of course, excellent reasons for asking folks to use the existing DoD infrastructure, but I can’t help but wonder what would happen if the doors were flung open, and the bar was lowered for participation.

This isn’t to say that I’m less enthusiastic about these experiments. I’m very excited at the idea of encouraging employees — in the Army, or anywhere else — to solve their own problems. That’s a goodness in and of itself. We just can’t forget that software isn’t a product — it’s a process that requires nurturing. The best way to nurture is to build a community, and that requires transparency and a low barrier to entry for participants. The larger and more active the community, the more likely the software will be better. The more closed, prescriptive, and limited the project, I think, the less likely that it will be viable in the long-term.

So these “Apps for…” competitions are instructive. Each project is building its own kind of community, and I’m eager to see how these projects fare in the months and years ahead.

Software isn’t a skycraper

Michael Daconta at Government Computer News has posted a brief call to arms for the software industry. Here’s the gist:

Although I am a believer in free markets and the benefits of competition, industry has a responsibility to work together on the foundational layers to build security, quality and reliability from the ground up to advance the professionalism of the field. In essence, the information technology industry must emulate other engineering disciplines, or technological disasters and cybersecurity holes will worsen.

Daconta is uneasy with the number of platforms and methods available to software developers, and sees ever-more options and disruptions in the near future; IPv6 and 64-bit computing seem to trouble him particularly. We’re already balkanized and disorganized, how can we possibly expect to produce reliable and useful software with all this messy innovation happening?

The answer, of course, is control. Lots of it. Specifically, three proposals:

  • Licenses for software developers
  • A new, reliable, layered software platform developed by the NSF and DARPA
  • Treat software like engineering, not art.

Gracious. I barely know where to start. Let’s try to imagine the software development world in five years, with these proposals in place.

Software development is now a licensed activity. Like an architect or a mechanical engineer, you have to pass an exam and perhaps post a bond to practice the discipline. There’s probably a professional association, like the America Bar Association or the American Medical Association, to administer the credentials.

This licensing regime is actually a pretty good idea, because all software has to be developed according to some very specific methods, with plenty of testing and documentation to back it up. So rather than letting any fool with a compiler write software, they’ll have to spend a year or two learning the right way to write code. The process is cumbersome, but every piece of software that gets compiled is perfect. At least, as perfect as we know how.

The licensing and formal methods are only possible, of course, because we have a government-directed platform that we must build on. Anyone who wants to run software in the government must do so on this stack.

It sounds as though Daconta would like a broad mandate for NASA-style code development. You can probably see where I’m going with this. Another way to tell the story might be:

There is now a government-owned platform that every government program is mandated to use, from clouds to mobile phones. Any software built on that platform must submit to rigorous, independent testing before it is deployed. Imagine ISO 9000 and Common Criteria having a baby with teeth. Anyone writing software must be licensed to do so in the United States. As a result, the pool of available programming talent is decimated. The costs of developing software for government rise, naturally.

The pool is further diminshed because developers who want to work with the latest hardware or software no longer work for agencies or contractors — they’ve wandered back to the private sector, where they can enjoy the fruits of the free market. The government software platform quickly begins to show its age, since the only developers on the platform are those that are paid to use it. Platforms that people truly want to use are in the open market, innovating at their leisure, out of the reach of government agencies.

As the government is no longer able to consume most commercially available software, it is now back in the business of writing software itself. More accurately, it is back in the business of hiring system integrators to write that software on its behalf. It’s the 1970s all over again. Budgets explode, and innovation grinds to a halt. It’s all the agencies can do just to tread water. System integrators, of course, are delighted. They’re now commanding outrageous salaries for the few programmers trained and willing to work on this mandated government platform.

Let’s hope there’s a better way.

This doesn’t diminish the problem that Daconta is hinting at, of course. Software reliability is certainly something to worry about. But there is no single solution or set of policy prescriptions that will solve the problem. I don’t think that imposing additional controls on the development of software makes sense, certainly not for all cases. There are already robust certification regimes and methods for software that does very important work: fly an airplane, control a nuclear reactor, and so forth. We don’t need that kind of scrutiny on my game console or desktop.

It’s important to note that these robust certifications only evaluate the software itself, not the people who make it. This is what’s great about software: we can examine the final product before it’s distributed. I don’t really mind if my software is written by a clever 7 year old. If it’s doing the job it’s supposed to, that’s fine with me.

This focus on ends, rather than the means, is something you can’t do with a building or an airplane. With software, we can change our minds with far fewer consequences. We can thoroughly test and scrutinize before it’s in a customer’s hands. When we do find a flaw, it’s easier to patch software than it is a 777 or a skyscraper. We should take advantage of that fact, not try to make software rigid and inflexible just because we know how to manage rigid and inflexible things. Because software has these unique properties, we have the freedom to bring more or less scrutiny to bear, depending on the circumstances. Which is what we’re doing already, through programs like the federally-funded Software Engineering Institute at Carnegie Mellon.

So where Daconta sees mayhem and chaos, I see creativity and innovation. There are many opportunities to improve the reliability of our software, but none of them have to do with the process by which we arrive at a particular piece of code. Projects like David Wheeler’s OpenProofs, for example, can provide the tools we need to be mathematically sure software is doing what we intended. The Linux Test Project does this for Linux, and it was inspired in no small part by governments’ Common Criteria mandates.

This is, in fact, how the process should work. The government should set the requirements for reliability and assurance, and allow the private sector to innovate its way toward those requirements. If we only create software that we can understand perfectly, we lose the ability to be creative, to innovate, and to take advantage of the collective intelligence and cleverness of millions of software developers. We will never eliminate risk in software, but we can manage that risk, not through more stringent controls, but by encouraging as many smart people as possible to address the problem.

What the Open Government Directive Means for Open Source

On the heels of the Open Government Memo of January 21st, 2009, the Obama Administration has issued the Open Government Directive. The Directive tells agencies what they must do to meet the expectations set by the Memo. The directive names many deadlines for agency compliance, most of them around reducing FOIA backlogs and increasing the amount of agency data released to the public. This isn’t surprising, since the Memo names transparency, collaboration, and participation as the guiding principles. Transparency is the easiest to articulate and implement — just get the data out there in a useful form. Josh Tauberer’s Open Data is Civic Capital: Best Practices for “Open Government Data” is an excellent handbook for doing this. If you want to track agencies’ progress, the Sunlight Labs folks have produced the outstanding Open Watcher.

What’s most interesting to me, and my friends at Open Source for America, though, are the more ambiguous orders. Although the Directive does not use the phrase ‘open source software’ at all, many of the principles and methodologies described are obvious references to open source. Many of these orders stand out as opportunities for open source developers, in the public and private sector, to demonstrate how our development model can help the Administration also make good on the last two principles: collaboration and participation. As Macon Phillips, the White House New Media Director said, “Open Source is… the best form of civic participation.”

Let’s take a look at the deadlines, helpfully produced by Daniel Schuman at the Sunlight Foundation.

45 days — January 22, 2010

“Each agency shall identify and publish online in an open format at least three high-value data sets and register those data sets via Data.gov” (p.2)

This is a wonderful opportunity for open source developers to demonstrate the power of citizen participation through software. The Administration has taken a great risk by pushing this data to the public. There are all kinds of reasons to not do it: privacy concerns, security issues, and the risk-averse culture in most of these organizations. Despite the instructions to be careful with citizens’ privacy, and the reminder to be sensitive to security issues, there’s still a chance that something could go wrong — plenty of reason to not follow through with this exercise. We need to help the Administration prove that this was a worthwhile cause. Just as we showed the power of citizen programmers in Apps for Democracy and Apps for America, we need to take these data sets and make them useful to the American public.

“The Deputy Director for Management at OMB, the Federal Chief Information Officer, and the Federal Chief Technology Officer will establish a working group that focuses on transparency, accountability, participation, and collaboration within the Federal Government. This group, with senior level representation from program and management offices throughout the Government, will serve several critical functions, including:

  • Providing a forum to share best practices on innovative ideas to promote transparency, including system and process solutions for information collection, aggregation, validation, and dissemination;
  • Coordinating efforts to implement existing mandates for Federal spending transparency, including the Federal Funding Accountability Transparency Act and the American Reinvestment and Recovery Act; and
  • Providing a forum to share best practices on innovative ideas to promote participation and collaboration, including how to experiment with new technologies, take advantage of the expertise and insight of people both inside and outside the Federal Government, and form high-impact collaborations with researchers, the private sector, and civil society.” (p.5)

Now here’s a working group I would like to speak with very much. If you read the language of the third subsection, it’s amazing how many words you have to use to not say the words “open source”: experiment with new technologies, using expertise inside and outside the government, high-impact collaborations with many communities of use… they’re all but begging to create open source software projects to support the release of this government data.

In this “forum for best practices” on open data initiatives, you can imagine how useful a recommendation of open source software might be. You can even imagine the working group recommending government open source projects to help handle data that may be in strange government-specific formats.

60 days — February 6, 2010

“Each agency shall create an Open Government Webpage located at http://www.[agency].gov/open to serve as the gateway for agency activities related to the Open Government Directive” (p.2)

“The Federal Chief Information Officer and the Federal Chief Technology Officer shall create an Open Government Dashboard on www.whitehouse.gov/open. The Open Government Dashboard will make available each agency’s Open Government Plan, together with aggregate statistics and visualizations designed to provide an assessment of the state of open government in the Executive Branch and progress over time toward meeting the deadlines for action outlined in this Directive.” (p.5)

Of course, if an agency is writing new software to support these new “/open” areas, I’d like to see that software made available under a open license. If there are any clever data analysis or visualization tools, those should be licensed as open source software, as well. That way, citizens would have the opportunity to help the agency with their own disclosures, and agencies could more easily share tools with each other.

90 days — March 8, 2010

“The Deputy Director for Management at OMB will issue, through separate guidance or as part of any planned comprehensive management guidance, a framework for how agencies can use challenges, prizes, and other incentive-backed strategies to find innovative or cost-effective solutions to improving open government.” (p.5)

This is a strangely oblique reference to Vivek Kundra’s Apps for Democracy project when he was CTO in Washington, DC, and the national-scale follow-on, Apps for America. Both of these contests asked that submissions be provided under OSI-approved licenses. This is important to keep these projects going. If contestant’s software is under a proprietary license, there is no momentum behind the contest, since nobody can contribute to it after the fact. You might as well hold no contest at all, and instead just bid the work out to a contractor.

120 days — April 7, 2010

“Each agency shall develop and publish on its Open Government Webpage an Open Government Plan that will describe how it will improve transparency and integrate public participation and collaboration into its activities. Additional details on the required content of this plan are attached. Each agency’s plan shall be updated every two years.” (p.4)

I would hope very much that these plans for additional public participation and collaboration include invitations to open source developers who would like to help an agency build tools that make them function more transparently and efficiently.

“The Administrator of the Office of Information and Regulatory Affairs (OIRA), in consultation with the Federal Chief Information Officer and the Federal Chief Technology Officer, will review existing OMB policies, such as Paperwork Reduction Act guidance and privacy guidance, to identify impediments to open government and to the use of new technologies and, where necessary, issue clarifying guidance and/or propose revisions to such policies, to promote greater openness in government.” (p.6)

I hope that this review would include an examination of FACA implementation guidelines, which is understood by many to prevent open source developers from directly participating with some Federal agencies, for fear of having offered the explicitly prohibited “volunteer help.” We believe this isn’t the case, and it would be great if OIRA published some clarifying language. If they were to provide an interpretation of OMB Circular 130-A that ensured it was safe for agencies to create open source software without running afoul of procurement regulations, that would be wonderful.

So here’s a tremendous opportunity for the open source community. We have been given an early Christmas gift: a pretty clear path for more open source software and (perhaps more importantly) more government-sponsored open source projects inside each agency. If you want to help take advantage of this opportunity, you can sign up at Open Source for America and join a working group. You’ll be glad you did.

A hearty thanks the Heather West of CDT and Melanie Chernoff of Red Hat for their invaluable comments.