It’s not often that developers have a chance to build something that is considered for a presidential State of the Union Address. Just ask Bob, Andrew, and Dave of Critical Juncture.
As it turns out, what was accomplished was even more remarkable. Two senior, long-time federal executives reached out to three developers from the Bay Area to build a completely new way for people to interact with one of the most important documents published in the US: The Federal Register. And the project went from first contact to API publication in a year a half.
Here’s the inside story of how it happened.
In 2010, with the 75th anniversary of the Federal Register approaching, Michael White was approached by some of the newcomers to the White House and encouraged to devise a way to make the information in the Federal Register easier to consume. Michael was also new to his post as Managing Editor of the Office of the Federal Register. As you’ll read, many of the people behind this extraordinary effort brought to their roles years of experience combined with the energy and excitement of a new position.
Although the Federal Register is unfamiliar to most people, it is “the official daily publication for rules, proposed rules, and notices of Federal agencies and organizations, as well as executive orders and other presidential documents.” As such, it has implications for all types of activities in this country and abroad. (See sidebar.)
With the high-profile backing of the White House—the project was even considered by the president’s speechwriters for inclusion in the state of the union address—Michael chose a radically different approach to bringing to life a “Federal Register 2.0.” (As President Obama and his speechwriters honed the speech, this section was eventually dropped because it didn’t fit into the flow of the address.)
During his 24 years experience as an attorney at the National Archives, where the Office of the Federal Register is located, he had dealt with every regulatory agency and major entity within the Federal government, eventually assuming the post of Chief Legal Counsel. Now he was stepping into an operating role as the Managing Editor of the Federal Register.
“I saw this as an opportunity to put ideas into action,” said White. “With the new responsibilities I’d have more leeway in focusing people on specific projects, something that, as Chief Counsel, you’re not able to do.”
Michael had been in his position for a relatively short time when David Ferriero, who had been the Archivist of the United States for an even shorter period, endorsed the concept of a Federal Register 2.0, re-thinking the publication with the aim of significantly increasing access and interaction.
Building the Federal Register 2.0, however, posed challenges on just about every front.
It posed an interagency challenge. “The people’s data should be freely available to the people” is a core value of the Federal Register 2.0 By contrast, Federal Register data in SGML format was a significant revenue source for the Government Printing Office (GPO). Since the GPO is statutorily required to generate enough revenue to fund most of its operations, any change that might decrease that revenue required agreement between the two agencies. Michael’s experience as a negotiator and the unique relationship with the GPO—“marital,” in Michael’s words— enabled them to work together to overcame this challenge
Next was the data feed. The Federal Register was available in SGML, which was the result of a strategic planning process begun in 1993. “That was better data [than paper], but it wasn’t smart data”, explained Michael, who had been part of that planning process. “What we needed now was XML with structured metadata for preservation and access.”
Access metadata contains familiar fields such as cross-reference tags and publication date. Preservation metadata contains everything they would need to perform a lossless migration from the current Documentum hosting platform in the future if they so choose.
The data feed challenge was solved by yet another executive who was also new to his post, Mike Wash, CIO of the Government Printing Office. He’d been an earlier champion of Federal Digital System (FDSys), a massive release of XML-structured data from across the Federal Government.
With these foundations in place and the support and encouragement of senior management, Michael decided to run the Federal Register 2.0 project differently. A 2.0 project needed a 2.0 style of operating, and in April 2010 he contacted the three Silicon Valley developers who today make up the core team of Critical Juncture: Andrew Carpenter, David Augustine, and Bob Burbach .
Andrew, David and Bob met while working at a large (600 employee) non-profit, and in August of 2009 they were looking for an interesting and challenging side project. Having just missed the Rails Rumble, a 48-hour competition, they came across information on the Apps for America 2 contest sponsored by Sunlight Labs, Google, O’Reilly Media, Craig Newmark, the Gov 2.0 Expo Showcase, and TechWeb, which was timed with the launch of Data.gov. (See Analysis.)
They chose to work with the then recently-released XML feed from the Federal Register, but with a late start they had just three weeks of nights and weekends. “We agreed we could work together at least that long. It was only three weeks” said Andrew. Little did they know that 3“only three weeks” preparing for a competition would lead to a new company.
Their development sprint produced GovPulse, which provided more user-friendly and engaging access to the Federal Register and landed them a second place finish in the contest.
GovPulse, which is still available, includes …
- the geographic impact of proposed and final regulations
- citation analysis
- comprehensive search and browse capabilities
Their second-place finish put them ahead of two other apps focused on the Federal Register, as well as several other teams working with different data sets.
A few months later they entered GovPulse, which they’d continued to develop, in the Consumer Electronics Association’s Apps for Innovation contest. In January 2011 they were named the winner, and a few months later they were contacted by Michael.
GovPulse represented roughly 90% of the functionality Michael was looking for in a first release of Federal Register 2.0. With expedited approvals and contracting, the work started in April and their target release date was July 26, timed with the 75th Anniversary of the Federal Register.
The pace, which was extraordinarily fast by Washington standards, raised eyebrows and concerns within some of the agencies which are required by law to publish pending and final regulations, among other things, in the Federal Register. (See Analysis.)
“We encountered a good deal of skepticism from some of the major regulatory agencies and from the Department of Justice, so we did some outreach and held open sessions, but it was hard because we didn’t have a working site to show them,” explained White.
Part of that outreach was the decision early on to separate the “official” publication of the Federal Register through FDSys from the beta rendition of Federal Register 2.0. “We have to be up front with what the consumer is getting,” says Michael. “We have to be 100% sure that we are harvesting all the data from FDSys and that we’re accurately rendering it before the oversight committees can eventually bless Federal Register 2.0 as official, and that will take time. It was an easier call legally when we moved from paper to PDF publication because it’s just a rendition of the printed format. For now we prefer to keep it in the beta-plus standpoint until we’re sure about the processes.”
Another decision that challenged the conventional way of doing business was the decision to host this in the cloud. This was only the second Federal project to move the cloud hosting (on Amazon S3 servers); the first was a project run out of the White House. After several meetings and technical due diligence, the others agreed that Amazon’s service met the Federal government’s requirements. (See Amazon’s announcement of their offering for Federal government agencies.)
As they got deep into the project, the three developers discovered that there had been no normalization of the data, even for things as significant as the names and abbreviations of agencies. “They’re document management, not database management,” explained Andrew. “They publish in two formats, one a direct representation of the document, and one metadata that is non-trivially linked.”
The code they developed to address this closely couples the document and metadata, and normalizes the document data. The ongoing task of normalizing agency references now includes an interface that allows the site editor to add table entries for new agencies and references to agencies as they occur.
As Michael explained, the project, on this time line, was only possible because of three factors:
- The expedited engagement process. It’s not unusual that the scoping and selection of a firm to complete a large Federal IT project can take a year or more, and that’s before any work on the actual implementation.
- Support from the senior levels of the Federal government. The project was on the radar screen of the new CIO and his staff. Just as importantly, the project had the full support of the head of the Office of the Federal Register and his colleague, the head of the Government Printing Office. They were both very experienced senior executives who were able to clear obstacles and keep the project on a fast track.
- The fact that the Government Printing Office was already generating an XML feed of the data from the Federal Register.
And what about working with a small team of developers? According to Michael White, “The experience has been nothing but positive. I’ve worked with some of the large Washington-based systems integrators, and they’re not as responsive as working with a small group of people.
“Plus, I felt that they had a lot of input from people in their community,” added Michael, “which meant that we were leveraging the intelligence of the larger open government community. The Beltway contractors are more proprietary—more insular.”
And the view of the developers? “In an era of smaller government, it’s important to find leaner ways to do things,” noted Andrew.
The project is all open source and available on github.
And what was involved in getting Federal officials comfortable with open source? As Andrew explained, “We think that, with open source, people write better code because they know that someone else might be essentially looking over his or her shoulder. It’s also more secure. If anyone can see all the code, you know there’s no back door access to data; no hidden APIs you don’t know about.”
Added Bob, “We love the contributions. If anyone sees anything, they let us know, and some experienced developers contribute code to the project.”
“So far, the use of the traditional Federal Register site far outpaces the Federal Register 2.0,” said Michael. “This is why publishing the API was significant. Issuing it and publishing it embraces the larger community, especially the open government community. We hope to see more applications developed that make use of the data in ways that reach a broader audience. After all, the agencies and their regulations have a big impact on American life.”
The recent release of an API to the Federal Register 2.0 site and their data set brings the development story full circle, from the release of raw data via FDSys to encourage the open government community to do creative things with the data, to an API and emerging developer community working with the normalized and enhanced data. (See the GPO’s press release [PDF])
And within a week of the release there were developers already writing Python clients in addition to the existing Ruby client that the API launched with.
The primary resources mentioned in this case study
About the Author: Issues and insights from developer program leaders.