Building an Internet archive system for the British Broadcasting Corporation.

ABSTRACT

Since its beginnings as the British Broadcasting Corporation (BBC) Networking Club in April 1994, the BBC's Web site has grown to over two million pages. While bbc.co.uk inarguably offers a valuable source of information, entertainment, and education for its users and provides an online arena for peer-to-peer communication, it also brings into focus the challenge of digital preservation. Apart from the sheer volume of material the site represents, the nature of that material is forever changing both to reflect editorial strategy and to benefit from new technologies and improved production techniques. To support its own internal business requirements and to satisfy external legislative requirements, the BBC's Information and Archives Department is building a Legal and Historical Internet Archive System to capture a selection of content as it is published to the "live" site. This article looks at how the design and development of that system supports the preservation of heterogeneous digital material in the wider context of archiving the BBC's new media output.

**********

The British Broadcasting Corporation's (BBC) (1) Information and Archives Department--a relatively recent amalgamation of research libraries, archives, and preservation services across the BBC's national and regional operational centers--manages much of the corporation's physical and electronic records and audio visual assets. One of its current projects is the introduction of a system for the automatic capture of the BBC's online services published to bbc.co.uk. This article describes the development of that system and its design and implementation in the context of the corporation's main business driver: the creation of distinctive programs and services.

Information and Archives (I&A) has always been responsible for a wide variety of material: documents, television and radio programs, and in recent years the digital and electronic equivalents. Preservation is an ongoing part of the remit; converting archived content into new formats has been an issue within the Television Archive since 1948, when nitrate was used for film production and archivists were already aware that alternative stock would need to be developed to save material in the long term. And in the BBC archives there is a vast amount of it.

Stored in multiple sites around the United Kingdom are over 600,000 hours of complete television programs, stockshots, and unedited or untransmitted material held on film, 2-inch and 1-inch videotape, and U-Matic and Beta formats. More than 300,000 hours of radio are available on wax cylinders, vinyl records, audio cassettes, CDs, and DAT or one-quarter-inch tape, with in excess of 25,000 sound effects captured on CD and vinyl. And then there are the 100 million documents held at the Written Archives Centre, 3 million photographs--hardcopy and electronic, 22 million newspaper cuttings, 1.2 million commercial music recordings, and 4 million items of sheet music. And all of that needs to be managed through processes for intake, cataloguing and indexing, research and access, and long-term preservation. So, although we are in an increasingly digital environment, the issues are not new--it is only the challenges that are different.

WHAT ARE THE DRIVERS FOR ARCHIVING AND PRESERVATION FOR THE BBC?

The BBC is beholden to several pieces of often contradictory UK legislation. One of these is the Broadcasting Act of 1996, which dictates that "it shall be the duty of each broadcasting body to retain a recording of every television or sound programme which is broadcast by that body--

(a) where it is of a television programme, during the period of 90 days beginning with the broadcast, and

(b) where it is of a sound programme, during the period of 42 days beginning with the broadcast" (Queen's Printer of Acts of Parliament, 1996a, Section 117, Part V).

In other words, the BBC is legally required to record its TV and radio output off-air to enable the corporation to answer complaints from the listening and viewing public.

This is reiterated by the 2003 Communications Act, which pays legal lip-service to the existence of platforms other than radio and television by applying the retention periods of 90 and 42 days to "every programme service" (Queen's Printer of Acts of Parliament, 2003a, Section 334). The act also includes a requirement "to comply with any request by OFCOM (2) to produce to them for examination or reproduction a recording retained in pursuance of the conditions in the licence" (Queen's Printer of Acts of Parliament, 2003a, Section 334).

Despite being relatively new legislation, disappointingly the act does not take into consideration the technical challenges of capturing new media services direct from broadcast, preserving them in their original form, and being able to re-create them as required in answer to legal claims. While neither act specifically mentions the Internet--or in fact any other new media platform--the BBC does not want to set any legal precedents but would rather demonstrate best endeavors to meet the letter of the law. To that end, an internal agreement was made in 2000 between the Head of Online and the BBC's Programme Complaints Unit that bbc.co.uk would be defined as another broadcast channel and its content captured off-air as with TV and radio.

But there are other legislative reasons for retaining the BBC's broadcast output that shape archive policy. The period of liability under the Defamation Act of 1996 is currently one year, which suggests that the BBC should be retaining content for longer than the Broadcasting Act's ninety-day requirement. Under section 2 of the act, any "offer to make amends" is an offer to "make a suitable correction of the statement complained of" and to " publish the correction and apology" (Queen's Printer of Acts of Parliament, 1996b). Consequently, at least one function of the archive is to provide a record of both the original defamatory material and any consequent official response broadcast on air.

Another raft of legislation centers on information and data management. Schedule 1 of the 1998 Data Protection Act outlines the data protection principles, which include that "Personal data shall be obtained only for one or more specified and lawful purposes" and that "appropriate technical and organisational measures shall be taken against unauthorised or unlawful processing of personal data" (Queen's Printer of Acts of Parliament, 1998, Schedule 1, Part 1).

BBC online publishes an increasing amount of user-generated content. "Talk" (http://www.bbc.co.uk/communicate/) calls for feedback on the BBC's output and provides a gateway to the message boards, while sites like "iCan" (http://www.bbc.co.uk/dna/ican/) and "Collective" (http://www .bbc.co.uk/dna/collective/) provide platforms for peer-to-peer debate on local and cultural issues. The personal data shared over the BBC's network needs to be managed in accordance with the requirements of the Data Protection Act; although this is more of a concern for content producers who directly receive and manipulate the data, there could be implications for data that is ingested into a long-term archiving system.

More specifically relevant to bbc.co.uk, the United Kingdom's Legal Deposit Libraries Act of 2003 is enabling legislation intended to extend the…