By Perry Willett, CDL Digital Preservation Services Manager

On October 5-6 2009, over 300 people from 22 countries attended iPRES 2009 at the Mission Bay Conference Center on the UCSF Mission Bay campus. iPRES 2009 was the sixth in an annual series of conferences devoted to digital preservation, and with the 300 attendees, the largest ever.  This year’s conference was coordinated by the UC Curation Center (the new name of CDL’s Digital Preservation Program), with the program committee chaired by Trisha Cruse.  Perry Willett was the project manager for the conference; Beaumont Yung and Rondy Epting-Day provided significant administrative support, as did Megan Amaral, CDL’s student intern from SJSU.

The theme of this year’s conference was "Moving into the mainstream.  Enabling our digital future."  The program was packed with thoughtful and thought-provoking presentations on all aspects of digital preservation. Some notable presentations included keynote addresses by David Kirsch (University of Maryland) on public interest in corporations’ business archives; Micah Altman (Harvard University) on public archives for scientific data; and a panel discussion by members of the Blue Ribbon Task Force on Sustainable Digital Preservation and Access.

The conference was extremely successful, and we’ve received much "iPraise." In addition to hosting the conference, this was an important opportunity for CDL to showcase our recent work on curation micro-services and web archiving, and to speak with current and potential partners interested in working with us.  The paper by Stephen Abrams, John Kunze and David Loy (delivered by Stephen) on curation micro-services was particularly well-received, with several highly positive Twitter tweets and blogposts during and after the conference.

CDL staff participated in many ways, with Stephen Abrams, Trisha Cruse, John Kunze, Tracy Seneca, and Perry Willett serving on the program committee.  Stephen and Tricia were also presenters at the conference, and Tracy and Heather Christenson gave poster sessions.  Many people served as reviewers for the program, including (in addition to the CDL staff members already mentioned) Scott Fisher, Martin Haye, Erik Hetzner, John Ober and Lisa Schiff.  Many people from UC campuses also served as reviewers and helped with the local arrangements.  Thanks to all of them for their contributions.

In addition to the scholarly program, we had a full slate of social events including the conference reception at the California Academy of Sciences on Monday evening.  During the conference, Rick Prelinger presented "Lost Landscapes of San Francisco," a film that includes rare footage of the city from newsreels, industrial documentaries and amateur films.  A special "après iPRES" event was held at the Hi Dive in San Francisco on Tuesday evening after the conference.  The conference was part of what was informally called "digital preservation week in San Francisco," with additional events and meetings later in the week sponsored by the International Internet Preservation Consortium, the JHOVE2 project, and Sun Microsystems.

The conference website at http://www.cdlib.org/iPres contains the complete program, along with an archive of Twitter tweets by conference attendees, photographs from the conference and social events on Flickr, and (eventually) video, PowerPoint and full papers from the presentations.  See the Amplified Conference page for photos, blogposts and tweets.

Special thanks go to the vendors who supported the conference: Sun Microsystems, Isilon Systems, ExLibris, Institute of Museum and Library Services (IMLS), Tessella, the Joint Information Systems Committee (JISC), NetApp, FileTek, Library of Congress and the National Digital Information Infrastructure and Preservation Program (NDIIPP), and DuraSpace.  Their support went a long way toward underwriting the costs of the conference.

The California Digital Library is pleased to announce public access to its Web Archives.  CDL’s Web Archives are built and published using its Web Archiving Service (WAS), which enables librarians to capture, curate, and preserve websites for the benefit of researchers and the general public.  New archives are continually being built and published, and will appear along with the current archives available at http://webarchives.cdlib.org/.

This first set of archives includes materials from the California state government agencies, and local government agencies from Orange County, San Diego, Los Angeles and more.  Also included are archives of Middle Eastern political organizations, American left-wing organizations, and web content related to events such as the 2007 Southern California Wildfires and the 2003 California Recall Election.

As government agencies and public policy organizations increasingly turn to the web as a primary means of publication, libraries are challenged to provide lasting access to the budgets, studies and reports that they have long collected for the benefit of the research community.  The WAS service also allows libraries to expand their collecting scope to more ephemeral materials such as press releases, local commission meeting minutes, public forum sites, and blogs.  All of these provide a glimpse of history in the making for future researchers.  The value of this service is described in a recent Chronicle of Higher Education article, Scholars Race to Preserve Guantánamo Records, which focuses on an archive currently being built by New York University using CDL’s Web Archiving Service.

The Web Archiving Service also enables the University of California (UC) libraries to work collaboratively on a monumental task: archiving the web sites of the State of California.   The State of California web domain (.ca.gov) represents the third largest subdomain of the U.S. government web presence.  The UC campuses have worked individually to capture and archive local information, and collectively to archive state publications.  Together, these archives represent a major achievement and a series of rich resources for California researchers.  These archives can also provide lasting access to the individual state publications that are catalogued and made available via UC’s Melvyl catalog.

The archives represent the culmination of the Web-at-Risk grant, funded by Library of Congress’ National Digital Information Infrastructure Preservation Program, and led by the California Digital Library.  With our grant partners at the University of North Texas and New York University, and our curatorial partners at the UC campuses, NYU and Stanford University, we are embarking on new era in collection building for libraries.

If you have any questions about the archives or about the Web Archiving Service, please contact washelp@ucop.edu.

Patricia Cruse, director of the CDL Digital Preservation Program, and Beth Sandore, associate university librarian for information technology planning and associate dean of libraries at the University of Illinois Library at Urbana-Champaign (who completed a rotation to the CDL in 2007) have edited a special issue of Library Trends.  The issue is comprised of sixteen articles that tell fascinating stories about the ground-breaking efforts of numerous partners within the Library of Congress National Digital Information Infrastructure and Preservation Program (NDIIPP).  Also included is an article by CDL’s Web Archiving Service Manager, Tracy Seneca, entitled, The Web-at-Risk at Three: Overview of an NDIIPP Web Archiving Initiative.

Since its inception in 2004, NDIIPP has grown from an experimental program into a true partnership of concerned organizations working together to sustain access to digital information critical to scholarship and cultural heritage nationwide.

Congratulations, Trisha and Tracy!

Will the Homeland Security and No Child Left Behind websites disappear on January 20th 2009?  The answer might surprise you.  January 20th 2009 will mark the beginning of a new presidential administration and the coincident end of the current administration, putting much of the online material related to its policies and initiatives at risk.  According to the Washington Post, “Many federal agency records exist only in digital form and are in danger of disappearing when the administration changes” (August 20, 2008).

The University of California community, not to mention scholars the world over, require perpetual access to these online materials in the normal conduct of research, teaching, and learning. Even without a change in administration, government records stored in digital form are notoriously volatile.  Web pages on government sites have an average life span of only 44 days.

To ensure that the historical record of the current administration is not lost, a partnership of government and nonprofit agencies has taken responsibility for its preservation.  The University of California – California Digital Library (CDL), in partnership with the Library of Congress, the Government Printing Office (GPO), the Internet Archive (IA), and the University of North Texas Libraries (UNTL) are planning the harvest and archival storage of more than 100 million US government web pages from the second George W. Bush administration.  This effort will involve the comprehensive harvest of the .gov domain as well as focused Web harvests of specific government agencies.  The goal is to conduct a broad capture of all Federal government Web sites, and a deep capture of specific high-priority sites that have been chosen by the project’s curators.  Each partner plays a critical role in the project.

The California Digital Library, a recipient of a Library of Congress National Digital Information Infrastructure and Preservation Program (NDIIPP) grant, leads the Web-at-Risk project, a goal of which is “to develop tools that enable librarians and archivists to capture, curate, preserve, and provide access to web-based government and political information.”  These tools will be put to use doing deep crawls of specific government agencies ranked as priority sites by the project’s curators.  In addition to CDL, UNTL will be responsible for conducting deep crawls.

The broad crawl will be the responsibility of the Internet Archive, a non-profit group providing universal and permanent access to digital information for educators, researchers, and the general public.  IA will use its advanced Web-crawling software, called Heritrix, to capture the intended sites.

In order to prioritize the vast list of URLs included in the scope of the crawl, the University of North Texas has designed a software tool that allows curators to nominate URLs for harvest and tag them with numeric rankings.

The Library of Congress, which has preserved congressional Web sites since December 2003, will focus on developing the overall harvesting plan.  The GPO and the libraries in its Federal Depository Library Program will assist in the curation process.

For more information on the End of Term project contact Patricia Cruse (patricia.cruse@ucop.edu), Director, Digital Preservation Program.

Margaret Low, software engineer in Digital Preservation, recently returned from the IFLA (International Federation of Library Associations and Institutions) Annual Conference in Quebec, where she presented a very well-received paper on the CDL preservation infrastructure.  The paper is available for download at: http://www.ifla.org/IV/ifla74/papers/084-Low-en.pdf.

CDL’s Preservation Program staff member Erik Hetzner presented a paper at JCDL (Joint Conference on Digital Libraries) in Pittsburgh, Pennsylvania last month.  His paper is entitled, "A simple method for citation metadata extraction using hidden Markov models" and is available at http://gales.cdlib.org/~egh/hmm-citation-extractor/sp181-hetzner.pdf.

UNIVERSITY OF CALIFORNIA, CALIFORNIA DIGITAL LIBRARY

TITLE: Digital Preservation Services Manager

CATEGORY: Full-Time

SALARY: Salary commensurate with qualifications and experience.
Excellent benefits.

TO APPLY:  http://jobs.ucop.edu/applicants/Central?quickFind=52447

POSITION DESCRIPTION:

Want to be part of a dynamic team that is working to preserve digital information for future generations?  At the California Digital Library (CDL), we’ve developed a world-class program to preserve digital material that supports the University of California’s research, teaching, and learning mission and you can be a part of it.  A key member of the team is the Digital Preservation Services Manager — reporting to the Director of the Digital Preservation Program the Manager is responsible for the day-to-day management of digital preservation services (production and development) through project management, the provision of support services (whether offered in person or online), and liaison with digital preservation service providers and support staff.  In addition, the Services Manager will be responsible for translating experience of users’ needs and perceptions of system capabilities in a manner that informs further refinement and extension of the digital preservation technology and service infrastructure.

This is an ideal opportunity for someone with solid people skills and a passion for working in a collaborative and dynamic environment.

The California Digital Library (CDL) supports the assembly and creative use of the world’s scholarship and knowledge for the UC libraries and the communities they serve.  In partnership with the UC libraries, the California Digital Library established the digital preservation program to ensure long-term access to the digital information that supports and results from research, teaching and learning at UC.

JOB REQUIREMENTS:

Bachelor’s degree in the social sciences, public administration, library and information science or a related field and at least three years’ relevant experience with development or delivery of online information services in educational, digital preservation, library, research, and/or cultural heritage settings or an equivalent combination of education and experience.

Demonstrated experience to plan, evaluate, budget for and manage complex projects from their inception through to their final delivery.

Plans projects and assignments and monitors performance according to priorities as demonstrated by regularly meeting established deadlines in an environment of multiple projects and changing priorities.

Strong logic and quantitative reasoning skills as demonstrated by ability to review and assess a range of variables to define key issues, evaluate reasonable alternatives and translate findings into recommended changes, actions or strategies.

Proven experience with and general understanding of the academic user community and the digital library/scholarly information services domain.

Demonstrated experience working with user community and technology/programming staff to build use cases, functional requirements and user interface design.

Excellent written and verbal communication skills as demonstrated by the ability to understand and articulate technical ideas and issues at a conceptual level and explain them clearly and concisely to non-technical staff.

Demonstrated ability to operate under general direction, able to develop creative solutions to problems, and tackle issues in a self-motivated manner in a service-oriented geographically distributed team environment.

Demonstrated ability to plan, evaluate, budget for and manage complex projects from their inception through to their final delivery.

The CDL Digital Preservation Group, under the leadership of John Kunze, has co-developed with the Library of Congress a format for transferring digital content.  “The BagIt format specification is based on the concept of ‘bag it and tag it,’ where digital content is packaged (the bag) along with a small amount of machine-readable text (the tag) to help automate the content’s receipt, storage and retrieval.  There is no software to install.  BagIt is an attempt to simplify large-scale data transfers between cultural institutions.”

Find at more from the Library of Congress press release:
http://www.digitalpreservation.gov/news/2008/20080602news_article_bagit.html

The full BagIt specification is available at http://www.cdlib.org/inside/diglib/bagit/bagitspec.html

The "CDL Guidelines for Digital Objects, Version 2.0" (CDL GDO) has been updated to include specifications for use of the METS File <file> element.  You can find the updated version at http://www.cdlib.org/inside/diglib/guidelines/ .

The revision applies to Sections 2.1, 2.2.2, 3.1, and 3.2.4 only:

• To support the orderly transmission and ingest of digital objects, the CDL recommends the inclusion of checksum (MD5, SHA-1, or CRC32) and byte size values in the METS File <file> element.  Note that this information is preferred, but not required.
• The subheadings within Sections 2.1 and 3.1 have been relabeled, and are now consistently based on METS element names.

Please contact the CDL at oacops@ucop.edu if you have any questions.

The CDL Digital Preservation Group has been busy with a variety of exciting activities, reported below.

Release 4 of the Web Archiving Service
On September 18th the Web Archiving Group released a new version of the Web Archiving Service – special thanks to Tracy Seneca, Scott Fisher, Margaret Low, Erik Hetzner, Mark Reyes, and Mike Wooldridge for getting this release out the door.  So far the group has received very positive feedback from users on the service’s functionality and the user interface.  We are also extremely pleased with the performance; we are up to 500 captures with relatively few hiccups.

We have also put together an overview of the service that is available on YouTube <http://tinyurl.com/2tdrwq>.  This brief overview explains why the content targeted for this project is at risk, how we plan to address this in the Web Archiving Service, and provides an explanation of the collections our curators are working on. Warning: the YouTube video quality is a bit sketchy so we have also made this presentation available in a high-quality video format; contact tracy.seneca at ucop dot edu for further information.

A kinder and gentler ARK page
Thanks to Kirsten Neilsen and John Kunze there is now a kinder, gentler introduction to ARK identifiers on Inside CDL <http://www.cdlib.org/inside/diglib/ark/>.  Don’t know what that is?  Then definitely take a look.  Our hope is that this will help others recognize and appreciate the true beauty and splendor of ARKs.  The new page has already been re-purposed in a German "technology watch" newsletter, <http://www.kim-forum.org/techwatch/kim-dini-technology-watch-report1_2007.pdf> which is the very first edition of a bi-annual publication from the Interoperable Metadata Center for Excellence and the German Networked Information Initiative.

Tidal wave of web data knocking on our door
For the past several years the Digital Preservation group has been working with Andreas Paepcke and Hector Garcia-Molina at Stanford University on web crawling activities.  Their research group has a wealth of experience collecting web data and while CDL’s Digital Preservation group was getting their “web crawling sea legs” they asked Stanford’s group to collect data on our behalf.  Over the years Stanford has collected over 100 TB of data ranging from dot.gov sites, election data, Katrina, Virginia Tech tragedy, etc.  However, they have been using a different crawler than the Web Archiving Service (WAS) crawler (Heritrix).  As a consequence their crawler output is incompatible with most web archiving services, including ours.  However, there is good news — they have recently created a tool that will turn the output of their crawler data into something that CDL’s service can understand.  Erik Hetzner, Mike Wooldridge, and Scott Fisher are just beginning to play around with this, but we are hoping for a positive outcome.

Contributing to the community by documenting Heritrix
As mentioned above, our Web Archiving Service uses Heritrix, the Internet Archive’s (IA) open-source, extensible, web-scale, archival-quality web crawler project.  "Heritrix" (often misspelled heretrix, heratrix, heritix, etc.) is an archaic word for "heiress", which the IA chose because the project seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations.  One of the challenges of using Heritrix is that there is a dearth of documentation.  Over the next several months Hunter Stern, CDL’s technical writer, will be working with Heritrix programmers at CDL and IA to better document the crawler.  This collaboration will help us tremendously and benefit the crawler community as well.

Moving big data: Mass Transit Project
Over the past couple of years the Digital Preservation Group has been working with the campuses to move large chunks of content into the Digital Preservation Repository (DPR).  In the process we have encountered a few speed bumps along the way. The issues are two-fold but related: the files are large and the network transfer rates have been unaccountably slow.  Though we have worked towards resolving this, we have more work to do in understanding the best transfer tools and in monitoring our networks to make sure there are no log jams and that they are ready to be used to their full potential bandwidth.  The goal is to make sure we’re making the best use of our Internet2 pathways to/from the campuses and the data centers for the benefit of all CDL projects.

The Digital Preservation group has embarked on two efforts to speed up movement of large files into the DPR.  First, they are collaborating with San Diego Supercomputer Center (SDSC) to understand how to transfer data across the network more quickly and efficiently.  Second, they are implementing (on a trial basis) a method of pulling in large numbers of external data objects into a kind of preservation holding tank in order to reduce the impact of network speed and latency on the overall DPR ingest process.  They are very excited about the collaboration with SDSC and Kirsten Neilsen will be leading the project for CDL – we’re calling the project “Mass Transit” and there is a project Wiki <http://masstransit.sdsc.edu/>.

If you want any additional information on any of these projects please contact Trisha Cruse (patricia.cruse@ucop.edu).