{"id":234,"date":"2018-09-27T19:40:38","date_gmt":"2018-09-27T19:40:38","guid":{"rendered":"http:\/\/wordpress.library.illinois.edu\/illinoisnewspaperproject\/?p=234"},"modified":"2019-09-26T18:27:20","modified_gmt":"2019-09-26T18:27:20","slug":"inp-and-newspaper-digitization","status":"publish","type":"post","link":"https:\/\/wordpress.library.illinois.edu\/illinoisnewspaperproject\/inp-and-newspaper-digitization\/","title":{"rendered":"The INP and Newspaper Digitization at Illinois"},"content":{"rendered":"

Newspaper digitization at University of Illinois began in 2004, under the leadership of Professor Mary Stuart, History Librarian and head of the History and Philosophy Library. At the time, Stuart was developing a plan to merge the History and Philosophy Library with the Newspaper Library, to form a single unit: the History, Philosophy, and Newspaper Library (HPNL<\/abbr>). As part of the proposed merger, the Illinois Newspaper Project (INP<\/abbr>)<\/a> would be brought under the umbrella of the new unit, with Stuart becoming the project’s Principal Investigator. While developing the proposal for the new unit, Stuart imagined that newspaper digitization would be a logical outgrowth of the INP<\/abbr>. To lay the groundwork for this future program, Stuart created the position of Research Information Specialist for the new unit. One of the responsibilities of the Research Information Specialist would be to provide technical support for the unit’s newspaper digitization program.<\/p>\n

By the time the new unit opened in 2005, it was already clear that newspaper digitization would transform historical research, and that public sector institutions had a role to play alongside the major private sector companies like ProQuest, which had unveiled its Historical New York Times<\/em> and Historical Wall Street Journal<\/em> just a few years earlier. In 2005, the National Digital Newspaper Program (NDNP<\/abbr>) <\/span><\/a> awarded its first two-year-cycle of grants to Virginia, California, Kentucky, New York, and Florida. Each state recipient digitized 100,000 pages of newspapers as part of the program. In 2007, the NDNP<\/abbr> awarded a second two-year cycle of grants and also unveiled Chronicling America <\/span><\/a>, the freely-available online collection of newspapers digitized by the NDNP partners. Meanwhile, the Brooklyn Public Library had digitized the Brooklyn Daily Eagle<\/em> (1841-1902), and although the newspaper itself held limited interest for most researchers, the technology demonstrated exciting possibilities for the future of newspaper digitization in public institutions. Around the same time, the Colorado State Library and the Colorado Historical Society jointly received a Library Services and Technology Act (LSTA<\/abbr>) <\/span><\/a> grant to digitize Colorado newspapers from 1859 to 1930. The University of Utah began digitizing Utah newspapers from the 1850s to the 1960s, and newspaper digitization projects were forming at Pennsylvania State University, Virginia, and Kentucky, to name just a few of the larger programs.<\/p>\n

Newspaper digitization at Illinois gained increasing momentum once the new unit opened. The Research Information Specialist position was filled in October, 2005, with the hiring of Nathan Yarasavage. The earliest obstacles to the creation of the program were, surprisingly, internal. There was a desire within the Library to centralize all digitization activities, and many regarded the work being done in the HPNL<\/abbr> as anomalous at best. The choice of a delivery platform became a flashpoint for these organizational tensions. In 2005-2006, there were few platform options. The Library of Congress was developing its own, open-source platform for the NDNP<\/abbr> (what later became known as “ChronAm”), but at the time local implementation would have been prohibitively labor- and resource-intensive, far beyond anything the Library could adapt for local in-house use. The two main proprietary platforms were Olive Software\u2019s Active Paper Archive (used by the Brooklyn Public Library, Penn State, and the Colorado State Library) and CONTENTdm <\/span><\/a> (used by Utah, which had a full-time programmer adapting CONTENTdm for newspapers). Olive offered the only out-of-the-box solution at the time. Even though the Library was already using CONTENTdm for some of its digitization projects, there was no interest in, or funding for, developing a newspaper application here.<\/p>\n

Active Paper Archive from Olive Software <\/span><\/a> was selected as the first platform for the digital newspaper collection, and the collection was titled the Illinois Digital Newspaper Collection (IDNC<\/abbr>). The IDNC<\/abbr> was unveiled to the public in 2007:<\/p>\n

\"IDNC<\/a>
IDNC Original Interface<\/figcaption><\/figure>\n

The HPNL<\/abbr>‘s newspaper digitization program was funded by a combination of grants, gifts, and support from the University Library. In 2006, Stuart received an LSTA<\/abbr> grant to digitize the Urbana Daily Courier<\/em> (1916-1925). Additional funding came from the Clifford Family Endowment. The launch of the Courier<\/em> was held on July 28, 2007 at the Urbana Free Library (view photographs from Urbana Courier launch event here <\/span><\/a>). Many former Courier<\/em> employees were present, as well as the former publisher, Byron Vedder (then in his mid-90s). In 2008, the HPNL<\/abbr> received a Special Heritage Award from Preservation and Conservation Association (PACA<\/abbr>) <\/span><\/a> for digitizing the Urbana Daily Courier<\/em>.<\/p>\n

In 2007, the Library Executive Committee gave the HPNL<\/abbr> an innovation seed grant to digitize the Daily Illini<\/em> (1916-1936). Additional funding for the digitization of the Daily Illini<\/em> came from the Clifford Family Endowment, the Stewart Howe Foundation Endowment, and the University of Illinois Library. The launch of the digitized Daily Illini<\/em> was held at the Illini Media building on April 17, 2008. (View photographs of the Daily Illini launch event here <\/span><\/a>.)<\/p>\n

Achieving a high quality digital image depends on a number of factors. One of the most important factors is the quality of the originals (print or microfilm). Both the Urbana Daily Courier<\/em> and the Daily Illini<\/em> were digitized directly from existing negative microfilm. Unfortunately, this film was created circa 1960, before best practices for preservation microfilming were established. Consequently, the microfilm suffers from bad lighting, incorrect exposure, uneven focus, and bleed-through. Furthermore, the originals from which the film was produced were often torn, soiled, faded, or stained, and many issues were missing pages. (Microfilming practices have improved considerably in the intervening decades, and microfilmed newspapers are of a much higher quality.) Tragically, there are no known original print copies of the Urbana Courier<\/em>; in order to free up space, local libraries\u2014including the University of Illinois Library\u2014destroyed the bound volumes of the original print newspaper after microfilm was produced. If original newsprint was found, it could be re-filmed and re-digitized, which would be the first step toward improving the legibility of the digital files for that title. (Please contact us if you have any information about extant back files of the Urbana Courier<\/em>.)<\/p>\n

Technical specifications for the first round of digitization: microfilm for the Urbana Daily Courier<\/em> and the Daily Illini<\/em> was scanned and digitally converted into bitonal 300 dpi TIFF files. PDFs and PNG derivatives were then created from the TIFFs. These derivatives are served over the web. TIFF files are stored offline on DVD. The microfilm was scanned in bitonal black-and-white rather than 8-bit gray scale because bitonal scans yield higher Optical Character Recognition (OCR<\/abbr>) accuracy rates than do grayscale scans. (OCR<\/abbr> <\/span><\/a> is the process through which scanned images of newspapers are made keyword searchable.)<\/p>\n

In August 2008, Stuart was awarded an LSTA<\/abbr> grant from the Illinois State Library to digitize approximately 100,000 pages of weekly farm newspapers published in Midwestern states from 1870 to 1923. This project, titled Farm, Field, and Fireside (FFF<\/abbr>) became freely available online in the summer of 2009. Like the IDNC<\/abbr>, FFF<\/abbr> used Olive Software\u2019s Active Paper Archive:<\/p>\n

\"Original<\/a>
Original Interface for the Farm, Field, and Fireside Digital Collection<\/figcaption><\/figure>\n

Thanks to additional grants and gifts, FFF<\/abbr> was expanded to include many of the leading farm newspapers. Other sources of funding for FFF<\/abbr> included the Douglas C. Roberts Family, the Norma Jean Johnston Estate, the Clifford Family Endowment, Lancaster Farming, Inc.<\/a>, the Minnesota Historical Society <\/span><\/a>, Pennsylvania State University <\/span><\/a>, the Wisconsin Historical Society <\/span><\/a>, Ohio State University <\/span><\/a>, and the T & C Schwartz Family Foundation.<\/p>\n

Even with the generous support of private donors and external granting agencies, FFF<\/abbr> includes only a small fraction of the entire farm newspaper output from the late nineteenth and early twentieth centuries. The University of Illinois Library has one of the world’s largest collections of farm newspapers in the original print editions. Many important farm newspapers remain to be filmed and digitized for inclusion in FFF<\/abbr>.<\/p>\n

Eventually, Stuart and Yarasavage had developed three separate digital newspaper collections: the Illinois Digital Newspaper Collection; Farm, Field and Fireside; and American Popular Entertainment (a collection of vaudeville trade newspapers, microfilmed and digitized with the support of Robert O. Endres, University of Illinois alumnus who worked as head film projectionist at Radio City Music Hall and later at Dolby Laboratories.). A fourth, pilot-project collection, called the Collegiate Chronicle, was to be a repository of college student newspapers, for the period 1875-1975. The goal of the project was both to digitize student newspapers for institutions that lacked the necessary IT<\/abbr> infrastructure, but also to aggregate already-digitized student newspapers from around the country into a single, searchable collection. Unfortunately, Stuart and Yarasavage were unable to secure sufficient funding to realize this project.<\/p>\n

In June 2009, Stuart was awarded the HPNL<\/abbr>‘s first National Endowment for the Humanities (NEH<\/abbr>) <\/span><\/a> grant for participation in the National Digital Newspaper Program (NDNP<\/abbr>). This grant funded the digitization of 100,000 pages of Illinois newspapers published between 1860 and 1922. Stuart applied for, and subsequently received, an additional two grants, extending the HPNL<\/abbr>‘s participation in the NDNP<\/abbr> through August of 2015 (the second grant was awarded in August, 2011, and the third grant in August, 2013). Like the first grant, the second and third grants each funded the digitization of 100,000 pages of Illinois newspapers. The Illinois Digital Newspaper Program (IDNP<\/abbr>) was the only state partner in this program to have all its batches of digital content accepted by the Library of Congress on first submission, without need for corrections or adjustments, thanks to the meticulous microfilm evaluation, quality review, metadata creation, and overall attention to detail by the Project Coordinator, Amy Sullivan, and the Metadata\/Quality Review Specialist, Tracy Nectoux.<\/p>\n

The following criteria were used when selecting newspapers for digitization by the IDNP<\/abbr>:<\/p>\n