Web archives projects are intended to strengthen the library’s research resources. As such, Web archives projects should reflect the University Library’s mission and policies in various collection development statements. Websites are selected for harvest to bolster, complement, and parallel existing library collections, meet administrative documentation retention requirements, assist researchers, and capture ephemeral materials valuable to subject specialties, including grey literature, blog posts, and other relevant and vulnerable content.<\/span><\/p>\n When creating a collection, units must be open to discuss roles and responsibilities of Web archives administration and maintenance, such as who will be responsible for creating metadata and doing quality assurance.<\/span><\/p>\n To be able to crawl seeds, add metadata, and description to the University of Illinois\u00a0<\/span>Urbana-Champaign web collection. One must be trained by someone in preservation service. To set up a training, please email Web Archives.<\/a><\/span><\/p>\n Copying materials is inherently part of the web archiving process. Intellectual property and copyright issues are an area where Web archives should be especially cognizant to respect the rights of rights holders without limiting libraries and archives’ rights to preserve important historical content.\u00a0\u00a0<\/span><\/p>\n Webmasters use robots.txt files, also known as the Robots Exclusion Standard, to tell Web robots whether they allow crawling or not. The robots.txt file can block out specific files, directories, or even entire sites from Web crawler harvest. Webmasters may implement robots.txt exclusions for any number of reasons, such as to ensure optimal server performance and privacy protection.\u00a0\u00a0<\/span><\/p>\n Archive-It\u2019s web crawler honors all robots.txt exclusion requests. However, the crawler can be set up to ignore these blocks in specific cases.\u00a0\u00a0<\/span><\/p>\n A robots.txt file is always located at the topmost level of a website and the file itself is always called robots.txt. To determine if a crawl may be blocked, view a website\u2019s robots.txt file by adding \u201c\/robots.txt\u201d to the end of the topmost level of a site\u2019s address.\u00a0\u00a0<\/span><\/p>\n Archive-It crawls are stored in the WARC (Web ARChive) file format, an<\/span> ISO standard (CD 28500)<\/span><\/a> for storing content harvested from the World Wide Web. Archive-It\u2019s primary crawler, Heritrix, and the Wayback Machine viewing software are open-source tools supported by an international community of institutions.\u00a0\u00a0<\/span><\/p>\n Content files are hosted on servers at the Internet Archive in San Francisco. A copy of Archive-It data is hosted and stored in a secure, controlled-access facility in Richmond, California, and mirrored in additional Internet Archive repositories. In addition, a dark copy of the Archive-It repository is replicated for preservation purposes at a university in the Eastern United States.\u00a0\u00a0<\/span><\/p>\n Note at this time the Archives do not maintain a local copy of WARC files.<\/span><\/p>\n You can capture password-protected pages if the crawler is provided with login credentials to access the site. Some login pages work differently from others and may be difficult to capture. If you encounter problems with password-protected sites, please contact<\/span> Archive-It Support<\/span><\/a>.\u00a0<\/span><\/p>\n To capture a password-protected site, add the login screen as the seed URL of the page you wish to capture. Under the Seeds <\/b> tab, check the box next to the login URL and click the Edit Settings<\/b> button. In the Edit Seed Settings<\/b> dialog box, enter the page\u2019s Login Name and Login Password, then click the Apply <\/b>button.\u00a0\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":" Collection Development Policy: Web archives projects are intended to strengthen the library’s research resources. As such, Web archives projects should reflect the University Library’s mission and policies in various collection development statements. Websites are selected for harvest to bolster, complement, and parallel existing library collections, meet administrative documentation retention requirements, assist researchers, and capture ephemeral […]<\/p>\n","protected":false},"author":853,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"footnotes":""},"class_list":["post-78","page","type-page","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/wordpress.library.illinois.edu\/born-digital\/wp-json\/wp\/v2\/pages\/78","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.library.illinois.edu\/born-digital\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/wordpress.library.illinois.edu\/born-digital\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.library.illinois.edu\/born-digital\/wp-json\/wp\/v2\/users\/853"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.library.illinois.edu\/born-digital\/wp-json\/wp\/v2\/comments?post=78"}],"version-history":[{"count":10,"href":"https:\/\/wordpress.library.illinois.edu\/born-digital\/wp-json\/wp\/v2\/pages\/78\/revisions"}],"predecessor-version":[{"id":200,"href":"https:\/\/wordpress.library.illinois.edu\/born-digital\/wp-json\/wp\/v2\/pages\/78\/revisions\/200"}],"wp:attachment":[{"href":"https:\/\/wordpress.library.illinois.edu\/born-digital\/wp-json\/wp\/v2\/media?parent=78"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Unit Responsibilities:<\/h3>\n
Archive-It Access:<\/h3>\n
\u00a0 \u00a0 \u00a0Intellectual Property\/Copyright:<\/h4>\n
\u00a0 \u00a0 \u00a0robots.txt exclusions:<\/h4>\n
\u00a0 \u00a0 \u00a0Storage and Contingency Planning:<\/h4>\n
\u00a0 \u00a0 \u00a0Password-Protected Content:<\/h4>\n
\u00a0 \u00a0 \u00a0Password-Protected Content:<\/h4>\n