A Report by Steve Hiller and Jim Self
Visiting Program Officers, Association of Research Libraries
The University of Illinois at Urbana-Champaign Library (UIUC Library) is one of an initial group of seven libraries participating in the Association of Research Libraries (ARL) project, “Making Library Assessment Work: Practical Approaches to for Developing and Sustaining Effective Assessment.” ( http://www.arl.org/stats/newmeas/VPO_Hiller_Self.html). This two year project is under the aegis of the ARL Statistics and Measurement Program, Martha Kyrillidou, Director. ARL Visiting Program Officers Steve Hiller and Jim Self conducted the evaluation of the UIUC Library assessment effort which was capped with a visit on May 23-24. In addition to a presentation on effective library assessment, they met with library faculty and staff and discussed a wide range of assessment and measurement related issues. The UIUC Library also responded to a request for information about assessment activities, needs, and issues before the visit. The following report summarizes the current environment and offers options and suggestions for developing effective and sustainable assessment at the UIUC Library.
THE VALUE OF LIBRARY ASSESSMENT
An ongoing, sustainable assessment program can help supply the type of information that enables the UIUC Library to make good decisions in uncertain environments. Assessment is not just the collection of data but a process that can assist in better understanding the teaching, learning and research needs of the academic community and the library resources and services that support them. In an era of tight or flat budgets the margin for error narrows and assessment can be a critical tool in assuring the wise allocation of resources that provide the most value and beneficial outcomes for the University. Assessment can help give the Library the necessary agility and nimbleness to maneuver quickly in this era of rapid changes.
Assessment should not be seen as the sole responsibility of one area in the organization or a single library. For assessment to work effectively it needs to be part of the overall library culture and integrated into planning, decision-making, and operations. Lakos and Phipps have defined a culture of assessment in libraries as:
A Culture of Assessment is an organizational environment in which decisions are based on facts, research, and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders. A Culture of Assessment exists in organizations where staff care to know what results they produce and how those results relate to customers’ expectations. Organizational mission, values, structures and systems support behavior that is performance and learning focused. (Lakos, A. and S. Phipps, “Creating a Culture of Assessment: A Catalyst for Organizational Change, portal: Libraries and the Academy, Vol. 4 (3), July 2004, pp 345-361, quote on p. 352).
We believe that assessment can flourish in a variety of organizational cultures as long as there is leadership and commitment. In a recent article we note how assessment has worked effectively in four academic research libraries (Universities of Arizona. Pennsylvania, Virginia and Washington) with very different organizational cultures:
Administrative leadership and support are critical to fostering an environment that equips staff with the vision, tools, and understanding necessary to make data-based decision making an integral organizational value. Yet there is no one way to attain this. Each of the four libraries discussed above takes a different approach to achieving this value, one that is aligned with the culture, mission, goals and objectives of each institution. It is also important to recognize that each of these libraries did not wait until all the organizational pieces were in place but started incrementally. In the long run, success will be measured by how effective each library is in using data wisely to provide the services and resources that support the learning, research and information needs of its community. (Hiller, S. and J. Self, “From Measurement to Management: Using Data Wisely for Planning and Decision-Making,” Library Trends, Vol. 53 (1), Summer 2004, pp. 129-155), quote on p. 150)
CURRENT ASSESSMENT ENVIRONMENT AND ACTIVITIES AT UIUC
The University of Illinois at Urbana-Champaign Library is one of the world’s great academic research libraries. The breadth and depth of its collections and information resources, the quality and dedication of the library faculty and staff, and the scope of services provided to the campus community all have been major factors in contributing to the prominence and excellence of the University. Yet the funding environment which supported the Library’s greatness has changed substantially in the past decade as well as the information environment and the way people find and use library resources and services. As Paula Kaufman, University Librarian, told library faculty and staff at her 2004 State of the Library address:
If we don’t change some of the ways we operate, and if we don’t change our expectations about support from the University, we will not continue to be successful.
Now the new demands and expectations are coming faster and more furiously than ever before, and just as we can’t muddle through financially hoping that the budget will reach levels that are truly unattainable, we can’t muddle through by clinging to traditions and modes of operating that no longer make sense.
Budget constraints and reductions during the past couple of years have led to the loss of 25 full-time positions and 50 FTE student workers and graduate assistants. The University Library Annual Report (January 28, 2005) noted:
It is clear that the Library cannot operate its system of 42 units as it has in the past. Our current strategy is moving us towards consolidating units and services, both physically and virtually, which, combined with strategies to leverage collection dollars is designed to use our financial resources as efficiently and effectively as possible.
The Advisory Committee to the Associate University Librarian for Services (Services Advisory Committee) provides guidance and advice on a number of service-related issues, including “How to identify with precision what expectations our users have for our services.” This group has taken the lead on several assessment initiatives including the recent large-scale surveys of graduate students (2004) and undergraduates (2005). The Services Advisory Committee made general survey data (frequency responses) available, and provided a brief summary of the graduate results and later a series of recommendations for library action based on the results. Previous general user surveys occurred in 1998 and 1999 and LibQUAL+™ was applied in 2001 and 2002. Service quality and satisfaction scores were exceptionally high in the LibQUAL+™ surveys, but it was felt that this survey instrument did not adequately capture data relevant to the 42 departmental libraries.
The Strategic Planning Committee (SPC) has also recognized the need for assessment in developing the UIUC Library’s Strategic Plan for 2005-09. In addition to articulating a specific goal of “assessing service quality”, they developed a number of action items under each goal, and called for ways to measure progress through assessment, evaluation, bench-marking, and measurement. While these action items were later removed from the plan for being too detailed at this stage, the Library plans to develop performance measures and benchmarks to track progress in achieving articulated goals.
Other assessment activities in the past few years have been undertaken by specific libraries and units or programs. These have included a number of studies on the use of physical spaces and the results have been used to adjust hours and staffing levels. Other efforts have reviewed use of collections, especially for serial retention decisions and more recently for relocation to the new Oak Street storage facility. It appears as though a number of libraries and units keep statistics of various activities, but except for those done for ARL, there seems to be little coordination or uniformity in what and how data is collected.
While UIUC library faculty conduct high quality research, in part due the rigor of their promotion, tenure and merit processes, much of the research and results pertaining to UIUC do not appear to be integrated into overall library planning and decision-making.
IDENTIFIED ISSUES AND CONCERNS
Prior to the site visit, the UIUC Library identified several assessment-related issues and concerns in the response to our request for information. These included:
Help in using data that’s collected
Ability to discuss data intelligently
Presenting and reporting assessment results
Creating an evidence-based culture
During the site visit, these topics were discussed with a number of different library groups. However, it quickly became apparent that a major issue was how to develop and sustain an assessment program in a library culture that traditionally has valued a decentralized operation with considerable autonomy for library faculty and the many library units. While some consolidation and reorganization of these service points have taken place recently, the unit perspective continues to be a major focal point for a number of library faculty and staff. Dean Kaufman noted in her state of the library address:
We must stop clinging to all of those service points we provide, especially in the Main Library. Whether we’re able to do some facility reconfiguration or not, we must find ways to offer our services at fewer locations with fewer staff. Our patrons are increasingly frustrated, our staff are increasingly stressed by high demands and bored by routine work, and it’s just too expensive. . . We must also look at the nature, number and mode of departmental libraries, retaining their traditional value and values but reconfiguring them to be as effective in the 21 st century as they were in the 20 th.
Reporting of assessment methodology and results tends to take place within the rigorous research and publication culture, where emphasis is placed on publication within peer-reviewed journals. This practice works against rapid and widespread dissemination within the library as the publication process can take considerable time. One library faculty member told us that she reviewed an excellent article for publication, and then realized that it was UIUC Library assessment-related research done by a colleague that she was unaware of. We were also told by several library faculty that the method for funding research makes it easier to collaborate with those outside the library than with library faculty colleagues although others said that wasn’t true. While we didn’t determine whether this was actually the case, the perception that it is by at least several faculty indicates a potential concern.
SUGGESTIONS AND OPTIONS FOR MOVING ASSESSMENT FORWARD
We offer the following suggestions and options for moving assessment forward at the UIUC Library. They are not intended to be prescriptive but rather outline best practices for establishing and sustaining a viable assessment program that can flourish at Illinois.
1. Provide leadership for assessment
Assessment cannot work effectively without a strong commitment from library leadership. Library administration has to step up and address that challenge and articulate assessment as a library priority for assessment to be effective and sustainable at Illinois. A critical first step would be the appointment of a person who can lead the program, help develop a culture of assessment and work with library faculty and staff to build a viable assessment program.
This person (hereafter referred to by the convenient name of “assessment coordinator”) would not necessarily conduct assessment activities, but would help coordinate efforts, advocate for assessment, serve as a resource person, know best practices, and promote understanding of the value of assessment throughout the organization. An assessment team or a committee with assessment as part of its charge (such as the Services Advisory Committee) can assist in support, building staff buy-in and helping to conduct a specific assessment project, but it cannot run a sustainable assessment program. An assessment coordinator can help in moving efforts forward, integrating data into the decision making stream, and providing both guidance and support to the assessment team.
Assessment should not be added on to existing responsibilities with no additional time or resources provided. Assessment responsibilities can be part-time but support should be provided with additional resources or a real reassignment of some or all of an individual’s current responsibilities. Given the UIUC library culture, we believe the assessment coordinator should be a tenured library faculty member, who has conducted and published assessment-related research and has earned the respect of colleagues. We recommend an initial half-time appointment to get the assessment program going, especially with the size and diversity of the UIUC Library. There are several UIUC library faculty who have an interest in assessment and also have requisite skills and expertise. Other characteristics that would be desirable for an assessment coordinator include:
Commitment to library users
A broad perspective on issues that affect academic libraries and higher education
Understanding the value of assessment in improving library services
Skill in summarizing and presenting results effectively to diverse audiences
Ability to work effectively with faculty, staff, managers and administrators throughout the organization
The assessment coordinator should be an ex officio member of the Services Advisory Committee.
2. Develop an assessment program with realistic assessment goals and activities that focus on how and where the library adds value to the academic community
It is increasingly important to measure how the library makes a positive difference to the success of the academic enterprise. Assessment goals and activities should help identify those areas and measure the value in ways that can be presented effectively to the University community and other stakeholders. Progress towards achieving these goals should be benchmarked and reviewed on a periodic basis. Priority for assessment activities should be given to efforts that provide this type of information. However, activities need to be realistic and aligned with organizational goals, resources, support and expertise.
The development and administration of large-scale surveys to graduate students (2004) and undergraduates (2005) and one planned for faculty in 2006 provide an excellent start to building a successful assessment program. Initiating a manageable and focused program lays the foundation for successful long-term assessment and avoids unreasonable expectations. A reasonable set of activities for the first two years would include:
Appoint an individual with responsibility for assessment
Work initially with an existing group such as the Services Advisory Committee
Take primary responsibility for survey analysis and reporting
Examine the feasibility of forming a separate assessment-related group
Identify and prioritize assessment needs through widespread consultation within the organization
Develop an assessment plan with clearly defined goals, objectives, support and benchmarks
Obtain training and skills necessary to understand and do good assessment
Promote a “culture of assessment” within the UIUC Library
Participate in appropriate professional venues
Help develop performance measures and benchmarks to support the Strategic Plan and other programs
Ascertain best methods and support needed for staff involvement and participation in assessment
Initiate assessment activities that address priorities and can be accomplished in a timely fashion
Evaluate the success of the program and resources needed
3. Utilize assessment to move the strategic planning process forward
Assessment will be a crucial component in fostering the success of the new UIUC Library Strategic Plan 2005-2009 (Unlocking Our Past Building Our Future). This strategic plan implicitly calls for a culture of assessment to inform and validate library programs, services, and vision. Assessment is a critical component of the first major area, “Expanding Our Service Orientation,” with a specific goal for assessing service quality but is also mentioned in other major areas. Additionally, assessment should be an important element in measuring progress toward achieving strategic planning goals. Measures and benchmarks should be clearly articulated, tied to specific actions and goals, and reviewed on a periodic basis
4. Leverage the research strengths of library faculty to better support assessment efforts
The UIUC Library faculty have established an enviable record of high quality research and their efforts are well represented in noteworthy peer-reviewed publications. While library faculty should be free to pursue research areas of interest, the Library should establish opportunities and incentives to encourage convergence of faculty research abilities with areas of organizational research interest and need. Faculty participation in this type of research would not affect their ability to publish findings.
5. Review how and why internal data are collected and used
Currently, a significant amount of time is spent collecting a variety of internal statistics and data not needed for reporting to external agencies. Many of these internal statistics are related to the traditional physical library. However, it’s not clear if there still is a need to collect these data or how they’re used. In addition to reviewing why they’re collected and used, an analysis of the costs involved in data collection with the benefits of use should be conducted. Examination of alternate methodologies such as sampling and automatic data capture should also be encouraged.
6. Establish a central repository for assessment-related data, statistics and research
A substantial amount of assessment-related data and research has been done by the Library, library units and library faculty. However, at this time there is no mechanism for archiving the data and making it accessible throughout the organization. It is difficult to discover and use what has been collected and done. An initial inventory of data collected centrally and by library units can be followed by one that pulls together library faculty research that pertains to the institution, including methodology used. Efforts should be made to standardize institutional data in a format that is accessible and usable by library faculty and staff. Another advantage of such an electronic repository is that the availability of data may preclude the need for additional data collection.
7. Report assessment-related results to internal and external communities and stakeholders
Library initiated assessment-related efforts should be reported in a timely and understandable manner to library faculty and staff, the campus community and external stakeholders. Making the data available is helpful, but it is more useful to present results in a manner that identifies key findings, what they mean, and how results will be used to improve library services. Planning for such major assessment activities as the 2006 faculty survey should include an outline of how and when results will be disseminated and presented. It is especially important that this information is readily available to those who participated in the assessment effort.
We believe that the UIUC Library is ready to move forward with developing an effective and sustainable assessment program. There is general recognition that the information and funding environments have changed substantially during the past five years and the Library needs to better understand how to support and partner with the campus community in maintaining institutional excellence. Because of the unique library culture it is essential that this program have not only strong administrative support but also the buy-in from the other library areas. The person selected as assessment coordinator must be able to work effectively with library administrators, faculty, staff and the many library units.